Make Cypress Run Faster by Splitting Specs

• The dominant spec
• Splitting the spec
  • Common utils
• Working with folders
  • Excluding the utils
  • Running the specs
• Speeding up

Let's take a look at the Cypress test run for Cypress' own documentation repository. The Cypress Dashboard shows that 4 CI machines have finished the run in just under 2 minutes 🎉.

Note: when looking at the top of the run information you see the "9m 05s" duration. This is the total test time added together. But because we split the tests across 4 machines using Cypress parallelization the wall clock time it took to finish all specs was 2 minutes.

The dominant spec

Notice that breakdown of specs per machine. Our test run used 4 machines. The first machine executed one spec and one spec only. The other three machines joined the run slightly later and each executed 5, 5, and 2 specs respectively. Those specs were much shorter than the first called examples_spec.js.

The examples_spec.js is dominating the run duration. If we add more test machines to the build, the total wall clock duration would not improve. You can see the computed durations for different numbers of machines by using the parallelization calculator button.

In our case, we are already at the fasted run time because a single machine is running the longest spec by itself. If we want to make our run faster, we need to split the spec. How much time could we gain? Click on the "Timeline" view and show the terminal output from the test runner.

The examples_spec.js has 11 individual tests. The longest test takes almost a minute. We could split this spec in half: one spec would have the test "displays large blog imgs" and the rest of the tests could go into another spec file. If these specs run in parallel, then the total run duration would be under 1 minute!

Splitting the spec

Let's split the examples_spec.js into several specs. I recommend placing these new specs under the common folder cypress/integration/examples. After all, these specs describe the examples documentation. The longest-running test is part of the suite called "Blogs". In fact the 3 tests in these suite are each quite long.

I will split them all into own spec file, leaving the rest of the existing specs in the original file. The final file structure of specs with their tests will be:

cypress/
  integration/
    examples/
      blogs/
        images_spec.js   // "lists small links" test
        links_spec.js    // "lists large blog urls" test
        urls_spec.js     // "displays large blog imgs" test
      examples_spec.js   // all other tests in one spec file

The "Blogs tests all share common preparation steps implemented using beforeEach hooks.

Our setup code is very simple. Before each test we read the list of blogs from an YML file - this is the file used by the Hexo static site generator to produce the documentation pages.

const YAML = require('yamljs')
describe('Blogs', function () {
  let blogs = []

  before(() => {
    cy.readFile('source/_data/blogs.yml')
    .then(function (yamlString) {
      blogs = YAML.parse(yamlString)
    })
  })

  beforeEach(() => {
    cy.visit('/examples/media/blogs-media.html')
    cy.contains('.article-title', 'Blogs').should('be.visible')
  })

  // the tests
})

💡 you can find the code changes in the pull request #3402

At first I can copy / paste this setup code into the three new spec files. Yes, the code is duplicated, but we will deal with this later. So now we have 3 new spec files, and the entire "examples" group of tests inside its own subfolder cypress/integration/examples. The Cypress Dashboard sees these specs for the first time - and runs them first. We run these specs first because we assume the developer is really interested in knowing if these new tests are passing or not.

How did it affect the testing time? It ... did not change much. It has dropped from 2 minutes to 1m 40s, but that's not the 2x improvement we expected

Note: the root cause of little change is because different machines on CircleCI take widely different time to attach the workspace and start executing tests. I will investigate this further in the future.

Common utils

Some of our specs in the cypress/integration/examples use the same utility methods. We can move them from each spec to the common file cypress/integration/examples/utils.js

cypress/integration/examples/utils.js

export const getAssetCacheHash = ($img) => {
  const src = $img.attr('src').split('.')

  return src.length >= 3 ? src.slice(-2, -1).pop() : ''
}

export const addAssetCacheHash = (assetSrc, hash) => {
  let parsedSrc = assetSrc.split('.')

  parsedSrc.splice(-1, 0, hash)

  return parsedSrc.join('.')
}

The rest of the specs import these functions as needed. For example

cypress/integration/examples/blogs/images_spec.js

import { getAssetCacheHash, addAssetCacheHash } from '../utils'

const YAML = require('yamljs')

describe('Blogs', function () {
  let blogs = []

  before(() => {
    cy.readFile('source/_data/blogs.yml')
    .then(function (yamlString) {
      blogs = YAML.parse(yamlString)
    })
  })

  beforeEach(() => {
    cy.visit('/examples/media/blogs-media.html')
    cy.contains('.article-title', 'Blogs').should('be.visible')
  })

  it('displays large blog imgs', () => {
    cy.get('.media-large .media img').each(($img, i) => {
      const assetHash = getAssetCacheHash($img)
      const imgSrc = assetHash.length
        ? addAssetCacheHash(blogs.large[i].img, assetHash)
        : blogs.large[i].img

      expect($img).to.have.attr('src', imgSrc)
      cy.request(Cypress.config('baseUrl') + imgSrc).its('status').should('equal', 200)
    })
  })
})

Notice that all example tests start the same way - they load the blogs YML file and visit the page. Let's move these utilities to the utils.js as well. I love simply moving them as functions to be called.

cypress/integration/examples/utils.js

...
/**
 * Reads the blogs YML file and converts it to the JavaScript array.
 */
export const getBlogs = () => {
  return cy.readFile('source/_data/blogs.yml').then(YAML.parse)
}

/**
 * Visits the blogs page and waits for it to load.
 */
export const visitBlogsPage = () => {
  cy.visit('/examples/media/blogs-media.html')
  cy.contains('.article-title', 'Blogs').should('be.visible')
}

The spec file simply imports and uses these utilities.

cypress/integration/examples/blogs/links_spec.js

import { getBlogs, visitBlogsPage } from '../utils'

describe('Blogs', function () {
  let blogs = []

  before(() => {
    getBlogs().then((list) => {
      blogs = list
    })
  })

  beforeEach(visitBlogsPage)

  it('lists small links', () => {
    cy.get('.media-small').each((blogEl, i) => {
      cy.wrap(blogs.small[i]).then((blog) => {
        cy.wrap(blogEl)
        .contains('a', blog.title)
        .should('have.attr', 'href', blog.sourceUrl)
      })
    })
  })
})

I like using JSDoc to document the utility methods, since modern code editors show nice popup when using them.

Question: would you consider the utils.js file a page object? Answer: maybe yes, but probably not. For me such common utilities without any internal state are just lightweight functions to call whenever the test needs. They avoid heavy abstractions and code duplication.

Working with folders

We have split an individual spec file into several files. Some of them are specs, and one file is the utility file.

Excluding the utils

Cypress considers every Javascript and TypeScript file found in the integration folder a spec file. Thus we need to exclude the utils.js file from this list. Let's set the test file pattern in the cypress.json file to only consider files that end with _spec.js as test files.

{
  "testFiles": "**/*_spec.js"
}

Notice that the utils.js file is no longer shown in the list of specs.

Running the specs

If we split the single spec into several files, how do we run them in the interactive mode all at once? By using the file search filter. If you want to run all "examples/*" specs at once, type "examples/" and click "Run 4 integration specs" button.

If you want to run then the same specs from the command line using cypress run use the --spec CLI argument with the wildcard pattern.

npx cypress run --spec 'cypress/integration/examples/**'

Note: Cypress still applies the testFiles filter to the found files, thus excluding the utils.js

Tip: use single quotes around the wildcard string to avoid the terminal's shell expanding it (which depends on the shell and the operating system), and instead let Cypress find the files correctly.

Speeding up

Finally, let's take a look a the slow test itself. It grabs all image urls from the page and checks each one by one.

it('displays large blog imgs', () => {
  cy.get('.media-large .media img').each(($img, i) => {
    const assetHash = getAssetCacheHash($img)
    const imgSrc = assetHash.length
      ? addAssetCacheHash(blogs.large[i].img, assetHash)
      : blogs.large[i].img

    expect($img).to.have.attr('src', imgSrc)
    cy.request(Cypress.config('baseUrl') + imgSrc).its('status').should('equal', 200)
  })
})

Because Cypress wraps each command in its chain to observe it, iterating over 200 links is slow. Since all we are interested is requesting resources to confirm they exist, let's shift the check into the Node code where we can quickly request them.

cypress/plugins/index.js

const got = require('got')

module.exports = (on, config) => {
  on('task', {
    async checkUrls(urls) {
      console.log('checking %d urls', urls.length)

      for await (const url of urls) {
        await got(url)
        console.log('✅ %s', url)
      }

      return null
    }
  })
}

From the test we can collect all links and then call the task to check them in a single Cypress command.

it('displays large blog imgs', () => {
  let urls = []

  cy.get('.media-large .media img')
    .should('have.length.gt', 10)
    .each(($img, i) => {
    const assetHash = getAssetCacheHash($img)
    const imgSrc = assetHash.length
      ? addAssetCacheHash(blogs.large[i].img, assetHash)
      : blogs.large[i].img

      expect($img).to.have.attr('src', imgSrc)
      const url = Cypress.config('baseUrl') + imgSrc
      urls.push(url)
    })
    .task('checkUrls', urls)
})

If any of the URLs is wrong, the got throws an error, the cy.task fails, and the test fails too. Previously, this test took almost a minute to run. With the change to the task, it takes 9 seconds.

So what's the final CI speed? Our test run finishes in under 1 minute - we are 🏎.

Happy fast testing!