products

Make Cypress Run Faster by Splitting Specs

Split the long-running spec into smaller specs in a subfolder.

Let's take a look at the Cypress test run for Cypress' own documentation repository. The Cypress Dashboard shows that 4 CI machines have finished the run in just under 2 minutes 🎉.

The specs have finished under two minutes

Note: when looking at the top of the run information you see the "9m 05s" duration. This is the total test time added together. But because we split the tests across 4 machines using Cypress parallelization the wall clock time it took to finish all specs was 2 minutes.

The total test duration vs wall clock duration

The dominant spec

Notice that breakdown of specs per machine. Our test run used 4 machines. The first machine executed one spec and one spec only. The other three machines joined the run slightly later and each executed 5, 5, and 2 specs respectively. Those specs were much shorter than the first called examples_spec.js.

The first spec takes the two minutes to finish

The examples_spec.js is dominating the run duration. If we add more test machines to the build, the total wall clock duration would not improve. You can see the computed durations for different numbers of machines by using the parallelization calculator button.

We compute the duration for different numbers of machines

In our case, we are already at the fasted run time because a single machine is running the longest spec by itself. If we want to make our run faster, we need to split the spec. How much time could we gain? Click on the "Timeline" view and show the terminal output from the test runner.

We can drill down into the single spec by opening its terminal output

The examples_spec.js has 11 individual tests. The longest test takes almost a minute. We could split this spec in half: one spec would have the test "displays large blog imgs" and the rest of the tests could go into another spec file. If these specs run in parallel, then the total run duration would be under 1 minute!

The individual tests inside the examples_spec.js and their duration

Splitting the spec

Let's split the examples_spec.js into several specs. I recommend placing these new specs under the common folder cypress/integration/examples. After all, these specs describe the examples documentation. The longest-running test is part of the suite called "Blogs". In fact the 3 tests in these suite are each quite long.

The three tests inside the Blogs suite

I will split them all into own spec file, leaving the rest of the existing specs in the original file. The final file structure of specs with their tests will be:

1
2
3
4
5
6
7
8
cypress/
  integration/
    examples/
      blogs/
        images_spec.js   // "lists small links" test
        links_spec.js    // "lists large blog urls" test
        urls_spec.js     // "displays large blog imgs" test
      examples_spec.js   // all other tests in one spec file

The "Blogs tests all share common preparation steps implemented using beforeEach hooks.

The Blogs tests before splitting

Our setup code is very simple. Before each test we read the list of blogs from an YML file - this is the file used by the Hexo static site generator to produce the documentation pages.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
const YAML = require('yamljs')
describe('Blogs', function () {
  let blogs = []

  before(() => {
    cy.readFile('source/_data/blogs.yml')
    .then(function (yamlString) {
      blogs = YAML.parse(yamlString)
    })
  })

  beforeEach(() => {
    cy.visit('/examples/media/blogs-media.html')
    cy.contains('.article-title', 'Blogs').should('be.visible')
  })

  // the tests
})

💡 you can find the code changes in the pull request #3402

At first I can copy / paste this setup code into the three new spec files. Yes, the code is duplicated, but we will deal with this later. So now we have 3 new spec files, and the entire "examples" group of tests inside its own subfolder cypress/integration/examples. The Cypress Dashboard sees these specs for the first time - and runs them first. We run these specs first because we assume the developer is really interested in knowing if these new tests are passing or not.

The new specs

How did it affect the testing time? It ... did not change much. It has dropped from 2 minutes to 1m 40s, but that's not the 2x improvement we expected

The total run time has not changed much

Note: the root cause of little change is because different machines on CircleCI take widely different time to attach the workspace and start executing tests. I will investigate this further in the future.

Common utils

Some of our specs in the cypress/integration/examples use the same utility methods. We can move them from each spec to the common file cypress/integration/examples/utils.js

cypress/integration/examples/utils.js
1
2
3
4
5
6
7
8
9
10
11
12
13
export const getAssetCacheHash = ($img) => {
  const src = $img.attr('src').split('.')

  return src.length >= 3 ? src.slice(-2, -1).pop() : ''
}

export const addAssetCacheHash = (assetSrc, hash) => {
  let parsedSrc = assetSrc.split('.')

  parsedSrc.splice(-1, 0, hash)

  return parsedSrc.join('.')
}

The rest of the specs import these functions as needed. For example

cypress/integration/examples/blogs/images_spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import { getAssetCacheHash, addAssetCacheHash } from '../utils'

const YAML = require('yamljs')

describe('Blogs', function () {
  let blogs = []

  before(() => {
    cy.readFile('source/_data/blogs.yml')
    .then(function (yamlString) {
      blogs = YAML.parse(yamlString)
    })
  })

  beforeEach(() => {
    cy.visit('/examples/media/blogs-media.html')
    cy.contains('.article-title', 'Blogs').should('be.visible')
  })

  it('displays large blog imgs', () => {
    cy.get('.media-large .media img').each(($img, i) => {
      const assetHash = getAssetCacheHash($img)
      const imgSrc = assetHash.length
        ? addAssetCacheHash(blogs.large[i].img, assetHash)
        : blogs.large[i].img

      expect($img).to.have.attr('src', imgSrc)
      cy.request(Cypress.config('baseUrl') + imgSrc).its('status').should('equal', 200)
    })
  })
})

Notice that all example tests start the same way - they load the blogs YML file and visit the page. Let's move these utilities to the utils.js as well. I love simply moving them as functions to be called.

cypress/integration/examples/utils.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
...
/**
 * Reads the blogs YML file and converts it to the JavaScript array.
 */
export const getBlogs = () => {
  return cy.readFile('source/_data/blogs.yml').then(YAML.parse)
}

/**
 * Visits the blogs page and waits for it to load.
 */
export const visitBlogsPage = () => {
  cy.visit('/examples/media/blogs-media.html')
  cy.contains('.article-title', 'Blogs').should('be.visible')
}

The spec file simply imports and uses these utilities.

cypress/integration/examples/blogs/links_spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import { getBlogs, visitBlogsPage } from '../utils'

describe('Blogs', function () {
  let blogs = []

  before(() => {
    getBlogs().then((list) => {
      blogs = list
    })
  })

  beforeEach(visitBlogsPage)

  it('lists small links', () => {
    cy.get('.media-small').each((blogEl, i) => {
      cy.wrap(blogs.small[i]).then((blog) => {
        cy.wrap(blogEl)
        .contains('a', blog.title)
        .should('have.attr', 'href', blog.sourceUrl)
      })
    })
  })
})

I like using JSDoc to document the utility methods, since modern code editors show nice popup when using them.

IntelliSense popup when using the utility method

Question: would you consider the utils.js file a page object? Answer: maybe yes, but probably not. For me such common utilities without any internal state are just lightweight functions to call whenever the test needs. They avoid heavy abstractions and code duplication.

Working with folders

We have split an individual spec file into several files. Some of them are specs, and one file is the utility file.

Excluding the utils

Cypress considers every Javascript and TypeScript file found in the integration folder a spec file. Thus we need to exclude the utils.js file from this list. Let's set the test file pattern in the cypress.json file to only consider files that end with _spec.js as test files.

1
2
3
{
  "testFiles": "**/*_spec.js"
}

Notice that the utils.js file is no longer shown in the list of specs.

Only _spec files are picked up by Cypress

Running the specs

If we split the single spec into several files, how do we run them in the interactive mode all at once? By using the file search filter. If you want to run all "examples/*" specs at once, type "examples/" and click "Run 4 integration specs" button.

Running just the filtered specs when in Cypress open mode

If you want to run then the same specs from the command line using cypress run use the --spec CLI argument with the wildcard pattern.

1
npx cypress run --spec 'cypress/integration/examples/**'

Note: Cypress still applies the testFiles filter to the found files, thus excluding the utils.js

Running just the examples specs

Tip: use single quotes around the wildcard string to avoid the terminal's shell expanding it (which depends on the shell and the operating system), and instead let Cypress find the files correctly.

Speeding up

Finally, let's take a look a the slow test itself. It grabs all image urls from the page and checks each one by one.

1
2
3
4
5
6
7
8
9
10
11
it('displays large blog imgs', () => {
  cy.get('.media-large .media img').each(($img, i) => {
    const assetHash = getAssetCacheHash($img)
    const imgSrc = assetHash.length
      ? addAssetCacheHash(blogs.large[i].img, assetHash)
      : blogs.large[i].img

    expect($img).to.have.attr('src', imgSrc)
    cy.request(Cypress.config('baseUrl') + imgSrc).its('status').should('equal', 200)
  })
})

Because Cypress wraps each command in its chain to observe it, iterating over 200 links is slow. Since all we are interested is requesting resources to confirm they exist, let's shift the check into the Node code where we can quickly request them.

cypress/plugins/index.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
const got = require('got')

module.exports = (on, config) => {
  on('task', {
    async checkUrls(urls) {
      console.log('checking %d urls', urls.length)

      for await (const url of urls) {
        await got(url)
        console.log('✅ %s', url)
      }

      return null
    }
  })
}

From the test we can collect all links and then call the task to check them in a single Cypress command.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
it('displays large blog imgs', () => {
  let urls = []

  cy.get('.media-large .media img')
    .should('have.length.gt', 10)
    .each(($img, i) => {
    const assetHash = getAssetCacheHash($img)
    const imgSrc = assetHash.length
      ? addAssetCacheHash(blogs.large[i].img, assetHash)
      : blogs.large[i].img

      expect($img).to.have.attr('src', imgSrc)
      const url = Cypress.config('baseUrl') + imgSrc
      urls.push(url)
    })
    .task('checkUrls', urls)
})

If any of the URLs is wrong, the got throws an error, the cy.task fails, and the test fails too. Previously, this test took almost a minute to run. With the change to the task, it takes 9 seconds.

The image test now runs really quickly

So what's the final CI speed? Our test run finishes in under 1 minute - we are 🏎.

The total parallelized run finishes under one minute

Happy fast testing!