products

Refactor Tests To Be Independent And Fast Using Cypress-Each Plugin

How to generate tests and even separate specs using cypress-each plugin.

I have made cypress-each to simplify generating tests from data. This plugin has already proved itself useful for API testing, and in this post I want to show a few more tricks it can help you do, like making the tests faster by generating separate spec files to be executed in parallel.

The initial test

Let's start with a single test that checks if elements are visible.

1
2
3
4
5
6
it('shows the expected elements', () => {
  cy.visit('/')
  cy.get('header').should('be.visible')
  cy.get('footer').should('be.visible')
  cy.get('.new-todo').should('be.visible')
})

You can watch me refactor this single test to avoid command duplication in the video below

The better test iterates through the list of selectors

1
2
3
4
5
6
7
it('shows the expected elements', () => {
  cy.visit('/')
  const selectors = ['header', 'footer', '.new-todo']
  selectors.forEach((selector) => {
    cy.get(selector).should('be.visible')
  })
})

The above test has a problem: if the first selector is wrong, the rest of the commands and assertions is skipped when it fails to find an element. Are the "footer" and the ".new-todo" selectors valid? Are those elements visible? We do not know, since the test has failed on the first command. We want to separate this test into 3 independent tests, and do it with minimal code duplication.

Using cypress-each to generate independent tests

This is where the cypress-each plugin comes in handy. It can generate separate it tests (or even separate describe suites) from a list of items.

1
2
3
4
$ npm i -D cypress-each
+ [email protected]
# install using Yarn
$ yarn add -D cypress-each

You can import the plugin directly from the spec file, or from the Cypress support file. If you import it from the support file, the it.each syntax becomes available in every spec.

1
2
3
4
5
6
7
import 'cypress-each'

const selectors = ['header', 'footer', '.new-todo']
it.each(selectors)('element %s is visible', (selector) => {
  cy.visit('/')
  cy.get(selector).should('be.visible')
})

You can watch me refactor the single test using cypress-each plugin in the video below

There are now three independent tests:

  • "element header is visible"
  • "element footer is visible"
  • "element .new-todo is visible"

If one test fails, the rest still runs and gives you the complete picture.

Tip: You can be quite flexible with the test title pattern. For example, you can use %k and %K placeholders to insert the item's index (0-based and 1-based respectively). You can even use your own test title format function like this:

1
2
3
4
5
import 'cypress-each'

const selectors = ['header', 'footer', '.new-todo']
const getTitle = (item, index, items) => `testing element %K "%s"`
it.each(selectors)(getTitle, (selector) => { ... })

The above syntax will create 3 tests with titles testing element 1 "header", testing element 2 "footer", and testing element 3 ".new-todo".

Parallel testing

If we have a lot of data items, and generate a test for each one, the entire spec can become quite long. For example, the following spec simulates a slow loading page using cy.wait(10000) command. With 9 element selectors, this small spec example runs for 90 seconds!

cypress/integration/visible-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import 'cypress-each'

describe('visible elements', () => {
  // simulate creating lots of tests by using the same selectors
  const selectors = [
    'header',
    'footer',
    '.new-todo',
    'header',
    'footer',
    '.new-todo',
    'header',
    'footer',
    '.new-todo',
  ]

  it.each(selectors)(
    (selector, k) => `testing ${k + 1} selector ${selector}`,
    (selector) => {
      cy.visit('/')
      // simulate slow-loading page
      cy.wait(10000)
      cy.get(selector).should('be.visible')
    },
  )
})

Spec takes 90 seconds to run through each test

Ughh, brutal. The best way to make the entire testing step faster is by running lots of tests in parallel, as described in the blog post Make Cypress Run Faster by Splitting Specs and Parallel end to end testing with Cypress, Docker and GitLab. But how do we split the tests generated using it.each into separate spec files?

First, instead of a single visible-spec.js spec file, create a folder and place several empty spec files there. For example, if we want to run the element tests using 3 test runners in parallel at a time, create 3 spec files there.

1
2
3
4
5
6
cypress/
  integration/
    visible-spec/
      spec1.js
      spec2.js
      spec3.js

Second, put the selector data into a JSON file to be imported from each spec.

cypress/integration/visible-spec/selectors.json
1
2
3
4
5
6
7
8
9
10
11
[
  "header",
  "footer",
  ".new-todo",
  "header",
  "footer",
  ".new-todo",
  "header",
  "footer",
  ".new-todo"
]

Third, move each test callback function into a separate JavaScript file. It is not a spec itself, it just exports the test callback to be imported by other specs.

cypress/integration/visible-spec/utils.js
1
2
3
4
5
6
7
8
9
export const testTitle = (selector, k) =>
  `testing ${k + 1} selector ${selector}`

export const testElementSelector = (selector) => {
  cy.visit('/')
  // simulate slow-loading page
  cy.wait(10000)
  cy.get(selector).should('be.visible')
}

Now let's finish the individual spec files spec1.js, spec2.js, and spec3.js. Each will handle just a subset of the generates tests by ... filtering the data items! Since we have three different spec files, we will use module 3 to split the items into tests for each spec.

cypress/integration/visible-spec/spec1.js
1
2
3
4
5
6
7
8
9
import selectors from './selectors.json'
import { testTitle, testElementSelector } from './utils'

describe('visible elements', () => {
  // there are 3 spec files testing the "selectors" list
  // this spec file will pick the selectors 0, 3, 6, etc.
  const filteredSelectors = selectors.filter((x, k) => k % 3 === 0)
  it.each(filteredSelectors)(testTitle, testElementSelector)
})

🎁 You can find these specs in the repo bahmutov/todo-graphql-example.

The spec1.js executes a third of the tests.

The first spec file only tests each third data case

Similarly, the other two spec files pick up a third of the test cases each. The spec2.js uses modulo 3 equals 1 to filter the test cases

cypress/integration/visible-spec/spec2.js
1
2
3
4
describe('visible elements', () => {
  const filteredSelectors = selectors.filter((x, k) => k % 3 === 1)
  it.each(filteredSelectors)(testTitle, testElementSelector)
})

The spec3.js picks the last third of the test cases using the module 3 equals 2.

cypress/integration/visible-spec/spec3.js
1
2
3
4
describe('visible elements', () => {
  const filteredSelectors = selectors.filter((x, k) => k % 3 === 2)
  it.each(filteredSelectors)(testTitle, testElementSelector)
})

You can also split the list of items into groups using Cypress._.chunk

cypress/integration/visible-spec/spec1.js
1
2
3
4
5
describe('visible elements', () => {
  // selectors has 9 items, split into chunks with max size 3
  const filteredSelectors = Cypress._.chunk(selectors, 3)[0]
  it.each(filteredSelectors)(testTitle, testElementSelector)
})

I have added the support for chunking the items into describe.each and it.each, see Chunking. For example, if we want to split all items across three spec files, the test files would use:

1
2
3
4
5
6
// spec1.js
it.each(selectors, 3, 0)(testTitle, testElementSelector)
// spec2.js
it.each(selectors, 3, 1)(testTitle, testElementSelector)
// spec3.js
it.each(selectors, 3, 2)(testTitle, testElementSelector)

Run specs in parallel

To show the time savings from running the generated tests in parallel, I have set up recording the tests results on Cypress Dashboard. The initial run used a single machine:

All tests including spec1, spec2, and spec3 ran on a single CI machine

Then I set up the CI to run the same tests using 3 containers on GH Actions, see the .github/workflows/ci.yml file:

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
name: ci
on: push
jobs:
  test:
    runs-on: ubuntu-20.04
    strategy:
      # when one test fails, DO NOT cancel the other
      # containers, because this will kill Cypress processes
      # leaving the Dashboard hanging ...
      # https://github.com/cypress-io/github-action/issues/48
      fail-fast: false
      matrix:
        # run 3 copies of the current job in parallel
        containers: [1, 2, 3]
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v2

      # https://github.com/cypress-io/github-action
      - name: Run tests 🧪
        uses: cypress-io/github-action@v2
        with:
          start: npm run dev
          wait-on: 'http://localhost:1234'
          record: true
          parallel: true
          group: 'All tests'
        env:
          # pass the Dashboard record key as an environment variable
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

The new test run took 1/3 of the time because our long specs were executed on 3 machines at once.

Running specs in parallel on CI using 3 machines

Much better.