products

Burn Cypress Tests on CircleCI

How to run a single test multiple times by using cypress-grep and CircleCI pipeline

Sometimes you add a new end-to-end test and ask yourself: "Is this test flake-free? Is it reliable?" If a new test is unreliable, you are bound to break other people's test runs. One way to determine if a test is robust, is to run it multiple times in a row. If the tests are all green, the test is good to go.

📺 If you would rather watch the explanation from this blog post, watch it here and subscribe to my YouTube channel.

Cypress can run a single test on demand using the cypress-grep plugin. For example, from the browser's DevTools Console we can execute the method Cypress.grep(<name of the test>, null, 10) to run a single test ten times in a row.

Run a single test by title ten times in a row from DevTools Console

You can grep the tests to run by title and run them N times in a row using Cypress headless mode. Pass the title and the burn number using --env ... argument. To execute the same test five times in a row:

1
2
3
4
5
6
7
8
9
10
11
12
13
$ npx cypress run --env grep='completes second',burn=5
cypress-grep: tests with "completes second" in their names
cypress-grep: running filtered tests 5 times
...
  - works
  ✓ completes second item: burning 1 of 5 (1677ms)
  ✓ completes second item: burning 2 of 5 (1228ms)
  ✓ completes second item: burning 3 of 5 (1196ms)
  ✓ completes second item: burning 4 of 5 (1192ms)
  ✓ completes second item: burning 5 of 5 (1188ms)

  5 passing (7s)
  1 pending

There is another way to pass the environment variables besides --env argument. You can pass the values using the environment variables prefixed with CYPRESS_ string. The command below is equivalent to the one above:

1
2
3
4
5
6
7
8
9
10
11
12
13
$ CYPRESS_grep='completes second' CYPRESS_burn=5 npx cypress run
cypress-grep: tests with "completes second" in their names
cypress-grep: running filtered tests 5 times
...
  - works
  ✓ completes second item: burning 1 of 5 (1677ms)
  ✓ completes second item: burning 2 of 5 (1228ms)
  ✓ completes second item: burning 3 of 5 (1196ms)
  ✓ completes second item: burning 4 of 5 (1192ms)
  ✓ completes second item: burning 5 of 5 (1188ms)

  5 passing (7s)
  1 pending

Sometimes we are not sure if a test is flaky or not when running it on a Continuous Integration (CI) server. If you are using a CircleCI to run Cypress tests, here is the way to be able to grep and burn a single test on demand.

🎁 You can find the full source code in the repo bahmutov/todomvc-tests-circleci.

Take your workflow that uses Cypress CircleCI Orb. It has a cypress/run job.

.circleci/config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
# to use orbs, must use version >= 2.1
version: 2.1
orbs:
  # import Cypress orb by specifying an exact version x.y.z
  # or the latest version 1.x.x using "@1" syntax
  # https://github.com/cypress-io/circleci-orb
  cypress: cypress-io/cypress@1
workflows:
  e2e:
    jobs:
      - cypress/run:
          name: Cypress E2E tests
          store_artifacts: true

We need to pass parameters to this workflow when launching it. We can use the pipeline parameters for this. Let's add a top-level section to the file.

1
2
3
4
5
6
7
8
9
10
parameters:
  # allow running selected tests once or multiple times
  # using the cypress-grep plugin
  # https://github.com/cypress-io/cypress-grep
  GREP:
    type: string
    default: ''
  BURN:
    type: integer
    default: 1

Before the cypress/run launches the Bash shell to execute npx cypress run we need to export the CYPRESS_grep and CYPRESS_burn environment variables. We can use the post-checkout option in the cypress/run job to do so, following the CircleCI env documentation. We can put the values from the pipeline parameters using a special CircleCI syntax:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
workflows:
  e2e:
    jobs:
      - cypress/run:
          name: Cypress E2E tests
          store_artifacts: true
          post-checkout:
            - run:
                name: Export grep parameters as environment variables
                # Exports GREP and BURN to environment variables
                # CYPRESS_GREP and CYPRESS_BURN
                # that Cypress will read automatically and
                # make available in Cypress.env() as Cypress.env('grep')
                # and Cypress.env('burn')
                command: |
                  echo 'export CYPRESS_grep="<< pipeline.parameters.GREP >>"' >> $BASH_ENV
                  echo 'export CYPRESS_burn=<< pipeline.parameters.BURN >>' >> $BASH_ENV

By default, the grep value is an empty string, thus all tests run once. But if we want to run just the second test to really stress-test it, we can launch the pipeline through the CircleCI web app.

Launch a pipeline run and provide parameters for cypress-grep

Tip: switch to the desired branch before running the pipeline if needed.

The pipeline parameters are exported as the environment variables for Cypress to read and pass to the plugin. The test "completes second item" then runs by itself 11 times.

Cypress receives the pipeline parameters through the environment variables

The selected test runs 11 times

Alternative: pass the environment variables as command prefix

Cypress CircleCI Orb allows you to define a command prefix that is concatenated with the full cypress run ... command formed by the orb itself. We can use this method to avoid the variable export step.

1
2
3
4
5
6
7
8
9
workflows:
  e2e:
    jobs:
      - cypress/run:
          name: Cypress E2E tests
          store_artifacts: true
          # set the environment variables before running Cypress
          # you can use "yarn" in place of "npx"
          command-prefix: CYPRESS_grep="<< pipeline.parameters.GREP >>" CYPRESS_burn=<< pipeline.parameters.BURN >> npx

Best alternative: use Cypress orb env parameter

I have added env: ... parameter to Cypress orb, released as v1.29.0 (see issue #355, PR #358), so now you can pass the burn parameter very simply.

1
2
3
4
5
6
7
workflows:
  e2e:
    jobs:
      - cypress/run:
          name: Cypress E2E tests
          store_artifacts: true
          env: grep="<< pipeline.parameters.GREP >>",burn=<< pipeline.parameters.BURN >>

Run a single test once

If you only provide the GREP parameter, without BURN, then the selected test(s) will run once. For example, let's run the test with "works" in the title.

We want to run just the tests with "works" in the title

Click the "Run Pipeline" button and observe only the selected test run.

Only the picked test ran

Two workflows

I usually have two workflows in my CircleCI config file. One to run just the grepped tests and another to run all the tests. We can control the workflows using the parameter GREP and the when / unless control statements. See chat.io config file for example.

.circleci/config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# if we want to run only some tests on CircleCI, we can call the workflow
# with parameters, as described in https://glebbahmutov.com/blog/burn-tests-on-circle/
parameters:
  # allow running selected tests once or multiple times
  # using the cypress-grep plugin
  # https://github.com/cypress-io/cypress-grep
  GREP:
    type: string
    default: ''
  BURN:
    type: integer
    default: 1

workflows:
  some-tests:
    # runs the Web tests when the user supplies a grep pattern
    when: << pipeline.parameters.GREP >>
    jobs:
      - cypress/run:
          name: Filtered E2E tests
          no-workspace: true
          group: 'Test grep: << pipeline.parameters.GREP >>'
          tags: << pipeline.parameters.GREP >>
          env: 'grep="<< pipeline.parameters.GREP >>",grepBurn=<< pipeline.parameters.BURN >>'

  all-tests:
    unless: << pipeline.parameters.GREP >>
    jobs:
      # normal build and test workflow

Bonus: check out the terminal utility run-cy-on-ci.

Read more