products

Parallel or not

This blog post shows how to configure CircleCI to run Cypress in parallel mode for internal branches, while only use a single machine to run Cypress tests for external pull requests.

Cypress.io test runner can run end-to-end tests in parallel if your continuous integration server can spin multiple agents. Agents in that case coordinate and split the tests amongst themselves. Most CI providers now allow you to use multiple machines, but CircleCI shines in this regard in my opinion. Here is how to run the same testing job on 4 machines.

1
2
3
4
5
jobs:
  test:
    parallelism: 4
    steps:
      # run test commands

Four is the default number of machines given to open source projects by the kind folks at Circle.

Cypress.io

When running Cypress.io end-to-end tests across multiple machines, you can use Cypress CircleCI Orb or write the test job steps yourself. Ultimately, each test machine has to call the cypress run command

1
npx cypress run --record --parallel

Each CI machine contacts the Cypress.io Dashboard API to pick the next spec to run. To authenticate with the Cypress Dashboard, each agent requires passing a private record key, usually by having an environment variable called CYPRESS_RECORD_KEY set. In the parallel mode, by adding more machines the test job can power through all tests very quickly, see my other blog post "Run Your End-to-end Tests 10 Times Faster with Automatic Test Parallelization".

The problem

Our friends at Spectrum have started using parallel running mode, which dropped their test run from 16 minutes to 2 minutes.

Before

1 machine

After

9 machines

This was great improvement, yet there was a problem. Often all 9 machines would be locked up running tests for code submitted by the outside contributors in their pull requests. Why was this happening?

To understand why all 9 machines would suddenly be busy, you need to know how continuous integration systems treat the project's environment variables like the private tokens or record keys. By default, as a security practice, CI providers do NOT pass the environment variables to the forked pull requests.

CircleCI project settings

You can even use this fact to stop some jobs for forked pull requests.

When Cypress test runner starts the command npx cypress run --record --parallel and the variable CYPRESS_RECORD_KEY is undefined, the test runner detects it and runs all the tests. So you, the project owner still get the test status: passes or fails, yet the private record key is not exposed to the untrusted 3rd party.

The test runner shows the following error message in the terminal, before running all tests:

1
2
3
4
5
Warning: It looks like you are trying to record this run from a forked PR.
The 'Record Key' is missing. Your CI provider is likely not passing private
environment variables to builds from forks.
These results will not be recorded.
This error will not alter the exit code.

So all forked pull requests to Spectrum project used all 9 machines to run all Cypress tests files on each machine, effectively using all resources.

Solution

I came up with the following solution to this problem. Here is the circle.yml from the demo project circleci-parallel-based-on-env:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
# we need version 2.1. to use job parameters
version: 2.1

jobs:
  run-it:
    parameters:
      parallelism:
        type: integer
        default: 1
        description: Number of boxes to use to run this job
    docker:
      - image: cypress/base:10
    parallelism: <<parameters.parallelism>>
    steps:
      - run: echo "runs with parallelism = <<parameters.parallelism>>"

  after-tests:
    docker:
      - image: cypress/base:10
    steps:
      - run: echo "all good"

workflows:
  version: 2.1
  main:
    # see how to filter jobs in workflow per branch
    # https://circleci.com/docs/2.0/configuration-reference/#jobs-1
    jobs:
      - run-it:
          name: parallel build job
          # pass parameter to the job
          parallelism: 2
          filters:
            branches:
              # for pull requests do not run this job
              ignore: /pull.*/

      - run-it:
          name: single build job
          # pass parameter to the job
          parallelism: 1
          filters:
            branches:
              # run pull requests with parallelism 1
              only: /pull.*/

      - after-tests:
          requires:
            - "parallel build job"
            - "single build job"

The above configuration file does not use Cypress, but shows in principle how you can configure the same job to run in parallel, using as many machines as you can for your own commits. At the same time the outside pull requests will only run on a single machine. Let me go through the configuration source block by block.

The test job definition

Our test job takes a parameter that specifies how many machines it should run on

1
2
3
4
5
6
7
8
9
10
11
12
jobs:
  run-it:
    parameters:
      parallelism:
        type: integer
        default: 1
        description: Number of boxes to use to run this job
    docker:
      - image: cypress/base:10
    parallelism: <<parameters.parallelism>>
    steps:
      - run: echo "runs with parallelism = <<parameters.parallelism>>"

Branch filters

CircleCI allows you to configure jobs to run for some branches, but not for others. For example, if we want to run a job in a workflow only for the master branch, we can write a workflow like

1
2
3
4
5
6
7
8
workflows:
  version: 2.1
  main:
    jobs:
      - run-it:
          filters:
            branches:
              only: master

We can also skip a job for a branch

1
2
3
4
5
6
7
8
9
workflows:
  version: 2.1
  main:
    jobs:
      - run-it:
          filters:
            branches:
              # do not run this job for branch "alpha"
              ignore: alpha

Important: all pull requests from external forkes are named pull/<number> on GitHub. We can use regular expressions to filter our jobs using pull prefix.

One machine for external pull requests

For external pull requests we can run our test job with parallelism: 1 parameter

1
2
3
4
5
6
- run-it:
    name: single build job
    parallelism: 1
    filters:
      branches:
        only: /pull.*/

Great, only 1 machine gets used for external pull requests.

Parallel tests for internal branches

For internal branches, we want to use all available machines, and our job in this case looks like this

1
2
3
4
5
6
7
- run-it:
    name: parallel build job
    parallelism: 2
    filters:
      branches:
        # for pull requests do not run this job
        ignore: /pull.*/

Note: do not name your internal branch with prefix "pull..." or it will match the filter and will run tests serially.

GitHub configuration

To make sure GitHub pull requests wait for the test results, we need to set a check somehow. But we have 2 jobs with two different names, and only one of them runs at a time. So how do we tell GitHub to wait for a status check for one of the two jobs? Here is a trick: create another dummy CircleCI job that requires both test jobs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
jobs:
  after-tests:
    docker:
      - image: cypress/base:10
    steps:
      - run: echo "all good"

workflows:
  version: 2.1
  main:
    # job "parallel build job"
    # job "single build job"
    - after-tests:
        requires:
          - "parallel build job"
          - "single build job"

Then on GitHub we can set this new job "after-tests" to be required.

Set "after-tests" as required GitHub check

CircleCI is smart enough to only require the job that can run for that branch. Thus checks for serial job from the outside forked pull request shows "single build job" + required "after-tests".

Jobs for forked pull request

But when there is a commit to an internal branch, the CircleCI runs tests on several machines, and GitHub shows "parallel build job" + again required "after-tests" job.

Jobs for forked pull request