- The problem
- GitHub configuration
Cypress.io test runner can run end-to-end tests in parallel if your continuous integration server can spin multiple agents. Agents in that case coordinate and split the tests amongst themselves. Most CI providers now allow you to use multiple machines, but CircleCI shines in this regard in my opinion. Here is how to run the same testing job on 4 machines.
Four is the default number of machines given to open source projects by the kind folks at Circle.
When running Cypress.io end-to-end tests across multiple machines, you can use Cypress CircleCI Orb or write the test job steps yourself. Ultimately, each test machine has to call the
cypress run command
npx cypress run --record --parallel
Each CI machine contacts the Cypress.io Dashboard API to pick the next spec to run. To authenticate with the Cypress Dashboard, each agent requires passing a private record key, usually by having an environment variable called CYPRESS_RECORD_KEY set. In the parallel mode, by adding more machines the test job can power through all tests very quickly, see my other blog post "Run Your End-to-end Tests 10 Times Faster with Automatic Test Parallelization".
This was great improvement, yet there was a problem. Often all 9 machines would be locked up running tests for code submitted by the outside contributors in their pull requests. Why was this happening?
To understand why all 9 machines would suddenly be busy, you need to know how continuous integration systems treat the project's environment variables like the private tokens or record keys. By default, as a security practice, CI providers do NOT pass the environment variables to the forked pull requests.
You can even use this fact to stop some jobs for forked pull requests.
When Cypress test runner starts the command
npx cypress run --record --parallel and the variable
CYPRESS_RECORD_KEY is undefined, the test runner detects it and runs all the tests. So you, the project owner still get the test status: passes or fails, yet the private record key is not exposed to the untrusted 3rd party.
The test runner shows the following error message in the terminal, before running all tests:
Warning: It looks like you are trying to record this run from a forked PR.
So all forked pull requests to Spectrum project used all 9 machines to run all Cypress tests files on each machine, effectively using all resources.
# we need version 2.1. to use job parameters
The above configuration file does not use Cypress, but shows in principle how you can configure the same job to run in parallel, using as many machines as you can for your own commits. At the same time the outside pull requests will only run on a single machine. Let me go through the configuration source block by block.
The test job definition
Our test job takes a parameter that specifies how many machines it should run on
CircleCI allows you to configure jobs to run for some branches, but not for others. For example, if we want to run a job in a workflow only for the
master branch, we can write a workflow like
We can also skip a job for a branch
Important: all pull requests from external forkes are named
pull/<number> on GitHub. We can use regular expressions to filter our jobs using
One machine for external pull requests
For external pull requests we can run our test job with
parallelism: 1 parameter
Great, only 1 machine gets used for external pull requests.
Parallel tests for internal branches
For internal branches, we want to use all available machines, and our job in this case looks like this
Note: do not name your internal branch with prefix "pull..." or it will match the filter and will run tests serially.
To make sure GitHub pull requests wait for the test results, we need to set a check somehow. But we have 2 jobs with two different names, and only one of them runs at a time. So how do we tell GitHub to wait for a status check for one of the two jobs? Here is a trick: create another dummy CircleCI job that requires both test jobs.
Then on GitHub we can set this new job "after-tests" to be required.
CircleCI is smart enough to only require the job that can run for that branch. Thus checks for serial job from the outside forked pull request shows "single build job" + required "after-tests".
But when there is a commit to an internal branch, the CircleCI runs tests on several machines, and GitHub shows "parallel build job" + again required "after-tests" job.