Running end-to-end tests takes time. The best way to cut the overall test duration is by running all tests in parallel by launching multiple test machines and splitting all spec files among them. Cypress has parallelization feature which works especially well when using Cypress CircleCI Orb. For example, in the repo bahmutov/todo-graphql-example I have about 20 spec files and the following CircleCI config file .circleci/config.yml:
All tests run using a single CircleCI machine. First, the install job runs, then the test job uses the workspace with the source code and the installed dependencies to start the application in the background and run the tests.
The Cypress test runner discovers all spec files to run
I am using Cypress Dashboard to record the test results. The recorded run shows a single machine executing all 19 specs, from the longest-running to the shortest specs.
We want to run the tests faster, but how many machines do we need? Let's click on the "Parallelization Calculator" button to find out.
Hmm, we could save a lot of time by running the tests on more machines. Should we tweak the
parallelism: 1 parameter in the
cypress/run job? I love flexibility: run the tests on the reasonable number of machines without overpaying, yet be able to use more machines when needed. To provide such flexibility we can configure workflow parameters and make the number of machines to use a parameter.
By default, we run two CircleCI machines in parallel, you can see the individual containers inside the "cypress/run" job on CircleCI
But we can start the workflow manually from the CircleCI web application and provide the workflow parameter
PARALLEL_TESTS. Thus whenever you want to work through a lot of tests, I suggest doing the following:
- Make a commit that skips the default CI execution by appending
[skip ci]to the commit's subject message
$ git commit --allow-empty -m "will run manually [skip ci]"
- Open the CircleCI project page and click "Run Pipeline" button
- A dialog appears. Add an integer parameter "PARALLEL_TESTS" and enter the number of CI machines to spin to run Cypress tests
- Click the "Run Pipeline" button and observe the workflow. A single "cypress/install" job will following by the "cypress/run" job that spawns N containers. In my case I see eight containers running the tests.
The Cypress Dashboard run shows the eight machines splitting the testing load and finishing in 30 seconds. Nice.
The testing time cannot be shortened any more by adding more machines, since it is now limited by the longest-running spec. We could try splitting the spec file. You can also notice that some testing machines joined the test run much later. You could optimize this by using CircleCI RAM disk, as described in my blog post Start CircleCI Machines Faster by Using RAM Disk.
The last observation: remember the parallelization calculator from the start of this blog post? It predicted that using 8 machines would finish all tests in 31 seconds. Well, the actual run finished in 32 seconds - so pretty spot on!