CircleCI Parallelism As A Parameter

Use workflow parameter to control the number of machines CircleCI is launching to run your tests faster when needed.

Running end-to-end tests takes time. The best way to cut the overall test duration is by running all tests in parallel by launching multiple test machines and splitting all spec files among them. Cypress has parallelization feature which works especially well when using Cypress CircleCI Orb. For example, in the repo bahmutov/todo-graphql-example I have about 20 spec files and the following CircleCI config file .circleci/config.yml:

.circleci/config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
version: 2.1
orbs:
# https://github.com/cypress-io/circleci-orb
cypress: cypress-io/cypres[email protected]
executors:
with-chrome:
docker:
# this Docker image is built by the Cypress team
# https://github.com/cypress-io/cypress-docker-images
- image: 'cypress/browsers:node16.5.0-chrome94-ff93'
workflows:
build:
jobs:
# first get the source code and install npm dependencies
- cypress/install:
executor: with-chrome
- cypress/run:
requires:
- cypress/install
executor: with-chrome
start: 'npm run server' # start server before running tests
wait-on: 'http://localhost:1234'
# run all tests using the Chrome browser
# installed in the executor. I have noticed that Chrome
# does not crash when running tests for this application
browser: chrome
record: true # record results on Cypress Dashboard
parallel: true # split all specs across machines
parallelism: 1 # how many CircleCI machines to use
group: 'all tests' # name this group "all tests" on the dashboard
tags: 'circleci'

All tests run using a single CircleCI machine. First, the install job runs, then the test job uses the workspace with the source code and the installed dependencies to start the application in the background and run the tests.

CircleCI workflow

The Cypress test runner discovers all spec files to run

Cypress starts the test run

I am using Cypress Dashboard to record the test results. The recorded run shows a single machine executing all 19 specs, from the longest-running to the shortest specs.

Cypress Dashboard recorded the run

We want to run the tests faster, but how many machines do we need? Let's click on the "Parallelization Calculator" button to find out.

Cypress Dashboard has a built-in parallelization calculator

Hmm, we could save a lot of time by running the tests on more machines. Should we tweak the parallelism: 1 parameter in the cypress/run job? I love flexibility: run the tests on the reasonable number of machines without overpaying, yet be able to use more machines when needed. To provide such flexibility we can configure workflow parameters and make the number of machines to use a parameter.

.circleci/config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
version: 2.1
orbs:
# https://github.com/cypress-io/circleci-orb
cypress: cypress-io/[email protected]
parameters:
# how many test machines to run in parallel?
PARALLEL_TESTS:
type: integer
default: 2
...
- cypress/run:
parallel: true # split all specs across machines
parallelism: << pipeline.parameters.PARALLEL_TESTS >>

By default, we run two CircleCI machines in parallel, you can see the individual containers inside the "cypress/run" job on CircleCI

Two machines running Cypress tests

But we can start the workflow manually from the CircleCI web application and provide the workflow parameter PARALLEL_TESTS. Thus whenever you want to work through a lot of tests, I suggest doing the following:

  1. Make a commit that skips the default CI execution by appending [skip ci] to the commit's subject message
1
2
3
$ git commit --allow-empty -m "will run manually [skip ci]"
[master 4b2b679] will run manually [skip ci]
$ git push
  1. Open the CircleCI project page and click "Run Pipeline" button

Click the Run Pipeline button

  1. A dialog appears. Add an integer parameter "PARALLEL_TESTS" and enter the number of CI machines to spin to run Cypress tests

Enter the PARALLEL_TESTS number

  1. Click the "Run Pipeline" button and observe the workflow. A single "cypress/install" job will following by the "cypress/run" job that spawns N containers. In my case I see eight containers running the tests.

Eight CI machines are running the end-to-end tests

The Cypress Dashboard run shows the eight machines splitting the testing load and finishing in 30 seconds. Nice.

Eight CI machines are split all the tests and finished in 30 seconds

The testing time cannot be shortened any more by adding more machines, since it is now limited by the longest-running spec. We could try splitting the spec file. You can also notice that some testing machines joined the test run much later. You could optimize this by using CircleCI RAM disk, as described in my blog post Start CircleCI Machines Faster by Using RAM Disk.

The last observation: remember the parallelization calculator from the start of this blog post? It predicted that using 8 machines would finish all tests in 31 seconds. Well, the actual run finished in 32 seconds - so pretty spot on!