CircleCI Parallelism As A Parameter

Running end-to-end tests takes time. The best way to cut the overall test duration is by running all tests in parallel by launching multiple test machines and splitting all spec files among them. Cypress has parallelization feature which works especially well when using Cypress CircleCI Orb. For example, in the repo bahmutov/todo-graphql-example I have about 20 spec files and the following CircleCI config file .circleci/config.yml:

.circleci/config.yml

version: 2.1
orbs:
  # https://github.com/cypress-io/circleci-orb
  cypress: cypress-io/cypress@1
executors:
  with-chrome:
    docker:
      # this Docker image is built by the Cypress team
      # https://github.com/cypress-io/cypress-docker-images
      - image: 'cypress/browsers:node16.5.0-chrome94-ff93'
workflows:
  build:
    jobs:
      # first get the source code and install npm dependencies
      - cypress/install:
          executor: with-chrome
      - cypress/run:
          requires:
            - cypress/install
          executor: with-chrome
          start: 'npm run server' # start server before running tests
          wait-on: 'http://localhost:1234'
          # run all tests using the Chrome browser
          # installed in the executor. I have noticed that Chrome
          # does not crash when running tests for this application
          browser: chrome
          record: true # record results on Cypress Dashboard
          parallel: true # split all specs across machines
          parallelism: 1 # how many CircleCI machines to use
          group: 'all tests' # name this group "all tests" on the dashboard
          tags: 'circleci'

All tests run using a single CircleCI machine. First, the install job runs, then the test job uses the workspace with the source code and the installed dependencies to start the application in the background and run the tests.

The Cypress test runner discovers all spec files to run

I am using Cypress Dashboard to record the test results. The recorded run shows a single machine executing all 19 specs, from the longest-running to the shortest specs.

We want to run the tests faster, but how many machines do we need? Let's click on the "Parallelization Calculator" button to find out.

Hmm, we could save a lot of time by running the tests on more machines. Should we tweak the parallelism: 1 parameter in the cypress/run job? I love flexibility: run the tests on the reasonable number of machines without overpaying, yet be able to use more machines when needed. To provide such flexibility we can configure workflow parameters and make the number of machines to use a parameter.

.circleci/config.yml

version: 2.1
orbs:
  # https://github.com/cypress-io/circleci-orb
  cypress: cypress-io/cypress@1
parameters:
  # how many test machines to run in parallel?
  PARALLEL_TESTS:
    type: integer
    default: 2
...
- cypress/run:
    parallel: true # split all specs across machines
    parallelism: << pipeline.parameters.PARALLEL_TESTS >>

By default, we run two CircleCI machines in parallel, you can see the individual containers inside the "cypress/run" job on CircleCI

But we can start the workflow manually from the CircleCI web application and provide the workflow parameter PARALLEL_TESTS. Thus whenever you want to work through a lot of tests, I suggest doing the following:

1. Make a commit that skips the default CI execution by appending [skip ci] to the commit's subject message

$ git commit --allow-empty -m "will run manually [skip ci]"
[master 4b2b679] will run manually [skip ci]
$ git push

2. Open the CircleCI project page and click "Run Pipeline" button

3. A dialog appears. Add an integer parameter "PARALLEL_TESTS" and enter the number of CI machines to spin to run Cypress tests

4. Click the "Run Pipeline" button and observe the workflow. A single "cypress/install" job will following by the "cypress/run" job that spawns N containers. In my case I see eight containers running the tests.

The Cypress Dashboard run shows the eight machines splitting the testing load and finishing in 30 seconds. Nice.

The testing time cannot be shortened any more by adding more machines, since it is now limited by the longest-running spec. We could try splitting the spec file. You can also notice that some testing machines joined the test run much later. You could optimize this by using CircleCI RAM disk, as described in my blog post Start CircleCI Machines Faster by Using RAM Disk.

The last observation: remember the parallelization calculator from the start of this blog post? It predicted that using 8 machines would finish all tests in 31 seconds. Well, the actual run finished in 32 seconds - so pretty spot on!