products

Split CI Jobs

How to execute multiple test tasks in parallel on CircleCI and TravisCI.

Who loves waiting on CI to finish testing multiple files? Not me. Luckily modern CIs support isolation and running multiple test jobs in many containers, splitting single sequence of test jobs into parallel runs. My favorite system for doing this is GitLabCI, but the same can be done in many other popular CIs. This blog post shows how to set up parallel test runs in CircleCI (v2) and on TravisCI. If I do not write this down I will forget and will have to reinvent this in the future, I am sure!

Example setup

Imagine you have a single repository with many folders with tests. The cypress/cypress-example-recipes is a good example. Different recipes are subfolders in the examples folder - there are more than 20 folders!

1
2
3
4
5
6
7
8
9
$ exa -l examples
drwxr-xr-x - irinakous 13 Dec 16:28 blogs__codepen-demo
drwxr-xr-x - irinakous 13 Dec 16:28 blogs__direct-control-angular
drwxr-xr-x - irinakous 28 Nov 11:16 blogs__e2e-api-testing
drwxr-xr-x - irinakous 13 Dec 10:12 blogs__e2e-snapshots
...
drwxr-xr-x - irinakous 28 Nov 11:16 testing-dom__tab-handling-links
drwxr-xr-x - irinakous 28 Nov 11:16 unit-testing__application-code
drwxr-xr-x - irinakous 28 Nov 11:16 unit-testing__react-enzyme

A single script going through every folder, running end-to-end tests takes about 15 minutes. There are two test services where we run tests: on Circle and on Travis. First, let us convert sequence to parallel jobs on Circle CI.

Circle

CircleCI v2 has introduces workflows that are extremely powerful. Think of a workflow as a series of jobs; for our purposes every job is independent. Here is how we can define a workflow - we will name each job after a folder.

1
2
3
4
5
6
7
8
9
10
workflows:
  version: 2
  # when adding new example subfolder with a recipe
  # add its name here to "create" CircleCI job
  all-recipes:
    jobs:
      - blogs__codepen-demo
      - blogs__direct-control-angular
      ...
      - unit-testing__react-enzyme

Now we need to define each job. We need to specify the Docker container the job should run in, any environment variables to set and the actual job steps. While we could copy and paste these steps, in reality all our test job are the same - only the folder is different. So we can use YAML template feature to create common job definition and only substitute the job name. Here is how it looks:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
defaults: &defaults
  parallelism: 1
  working_directory: ~/app
  docker:
    - image: cypress/browsers:chrome63
      environment:
        ## this enables colors + fixes failing unit tests
        TERM: xterm
        # avoid million NPM install messages
        npm_config_loglevel: warn
        # allow installing when the main user is root
        npm_config_unsafe_perm: true
  steps:
    - checkout
    # npm module caching
    - restore_cache:
        key: root-deps
    - run: npm install
    - run: npm prune
    - save_cache:
        key: root-deps-{{ checksum "package.json" }}
        paths:
          - node_modules
    # starting server and running tests
    - run:
        name: start the server
        background: true
        command: |
          cd examples/$CIRCLE_JOB
          npm run start
    - run:
        name: Cypress tests
        command: |
          cd examples/$CIRCLE_JOB
          npm run cypress:run -- --record

jobs:
  # define a new job with defailts for each "examples/*" subfolder
  blogs__codepen-demo:
    <<: *defaults
  blogs__direct-control-angular:
    <<: *defaults
  ...
  unit-testing__react-enzyme:
    <<: *defaults

Note that every job in the workflow automatically receives the name (as we defined it) in the environment variable CIRCLE_JOB - which we used to change into the right folder. You can see the exact cicle.yml file at the moment of this writing.

Travis

TravisCI has recently introduced build stages. Personally I do not think they are as easy to use as CircleCI workflows, they are suitable for our task. Again we will put all common test steps into a template. Because there is no predefined environment variable with the folder name, I will pass it as an environment variable DIR.

1
2
3
4
5
6
# common build steps for each recipe
defaults: &defaults
  script:
    - cd examples/$DIR
    - npm start -- --silent &
    - npm run cypress:run -- --record

Now to the test job definitions. They all belong to the same stage named test, thus they will run in parallel. Instead of job's name, I am setting environment variable DIR to the folder name.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
jobs:
  include:
    # define a separate script for each "examples/*" folder
    # this will run it in a separate job on TravisCI
    - stage: test
      env:
        - DIR=blogs__codepen-demo
      <<: *defaults
    - stage: test
      env:
        - DIR=blogs__direct-control-angular
      <<: *defaults
    - stage: test
      env:
        - DIR=blogs__e2e-api-testing
      <<: *defaults
    ...
    - stage: test
      env:
        - DIR=unit-testing__react-enzyme
      <<: *defaults

You can find the exact .travis.yml file here. Here is how TravisCI shows the progress of the test run - some jobs have finished, others are running and more jobs are queued up.

Travis jobs