products

Run Cypress Specs In Parallel For Free Using Spec Timings

Use previous spec timings to efficiently split specs across multiple machines using the `cypress-split` plugin.

Recently Cypress-the-company blocked projects that use 3rd party dashboard plugins like sorry-cypress.

Cypress blocked 3rd party dashboard plugins

For more details see Cypress.io Blocking of Sorry Cypress and Currents.

Free solution is available

If you want to split your multiple specs across multiple machines for free, I have created cypress-split plugin and described how to Run Cypress Specs In Parallel For Free. This solution does not "mimic" the Cypress Cloud API, thus it should not be banned.

☢️ If Cypress team DOES block cypress-split plugin, then I will have no choice but to block all my Cypress plugins (I have written 75+ Cypress plugins) if Cypress is run with --record flag.

All we need to do to use this plugin is to add it to the setupNodeEvents callback.

cypress.config.mjs
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import { defineConfig } from 'cypress'
import cypressSplit from 'cypress-split'

module.exports = defineConfig({
  e2e: {
    // baseUrl, etc
    supportFile: false,
    fixturesFolder: false,
    setupNodeEvents(on, config) {
      cypressSplit(on, config)
      // IMPORTANT: return the config object
      return config
    },
  },
})

The specs were split purely based on their names in the list, which can create unbalanced lists for machines. For example, let's take a project with five specs split across two machines. The first run splits the specs alphabetically:

Alphabetical spec timings on two machines

Here is our GitHub Actions workflow file

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
name: ci
on: push
jobs:
  test-split:
    runs-on: ubuntu-22.04
    # use two containers to run the tests
    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2]
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v4

      - name: Run split Cypress tests 🧪
        # https://github.com/cypress-io/github-action
        uses: cypress-io/github-action@v6
        with:
          # we don't need Cypress own runner summary
          publish-summary: false
        env:
          SPLIT: ${{ strategy.job-total }}
          SPLIT_INDEX: ${{ strategy.job-index }}

🎁 You can find the example application in the repo cypress-split-timings-example. In this examples I am using version v1 the cypress-split plugin.

Ughh, there is a spec spec-d.cy.js that is much longer than others, and it makes the second machine take much longer. While the second machine is running spec-d.cy.js and spec-e.cy.js, the first machine is idle. It would be better to shift the spec spec-e.cy.js to the first machine. Then we would save 10 seconds.

Timings

Let's tell cypress-split to split specs based on previous run timings. Unfortunately, we do not have 3rd party service to keep track of spec timings, thus we need to do it ourselves. Let's set another environment variable SPLIT_FILE on the test runners. Right now, it points at a non-existent file timings.json in the root folder of the repo.

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
...
- name: Run split Cypress tests 🧪
  # https://github.com/cypress-io/github-action
  uses: cypress-io/github-action@v6
  with:
    # we don't need Cypress own runner summary
    publish-summary: false
  env:
    SPLIT: ${{ strategy.job-total }}
    SPLIT_INDEX: ${{ strategy.job-index }}
    SPLIT_FILE: 'timings.json'

Let's run the workflow again. The timings.json file is not found yet, no big deal. The specs are split alphabetically.

1
2
3
4
5
6
7
8
9
10
cypress-split: there are 5 found specs
cypress-split: chunk 1 of 2
cypress-split: Could not split specs by duration
ENOENT: no such file or directory, open 'timings.json'
cypress-split: splitting as is by name
k  spec
-  ------------------------
1  cypress/e2e/spec-a.cy.js
2  cypress/e2e/spec-b.cy.js
3  cypress/e2e/spec-c.cy.js

At the end of the run, the plugin prints JSON object with spec durations for the current machines.

Spec timings from the first machine

The second machine prints its timings JSON

Spec timings from the second machine

Note: the timings are logged to the terminal and saved in the local file on CI. Thus at the end of the test-run each machine has its own uncommitted SPLIT_FILE file on disk.

You manually copy / paste both timings and merge them into a single timings.json file and add to the repo.

timings.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
  "durations": [
    {
      "spec": "cypress/e2e/spec-a.cy.js",
      "duration": 10065
    },
    {
      "spec": "cypress/e2e/spec-b.cy.js",
      "duration": 10054
    },
    {
      "spec": "cypress/e2e/spec-c.cy.js",
      "duration": 10061
    },
    {
      "spec": "cypress/e2e/spec-d.cy.js",
      "duration": 60124
    },
    {
      "spec": "cypress/e2e/spec-e.cy.js",
      "duration": 10053
    }
  ]
}

With this file present in the repo, the CI runs and cypress-split splits the specs based on durations. The first machine runs only the spec cypress/e2e/spec-d.cy.js which takes by itself 60 seconds. The second machine runs the rest of the specs that together take up 40 seconds.

Specs split based on timings

Nice. The spec split implementation can be found in the src/timings.js file in the cypress-split plugin. Right now it is a simple greedy algorithm, but it seems to work just fine.

Updating timings.json file: single CI job

If specs change or new specs are added, the timings file becomes out-of-date. I have a couple of ideas how to update it periodically without relying on 3rd party service.

Note: if the spec is new and does not have duration in the timings file yet, the split algorithm assigns it an average duration of all existing specs.

The first strategy is to use a single long CI job that runs all specs and then commits the updated timings.json file. It is very easy to commit changed files when using GitHub Actions CI.

.github/workflows/nightly.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
name: nightly
on:
  # run this workflow every night at 3am
  schedule:
    - cron: '0 3 * * *'
  # or when the user triggers it from GitHub Actions page
  workflow_dispatch:
jobs:
  all-tests:
    runs-on: ubuntu-22.04
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v4

      - name: Run all Cypress tests 🧪
        # https://github.com/cypress-io/github-action
        uses: cypress-io/github-action@v6
        with:
          # we don't need Cypress own runner summary
          publish-summary: false
        env:
          # pretend we want to split all specs
          # just so we can write all spec timings into the file
          SPLIT: 1
          SPLIT_INDEX: 0
          SPLIT_FILE: 'timings.json'

      - name: Commit changed spec timings ⏱️
        if: github.ref == 'refs/heads/main'
        # https://github.com/stefanzweifel/git-auto-commit-action
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated spec timings
          branch: main
          file_pattern: timings.json

The above workflow runs every day or when I start it manually from GitHub repo Actions tab. If there are any changes to the timings, the timings.json file is committed and pushed to the repository using the wonderful reusable GitHub Action stefanzweifel/git-auto-commit-action.

Note: cypress-split writes new timings file only if there are new specs or any durations are off by more than 10% from the existing ones.

Updating timings.json file: merge timings

Instead of running a single job with all specs to update the timings file, we can use my split workflow from cypress-workflows repo. It works with cypress-split alias cypress-split-merge to merge downloaded timings files from parallel runs and output a single combined JSON as a GitHub Actions output. Here is an example workflow from cypress-split-timings-example repo.

.github/workflows/split.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
name: split
on:
  # launch this workflow from GitHub Actions page
  workflow_dispatch:
jobs:
  split:
    # https://github.com/bahmutov/cypress-workflows
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 2
      # use timings to split E2E specs across 2 machines efficiently
      split-file: 'timings.json'

  # this job grab the output for the `split` workflow
  # and writes it into a JSON file "timings.json"
  # and then commits the updated file to the repository
  commit-updated-timings:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-22.04
    needs: split
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v4

      - name: Show merged timings 🖨️
        run: |
          echo off
          echo '${{ needs.split.outputs.merged-timings }}'

      - name: Write updated timings 💾
        # pretty-print json string into a file
        run: echo '${{ toJson(fromJson(needs.split.outputs.merged-timings)) }}' > timings.json

      - name: Commit changed spec timings ⏱️
        # https://github.com/stefanzweifel/git-auto-commit-action
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated spec timings
          branch: main
          file_pattern: timings.json

The above workflow in action

The reusable workflow split expands into multiple jobs. At the end, the merge-split-timings job creates the merged-timings output with all combined timings. We can commit the timings into the repo to be used for next CI run.

Showing merged timings

If you have any problems using cypress-split or cypress-workflows do not hesitate opening a GitHub issue. I will be happy to help.

Running all E2E tests should be fast and easy.

See also