products

Quickly Run The Changed Specs on GitHub Actions

Improve the testing speed when testing pull requests using Cypress.

In the previous blog post Run Changed Traced Specs On GitHub Actions I have explained how you can determine the Cypress specs that changed in the current pull request. In this post, I will show how to run them even quicker using GitHub Actions.

🎁 You can find the source code for this blog post in the repo bahmutov/changed-specs-quickly-example.

Precompute spec dependencies

If you have multiple specs plus utilities, it makes sense to precompute their import graph and store it in a JSON file in the repo. We will use my tool spec-change to compute dependencies among spec files.

package.json
1
2
3
4
5
{
  "scripts": {
    "trace-deps": "spec-change --folder cypress --save-deps deps.json --time"
  }
}

We can run the npm run trace-deps command locally. Here is its partial outut

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
$ npm run trace-deps

> [email protected] trace-deps
> spec-change --folder cypress --save-deps deps.json --time

spec-change took 1523ms
{
  "e2e/utils.ts": [
    "e2e/adding-spec.ts",
    "e2e/clear-completed-spec.ts",
    "e2e/complete-all-spec.ts",
    "e2e/editing-spec.ts",
    "e2e/item-spec.ts",
    "e2e/persistence-spec.ts",
    "e2e/routing-spec.ts",
    "e2e/spec.ts"
  ],
  "e2e/model.d.ts": [
    "e2e/cast-fixture-spec.ts",
    "e2e/fixture-spec.ts",
    "e2e/import-fixture-spec.ts",
    "e2e/using-fixture-spec.ts"
  ],
  "fixtures/todos.json": [
    "e2e/import-fixture-spec.ts"
  ],
  ...
}

The file cypress/e2e/utils.ts is popular. Several specs depend on it. If this file changes, we would consider e2e/adding-spec.ts, e2e/clear-completed-spec.ts and others changed too.

We can periodically update the deps.json file on CI using a job like this:

.github/workflows/spec-dependencies.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# this workflow computes the dependencies between Cypress spec
# source files every day and saves it to the file deps.json
# and pushes any changes back to the origin.
# This lets the pull requests quickly determine which spec
# files to re-run by looking at the changes
name: spec dependencies
on:
  schedule:
    - cron: '0 4 * * *'
  # trigger this workflow from GitHub Actions UI
  workflow_dispatch:

jobs:
  find-dependencies:
    name: Find source dependencies
    runs-on: ubuntu-22.04
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v4

      - name: Install spec-change 📦
        # https://github.com/bahmutov/npm-install
        uses: bahmutov/npm-install@v1
        with:
          # use just package.json checksum
          useLockFile: false
          install-command: 'npm install spec-change'
          cache-key-prefix: 'spec-change'

      - name: Find Cypress spec dependencies 🧱
        run: npm run trace-deps

      - name: Commit any changed files 💾
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated Cypress spec dependencies
          branch: master
          file_pattern: deps.json

The install step installs only the spec-change NPM dependency and caches it by caching the ~/.npm folder. We don't want to install all dependencies, and we don't want to download the Cypress binary.

1
2
3
4
5
6
7
8
- name: Install spec-change 📦
  # https://github.com/bahmutov/npm-install
  uses: bahmutov/npm-install@v1
  with:
    # use just package.json checksum
    useLockFile: false
    install-command: 'npm install spec-change'
    cache-key-prefix: 'spec-change'

Run changed specs first

When a user opens a pull request, they probably have modified some specs. We want to run the modified specs first, and if they pass, we would like to run all specs. We can find the changed between the current branch and the main branch using the find-cypress-specs utility. We can use either simple Git file change or trace dependencies.

package.json
1
2
3
4
5
6
7
{
  "scripts":{
    "trace-deps": "spec-change --folder cypress --save-deps deps.json --time",
    "changed-specs": "find-cypress-specs --branch main --parent",
    "trace-changed-specs": "find-cypress-specs --branch main --parent --trace-imports cypress --cache-trace"
  }
}

I prefer using the trace so that any change to the utils.ts forces the spec files that import it to run. Here is our workflow to determine the changed spec files.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
name: PR checks
on: [pull_request]
jobs:
  find-changed-specs:
    runs-on: ubuntu-22.04
    outputs:
      changedSpecs: ${{ steps.trace.outputs.changedSpecs }}
      changedSpecsN: ${{ steps.trace.outputs.changedSpecsN }}
      machinesNeeded: ${{ steps.trace.outputs.machinesNeeded }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          # need to fetch info about all branches
          # to determine the changed spec files
          # https://glebbahmutov.com/blog/faster-ci-feedback/
          fetch-depth: 0

      - name: Install find-cypress-specs 📦
        # https://github.com/bahmutov/npm-install
        uses: bahmutov/npm-install@v1
        with:
          # use just package.json checksum
          useLockFile: false
          install-command: 'npm install find-cypress-specs'
          cache-key-prefix: 'find-cypress-specs'

      - name: Trace changed specs
        # against the "main" branch
        # and including changed files imported by specs
        # and compute how many machines we need to
        # run changed the changed specs in parallel
        # https://github.com/bahmutov/find-cypress-specs
        id: trace
        run: |
          npm run trace-changed-specs -- \
            --set-gha-outputs --gha-summary \
            --specs-per-machine 2 \
            --max-machines 3

Notice the last job "Trace changed specs" has an id "trace". This job produces outputs, like the list of changed specs, the number of changed specs, and even the recommended number of machines to use to run the changed specs in parallel. We expose these outputs from the job find-changed-specs:

1
2
3
4
5
6
7
jobs:
  find-changed-specs:
    runs-on: ubuntu-22.04
    outputs:
      changedSpecs: ${{ steps.trace.outputs.changedSpecs }}
      changedSpecsN: ${{ steps.trace.outputs.changedSpecsN }}
      machinesNeeded: ${{ steps.trace.outputs.machinesNeeded }}

Let's pretend we modified the utils.ts file and opened a pull request. The workflow runs and detects all the specs that rely on utils.ts file

We found 8 specs that are changed in this pull request

The workflow

After finding the changed specs, we should quickly run them. First, we run all changed specs across the N recommended machines. Then we want ti run the other specs, also in parallel. Here is how we can do it using bahmutov/cypress-workflows reusable GitHub workflows. This is the rest of the PR workflow shown above; two jobs are after the "find-changed-specs" job.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# run all changed specs in parallel
changed-specs:
  needs: find-changed-specs
  if: ${{ needs.find-changed-specs.outputs.changedSpecsN > 0 }}
  # https://github.com/bahmutov/cypress-workflows
  uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
  with:
    spec: ${{ needs.find-changed-specs.outputs.changedSpecs }}
    # convert the output string to a number
    nE2E: ${{ fromJson(needs.find-changed-specs.outputs.machinesNeeded) }}
    start: 'npm start'

# run other specs in parallel IF changes specs passed
other-specs:
  needs: [find-changed-specs, changed-specs]
  uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
  with:
    # we already have done the changed specs
    # now let's run all other specs
    skip-spec: ${{ needs.find-changed-specs.outputs.changedSpecs }}
    nE2E: 2
    start: 'npm start'

We take the find-changed-specs job's outputs and make two jobs, both using the split workflow from bahmutov/cypress-workflows. I used just two machines to run the rest of the specs. In the future I hope to compute a good number of machines based on the specs.

The output from the machines running the changed specs

You can find the full workflow and source code in the repo bahmutov/changed-specs-quickly-example.

See more