products

Run Changed Traced Specs On GitHub Actions

How to accurately Cypress.io specs to run first when using GitHub Actions

Imagine you have a lot of Cypress spec files, and you want to refactor some code. You probably want to run the changed specs first to get faster CI feedback, right? Let's say you have a situation like I have in the repo bahmutov/test-todomvc-using-app-actions. You have 10+ spec files, and a few utilities shared across specs.

Specs and utilities in cypress/e2e folder

If you open a pull request with edited spec files, you can find which specs have changed and should run first using Git commands. I have even implemented code in find-cypress-specs utility to make finding changed specs simple in my projects. For example, if we modify the spec file cypress/e2e/fixture-spec.ts:

cypress/e2e/fixture-spec.ts
1
2
3
import type { Todo } from './model'

// change something in this spec file

We can run the following command to find this spec

1
2
3
> branch: f1
$ npx find-cypress-specs --branch master --parent
cypress/e2e/fixture-spec.ts

What happens when we modify the model.d.ts file imported by cypress/e2e/fixture-spec.ts? Several specs import this .d.ts file, so we should re-run those specs for a pull request that modifies model.d.ts file.

Spec files that import the model.d.ts file

My goal today is to find these specs affected by the change to the source file model.d.ts they all import. If I can find them, my pull request workflow can run the affected specs only, or as the first step when verifying the tests are still working correctly.

Trace the changed specs

In the latest find-cypress-specs release 1.21.0 there is a new feature that uses spec-change to trace all import and require statements in the source files to build a graph of file dependencies (excluding 3rd party code from node_modules). From the graph, it can find the spec files affected by the model.d.ts file change, and you can set the GitHub Actions workflow to run them. Here is a command to trace the source file dependencies in the cypress folder.

1
2
3
4
5
> branch: f1
$ npx find-cypress-specs --branch master --parent --trace-imports cypress

cypress/e2e/cast-fixture-spec.ts,cypress/e2e/fixture-spec.ts,
cypress/e2e/import-fixture-spec.ts,cypress/e2e/using-fixture-spec.ts

The tool find four specs affected by the change. If we are using GitHub Actions, we can test the affected spec files. Here is the pull request workflow we can use.

.github/workflows/pr.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
name: pr changed tests
on: [pull_request]
jobs:
  changed-tests:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          # need to fetch info about all branches
          # to determine the changed spec files
          # https://glebbahmutov.com/blog/faster-ci-feedback/
          fetch-depth: 0

      # Install NPM dependencies, cache them correctly
      # https://github.com/cypress-io/github-action
      - name: Cypress run
        uses: cypress-io/github-action@v5
        with:
          # only install everything
          runTests: false

      - name: Changed specs
        # see changed Cypress specs in this branch
        run: npm run changed-specs

      - name: Trace changed specs
        # against the "master" branch
        # and including changed files imported by specs
        # https://github.com/bahmutov/find-cypress-specs
        id: trace
        run: npx find-cypress-specs --branch master --parent \
          --trace-imports cypress --set-gha-outputs

      - name: Run changed Cypress specs 🏎
        if: ${{ steps.trace.outputs.changedSpecsN > 0 }}
        uses: cypress-io/github-action@v5
        with:
          # we have already installed all dependencies above
          install: false
          spec: ${{ steps.trace.outputs.changedSpecs }}
          # start the app before running the tests
          start: npm start

The option --set-gha-outputs uses GitHub Actions SDK library to save the found spec files and their number as the step's outputs, allowing us to decide if we need to run the end-to-end tests steps next.

1
2
3
4
5
6
7
8
9
- name: Run changed Cypress specs 🏎
  if: ${{ steps.trace.outputs.changedSpecsN > 0 }}
  uses: cypress-io/github-action@v5
  with:
    # we have already installed all dependencies above
    install: false
    spec: ${{ steps.trace.outputs.changedSpecs }}
    # start the app before running the tests
    start: npm start

Tracing the changed specs in action

I have opened the pull request #334 and the PR workflow has finished successfully.

The pull request traced changed specs workflow

Let's drill into the workflow to see what specs it traced and executed. Remember: we only modified 3 different files, without touching any of the Cypress specs.

The pull request touched 3 non-spec files

Yet, the trace detected that the changes do affect 4 spec files.

The source changes do affect 4 Cypress specs

The Cypress GitHub action then executes only those affected four specs.

The workflow quickly executes the affected 4 specs

Finding the affected spec files and running them first on pull request lets you dramatically cut down on the time it takes to verify changes to the testing code, especially if you run run A LOT of Cypress tests.

Update 1: limit the added spec files

If you change a common utils.js file that many specs import, you might end up with a giant pull request with hundreds of specs. You can prevent it by using the CLI parameter --max-added-traced-specs <N>, for example to add at most 10 extra specs by tracing:

1
npx find-cypress-specs --branch main --parent --trace-imports cypress --max-added-traced-specs 10

Update 2: speed up tracing using cached dependencies file

If you have a lot of source files, finding dependencies takes time. You can see how long it takes by adding --time-trace flag.

1
2
$ npx find-cypress-specs --branch main --parent --trace-imports cypress --time-trace
tracing took 1234ms

You can save the found dependencies into a JSON file deps.json and then reuse it

1
2
3
4
5
# get the number of affected specs
$ npx find-cypress-specs --branch main --parent --trace-imports cypress --cache-trace --count

# quickly get the affected specs without recomputing the dependencies
$ npx find-cypress-specs --branch main --parent --trace-imports cypress --cache-trace

Since the E2E test specs don't change too often, you can compute the dependencies once a day and commit to you source repository. Here is an example GitHub Actions workflow from bahmutov/test-todomvc-using-app-actions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# this workflow computes the dependencies between Cypress spec
# source files every day and saves it to the file deps.json
# and pushes any changes back to the origin.
# This lets the pull requests quickly determine which spec
# files to re-run by looking at the changes
name: spec dependencies
on:
  schedule:
    - cron: '0 4 * * *'

jobs:
  find-dependencies:
    name: Find source dependencies
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v3

      - name: Cypress run
        uses: cypress-io/github-action@v5
        with:
          # only install everything
          runTests: false

      - name: Find Cypress spec dependencies 🧱
        run: npm run trace-deps

      - name: Commit any changed files 💾
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated Cypress spec dependencies
          branch: master
          file_pattern: deps.json

The commands to recompute the traced dependencies and use them are in package.json file

package.json
1
2
3
4
5
6
7
{
  "scripts": {
    "trace-deps": "spec-change --folder cypress --save-deps deps.json --time",
    "trace-changed-specs":
      "find-cypress-specs --branch master --parent --trace-imports cypress --cache-trace --set-gha-outputs"
  }
}

Make sure to commit the deps.json file into the repo, and the above workflow will keep it updated every night. Even more realistic is to use your personal GitHub token to be able to force push to the protected main branch and to trigger the workflow on schedule AND manually. The above workflow would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
name: spec dependencies
on:
  # update dependencies every night OR manually
  schedule:
    - cron: '0 4 * * *'
  workflow_dispatch:

jobs:
  find-dependencies:
    name: Find source dependencies
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v3
        with:
          # the workflow needs to push the updated files back
          # to the protected branch "master", thus we need to use
          # a personal GH token that allows to do it
          token: ${{ secrets.MY_GITHUB_TOKEN }}

      - name: Cypress run
        uses: cypress-io/github-action@v5
        with:
          # only install everything
          runTests: false

      - name: Find Cypress spec dependencies 🧱
        run: npm run trace-deps

      - name: Commit any changed files 💾
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated Cypress spec dependencies
          branch: master
          file_pattern: deps.json
          # because we have "master" branch protected, have to force it
          push_options: --force