products

Get Faster Feedback From Your Cypress Tests Running On GitHub Actions

How to run the modified Cypress.io specs first when using GitHub Actions

As your project grows, the end-to-end tests take longer and longer to finish. You open a pull request and ... wait for 10-20 minutes for the tests to finish. Then you search the Cypress Dashboard to find the spec with the modified test, just to see if it has failed or passed. All this time, you are thinking to yourself - why can't Cypress run the modified specs first? While there is no built-in way in Cypress as of October 2021, it is not hard to implement it yourself. In this blog post, I will show how to run new and changed Cypress.io specs first if you are using GitHub Actions. Similar approach could be used with any CI provider, like CircleCI.

🎁 You can find the source code for this blog post in the repo bahmutov/chat.io

The initial workflow

At first, our GitHub workflow file checks out the source code and runs tests using Cypress GH Action I have written:

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

name: ci
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-20.04
    steps:
      - name: Check out code 🛎
        uses: actions/checkout@v2

      # run utility services in the background
      # using docker-compose (see docker-compose.yml file)
      - name: Run docker-compose
        run: docker-compose up -d

      # install and cache dependencies, start the server
      # and run all Cypress.io tests
      # https://github.com/cypress-io/github-action
      - name: Cypress tests 🧪
        uses: cypress-io/github-action@v2
        with:
          start: npm start
          wait-on: 'http://localhost:3000'
          record: true
        env:
          # for recording test results and videos to Cypress Dashboard
          CYPRESS_RECORD_KEY: ${{secrets.CYPRESS_RECORD_KEY}}

The above workflow runs on every commit and on every pull request. We still want to run all the tests for every commit pushed to the main branch. But for the pull requests, we want to run the modified specs first before running all tests. Thus I modify the above ci workflow to only run on commits pushed to the main branch.

1
2
3
4
5
6
7
8
name: ci
# run all tests on the main branch
on:
  push:
    branches:
      - main
...
# rest of the workflow

You can find this workflow in .github/workflows/ci.yml. This workflow is tied to the README badge, showing the current test status of the project:

1
2
3
4
[![ci status][ci image]][ci url]

[ci image]: https://github.com/bahmutov/chat.io/workflows/ci/badge.svg?branch=main
[ci url]: https://github.com/bahmutov/chat.io/actions

The pull request workflow

I will use a separate workflow file for CI steps to run for the pull requests. You can find the finished workflow file at .github/workflows/pr.yml. Let's start by cloning the ci.yml and just modifying the on trigger.

.github/workflows/pr.yml
1
2
3
4
5
name: pr
# on pull request, determine changed or added Cypress specs
# if there are any (but not too many), run them first
# then run all Cypress specs
on: [pull_request]

First, we will need to check out the source code. Because we want to determine the files changed between the PR branch and the default main branch, we need to fetch this information. Thus I will use the parameter fetch-depth: 0 with actions/checkout action:

1
2
3
4
5
6
7
# https://github.com/actions/checkout
- name: Check out main branch 🛎
  uses: actions/checkout@v2
  with:
    # need to fetch info about all branches
    # to determine the changed spec files
    fetch-depth: 0

Here is how we can find all changed (added and modified) files between the current branch and the main branch

1
2
3
- name: List changed files 🗂
  # should we get the branch names from the PR?
  run: git diff --name-only origin/main

For example, I have started a new branch example-branch and modified the spec rooms.js and added a new spec ``

Git local status

Let's commit and push this branch to the remote origin.

1
2
3
$ git add .
$ git commit -m "new spec and small tweaks"
$ git push -u

Even when working locally, we can see the changed files between the current branch example-branch and the main one.

1
2
3
$ git diff --name-only origin/main
cypress/integration/register-using-task2.js
cypress/integration/rooms.js

The pull request might have other modified files besides the Cypress specs. For example, I will touch the README file too. Here is how we can filter the specs

1
2
3
4
5
6
7
8
$ git diff --name-only origin/main
README.md
cypress/integration/register-using-task2.js
cypress/integration/rooms.js

$ git diff --name-only origin/main | grep cypress/integration
cypress/integration/register-using-task2.js
cypress/integration/rooms.js

Super. Later we will need to know the number of modified specs - we can use wc -l to count the lines with the modified Cypress specs

1
2
$ git diff --name-only origin/main | grep cypress/integration | wc -l
       2

Ughh, why is there whitespace around 2, let's trim it

1
2
$ git diff --name-only origin/main | grep cypress/integration | wc -l | tr -d ' '
2

Now that we know the number of changed specs, let's also join them into a single string to be passed to the cypress run --spec ... parameter.

1
2
$ git diff --name-only origin/main | grep cypress/integration | tr '\n' ','
cypress/integration/register-using-task2.js,cypress/integration/rooms.js,

Super. We can compute the number and the spec parameter in the workflow, and even hide the details from other specs by using the output parameters.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
- name: List changed files 🗂
  # should we get the branch names from the PR?
  run: git diff --name-only origin/main

- name: List changed specs ✨
  id: list-changed-specs
  run: |
    n=$(git diff --name-only origin/main | grep cypress/integration | wc -l | tr -d ' ')
    specs=$(git diff --name-only origin/main | grep cypress/integration | tr '\n' ',')
    echo "Changed and added Cypress specs"
    echo ${specs}
    echo "number of added or changed Cypress specs ${n}"

    # output the number of specs and the specs list
    echo "::set-output name=specsN::${n}"
    echo "::set-output name=specs::${specs}"

Other workflow steps can access the number of changed specs using ${{ steps.list-changed-specs.outputs.specsN }} expression syntax. Let's set up two test jobs - the first one will run if there are changed specs, but not more than 5. If there are lots of modified specs, it makes sense to simply run all of them.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# https://github.com/cypress-io/github-action
- name: Run changed Cypress specs first 🌲
  # it makes sense to run changed specs only if there are a few
  # otherwise just run all specs in the next step
  if: ${{ steps.list-changed-specs.outputs.specsN > 0 && steps.list-changed-specs.outputs.specsN < 5 }}
  uses: cypress-io/github-action@v2
  with:
    start: npm start
    wait-on: 'http://localhost:3000'
    record: true
    group: '1. Changed specs'
    spec: ${{ steps.list-changed-specs.outputs.specs }}
  env:
    # for recording test results and videos to Cypress Dashboard
    CYPRESS_RECORD_KEY: ${{secrets.CYPRESS_RECORD_KEY}}

# if the changed / added Cypress tests passed
# run all Cypress tests to confirm the app is working
- name: All Cypress tests 🧪
  uses: cypress-io/github-action@v2
  with:
    # hmm, is the application running?
    start: npm start
    wait-on: 'http://localhost:3000'
    record: true
    group: '2. All Cypress tests'
  env:
    # for recording test results and videos to Cypress Dashboard
    CYPRESS_RECORD_KEY: ${{secrets.CYPRESS_RECORD_KEY}}

We have two problems in the above workflow:

  • we install the NPM dependencies twice (potentially)
  • we are trying to run the application using npm start twice (potentially)

Thus we can optimize the workflow by installing the dependencies just once, and starting the application before running any Cypress tests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# install dependencies
- name: Install dependencies 📦
  uses: cypress-io/github-action@v2
  with:
    # just perform install
    runTests: false

- name: Start the app 🏁
  run: npm start &

# https://github.com/cypress-io/github-action
- name: Run changed Cypress specs first 🌲
  # it makes sense to run changed specs only if there are a few
  # otherwise just run all specs in the next step
  if: ${{ steps.list-changed-specs.outputs.specsN > 0 && steps.list-changed-specs.outputs.specsN < 5 }}
  uses: cypress-io/github-action@v2
  with:
    # we have already installed all dependencies above
    install: false
    # the server is running already, but just wait for it
    wait-on: 'http://localhost:3000'
    record: true
    group: '1. Changed specs'
    spec: ${{ steps.list-changed-specs.outputs.specs }}
  env:
    # for recording test results and videos to Cypress Dashboard
    CYPRESS_RECORD_KEY: ${{secrets.CYPRESS_RECORD_KEY}}

- name: All Cypress tests 🧪
  uses: cypress-io/github-action@v2
  with:
    # we have already installed all dependencies above
    install: false
    wait-on: 'http://localhost:3000'
    record: true
    group: '2. All Cypress tests'
  env:
    # for recording test results and videos to Cypress Dashboard
    CYPRESS_RECORD_KEY: ${{secrets.CYPRESS_RECORD_KEY}}

Tip: Cypress GH Action can do a lot, find all examples in the cypress-io/github-action repo.

Limit the Git output

Two words of caution: the command git diff --name-only origin/main outputs all file names, including the names of the deleted files. Thus I limit the list of the modified and added files only using

1
git diff --name-only --diff-filter=AM origin/main

You should also be careful about printing the list of changed files. By itself the above command will page the output and pause after N lines. This will halt the CI job usually:

1
2
3
4
5
6
7
8
9
10
$ git diff --name-only --diff-filter=AM origin/main
.circleci/config.yml
README.md
cypress/integration/spec-a.js
cypress/integration/spec-b.js
cypress/integration/spec-c.js
...
--More--

Too long with no output (exceeded 10m0s): context deadline exceeded

Thus you want to pipe the output through the filters first - the filters do not get paginated. For example, you can get the raw number of changed files.

1
2
3
4
5
n=$(git diff --name-only --diff-filter=AM origin/main | grep cypress/integration | wc -l | tr -d ' ')
if [ ${n} -gt 20 ]; then
  echo "Too many files changes..."
  # stop the changed specs job
fi

Workflow in action

Let's see how the above workflow performs. I have opened the pull request #12.

The GH workflow shows the steps, and that the modified specs task was executed

GitHub workflow

The list-changed-spec step has calculated the two changed Cypress test files correctly

The changed Cypress specs

The changed tests have finished successfully, while all tests have failed in an unrelated spec group-chat.js. Notice how fast the modified specs have finished vs waiting for all the tests: 30 seconds vs 5 minutes.

The recorded Dashboard run

The group-chat.js shows the test fails to log in the first user A

The failed test screenshot

Let's modify the group-chat.js - something goes wrong there, let's change the user name to be a userA instead of just A. Once I push the commit with the username change, the 3 changes specs run:

The three modified specs ran first

Our fix has solved the problem, and all the Cypress specs have passed. We are good to merge.

See also

Update 1: pick the changed specs better

What happens if you modify a non-spec file in the cypress/integration folder? For example, if you touch the cypress/integration/utils.js that is excluded by the cypress.json file, our crude "give me the changed files" filter tries to run this file by itself, causing the Cypress to exit with an error code "No specs found".

The job fails when trying to run with non-spec file only

This is where we can use my utility find-cypress-specs to find the changed spec files. You can ask this utility for only the specs changed against a Git branch.

1
2
specs=$(npx find-cypress-specs --branch main)
n=$(npx find-cypress-specs --branch main --count)

This number and the list are computed using the Git and the cypress.json file settings.

Update 2: pick the changed specs better part 2

When using find-cypress-specs in a busy repository, I found a shortcoming: the added and changed specs against the main branch picked up specs changed on the main branch in addition to the specs changed in the pull request branch. So I added another option to the CLI tool: pick the changed specs only between the current commit and its parent merge commit. The best way to show the difference is via a diagram below.

The specs changed by the commits on the main and featureA branches

Let's say we want to find the changed specs between the last commit on the branch "FeatureA" and the current head of branch "main". It would give us the changed files between the two branches.

1
2
3
# from the branch "FeatureA"
$ npx find-cypress-specs --branch main
spec1.js,spec2.js,spec3.js,spec4.js

We are probably not interested in the specs spec3.js and spec4.js added on the "main" branch - they are not part of the feature work. To limit the changed specs to search the current branch up to the merge commit "C1" we can use the --parent flag.

1
2
3
# from the branch "FeatureA"
$ npx find-cypress-specs --branch main --parent
spec1.js,spec2.js

I think it makes a lot more sense, doesn't it?

Tip: if you use cypress-grep to tag and selectively run tests, you can filter the changed specs by the presence of tag(s). For example, to run only the changed specs that have the tag "@user" inside, use npx find-cypress-specs --branch main --parent --tagged @user.

See also