Get Faster Feedback From Your Cypress Tests Running On GitHub Actions

How to run the modified Cypress.io specs first when using GitHub Actions

As your project grows, the end-to-end tests take longer and longer to finish. You open a pull request and ... wait for 10-20 minutes for the tests to finish. Then you search the Cypress Dashboard to find the spec with the modified test, just to see if it has failed or passed. All this time, you are thinking to yourself - why can't Cypress run the modified specs first? While there is no built-in way in Cypress as of October 2021, it is not hard to implement it yourself. In this blog post, I will show how to run new and changed Cypress.io specs first if you are using GitHub Actions. Similar approach could be used with any CI provider, like CircleCI.

๐ŸŽ You can find the source code for this blog post in the repo bahmutov/chat.io

The initial workflow

At first, our GitHub workflow file checks out the source code and runs tests using Cypress GH Action I have written:

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

name: ci
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-20.04
steps:
- name: Check out code ๐Ÿ›Ž
uses: actions/[email protected]

# run utility services in the background
# using docker-compose (see docker-compose.yml file)
- name: Run docker-compose
run: docker-compose up -d

# install and cache dependencies, start the server
# and run all Cypress.io tests
# https://github.com/cypress-io/github-action
- name: Cypress tests ๐Ÿงช
uses: cypress-io/[email protected]
with:
start: npm start
wait-on: 'http://localhost:3000'
record: true
env:
# for recording test results and videos to Cypress Dashboard
CYPRESS_RECORD_KEY: ${{secrets.CYPRESS_RECORD_KEY}}

The above workflow runs on every commit and on every pull request. We still want to run all the tests for every commit pushed to the main branch. But for the pull requests, we want to run the modified specs first before running all tests. Thus I modify the above ci workflow to only run on commits pushed to the main branch.

1
2
3
4
5
6
7
8
name: ci
# run all tests on the main branch
on:
push:
branches:
- main
...
# rest of the workflow

You can find this workflow in .github/workflows/ci.yml. This workflow is tied to the README badge, showing the current test status of the project:

1
2
3
4
[![ci status][ci image]][ci url]

[ci image]: https://github.com/bahmutov/chat.io/workflows/ci/badge.svg?branch=main
[ci url]: https://github.com/bahmutov/chat.io/actions

The pull request workflow

I will use a separate workflow file for CI steps to run for the pull requests. You can find the finished workflow file at .github/workflows/pr.yml. Let's start by cloning the ci.yml and just modifying the on trigger.

.github/workflows/pr.yml
1
2
3
4
5
name: pr
# on pull request, determine changed or added Cypress specs
# if there are any (but not too many), run them first
# then run all Cypress specs
on: [pull_request]

First, we will need to check out the source code. Because we want to determine the files changed between the PR branch and the default main branch, we need to fetch this information. Thus I will use the parameter fetch-depth: 0 with actions/checkout action:

1
2
3
4
5
6
7
# https://github.com/actions/checkout
- name: Check out main branch ๐Ÿ›Ž
uses: actions/[email protected]
with:
# need to fetch info about all branches
# to determine the changed spec files
fetch-depth: 0

Here is how we can find all changed (added and modified) files between the current branch and the main branch

1
2
3
- name: List changed files ๐Ÿ—‚
# should we get the branch names from the PR?
run: git diff --name-only origin/main

For example, I have started a new branch example-branch and modified the spec rooms.js and added a new spec

Git local status

Let's commit and push this branch to the remote origin.

1
2
3
$ git add .
$ git commit -m "new spec and small tweaks"
$ git push -u

Even when working locally, we can see the changed files between the current branch example-branch and the main one.

1
2
3
$ git diff --name-only origin/main
cypress/integration/register-using-task2.js
cypress/integration/rooms.js

The pull request might have other modified files besides the Cypress specs. For example, I will touch the README file too. Here is how we can filter the specs

1
2
3
4
5
6
7
8
$ git diff --name-only origin/main
README.md
cypress/integration/register-using-task2.js
cypress/integration/rooms.js

$ git diff --name-only origin/main | grep cypress/integration
cypress/integration/register-using-task2.js
cypress/integration/rooms.js

Super. Later we will need to know the number of modified specs - we can use wc -l to count the lines with the modified Cypress specs

1
2
$ git diff --name-only origin/main | grep cypress/integration | wc -l
2

Ughh, why is there whitespace around 2, let's trim it

1
2
$ git diff --name-only origin/main | grep cypress/integration | wc -l | tr -d ' '
2

Now that we know the number of changed specs, let's also join them into a single string to be passed to the cypress run --spec ... parameter.

1
2
$ git diff --name-only origin/main | grep cypress/integration | tr '\n' ','
cypress/integration/register-using-task2.js,cypress/integration/rooms.js,

Super. We can compute the number and the spec parameter in the workflow, and even hide the details from other specs by using the output parameters.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
- name: List changed files ๐Ÿ—‚
# should we get the branch names from the PR?
run: git diff --name-only origin/main

- name: List changed specs โœจ
id: list-changed-specs
run: |
n=$(git diff --name-only origin/main | grep cypress/integration | wc -l | tr -d ' ')
specs=$(git diff --name-only origin/main | grep cypress/integration | tr '\n' ',')
echo "Changed and added Cypress specs"
echo ${specs}
echo "number of added or changed Cypress specs ${n}"

# output the number of specs and the specs list
echo "::set-output name=specsN::${n}"
echo "::set-output name=specs::${specs}"

Other workflow steps can access the number of changed specs using ${{ steps.list-changed-specs.outputs.specsN }} expression syntax. Let's set up two test jobs - the first one will run if there are changed specs, but not more than 5. If there are lots of modified specs, it makes sense to simply run all of them.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# https://github.com/cypress-io/github-action
- name: Run changed Cypress specs first ๐ŸŒฒ
# it makes sense to run changed specs only if there are a few
# otherwise just run all specs in the next step
if: ${{ steps.list-changed-specs.outputs.specsN > 0 && steps.list-changed-specs.outputs.specsN < 5 }}
uses: cypress-io/[email protected]
with:
start: npm start
wait-on: 'http://localhost:3000'
record: true
group: '1. Changed specs'
spec: ${{ steps.list-changed-specs.outputs.specs }}
env:
# for recording test results and videos to Cypress Dashboard
CYPRESS_RECORD_KEY: ${{secrets.CYPRESS_RECORD_KEY}}

# if the changed / added Cypress tests passed
# run all Cypress tests to confirm the app is working
- name: All Cypress tests ๐Ÿงช
uses: cypress-io/[email protected]
with:
# hmm, is the application running?
start: npm start
wait-on: 'http://localhost:3000'
record: true
group: '2. All Cypress tests'
env:
# for recording test results and videos to Cypress Dashboard
CYPRESS_RECORD_KEY: ${{secrets.CYPRESS_RECORD_KEY}}

We have two problems in the above workflow:

  • we install the NPM dependencies twice (potentially)
  • we are trying to run the application using npm start twice (potentially)

Thus we can optimize the workflow by installing the dependencies just once, and starting the application before running any Cypress tests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# install dependencies
- name: Install dependencies ๐Ÿ“ฆ
uses: cypress-io/[email protected]
with:
# just perform install
runTests: false

- name: Start the app ๐Ÿ
run: npm start &

# https://github.com/cypress-io/github-action
- name: Run changed Cypress specs first ๐ŸŒฒ
# it makes sense to run changed specs only if there are a few
# otherwise just run all specs in the next step
if: ${{ steps.list-changed-specs.outputs.specsN > 0 && steps.list-changed-specs.outputs.specsN < 5 }}
uses: cypress-io/[email protected]
with:
# we have already installed all dependencies above
install: false
# the server is running already, but just wait for it
wait-on: 'http://localhost:3000'
record: true
group: '1. Changed specs'
spec: ${{ steps.list-changed-specs.outputs.specs }}
env:
# for recording test results and videos to Cypress Dashboard
CYPRESS_RECORD_KEY: ${{secrets.CYPRESS_RECORD_KEY}}

- name: All Cypress tests ๐Ÿงช
uses: cypress-io/[email protected]
with:
# we have already installed all dependencies above
install: false
wait-on: 'http://localhost:3000'
record: true
group: '2. All Cypress tests'
env:
# for recording test results and videos to Cypress Dashboard
CYPRESS_RECORD_KEY: ${{secrets.CYPRESS_RECORD_KEY}}

Tip: Cypress GH Action can do a lot, find all examples in the cypress-io/github-action repo.

In action

Let's see how the above workflow performs. I have opened the pull request #12.

The GH workflow shows the steps, and that the modified specs task was executed

GitHub workflow

The list-changed-spec step has calculated the two changed Cypress test files correctly

The changed Cypress specs

The changed tests have finished successfully, while all tests have failed in an unrelated spec group-chat.js. Notice how fast the modified specs have finished vs waiting for all the tests: 30 seconds vs 5 minutes.

The recorded Dashboard run

The group-chat.js shows the test fails to log in the first user A

The failed test screenshot

Let's modify the group-chat.js - something goes wrong there, let's change the user name to be a userA instead of just A. Once I push the commit with the username change, the 3 changes specs run:

The three modified specs ran first

Our fix has solved the problem, and all the Cypress specs have passed. We are good to merge.

See also