How to Keep Cypress Tests in Another Repo While Using CircleCI

How to keep the tests in sync with the features, while keeping them in separate repositories and running CircleCI workflows.

Imagine you are developing a web application and deploying it to preview environments using Vercel. How do you run the tests that reside in a separate repo? This blog post teaches you how to trigger CircleCI workflows after deployment.

Note: I have written a similar blog post How to Keep Cypress Tests in Another Repo While Using GitHub Actions that shows the solution when using Netlify and GitHub Actions.

🎁 You can find the example application in the repository bahmutov/todomvc-no-tests-vercel and its end-to-end tests in the repository bahmutov/todomvc-tests-circleci. You can see the deployed application at https://todomvc-no-tests-vercel.vercel.app/ and see CircleCI workflows at https://app.circleci.com/pipelines/github/bahmutov/todomvc-tests-circleci. You can find the recorded tests on Cypress Dashboard here.

The application

Every pull request opened in the bahmutov/todomvc-no-tests-vercel is automatically deployed to Vercel at the URL that follows the pattern

1
https://${VERCEL_PROJECT_NAME}-git-${GITHUB_HEAD_REF}-${VERCEL_TEAM_NAME}.vercel.app/

For example, pull request #3 from branch named pr3 shows the following Vercel comment after the deploy:

Vercel deployment comment

The tests

The developer would normally run the end-to-end tests against the application running locally. Thus the baseUrl in the cypress.json file points at the local app by default.

cypress.json
1
2
3
4
5
6
{
"baseUrl": "http://localhost:3000",
"fixturesFolder": false,
"supportFile": false,
"pluginsFile": false
}

I wrote a simple end-to-end test that you can find in cypress/integration folder.

spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
/// <reference types="cypress" />

it('works', () => {
cy.visit('/')
// application starts with 3 todos
cy.get('.todo').should('have.length', 3)
cy.get('[data-cy=new-todo]').type('Add tests!{enter}')
cy.get('.todo')
.should('have.length', 4)
.eq(3)
.should('include.text', 'Add tests!')

cy.contains('.todo', 'Learn about React')
.contains('[data-cy=complete]', 'Complete')
.click()
cy.contains('.todo', 'Learn about React').find('[data-cy=remove]').click()
cy.get('.todo').should('have.length', 3)
cy.contains('.todo', 'Learn about React').should('not.exist')

cy.screenshot('finished', { capture: 'runner' })
})

The screenshot finished.png

The tests workflow

We want to run our tests on CircleCI using Cypress CircleCI Orb and we want to trigger the pipeline using CircleCI API. To pass the URL to test, we can use pipeline parameters. The workflow file is shown below. I like having a separate "info" just just to print the received pipeline parameters.

.circleci/config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# to use orbs, must use version >= 2.1
version: 2.1
orbs:
# import Cypress orb by specifying an exact version x.y.z
# or the latest version 1.x.x using "@1" syntax
# https://github.com/cypress-io/circleci-orb
cypress: cypress-io/[email protected]

parameters:
TEST_BRANCH:
type: string
default: 'main'
# by default, test the production deployment
TEST_URL:
type: string
default: 'https://todomvc-no-tests-vercel.vercel.app/'

jobs:
info:
machine:
image: ubuntu-2004:202104-01
steps:
- run:
name: print variables
command: |
echo "TEST_BRANCH is << pipeline.parameters.TEST_BRANCH >>"
echo "TEST_URL is << pipeline.parameters.TEST_URL >>"

# preview deploys might take a little bit to be ready
# this job pings the TEST_URL to check if the deployment has finished
wait-for-deploy:
machine:
image: ubuntu-2004:202104-01
steps:
# we don't really need to check TEST_URL,
# since we only run the entire workflow when it is present
# but I like to remember how to use the Circle halt command
- unless:
condition: << pipeline.parameters.TEST_URL >>
steps:
# https://circleci.com/docs/2.0/configuration-reference/#ending-a-job-from-within-a-step
- run: circleci-agent step halt
- run:
name: wait for deployment
command: |
echo "Using wait-on to check if the URL << pipeline.parameters.TEST_URL >> responds"
echo "See https://www.npmjs.com/package/wait-on"
npx wait-on --verbose \
--interval 10000 --timeout 60000 \
<< pipeline.parameters.TEST_URL >>

workflows:
e2e:
# only run the workflow when TEST_URL is set
when: << pipeline.parameters.TEST_URL >>
jobs:
- info
- wait-for-deploy

I am including a job "wait-for-deploy" to ping the TEST_URL every 10 seconds until it responds. When this job finishes successfully, the preview deploy is ready to be tested. Let's add another job to the workflow to run after wait-for-deploy is done.

1
2
3
4
5
6
7
8
9
10
11
12
13
jobs:
- info
- wait-for-deploy

- cypress/run:
name: Cypress E2E tests
requires:
- wait-for-deploy
config: 'baseUrl=<< pipeline.parameters.TEST_URL >>'
# save videos and screenshots on Circle as artifacts
store_artifacts: true
# we do not need to save the workspace after the tests are done
no-workspace: true

We are using the job run defined in the imported orb cypress: cypress-io/[email protected] and control it via parameters like config and store_artifacts. Let's run this workflow on CircleCI.

The finished CircleCI workflow

You can click on the "Cypress E2E tests" job to see the stored test artifacts: movies and screenshots.

The stored test artifacts

Tip: while storing test artifacts is possible, Cypress Dashboard does a much better job showing them.

Tip 2: it is easy to mess up YML CI configuration syntax. Luckily, you can use CircleCI CLI utility to validate the config file syntax before pushing the code to the remote repository.

1
2
3
4
5
6
7
$ circleci config validate .circleci/config.yml
Error: Error calling workflow: 'e2e'
Error calling job: 'test'
Unknown variable(s): TEST_BRANCH
# fix the syntax error and verify again
$ circleci config validate .circleci/config.yml
Config file at .circleci/config.yml is valid.

Trigger CircleCI pipeline

Let's get back to the application repository. We need to trigger the testing pipeline. The first solution is to use a GitHub Actions workflow.

.github/workflows/pr.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# every time we open a pull request, or a commit is pushed to it
# Vercel deploys the site to a preview environment
name: pull
on: pull_request
jobs:
trigger-tests:
runs-on: ubuntu-20.04
steps:
# trigger CircleCI pipeline to run E2E tests
# https://circleci.com/docs/api/v2/#operation/triggerPipeline

# the preview URL follows the format:
# https://<project name>-git-<branch name>-<team name>.vercel.app/
# https://docs.github.com/en/actions/reference/environment-variables#default-environment-variables
- name: Trigger CircleCI
run: |
export VERCEL_PROJECT_NAME=todomvc-no-tests-vercel
export VERCEL_TEAM_NAME=gleb-bahmutov
export PREVIEW_URL=https://${VERCEL_PROJECT_NAME}-git-${GITHUB_HEAD_REF}-${VERCEL_TEAM_NAME}.vercel.app/
echo "Vercel deployment URL is ${PREVIEW_URL}"

# --silent option to not print request progress
curl -u ${{ secrets.CIRCLE_CI_API_TOKEN }}: \
--silent \
--data parameters[TEST_BRANCH]=${GITHUB_HEAD_REF} \
--data parameters[TEST_URL]=${PREVIEW_URL} \
https://circleci.com/api/v2/project/gh/bahmutov/todomvc-tests-circleci/pipeline

We will need a project or a personal CircleCI API token to trigger the pipeline. We can store it privately using GitHub Actions secrets tab. The parameters fields will be set as pipeline params on CircleCI. We are passing the current branch name and Vercel PR preview URL we have formed ourselves.

Trigger CircleCI pipeline from GitHub Actions

The trigger works. We can look at the list of pipelines to see the pipeline #13. The triggered pipelines do not have a commit message.

Pipeline #13 on CircleCI

The info job shows the parameters passed from GitHub.

Pipeline parameters printed by the "info" job

The screenshot image in the test artifacts in the "Cypress E2E tests" job shows the preview URL was tested.

Cypress tests ran against the preview environment

Trigger the tests on the different branch

When developing the application feature, the programmer probably has updated or new tests in the corresponding branch in the test repo. Thus when testing the pull request against the branch feature-X we want to check out a branch with the same name from the test repo before running tests. Here is how we can do this via reusable CircleCI commands and Cypress Orb post-checkout parameter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
# to use orbs, must use version >= 2.1
version: 2.1
orbs:
# import Cypress orb by specifying an exact version x.y.z
# or the latest version 1.x.x using "@1" syntax
cypress: cypress-io/[email protected]

parameters:
# by default test everything using the current default branch
# but if the pipeline is triggered via API, you can pass the branch name
# to check out and run tests from.
TEST_BRANCH:
type: string
default: 'main'

# https://circleci.com/docs/2.0/reusing-config/#authoring-reusable-commands
commands:
switch_branch:
description: |
Changes the current branch to the latest commit on the specific branch.
NOTE: if the branch does not exist, does nothing.
parameters:
BRANCH_NAME:
type: string
default: ''
steps:
- when:
condition: << parameters.BRANCH_NAME >>
steps:
- run:
name: Checkout branch << parameters.BRANCH_NAME >>
command: |
echo "Switching to branch << parameters.BRANCH_NAME >> if it exists"
git checkout << parameters.BRANCH_NAME >> || true
git pull origin << parameters.BRANCH_NAME >> || true
print_git:
description: |
Prints the current git branch and the commit hash.
steps:
- run:
name: Print current Git info
# looks like Cypress default executor does not have
# a very recent Git version, thus we cannot use "--show-current"
command: |
echo "current branch is: $(git branch -a)"
echo "current commit is: $(git rev-parse --short HEAD)"

workflows:
e2e:
jobs:
- cypress/run:
name: Cypress E2E tests
# switch to the test branch before installing and running tests
post-checkout:
- switch_branch:
BRANCH_NAME: << pipeline.parameters.TEST_BRANCH >>
- print_git
# we do not need to keep the workspace around
# since there are no other jobs that depend on it
no-workspace: true

Note: the above fragment comes from the repository bahmutov/circleci-checkout-experiment where I experimented with CircleCI to ensure this specific way of running tests from the test branch works.

Shortcomings

The approach to testing the preview deployments described above works, but has a bad drawback. It tests the branch preview URL, and not the individual deploys. Imagine the pull request has several commits, each triggering a test run. Imagine that building and deploying the preview URL takes 5 minutes, while the tests only take 1 minute. The sequence of events can be see in the table below.

wall clock event
00:00 very first deploy to the branch "my-feature" with commit A
00:01 open pull request "merge-my-feature" from branch "my-feature"
00:01 Vercel starts building and deploying "https://merge-my-feature..." preview
00:01 GitHub Actions trigger CircleCI pipeline with TEST_URL=https://merge-my-feature...
00:01 Job wait-for-deploy starts pinging "https://merge-my-feature..." url
00:06 Vercel deploys "https://merge-my-feature..." url
00:06 Job wait-for-deploy finishes after receiving 200 response from "https://merge-my-feature..." url
00:06 Job Cypress E2E tests runs tests against the "https://merge-my-feature..." url
00:10 user makes another push to the branch "my-feature" with commit B
00:10 Vercel starts building and deploying "https://merge-my-feature..." preview
00:10 GitHub Actions trigger CircleCI pipeline with TEST_URL=https://merge-my-feature...
00:10 Job wait-for-deploy starts pinging "https://merge-my-feature..." url.
00:10 Job wait-for-deploy finishes almost immediately because the url responds. The preview still has commit A code.
00:10 Job Cypress E2E tests runs tests against the "https://merge-my-feature..." url
00:16 Vercel deploys "https://merge-my-feature..." url, but nothing tests the deployed commit B code

If we trigger the CircleCI pipeline immediately from GitHub Actions, then the second commit will trigger running the tests. The tests will start quickly because the job wait-for-deploy hits the already deployed branch preview URL, the tests pass. Then the Vercel preview happens - and it never gets tested!

Thus we need something better - we need to trigger the CircleCI pipeline after Vercel successfully deploys.

The deployment event

Luckily, Vercel GitHub integration delivers a deployment_status event as I described in Test the Preview Vercel Deploys. Let's first look at the events delivered by Vercel to GitHub Actions.

info.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
# print info from deployment events sent by Vercel
# https://glebbahmutov.com/blog/develop-preview-test/
name: info
# https://docs.github.com/en/actions/reference/events-that-trigger-workflows
on: [deployment, deployment_status]
jobs:
show-event:
runs-on: ubuntu-20.04
steps:
- name: Dump GitHub context
env:
GITHUB_CONTEXT: ${{ toJson(github) }}
run: echo "$GITHUB_CONTEXT"

The deployment_status event is delivered twice: first with the status pending, then with the status success. The event also has the unique URL of that deploy - because Vercel does immutable deploys.

The deployment_status event has immutable deployment URL

Thus we can trigger the CircleCI pipelines using a unique URL after the deployment has finished, see deploy.yml

.github/workflows/deploy.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
name: deploy
on: deployment_status
jobs:
trigger-tests-after-deploy:
# only runs this job on successful deploy
# https://glebbahmutov.com/blog/develop-preview-test/
if: github.event.deployment_status.state == 'success'
runs-on: ubuntu-20.04
steps:
# trigger CircleCI pipeline to run E2E tests
# https://circleci.com/docs/api/v2/#operation/triggerPipeline
- name: Trigger CircleCI
run: |
echo "Vercel unique deployment URL is ${{ github.event.deployment_status.target_url }}"

# --silent option to not print request progress
curl -u ${{ secrets.CIRCLE_CI_API_TOKEN }}: \
--silent \
--data parameters[TEST_BRANCH]=${GITHUB_HEAD_REF} \
--data parameters[TEST_URL]=${{ github.event.deployment_status.target_url }} \
https://circleci.com/api/v2/project/gh/bahmutov/todomvc-tests-circleci/pipeline

The above GitHub Action workflow triggers the CircleCI pipeline with unique URL which you can see in the test screenshot.

The deployment preview URL

Merging the pull requests

When we modify the code and the tests in two repos, we have two open pull requests. Which one do we merge first? It is a little bit of a chicken and an egg problem.

  1. We cannot merge the application pull request first - if it runs the tests before we merge the pull request in the test repo, the tests will fail, since they are still original tests.
  2. We cannot merge the test pull request first, since the application code is still the original source, and not what the tests expect to see.

We can try to time it and merge the code first, then while it is building merge the test pull request, hoping it would hit the the deployed updated application. But I would suggest a simpler approach.

  1. Merge the tests first, but skip the build using the [skip ci] text in the commit subject, see CircleCI docs for example. In the screenshot below I am squashing 3 commits in the test repo into a single commit that should not trigger the tests.

Merging the test pull request first while skipping the CI build

The CircleCI shows the test commit was noted, but did not trigger the workflow.

CircleCI did not build the commit with the message "[skip ci]"

  1. Merge the code change pull request. It will trigger the tests that are now match the code.

Problem solved.

The remaining problem

Our implementation is almost perfect. The tests are triggered correctly, the deployment URLs are unique - but we cannot pass the branch to the CircleCI pipeline! The TEST_BRANCH parameter is empty because the deployment_status event has no "memory" of the branch that has triggered the deployment, and the GitHub environment has no GITHUB_HEAD_REF set - because GH does not "know" which branch you are testing or deploying.

I wish Vercel included this information in the event details, since their system knows which branch has been deployed. We could use this information to checkout or trigger the right branch in the test repository. Then we could modify the application in branch feature-X while updating the tests in the separate repo using the same branch name feature-X. When the preview URL has been deployed we would run the tests from branch feature-X.

Workaround

The Vercel deployment event has the commit SHA. If you dump the GitHub event object, check the sha property:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"token": "***",
"job": "show-event",
"ref": "",
"sha": "7202eca208589fabacb2f35ac5d8dd46fcb8a12f",
"repository": "bahmutov/todomvc-no-tests-vercel",
"repository_owner": "bahmutov",
"repositoryUrl": "git://github.com/bahmutov/todomvc-no-tests-vercel.git",
"run_id": "1004603109",
"run_number": "24",
"retention_days": "90",
"actor": "vercel[bot]",
"workflow": "info",
"head_ref": "",
"base_ref": "",
"event_name": "deployment_status",
...
}

We can check out the full repository and find the branch name the commit belongs to (assuming the pull request commit really belongs to one branch)

1
2
3
echo "Deployed commit ${{ github.sha }}"
export BRANCH_NAME=$(git show -s --pretty=%D HEAD | tr -s ',' '\n' | sed 's/^ //' | grep -e 'origin/' | head -1 | sed 's/\origin\///g')
echo "Deployed branch ${BRANCH_NAME}"

Update 1: record tests on Cypress Dashboard

Cypress Dashboard is very useful for showing the results of tests and quickly diagnosing the failed ones. I have set up the test recording, you can see the run results here. The CircleCI cypress/run job gets a few extra parameters:

1
2
3
4
5
6
7
8
- cypress/run:
name: Cypress E2E tests
...
# record test results on Cypress Dashboard
record: true
group: e2e
# tag the recording with the branch name to make it easier to find
tags: '<< pipeline.parameters.TEST_BRANCH >>'

We can include the results Markdown badge in both the tests repo's README, and in in the TodoMVC application's README file.

Recorded test runs

Note: when recording test results on Cypress Dashboard, I usually disable storing the videos and screenshots as test artifacts on CI, since it is no longer useful.

Update 2: add the Cypress GitHub integration

After recording tests on Cypress Dashboard, I have installed Cypress GitHub Integration App in the test repo. You can give the app access to all your repos, or just select ones.

Cypress GH App repo access

🔐 Note that the app only needs read and write access to the commit statuses and pull requests, no source code access. For more, read What does Cypress record? and Cypress Security page.

Once we gave the Cypress GH app access to our repository, we can link the Dashboard project back to the repository and enable pull request comments and status checks.

Enable the Cypress GH integration PR checks

Let's work on a new application feature - let's change a selector for the "remove todo" button.

1
2
- <button data-cy="remove" ...>x</button>
+ <button data-cy="destroy" ...>x</button>

We have opened a new pull request in the application repo with this change. The status checks on the application repo are all green - because our GitHub integration is connected to the separate repo. On the separate repo we see a status check on the last commit to the main branch - because that's the commit the CircleCI pipeline has run against when triggered.

A new commit status check appears in the test repo

Let's open a repo with the same branch name as the branch in the application.

1
2
$ git checkout -b rename-attribute
Switched to a new branch 'rename-attribute'

We can update the test to pass against the updated application.

1
2
- cy.contains('.todo', 'Learn about React').find('[data-cy=remove]').click()
+ cy.contains('.todo', 'Learn about React').find('[data-cy=destroy]').click()

But now we have a problem - we need to trigger the pipeline with the tests on the branch rename-attribute in the todomvc-tests-circleci repo in order for the Cypress GH Integration to tie the test results to the right pull request. We can trigger the pipeline using the branch name, plus we can use a fallback - if there is no branch with the given test name, trigger the default pipeline. Our deployment status GitHub Actions workflow is thus:

.github/workflows/deploy.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
...
# trigger CircleCI pipeline to run E2E tests
# https://circleci.com/docs/api/v2/#operation/triggerPipeline
- name: Trigger CircleCI
run: |
echo "Vercel unique deployment URL is ${{ github.event.deployment_status.target_url }}"

echo "Deployed commit ${{ github.sha }}"
export BRANCH_NAME=$(git show -s --pretty=%D HEAD | tr -s ',' '\n' | sed 's/^ //' | grep -e 'origin/' | head -1 | sed 's/\origin\///g')
echo "Deployed branch ${BRANCH_NAME}"

export TEST_PIPELINE_URL=https://circleci.com/api/v2/project/gh/bahmutov/todomvc-tests-circleci/pipeline
# --silent option to not print request progress
# if the test repo does not have branch with the same name
# trigger the default branch pipeline
curl -u ${{ secrets.CIRCLE_CI_API_TOKEN }}: \
--silent \
--data branch=${BRANCH_NAME} \
--data parameters[TEST_BRANCH]=${BRANCH_NAME} \
--data parameters[TEST_URL]=${{ github.event.deployment_status.target_url }} \
${TEST_PIPELINE_URL} || \
curl -u ${{ secrets.CIRCLE_CI_API_TOKEN }}: \
--silent \
--data parameters[TEST_BRANCH]=${BRANCH_NAME} \
--data parameters[TEST_URL]=${{ github.event.deployment_status.target_url }} \
${TEST_PIPELINE_URL}

Let's say a test fails, because the application renamed the data-cy attribute back to remove. Then the pull request #3 shows the test results and the details for the failed test, including a screenshot thumbnail and a link to the test result on the Dashboard.

Failed status checks

Cypress comments with results on the PR

My favorite part in all of this, is clicking on the failed test's thumbnail, seeing the application screenshot at the moment of failure, then inspecting the test history to see if the test was modified recently (it was).

Going from the PR comment to the test screenshot and history

Maybe we need to change it back to make it work... I have changed the test command back to .find('[data-cy=remove]') and committed the test while skipping the CI.

1
2
$ g done "change test back [skip ci]"
$ git push

Let's re-run the CircleCI pipeline - but we can re-run the failed E2E test job only.

Re-running the failed E2E test job in the pipeline

The updated test fixes the pipeline and the test comment is updated.

The Cypress tests have been fixed

Again, we can click on the comment to inspect the test run, including the test change history.

The test has been updated twice

Update 3: trigger the CircleCI pipeline correctly

After experimenting with triggering the CircleCI pipeline, I found that using curl with the fallback branch is really tricky. If the branch is not found, the request fails but the status code is 200, since curl receives an object. To make it robust I have written a little utility trigger-circleci-pipeline that you can use to trigger a pipeline run on a given branch with fallback to the default branch.

1
2
3
4
5
6
7
# assuming the environment variable CIRCLE_CI_API_TOKEN
# has your personal CircleCI token, trigger a workflow
# in the CircleCI project bahmutov/todomvc-tests-circleci
# and pass pipeline parameters
$ npx trigger-circleci-pipeline \
--org bahmutov --project todomvc-tests-circleci --branch ${BRANCH_NAME} \
--parameters TEST_URL=${TEST_URL},TEST_BRANCH=${BRANCH_NAME}

See also