How to Keep Cypress Tests in Another Repo While Using CircleCI

How to keep the tests in sync with the features, while keeping them in separate repositories and running CircleCI workflows.

Imagine you are developing a web application and deploying it to preview environments using Vercel. How do you run the tests that reside in a separate repo? This blog post teaches you how to trigger CircleCI workflows after deployment.

Note: I have written a similar blog post How to Keep Cypress Tests in Another Repo While Using GitHub Actions that shows the solution when using Netlify and GitHub Actions.

🎁 You can find the example application in the repository bahmutov/todomvc-no-tests-vercel and its end-to-end tests in the repository bahmutov/todomvc-tests-circleci. You can see the deployed application at https://todomvc-no-tests-vercel.vercel.app/ and see CircleCI workflows at https://app.circleci.com/pipelines/github/bahmutov/todomvc-tests-circleci.

The application

Every pull request opened in the bahmutov/todomvc-no-tests-vercel is automatically deployed to Vercel at the URL that follows the pattern

1
https://${VERCEL_PROJECT_NAME}-git-${GITHUB_HEAD_REF}-${VERCEL_TEAM_NAME}.vercel.app/

For example, pull request #3 from branch named pr3 shows the following Vercel comment after the deploy:

Vercel deployment comment

The tests

The developer would normally run the end-to-end tests against the application running locally. Thus the baseUrl in the cypress.json file points at the local app by default.

cypress.json
1
2
3
4
5
6
{
"baseUrl": "http://localhost:3000",
"fixturesFolder": false,
"supportFile": false,
"pluginsFile": false
}

I wrote a simple end-to-end test that you can find in cypress/integration folder.

spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
/// <reference types="cypress" />

it('works', () => {
cy.visit('/')
// application starts with 3 todos
cy.get('.todo').should('have.length', 3)
cy.get('[data-cy=new-todo]').type('Add tests!{enter}')
cy.get('.todo')
.should('have.length', 4)
.eq(3)
.should('include.text', 'Add tests!')

cy.contains('.todo', 'Learn about React')
.contains('[data-cy=complete]', 'Complete')
.click()
cy.contains('.todo', 'Learn about React').find('[data-cy=remove]').click()
cy.get('.todo').should('have.length', 3)
cy.contains('.todo', 'Learn about React').should('not.exist')

cy.screenshot('finished', { capture: 'runner' })
})

The screenshot finished.png

The tests workflow

We want to run our tests on CircleCI using Cypress CircleCI Orb and we want to trigger the pipeline using CircleCI API. To pass the URL to test, we can use pipeline parameters. The workflow file is shown below. I like having a separate "info" just just to print the received pipeline parameters.

.circleci/config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# to use orbs, must use version >= 2.1
version: 2.1
orbs:
# import Cypress orb by specifying an exact version x.y.z
# or the latest version 1.x.x using "@1" syntax
# https://github.com/cypress-io/circleci-orb
cypress: cypress-io/[email protected]

parameters:
TEST_BRANCH:
type: string
default: 'main'
# by default, test the production deployment
TEST_URL:
type: string
default: 'https://todomvc-no-tests-vercel.vercel.app/'

jobs:
info:
machine:
image: ubuntu-2004:202104-01
steps:
- run:
name: print variables
command: |
echo "TEST_BRANCH is << pipeline.parameters.TEST_BRANCH >>"
echo "TEST_URL is << pipeline.parameters.TEST_URL >>"

# preview deploys might take a little bit to be ready
# this job pings the TEST_URL to check if the deployment has finished
wait-for-deploy:
machine:
image: ubuntu-2004:202104-01
steps:
# we don't really need to check TEST_URL,
# since we only run the entire workflow when it is present
# but I like to remember how to use the Circle halt command
- unless:
condition: << pipeline.parameters.TEST_URL >>
steps:
# https://circleci.com/docs/2.0/configuration-reference/#ending-a-job-from-within-a-step
- run: circleci-agent step halt
- run:
name: wait for deployment
command: |
echo "Using wait-on to check if the URL << pipeline.parameters.TEST_URL >> responds"
echo "See https://www.npmjs.com/package/wait-on"
npx wait-on --verbose \
--interval 10000 --timeout 60000 \
<< pipeline.parameters.TEST_URL >>

workflows:
e2e:
# only run the workflow when TEST_URL is set
when: << pipeline.parameters.TEST_URL >>
jobs:
- info
- wait-for-deploy

I am including a job "wait-for-deploy" to ping the TEST_URL every 10 seconds until it responds. When this job finishes successfully, the preview deploy is ready to be tested. Let's add another job to the workflow to run after wait-for-deploy is done.

1
2
3
4
5
6
7
8
9
10
11
12
13
jobs:
- info
- wait-for-deploy

- cypress/run:
name: Cypress E2E tests
requires:
- wait-for-deploy
config: 'baseUrl=<< pipeline.parameters.TEST_URL >>'
# save videos and screenshots on Circle as artifacts
store_artifacts: true
# we do not need to save the workspace after the tests are done
no-workspace: true

We are using the job run defined in the imported orb cypress: cypress-io/[email protected] and control it via parameters like config and store_artifacts. Let's run this workflow on CircleCI.

The finished CircleCI workflow

You can click on the "Cypress E2E tests" job to see the stored test artifacts: movies and screenshots.

The stored test artifacts

Tip: while storing test artifacts is possible, Cypress Dashboard does a much better job showing them.

Tip 2: it is easy to mess up YML CI configuration syntax. Luckily, you can use CircleCI CLI utility to validate the config file syntax before pushing the code to the remote repository.

1
2
3
4
5
6
7
$ circleci config validate .circleci/config.yml
Error: Error calling workflow: 'e2e'
Error calling job: 'test'
Unknown variable(s): TEST_BRANCH
# fix the syntax error and verify again
$ circleci config validate .circleci/config.yml
Config file at .circleci/config.yml is valid.

Trigger CircleCI pipeline

Let's get back to the application repository. We need to trigger the testing pipeline. The first solution is to use a GitHub Actions workflow.

.github/workflows/pr.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# every time we open a pull request, or a commit is pushed to it
# Vercel deploys the site to a preview environment
name: pull
on: pull_request
jobs:
trigger-tests:
runs-on: ubuntu-20.04
steps:
# trigger CircleCI pipeline to run E2E tests
# https://circleci.com/docs/api/v2/#operation/triggerPipeline

# the preview URL follows the format:
# https://<project name>-git-<branch name>-<team name>.vercel.app/
# https://docs.github.com/en/actions/reference/environment-variables#default-environment-variables
- name: Trigger CircleCI
run: |
export VERCEL_PROJECT_NAME=todomvc-no-tests-vercel
export VERCEL_TEAM_NAME=gleb-bahmutov
export PREVIEW_URL=https://${VERCEL_PROJECT_NAME}-git-${GITHUB_HEAD_REF}-${VERCEL_TEAM_NAME}.vercel.app/
echo "Vercel deployment URL is ${PREVIEW_URL}"

# --silent option to not print request progress
curl -u ${{ secrets.CIRCLE_CI_API_TOKEN }}: \
--silent \
--data parameters[TEST_BRANCH]=${GITHUB_HEAD_REF} \
--data parameters[TEST_URL]=${PREVIEW_URL} \
https://circleci.com/api/v2/project/gh/bahmutov/todomvc-tests-circleci/pipeline

We will need a project or a personal CircleCI API token to trigger the pipeline. We can store it privately using GitHub Actions secrets tab. The parameters fields will be set as pipeline params on CircleCI. We are passing the current branch name and Vercel PR preview URL we have formed ourselves.

Trigger CircleCI pipeline from GitHub Actions

The trigger works. We can look at the list of pipelines to see the pipeline #13. The triggered pipelines do not have a commit message.

Pipeline #13 on CircleCI

The info job shows the parameters passed from GitHub.

Pipeline parameters printed by the "info" job

The screenshot image in the test artifacts in the "Cypress E2E tests" job shows the preview URL was tested.

Cypress tests ran against the preview environment

Trigger the tests on the different branch

When developing the application feature, the programmer probably has updated or new tests in the corresponding branch in the test repo. Thus when testing the pull request against the branch feature-X we want to check out a branch with the same name from the test repo before running tests. Here is how we can do this via reusable CircleCI commands and Cypress Orb post-checkout parameter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
# to use orbs, must use version >= 2.1
version: 2.1
orbs:
# import Cypress orb by specifying an exact version x.y.z
# or the latest version 1.x.x using "@1" syntax
cypress: cypress-io/[email protected]

parameters:
# by default test everything using the current default branch
# but if the pipeline is triggered via API, you can pass the branch name
# to check out and run tests from.
TEST_BRANCH:
type: string
default: 'main'

# https://circleci.com/docs/2.0/reusing-config/#authoring-reusable-commands
commands:
switch_branch:
description: |
Changes the current branch to the latest commit on the specific branch.
NOTE: if the branch does not exist, does nothing.
parameters:
BRANCH_NAME:
type: string
default: ''
steps:
- when:
condition: << parameters.BRANCH_NAME >>
steps:
- run:
name: Checkout branch << parameters.BRANCH_NAME >>
command: |
echo "Switching to branch << parameters.BRANCH_NAME >> if it exists"
git checkout << parameters.BRANCH_NAME >> || true
git pull origin << parameters.BRANCH_NAME >> || true
print_git:
description: |
Prints the current git branch and the commit hash.
steps:
- run:
name: Print current Git info
# looks like Cypress default executor does not have
# a very recent Git version, thus we cannot use "--show-current"
command: |
echo "current branch is: $(git branch -a)"
echo "current commit is: $(git rev-parse --short HEAD)"

workflows:
e2e:
jobs:
- cypress/run:
name: Cypress E2E tests
# switch to the test branch before installing and running tests
post-checkout:
- switch_branch:
BRANCH_NAME: << pipeline.parameters.TEST_BRANCH >>
- print_git
# we do not need to keep the workspace around
# since there are no other jobs that depend on it
no-workspace: true

Note: the above fragment comes from the repository bahmutov/circleci-checkout-experiment where I experimented with CircleCI to ensure this specific way of running tests from the test branch works.

Shortcomings

The approach to testing the preview deployments described above works, but has a bad drawback. It tests the branch preview URL, and not the individual deploys. Imagine the pull request has several commits, each triggering a test run. Imagine that building and deploying the preview URL takes 5 minutes, while the tests only take 1 minute. The sequence of events can be see in the table below.

wall clock event
00:00 very first deploy to the branch "my-feature" with commit A
00:01 open pull request "merge-my-feature" from branch "my-feature"
00:01 Vercel starts building and deploying "https://merge-my-feature..." preview
00:01 GitHub Actions trigger CircleCI pipeline with TEST_URL=https://merge-my-feature...
00:01 Job wait-for-deploy starts pinging "https://merge-my-feature..." url
00:06 Vercel deploys "https://merge-my-feature..." url
00:06 Job wait-for-deploy finishes after receiving 200 response from "https://merge-my-feature..." url
00:06 Job Cypress E2E tests runs tests against the "https://merge-my-feature..." url
00:10 user makes another push to the branch "my-feature" with commit B
00:10 Vercel starts building and deploying "https://merge-my-feature..." preview
00:10 GitHub Actions trigger CircleCI pipeline with TEST_URL=https://merge-my-feature...
00:10 Job wait-for-deploy starts pinging "https://merge-my-feature..." url.
00:10 Job wait-for-deploy finishes almost immediately because the url responds. The preview still has commit A code.
00:10 Job Cypress E2E tests runs tests against the "https://merge-my-feature..." url
00:16 Vercel deploys "https://merge-my-feature..." url, but nothing tests the deployed commit B code

If we trigger the CircleCI pipeline immediately from GitHub Actions, then the second commit will trigger running the tests. The tests will start quickly because the job wait-for-deploy hits the already deployed branch preview URL, the tests pass. Then the Vercel preview happens - and it never gets tested!

Thus we need something better - we need to trigger the CircleCI pipeline after Vercel successfully deploys.

The deployment event

Luckily, Vercel GitHub integration delivers a deployment_status event as I described in Test the Preview Vercel Deploys. Let's first look at the events delivered by Vercel to GitHub Actions.

info.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
# print info from deployment events sent by Vercel
# https://glebbahmutov.com/blog/develop-preview-test/
name: info
# https://docs.github.com/en/actions/reference/events-that-trigger-workflows
on: [deployment, deployment_status]
jobs:
show-event:
runs-on: ubuntu-20.04
steps:
- name: Dump GitHub context
env:
GITHUB_CONTEXT: ${{ toJson(github) }}
run: echo "$GITHUB_CONTEXT"

The deployment_status event is delivered twice: first with the status pending, then with the status success. The event also has the unique URL of that deploy - because Vercel does immutable deploys.

The deployment_status event has immutable deployment URL

Thus we can trigger the CircleCI pipelines using a unique URL after the deployment has finished, see deploy.yml

.github/workflows/deploy.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
name: deploy
on: deployment_status
jobs:
trigger-tests-after-deploy:
# only runs this job on successful deploy
# https://glebbahmutov.com/blog/develop-preview-test/
if: github.event.deployment_status.state == 'success'
runs-on: ubuntu-20.04
steps:
# trigger CircleCI pipeline to run E2E tests
# https://circleci.com/docs/api/v2/#operation/triggerPipeline
- name: Trigger CircleCI
run: |
echo "Vercel unique deployment URL is ${{ github.event.deployment_status.target_url }}"

# --silent option to not print request progress
curl -u ${{ secrets.CIRCLE_CI_API_TOKEN }}: \
--silent \
--data parameters[TEST_BRANCH]=${GITHUB_HEAD_REF} \
--data parameters[TEST_URL]=${{ github.event.deployment_status.target_url }} \
https://circleci.com/api/v2/project/gh/bahmutov/todomvc-tests-circleci/pipeline

The above GitHub Action workflow triggers the CircleCI pipeline with unique URL which you can see in the test screenshot.

The deployment preview URL

Merging the pull requests

When we modify the code and the tests in two repos, we have two open pull requests. Which one do we merge first? It is a little bit of a chicken and an egg problem.

  1. We cannot merge the application pull request first - if it runs the tests before we merge the pull request in the test repo, the tests will fail, since they are still original tests.
  2. We cannot merge the test pull request first, since the application code is still the original source, and not what the tests expect to see.

We can try to time it and merge the code first, then while it is building merge the test pull request, hoping it would hit the the deployed updated application. But I would suggest a simpler approach.

  1. Merge the tests first, but skip the build using the [skip ci] text in the commit subject, see CircleCI docs for example. In the screenshot below I am squashing 3 commits in the test repo into a single commit that should not trigger the tests.

Merging the test pull request first while skipping the CI build

The CircleCI shows the test commit was noted, but did not trigger the workflow.

CircleCI did not build the commit with the message "[skip ci]"

  1. Merge the code change pull request. It will trigger the tests that are now match the code.

Problem solved.

The remaining problem

Our implementation is almost perfect. The tests are triggered correctly, the deployment URLs are unique - but we cannot pass the branch to the CircleCI pipeline! The TEST_BRANCH parameter is empty because the deployment_status event has no "memory" of the branch that has triggered the deployment, and the GitHub environment has no GITHUB_HEAD_REF set - because GH does not "know" which branch you are testing or deploying.

I wish Vercel included this information in the event details, since their system knows which branch has been deployed. We could use this information to checkout or trigger the right branch in the test repository. Then we could modify the application in branch feature-X while updating the tests in the separate repo using the same branch name feature-X. When the preview URL has been deployed we would run the tests from branch feature-X.

Workaround

The Vercel deployment event has the commit SHA. If you dump the GitHub event object, check the sha property:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"token": "***",
"job": "show-event",
"ref": "",
"sha": "7202eca208589fabacb2f35ac5d8dd46fcb8a12f",
"repository": "bahmutov/todomvc-no-tests-vercel",
"repository_owner": "bahmutov",
"repositoryUrl": "git://github.com/bahmutov/todomvc-no-tests-vercel.git",
"run_id": "1004603109",
"run_number": "24",
"retention_days": "90",
"actor": "vercel[bot]",
"workflow": "info",
"head_ref": "",
"base_ref": "",
"event_name": "deployment_status",
...
}

We can check out the full repository and find the branch name the commit belongs to (assuming the pull request commit really belongs to one branch)

1
2
3
echo "Deployed commit ${{ github.sha }}"
export BRANCH_NAME=$(git show -s --pretty=%D HEAD | tr -s ',' '\n' | sed 's/^ //' | grep -e 'origin/' | head -1 | sed 's/\origin\///g')
echo "Deployed branch ${BRANCH_NAME}"

See also