process

Test the Preview Vercel Deploys

Run end-to-end Cypress tests against Vercel preview deploys using GitHub Actions

The site

Let's play with a personal site made using 11ty. You can find the source code in the repo bahmutov/eleventy-example. We start with a basic page in the README.md file.

README.md
1
2
# My site
> A static site using [11ty](https://www.11ty.dev/)

Install the 11ty NPM package

1
2
$ npm i -D @11ty/eleventy
+ @11ty/[email protected]

And start the local site

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ npx eleventy --serve
Writing _site/README/index.html from ./README.md.
Writing _site/index.html from ./index.html.
Wrote 2 files in 0.08 seconds (v0.11.0)
Watching…
[Browsersync] Access URLs:
 -----------------------------------
       Local: http://localhost:8081
    External: http://10.0.0.141:8081
 -----------------------------------
          UI: http://localhost:3001
 UI External: http://localhost:3001
 -----------------------------------
[Browsersync] Serving files from: _site

The static page does not look like much - but the generator is fast and simple to use.

README page

Tip: by default eleventy --start serves the page at port 8080. If that port is busy, it automatically serves at the next available port, in this case the site was serves at port 8081.

Vercel deployment

We need the entire world to see our awesome site. Let's deploy it using Vercel platform. I have created a new project and picked "11ty" application's default settings for the build command and output folder.

Vercel project settings page

I have linked the Vercel project with the GitHub repository bahmutov/eleventy-example

Vercel project is linked to the GitHub repository

Every time we push a new commit to the main branch, the static site is built and deployed globally under a subdomain of vercel.app.

Production site lives at this domain

Here is the production site deployed from the main branch.

Deployed main branch is the production

Index page

We probably want to deploy the top level index page as well, and navigate to the /README page. Let's throw a root index.html there

index.html
1
2
3
4
5
6
7
8
9
10
<!DOCTYPE html>
<html>
  <head>
    <title>11ty Example</title>
  </head>
  <body>
    <p>Hi there</p>
    <p>Check out <a href="/README">the README</a></p>
  </body>
</html>

The navigation works: when the users clicks on the link, the browser goes to the /README page. When the user clicks the browser's "back" button, it navigates back to the index page.

Navigation is working locally

Preview deploys

I want to deploy the new index page - but I am not sure if the top level navigation is going to work with an actual domain (because I seriously doubt my programming skills). It would be a nice idea to deploy the full site to a temporary preview domain first, test it out, and if it works correctly, then merge the pull request to the main branch, which deploys to production.

I will create a new branch add-index-page and commit the new code there:

1
2
3
~/git/eleventy-example on add-index-page
$ git log --oneline
1407be3 (HEAD -> add-index-page) add index page with navigation

In my GitHub repository I have installed the Vercel GitHub App which automatically deploys every pull request.

Vercel for GitHub

By pushing the new branch to the repository and opening a pull request #2 I trigger the deployment. The Vercel bot comments on the PR with the deployed URL

Preview deployment comment

We can click on the preview URL to visit the deployed site and confirm manually that the navigation works

Preview deployment shows the navigation is working

The PR preview URL https://eleventy-example-git-add-index-page.bahmutov.vercel.app/ is a concatenation of the project name "eleventy-example", the source type "git", the branch name "add-index-page", my user name "bahmutov", and the top level domain name "vercel.app". In addition, there is a unique preview URL for every commit. You can find these URLs at the Vercel deploy page.

PR preview URLs

If we push more commits to the branch add-index-page, the new deploys will get their new unique eleventy-example-<HASH>.vercel.app URLs, while the latest branch preview will still have the top level <project>-git-<branch>-<username>.vercel.app URL.

Testing

We have tried the PR preview manually, but a better idea to prevent bugs in the deployed web applications is to write automated end-to-end tests using Cypress. Let's install Cypress and write a test in our pull request.

1
$ npm i -D cypress

When we ran the site locally, we tested it at the localhost:8080. Let's put this setting into cypress.json file.

cypress.json
1
2
3
{
  "baseUrl": "http://localhost:8080"
}

Our spec file will perform what we have done manually - it will navigate by using the link to the /README page and back.

cypress/integration/spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
/// <reference types="cypress" />
describe("11ty", () => {
  it("navigates", () => {
    // find more Cypress commands at
    // https://on.cypress.io/api
    cy.visit("/");
    cy.contains("Hi there");
    cy.contains("a", "the README").click();
    cy.location("pathname").should("match", /\/README\/$/);
    cy.go("back");
    cy.contains("Hi there"); // back on the index page
  });
});

Start Cypress with npx cypress open while the application is running and observe the test passing for the right reason. Hover over each command to observe the site's DOM snapshot and how it changed in response to the anchor click or cy.go('back') command.

Navigation test passing locally

Tip: read how to write flake-free Cypress tests when navigating from page to page in the blog post When Can The Test Navigate?.

Testing previews

We ran the above test locally. Let's run the same test automatically against the deployed preview URL. Luckily for us, Vercel for GitHub dispatches deployment events to GitHub, and we can write a GitHub Action that would execute in response to this event. Let's first simply print the event object to see if it has the target preview URL to test.

Tip: if you have never used GitHub Actions, read my Trying GitHub Actions blog post.

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
name: ci
# https://docs.github.com/en/actions/reference/events-that-trigger-workflows
on: [push, deployment, deployment_status]
jobs:
  show-event:
    runs-on: ubuntu-latest
    steps:
      - name: Dump GitHub context
        env:
          GITHUB_CONTEXT: ${{ toJson(github) }}
        run: echo "$GITHUB_CONTEXT"
  • push event happens for every commit. This would give us a chance to test the site separately from the deployment. For example, we could lint the site's source text and run the Cypress tests against the site running locally.
  • deployment event happens when Vercel starts the preview deploy - it does not have the deploy URL
  • deployment_status event is sent by Vercel twice. First, when the preview deployment starts, and second time when the preview deployment finishes.

Notice that the deployment_status is not listed in the PR checks - they are "hidden" or overwritten by the "deployment" event, which to me seems like a bad user interface.

GitHub pull request checks

Instead, we need to look at the "Actions" tab to see the ci workflows run, and by clicking inside figure out they ran triggered by the deployment_status events.

CI events for every commit

The pending deployment_status event has the following information inside the github JSON event

1
2
3
4
"description": "Vercel is deploying your app",
"environment": "Preview",
"state": "pending",
"target_url": "https://eleventy-example-5ccl0n3a7.vercel.app"

The second deployment_status event inside the GitHub CI action has status

1
2
3
4
"description": "Deployment has completed",
"environment": "Preview",
"state": "success",
"target_url": "https://eleventy-example-5ccl0n3a7.vercel.app"

We can limit the GitHub Action to only run a job when an expression is true. In our case we want to run end-to-end tests only after successful deploy - and we want to pass the "target_url" as baseUrl parameter. We can pass the base url using the CYPRESS_BASE_URL environment variable. Here is our updated workflow file

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
name: ci
# https://docs.github.com/en/actions/reference/events-that-trigger-workflows
on: [push, deployment, deployment_status]
jobs:
  e2e:
    # only runs this job on successful deploy
    if: github.event_name == 'deployment_status' && github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - name: Dump GitHub context
        env:
          GITHUB_CONTEXT: ${{ toJson(github) }}
        run: |
          echo "$GITHUB_CONTEXT"
      - name: Checkout 🛎
        uses: actions/checkout@v1
      - name: Run Cypress 🌲
        uses: cypress-io/github-action@v2
        env:
          CYPRESS_BASE_URL: ${{ github.event.deployment_status.target_url }}

When we push the commit, we can see the CI jobs skipped - except for the last job.

Skip all jobs but one - which runs on successful deployment

I love requiring certain test jobs to pass before a pull request can be merged. Thus I set up protected branches with e2e job required to allow merging.

Protected branch `main` requires `e2e` job to pass successfully

The E2E job runs ... and fails!

Our test against the preview URL fails

Tip: debugging realistic test failures is much simpler when you have access to the screenshots and test run videos, I recommend using Cypress Dashboard to record the test artifacts from any CI.

Hmm, seems Vercel does not add a trailing slash when serving the production version of the code, while 11ty running locally does add one.

The difference in trailing slashes

Testing against the deployed preview URLs just showed its usefulness. Let's update the test to be less strict.

1
2
- cy.location("pathname").should("match", /\/README\/$/);
+ cy.location("pathname").should("include", "/README");

Perfect. The test passes locally and against the preview URL.

Passing E2E test

Once the checks are green, I am confident and merge the pull request into the main branch. Vercel then deploys the production site. I can also remove extra CI events and only run the GitHub workflow on successful deployment. To install, cache and run Cypress we can use Cypress GitHub Action:

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
name: ci
# https://docs.github.com/en/actions/reference/events-that-trigger-workflows
on: [deployment_status]
jobs:
  e2e:
    # only runs this job on successful deploy
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v1
      - name: Run Cypress 🌲
        uses: cypress-io/github-action@v2
        env:
          CYPRESS_BASE_URL: ${{ github.event.deployment_status.target_url }}

Bonus: print the GitHub event and values

In the previous workflow file we used the github.event.deployment_status.target_url value. The github.event contains lots of other properties. We can also print the GitHub environment variables by using @bahmutov/print-env. Then we get things like the branch name that are not present in the github.event object.

1
2
3
4
5
6
7
- name: Print GitHub event 🖨
  env:
    GITHUB_EVENT: ${{ toJson(github.event) }}
  run: echo "$GITHUB_EVENT"

- name: Print GitHub env variables 🖨
  run: npx @bahmutov/print-env GITHUB

For example pull request deploy we will see print out like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
echo "$GITHUB_EVENT"
shell: /bin/bash -e {0}
env:
GITHUB_EVENT: {
"action": "created",
"deployment": {
  "created_at": "2020-11-27T18:20:25Z",
  "description": null,
  "environment": "Preview",
  "id": 295433471,
  "original_environment": "Preview",
  "payload": {},
  "performed_via_github_app": null,
  "ref": "3019a87169a5a9df28c88561fc21e71dd4c18ab2",
  "repository_url": "api.github.com/repos/bahmutov/eleventy-example",
  "sha": "3019a87169a5a9df28c88561fc21e71dd4c18ab2",
  "statuses_url": "api.github.com/repos/bahmutov/eleventy-example/deployments/295433471/statuses",
  "task": "deploy",
  "updated_at": "2020-11-27T18:20:35Z",
  "url": "api.github.com/repos/bahmutov/eleventy-example/deployments/295433471"
},
"deployment_status": {
  "created_at": "2020-11-27T18:20:35Z",
  "deployment_url": "api.github.com/repos/bahmutov/eleventy-example/deployments/295433471",
  "description": "Deployment has completed",
  "environment": "Preview",
  "id": 436572748,
  "performed_via_github_app": null,
  "repository_url": "api.github.com/repos/bahmutov/eleventy-example",
  "state": "success",
  "target_url": "eleventy-example-7wrr5ezkh.vercel.app",
  "updated_at": "2020-11-27T18:20:35Z",
  "url": "api.github.com/repos/bahmutov/eleventy-example/deployments/295433471/statuses/436572748"
},
...

Run npx @bahmutov/print-env GITHUB
npx: installed 8 in 1.684s
Found environment variables that start with GITHUB:
GITHUB_ACTION=run2
GITHUB_ACTIONS=true
GITHUB_ACTOR=vercel[bot]
GITHUB_API_URL=api.github.com
GITHUB_BASE_REF=
GITHUB_ENV=/home/runner/work/_temp/_runner_file_commands/set_env_67b033b7-e4d5-452a-838a-3ceabd989b00
GITHUB_EVENT_NAME=deployment_status
GITHUB_EVENT_PATH=/home/runner/work/_temp/_github_workflow/event.json
GITHUB_GRAPHQL_URL=api.github.com/graphql
GITHUB_HEAD_REF=
GITHUB_JOB=print
GITHUB_PATH=/home/runner/work/_temp/_runner_file_commands/add_path_67b033b7-e4d5-452a-838a-3ceabd989b00
GITHUB_REF=
GITHUB_REPOSITORY=bahmutov/eleventy-example
GITHUB_REPOSITORY_OWNER=bahmutov
GITHUB_RETENTION_DAYS=90
GITHUB_RUN_ID=387593912
GITHUB_RUN_NUMBER=4
GITHUB_SERVER_URL=github.com
GITHUB_SHA=3019a87169a5a9df28c88561fc21e71dd4c18ab2
GITHUB_WORKFLOW=print
GITHUB_WORKSPACE=/home/runner/work/eleventy-example/eleventy-example

Note: I could not find the branch name in the deployment event or GitHub environment variables, thus to print the branch name I did the following

1
2
3
4
5
6
7
8
# let's get the source code and find branch using the commit SHA
- name: Checkout 🛎
  uses: actions/checkout@v2
  with:
    # fetch all commits so we can find the branch
    fetch-depth: 0
- name: Print branch 🌳
  run: git name-rev --name-only $GITHUB_SHA

Which prints for branch print-event the following:

1
2
Run git name-rev --name-only $GITHUB_SHA
remotes/origin/print-event

GitHub Checks

There is one more detail that GitHub gets wrong when running tests on deployment_status event. It seems to be confused about which commit is tested, so it never reports the status back to the Pull Request. For example in #3 we see a pending check.

The e2e CI check is pending even after the tests have finished

To fix this, we can send the commit status ourselves using GitHub REST API call. Here is the added section of the GitHub workflow file. If the Cypress tests pass, we post success. If the Cypress tests fail, or any job step fails, we post the failure status.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# ci.yml file
...
- name: Run Cypress 🌲
  uses: cypress-io/github-action@v2
  env:
    CYPRESS_BASE_URL: ${{ github.event.deployment_status.target_url }}

# https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#job-status-check-functions
- name: Cypress tests ✅
  if: ${{ success() }}
  # set the merge commit status check
  # using GitHub REST API
  # see https://docs.github.com/en/rest/reference/repos#create-a-commit-status
  run: |
    curl --request POST \
    --url https://api.github.com/repos/${{ github.repository }}/statuses/${{ github.sha }} \
    --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' \
    --header 'content-type: application/json' \
    --data '{
      "context": "e2e",
      "state": "success",
      "description": "Cypress tests passed"
    }'

- name: Cypress tests 🚨
  if: ${{ failure() }}
  run: |
    curl --request POST \
    --url https://api.github.com/repos/${{ github.repository }}/statuses/${{ github.sha }} \
    --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' \
    --header 'content-type: application/json' \
    --data '{
      "context": "e2e",
      "state": "failure",
      "description": "Cypress tests failed"
    }'

You can see the example run in PR #4. The check "e2e" matches the required check name, thus the pull request is all green.

Posted commit checks

When we look at the Action steps, we can see the "Post success" step ran, while the "Post failure" step was skipped.

Job steps

We can even provide a link that would take us from the status check straight to the workflow run by forming the full target_url property when posting the status check.

1
2
3
4
5
6
--data '{
  "context": "e2e",
  "state": "success",
  "description": "Cypress tests passed",
  "target_url": "https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
}'

This creates the little "Details" link on the right.

Run URL in the status check

Cypress Dashboard

I have mentioned before that every Cypress run generates a video, and every failed Cypress test automatically saves the screenshot of the failure. You can store these test artifacts on GitHub, or send them to the Cypress Dashboard where they can be easily viewed. Let's set up our project for recording.

In the Cypress Desktop GUI select the "Runs" tab.

Cypress Runs tab

Click the "Set up project to record" button.

I will place the "eleventy-example" project under my personal "Gleb OSS" organization that has the Cypress Open Source Software billing plan. The test recordings will be public - everyone should be able to see them.

Setting up the project

Once I click "Set up project" button, its project id will be added to the cypress.json file and the recording key is shown. Please keep this key private.

Project recording key

I will set the recording key as a GitHub Action Secret in my repository.

Set the record key as GH Action Secret

Let's update the GitHub workflow file to record test results and artifacts. We are using Cypress GitHub Action, thus we need to add record: true parameter and pass the recording secret as an environment variable.

1
2
3
4
5
6
7
8
# updated ci.yml file
- name: Run Cypress 🌲
  uses: cypress-io/github-action@v2
  with:
    record: true
  env:
    CYPRESS_BASE_URL: ${{ github.event.deployment_status.target_url }}
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

The tests run on GitHub and show the recorded run URL https://dashboard.cypress.io/projects/y2sysj/runs/1

Recorded test runs show the Dashboard URL

At the Dashboard page you can see every spec, every video, every screenshot, and lots of test analytics.

First recorded run

Cypress GH Integration

If we are using GitHub and Cypress Dashboard, we might as well use the Cypress GitHub Integration App. I can add it to all my repositories, or just install it for selected repository "eleventy-example".

Installing Cypress GitHub App

Once installed, we can configure how the Cypress GH App comments on pull requests and sets status checks.

Configure Cypress GitHub App

Let's try a new pull request #6 - we will see Vercel's comment and Cypress' GH comment.

Cypress GitHub App comments on the pull requests

Cypress GH App adds its own status check to the commit.

Full set of commit status checks

Because we have the Cypress GH status check, we could remove our custom test job check implemented using curl commands. I will leave them in for completeness sake.

More info

If you want to test a site deployed to GitHub Pages, read Triple Tested Static Site Deployed to GitHub Pages Using GitHub Actions.

You can test deployed previews using Netlify + CircleCI combination

For full confidence, we do recommend adding visual testing using open source or commercial service to your functional end-to-end tests. Visual testing will prevent any CSS or style regressions from creeping into your site.

We also recommend for complex web applications to measure the code coverage to ensure all implemented features are covered by end-to-end tests. E2E tests are extremely effective at covering a lot of code.