products

Test Sites Deployed To Netlify Using netlify-plugin-cypress

Test a site after deploying it to Netlify preview or production environment.

Example repo

I have cloned the Eleventy starter with serverless functions under my GitHub repo bahmutov/eleventyone. The site is deployed automatically from GitHub to Netlify and you can see it under https://eleventyone-test.netlify.app/.

The first test

Any time I open a Pull Request a new preview is deployed at Netlify automatically. I would like to run end-to-end tests against the deployed URL. Let's see how to do this.

The first test

First, install Cypress test runner as a dev dependency

1
2
$ npm i -D cypress
+ [email protected]

Second, let's write a few tests. We have the page itself which can be served locally using npm run dev

1
2
3
4
5
6
7
8
9
10
11
12
$ npm run dev
> cross-env ELEVENTY_ENV=dev eleventy --serve
...
[Browsersync] Access URLs:
 -----------------------------------
       Local: http://localhost:8080
    External: http://10.0.0.126:8080
 -----------------------------------
          UI: http://localhost:3001
 UI External: http://localhost:3001
 -----------------------------------
[Browsersync] Serving files from: dist

Great, we need to make sure this page loads and at least displays the heading text.

cypress/integration/spec.js
1
2
3
4
5
6
7
8
/// <reference types="cypress" />

describe('Demo site', () => {
  it('loads', () => {
    cy.visit('/')
    cy.contains('h1', 'EleventyOne').should('be.visible')
  })
})

Our Cypress settings file cypress.json specifies the base url when running locally. It also says that our project uses no fixtures, no plugins, and no support file.

cypress.json
1
2
3
4
5
6
{
  "baseUrl": "http://localhost:8080",
  "fixturesFolder": false,
  "supportFile": false,
  "pluginsFile": false
}

Run Cypress from the second terminal to run the test with npx cypress open.

The first test

Run tests on Netlify

Let's plugin into the Netlify build system to run our Cypress test after the deploy. We will install the netlify-plugin-cypress as another dev dependency.

1
2
$ npm i -D netlify-plugin-cypress
+ [email protected]

Update the netlify.toml file to include the netlify-plugin-cypress and tell it to skip tests before the deployment and run them after the deployment using onSuccess event.

netlify.toml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

# run Cypress E2E tests
[[plugins]]
package = "netlify-plugin-cypress"
  [plugins.inputs]
    # skip the default tests before the deploy
    skip = true
  [plugins.inputs.onSuccess]
    # run tests after the deploy
    enable = true

When Netlify runs its deploy, it will show the new messages from the Cypress plugin. The log shows the tests executing against the deployed URL and successfully passing.

Passing tests against the preview URL

Let's merge this pull #1 request since it is passing.

Netlify status check on GitHub pull request

After GitHub has finished merging the code, Netlify builds the production version of the site, again running the E2E tests.

Testing Netlify functions

Our site includes a couple Netlify functions. For example, we can ask for a joke and functions/fetch-joke.js delivers one. To run both the static site and the Netlify functions locally we need to install netlify CLI:

1
2
$ npm i -D netlify-cli
+ [email protected]

Let's change our package.json to include the dev:netlify script:

package.json
1
2
3
4
5
6
7
8
9
{
  "scripts": {
    "start": "npm run dev",
    "dev": "cross-env ELEVENTY_ENV=dev eleventy --serve",
    "build": "cross-env ELEVENTY_ENV=prod eleventy",
    "seed": "cross-env ELEVENTY_ENV=seed eleventy",
    "dev:netlify": "netlify dev"
  }
}

Start the local Netlify environment from the first terminal using npm run dev:netlify:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
> netlify dev

◈ Netlify Dev ◈
◈ Injected netlify.toml file env var: CYPRESS_CACHE_FOLDER
◈ Ignored netlify.toml file env var: TERM (defined in process)
◈ Ignored general context env var: LANG (defined in process)
◈ Ignored general context env var: LC_ALL (defined in process)
◈ Overriding command with setting derived from netlify.toml [dev] block: npm run start
◈ Functions server is listening on 49254
◈ Starting Netlify Dev with eleventy
...
   ┌─────────────────────────────────────────────────┐
   │                                                 │
   │   ◈ Server now ready on http://localhost:8888   │
   │                                                 │
   └─────────────────────────────────────────────────┘

Notice that Netlify proxies Eleventy to :8888 locally. Thus we should update the local base url to http://localhost:8888 in cypress.json file. The netlify.toml file defines an API redirect to the function:

netlify.toml
1
2
3
4
[[redirects]]
  from = "/api/*"
  to = "/.netlify/functions/:splat"
  status = 200

Thus our Cypress test can use cy.request like this:

1
2
3
4
it.only('delivers jokes', () => {
  cy.visit('/')
  cy.request('/api/fetch-joke')
})

I like working on a single test using it.only while writing the test. Cypress watches the spec file automatically and reruns the test on any changes. The Command Log shows the Netlify function returning a joke:

Inspecting the response from the Netlify function

We need to grab the body of the response object, then parse it as JSON, and then confirm the msg property is an non-empty string.

1
2
3
4
5
6
7
8
9
it.only('delivers jokes', () => {
  cy.visit('/')
  cy.request('/api/fetch-joke')
    .its('body')
    .then(JSON.parse)
    .its('msg')
    .should('be.a', 'string')
    .and('be.not.empty')
})

Great, the test passes.

Testing the joke returned by the Netlify function

The PR pull/2 is green.

The new API test passes against the preview URL

Let's ship it 🚢

Tip: for more on how Cypress can run API tests read Add GUI to your E2E API tests and Black box API testing with server logs blog posts.

Failing tests and artifacts

Now let's imagine a pull request that fails its tests. For example, if we change the landing page to have a different title "Netlify Cypress Tests" the tests would fail. Yet the pull/3 still shows the passing Netlify check!

Green Netlify status check on PR 3

The Netlify tab shows a warning

Failed E2E tests show up as a warning

We can see the failed tests in the log, yet Netlify does not change the status of the deploy - after all, the deployment was successful! Related question: Cypress captures screenshots on test failures and a video of the test run, where can we see these test artifacts, marked with green arrows the screenshot below?

Deployment log shows saved error screenshot and video files

Let's see those test artifacts.

Recording to Cypress Dashboard

Most CI systems, including the Netlify Build, struggle to store and show the test artifacts well enough. Thus the simplest solution to record test results and store the test artifacts is to use the Cypress Dashboard. Let's set up the test recording for this project. We can set up recording by going to the "Runs" tab in the Test Runner

Click "Connect to Dashboard" button to set up project recording

We should set the displayed record key as an environment variable CYPRESS_RECORD_KEY in the deploy environment variables.

Adding the record key to the Netlify deployment settings

Then just add the record = true input parameter to our netlify-plugin-cypress's settings:

netlify
1
2
3
4
5
6
7
8
9
10
# run Cypress E2E tests
[[plugins]]
package = "netlify-plugin-cypress"
  [plugins.inputs]
    # skip the default tests before the deploy
    skip = true
  [plugins.inputs.onSuccess]
    # run tests after the deploy
    enable = true
    record = true

Tip: grab the project's dashboard badge and add it to the README's markdown. At first it shows "no tests found", but will be updated after the first test run in the selected branch.

Cypress badge settings

After we push the code we can see the test results including the error screenshot and the video of the test run.

Cypress badge settings

Cypress GitHub App and status checks

Now let's make sure the pull request does not have all green status checks when the E2E tests fail. Install the Cypress GitHub App and give it access to the repo.

Cypress GitHub App settings

Tip: Cypress has the GitLab and BitBucket integrations in beta testing.

Let's trigger a new deployment - we can can do it right from the Netlify page.

Trigger the deploy from the Netlify page

After the deploy runs, you will see a GitHub comment posted by the Cypress app. It shows a failing test with a screenshot thumbnail. You also now have a red status check from the Cypress app - you know the pull request is breaking its tests.

Cypress GitHub App

Let's require the passing tests before the pull request can be merged. Under the GitHub repo's settings check these options and mark both status checks required.

Require successful deploy and passing tests for pull requests

Now switch back to the pull request #3 to see the updated message - the PR cannot be merged until we fix the tests.

Pull request with failing tests cannot be merged

Let's update the test and push it.

cypress/integration/spec.js
1
cy.contains('h1', 'Netlify Cypress Tests').should('be.visible')

Beautiful - our site is good again.

The tests have been fixed

Bonus: automate dependency updates

This blog post describes the testing of this project as of February 2021. I would like the project and its tests to keep working without becoming obsolete for the future readers. Thus I recommend setting up Renovate bot to Keep Examples Up To Date. In the repo's root create renovate.json file that once a week checks for new versions and opens a pull request. If the checks are green, the version update pull request gets automatically merged. We are only interested in a few dependencies, like Cypress, netlify-plugin-cypress and the @11ty/eleventy package.

renovate.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "extends": [
    "config:base"
  ],
  "automerge": true,
  "prHourlyLimit": 2,
  "updateNotScheduled": false,
  "timezone": "America/New_York",
  "schedule": [
    "every weekend"
  ],
  "masterIssue": true,
  "labels": [
    "type: dependencies",
    "renovate"
  ],
  "packageRules": [
    {
      "packagePatterns": [
        "*"
      ],
      "excludePackagePatterns": [
        "@11ty/eleventy",
        "cypress",
        "netlify-plugin-cypress"
      ],
      "enabled": false
    }
  ]
}

Of course, to automatically upgrade dependencies we need to have more tests, thus I will add a few E2E tests to check the page navigation.

cypress/integration/spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
/// <reference types="cypress" />

describe('Demo site', () => {
  it('loads', () => {
    cy.visit('/')
    cy.contains('h1', 'Netlify Cypress Tests').should('be.visible')
  })

  it('delivers jokes', () => {
    cy.visit('/')
    cy.request('/api/fetch-joke')
      .its('body')
      .then(JSON.parse)
      .its('msg')
      .should('be.a', 'string')
      .and('be.not.empty')
  })

  it('navigates directly', () => {
    cy.visit('/about')
    cy.contains('h1', 'About').should('be.visible')
  })

  it('navigates via links', () => {
    cy.visit('/')
    cy.get('.nav li').should('have.length.gt', 1)
      .contains('a', 'about')
      .should('have.attr', 'href', '/about').click()
    cy.location('pathname').should('include', '/about')
    cy.contains('h1', 'About').should('be.visible')

    cy.get('.nav li').should('have.length.gt', 1)
      .contains('a', 'home')
      .should('have.attr', 'href', '/').click()
    cy.contains('h1', 'Netlify Cypress Tests').should('be.visible')
  })
})

I activated the RenovateApp and it will keep my repo bahmutov/eleventyone up-to-date from now on. You can immediately see the master issue the RenovateApp opens that describes the actions it wants to take:

RenovateApp keeps master issue open

Most of the time the RenovateApp's actions are automatic and it can be trusted to merge its own pull requests, since our E2E tests against the deployed URLs give me enough confidence.

RenovateApp opens a pull request to pin dependencies

Version badges

Because the versions of Cypress and netlify-plugin-cypress are important to anyone reading this blog post and looking at the source code, I like showing the current versions right in the README. To create a badge and update it after the RenovateApp upgrades dependencies, I use my own dependency-version-badge utility. I like running it using GitHub Actions. Here is my workflow file

.github/workflows/badges.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
name: badges
on:
  push:
    # update README badge only if the README file changes
    # or if the package.json file changes, or this file changes
    branches:
      - master
    paths:
      - README.md
      - package.json
      - .github/workflows/badges.yml
  schedule:
    # update badges every night
    # because we have a few badges that are linked
    # to the external repositories
    - cron: '0 3 * * *'

jobs:
  badges:
    name: Badges
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v2

      - name: Update version badges 🏷
        run: npx -p dependency-version-badge update-badge cypress netlify-plugin-cypress

      - name: Commit any changed files 💾
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Updated badges
          branch: master
          file_pattern: README.md

The above CI job runs when the listed files change and once at night. It keeps the version badges up-to-date in the README file.

README badges for the project

Happy Testing!

See more