products

Code Coverage For Nextjs Application

How to collect TypeScript code coverage using Cypress End-to-End tests.

📦 you can find the source code for this blog post in the repository bahmutov/next-ts-app and the deployed application at https://next-ts-app-swart.vercel.app/. You can find the tests in the separate repo bahmutov/next-ts-app-tests.

The application

I have scaffolded the Next.js application using the recommended command

1
2
$ npx create-next-app@latest --typescript
+ [email protected]

There are two modes for running the application: the dev and the prod. I would like to instrument the application in both modes. Thus I have added the following .babelrc file

1
2
3
4
{
  "presets": ["next/babel"],
  "plugins": ["istanbul"]
}

I have installed the babel-plugin-istanbul@6 NPM module and if everything works, then starting npm run dev and opening localhost:3000 shows the code coverage counters under window.__coverage__ object

Code was instrumented successfully

Instrument when necessary

We want to instrument the app when necessary, thus the simplest way is to look at an environment variable. I have renamed the .babelrc file into .babelrc.js file to include the Istanbul plugin only when the environment variable INSTRUMENT_CODE is present.

.babelrc.js
1
2
3
4
5
6
7
8
9
const shouldInstrumentCode = 'INSTRUMENT_CODE' in process.env
console.log('shouldInstrumentCode', shouldInstrumentCode)

module.exports = {
  "presets": ["next/babel"],
  "plugins": shouldInstrumentCode ? ["istanbul"] : []
}

console.dir(module.exports, {depth: null})

I am using Vercel to run the application, and I set this variable to have the code coverage counters present in the deployed code.

Set the INSTRUMENT_CODE variable to instrument the build on Vercel

Note: code coverage can add some overhead to the production application, so decide if it is worth it. You can still instrument the dev builds and run the end-to-end tests to collect the code coverage, while keeping the production build lean.

Tip: to make sure the instrumentation really regenerates the bundles, delete the .next folder before running. In my package.json I have the following scripts for running locally

package.json
1
2
3
4
5
6
7
8
9
{
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "predev:instrumented": "rm -rf .nyc_output coverage .next",
    "dev:instrumented": "INSTRUMENT_CODE=1 next dev"
  }
}

Locally I use npm run dev:instrumented to launch the instrumented application.

Cypress tests with code coverage report

Let's install Cypress test runner and its code coverage plugin

1
2
3
$ npm i -D cypress @cypress/code-coverage
+ [email protected]
+ @cypress/[email protected]

I have registered the code coverage report in the plugins file

cypress/plugins/index.ts
1
2
3
4
5
6
7
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config
  require('@cypress/code-coverage/task')(on, config)

  return config
}

and loaded the plugin from the support file

cypress/support/index.ts
1
import '@cypress/code-coverage/support'

My test is simple: just visiting the site defined in the cypress.json as baseUrl: http://localhost:3000

cypress/integration/spec.ts
1
2
3
it('loads the home page', () => {
  cy.visit('/')
})

In the Cypress Command Log I see the code coverage report messages.

The code coverage plugin logs its messages

If you are not sure where the generated report is saved, open the DevTools console and click on the last message. It shows the report was written in the "coverage" folder.

We should look for the coverage report in the coverage folder

There are coverage reports in various formats

1
2
3
4
5
6
7
8
9
$ ls -la coverage
total 32
drwxr-xr-x   7 glebbahmutov  staff   224 Mar  8 15:05 .
drwxr-xr-x  21 glebbahmutov  staff   672 Mar  8 15:29 ..
-rw-r--r--   1 glebbahmutov  staff  1022 Mar  8 16:55 clover.xml
-rw-r--r--   1 glebbahmutov  staff  1394 Mar  8 16:55 coverage-final.json
-rw-r--r--   1 glebbahmutov  staff   883 Mar  8 16:55 coverage-summary.json
drwxr-xr-x  12 glebbahmutov  staff   384 Mar  8 15:05 lcov-report
-rw-r--r--   1 glebbahmutov  staff   256 Mar  8 16:55 lcov.info

I am interested in the HTML report showing code coverage on top of the source files.

1
$ open coverage/lcov-report/index.html

The top level report shows 75% of all instrumented statements executed by the cy.visit('/') command.

The top level report shows coverage by file

We can click on the filename to see the individual coverage report

The function "add" was never called by the application

Deployment and testing

I have set up my Next.js application to deploy on Vercel. You can find the production version of the application at https://next-ts-app-swart.vercel.app/. The INSTRUMENT_CODE environment variable is set during the Vercel build, thus you can see the code coverage object if you open the DevTools.

The deployed production code has the code coverage object

Take a look that the source paths in the code coverage object in the deployed production application (marked with an orange arrow). The source paths are different from the source paths to the files when running locally. Let's run the tests to see if we can correctly generate the test coverage report from this coverage object. I will open Cypress test runner pointing at the deployed URL

1
$ CYPRESS_baseUrl=https://next-ts-app-swart.vercel.app/ npx cypress open

The tests finish and generate the code coverage report.

The E2E test visited the production site

The code coverage plugin has successfully mapped the production code paths to the local source files and generated the report

The code coverage report for the production app

We can see the source code "search" and mapping from the production paths to the local application source paths by enabling the debug logs when starting Cypress

1
$ DEBUG=code-coverage CYPRESS_baseUrl=https://next-ts-app-swart.vercel.app/ npx cypress open

The logs show how the plugin is looking for a parent folder so that all paths in the code coverage object map to the existing file paths.

The production source paths were mapped to the local source files

To generate the report we need to code coverage information and the application source files.

Tests in a separate repo

In some situations, the tests live in a repository separate from the application. I have described such situation in the blog posts How to Keep Cypress Tests in Another Repo While Using GitHub Actions and How to Keep Cypress Tests in Another Repo While Using CircleCI. For this blog post, I have created repository bahmutov/next-ts-app-tests with a copy of Cypress tests. We can run these tests against the deployed application

1
$ DEBUG=code-coverage CYPRESS_baseUrl=https://next-ts-app-swart.vercel.app/ npx cypress open

This time, the code coverage cannot be mapped to the source files, since there are no local files to find

The code coverage plugin could not find source files referenced in the coverage object

The plugin has still generated the overall report, but you cannot drill down into the individual source file reports

Without the source files, you cannot see the code coverage report per file

If we copy just the pages folder from the next-ts-app into the "next-ts-app-tests" folder before running the Cypress tests, then it finds it and can generate the report.

Tip: instead of copying the pages folder from the application's folder to the test folder, I create a symbolic link

1
2
3
4
5
6
7
# assuming the following structure
#   next-ts-app/
#     the application with "pages" folder
#   next-ts-app-tests/
#     the folder with the tests
# from the "next-ts-app-tests" folder call
$ ln -s ../next-ts-app/pages

Now the "pages" folder is linked to the tests folder

1
2
3
$ ls -la
...
pages -> ../next-ts-app/pages

Cloning the app repo into the tests repo

Let's give our tests the application's source code so it can generate the file code coverage reports. I will use GitHub Actions to checkout out both repos and copy the "pages" folder from the application folder into the tests folder.

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
name: ci
on: push
jobs:
  test:
    runs-on: ubuntu-20.04
    steps:
      # https://github.com/actions/checkout
      - name: Check out this repo 🛎
        uses: actions/checkout@v3

      - name: Check out the application repo 🛎
        uses: actions/checkout@v3
        with:
          repository: bahmutov/next-ts-app
          path: next-ts-app

      # help the code coverage tool find the source files
      # can also move or link the source files
      - name: Copy application source files 💾
        run: cp -r next-ts-app/pages .

      - name: Run tests against the production site 🧪
        # https://github.com/cypress-io/github-action
        uses: cypress-io/github-action@v3
        with:
          config: 'baseUrl=https://next-ts-app-swart.vercel.app/'
        env:
          DEBUG: code-coverage

      # https://github.com/marketplace/actions/github-pages-action
      - name: Deploy code coverage report 🚀
        uses: peaceiris/actions-gh-pages@v3
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./coverage/lcov-report

Tip: I have used actions/github-pages-action step at the end to publish the generated HTML code coverage report to GitHub Pages. You can find it at https://glebbahmutov.com/next-ts-app-tests/.

The code coverage report produced on CI and hosted on GitHub Pages

Fetching the right application source code

Imagine you are deploying an instrumented application to some environment, like https://instrumented.acme.co once per day. Then you run the tests against it to generate the full code coverage report. You only do this once per day because instrumenting and running the tests is slow, but there might be multiple commits to the application source code itself. How do you use the right source code when generating the coverage reports? By checking out the right source code commit for the deployed application.

Next.js applications embed the buildId in the pages, and you can control the ID. For example, you can concatenate the branch and the source code commit SHA, like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
// next.config.js
// https://github.com/cypress-io/commit-info
const { getBranch, getSha } = require('@cypress/commit-info')

/** @type {import('next').NextConfig} */
const nextConfig = {
  reactStrictMode: true,
  generateBuildId: async () => {
    // make sure to use Vercel variables if available
    // https://vercel.com/docs/concepts/projects/environment-variables
    const branch =
      process.env.VERCEL_GIT_COMMIT_REF ||
      (await getBranch()) ||
      'unknown branch'
    const sha =
      process.env.VERCEL_GIT_COMMIT_SHA || (await getSha()) || 'unknown sha'
    const buildId = `${branch}:::${sha}`
    console.log('generated build id "%s"', buildId)
    return buildId
  },
}

module.exports = nextConfig

I wrote a little GitHub action to query the HTML page and extract the build ID value and split it into branch and commit SHA. Then you can check out the right source code commit when checking out the application on CI.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# https://github.com/actions/checkout
- name: Check out this repo 🛎
  uses: actions/checkout@v3

- name: Get the build info 🖨
  uses: bahmutov/get-build-id@v1
  id: get-build-id
  with:
    url: 'https://next-ts-app-swart.vercel.app/'

- name: Print the build outputs 🖨
  run: |
    echo "Next.js build ID: ${{ steps.get-build-id.outputs.buildId }}"
    echo "Next.js build branch: ${{ steps.get-build-id.outputs.branch }}"
    echo "Next.js build commit: ${{ steps.get-build-id.outputs.commit }}"

- name: Check out the application repo 🛎
  uses: actions/checkout@v3
  with:
    repository: bahmutov/next-ts-app
    # from the build ID, we get the commit matching the deployed site
    # so let's fetch just that commit to make sure our report
    # uses the correct source files
    ref: ${{ steps.get-build-id.outputs.commit }}
    path: next-ts-app

# help the code coverage tool find the source files
- name: Copy application source files 💾
  run: cp -r next-ts-app/pages .

Code coverage on CircleCI

I have set up an equivalent code coverage collection on CircleCI. The tests project logs in using a machine user account and checks out the application source code before running the tests and generating the report. See .circleci/config.yml file for the current code

.circleci/config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
version: 2.1
orbs:
  # https://github.com/cypress-io/circleci-orb
  cypress: cypress-io/cypress@1

workflows:
  build:
    jobs:
      - cypress/run:
          filters:
            branches:
              ignore:
                - gh-pages
          post-checkout:
            - run: echo "Checking out the application"
            - run: git clone [email protected]:bahmutov/next-ts-app.git --depth 1
            - run:
                name: Link source pages to this repo
                # syntax is: "ln <existing folder> <link path>"
                command: |
                  ln -s next-ts-app/pages pages
                  ls -la
          config: 'baseUrl=https://next-ts-app-swart.vercel.app/'
          no-workspace: true
          post-steps:
            - store_artifacts:
                path: coverage/lcov-report

The source code report is stored as a test artifact on CircleCI

The code coverage report on CircleCI

Tip: if you do not want to set up SSH key to check out the second repository, you could use a GitHub token

1
git clone https://${GITHUB_TOKEN}:[email protected]/owner/repo <local folder name> --depth 1

See more