products

Code Coverage for End-to-end Tests

How to instrument application code and collect code coverage during Cypress E2E tests.

In my previous blog post "Stub navigator API in end-to-end tests" I have shown a small web application that shows the current battery charge information. Then I have shown several Cypress end-to-end tests and found an edge case that caused the application to crash.

Application crashes trying to attach the event listener

The crash was caused by the failure of the program src/index.js to account for the final - else code path when dealing with the browser capabilities.

src/index.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
var battery
if (navigator.battery) {
  readBattery(navigator.battery)
} else if (navigator.getBattery) {
  navigator.getBattery().then(readBattery)
} else {
  // hmm, the "battery" variable remains undefined
  document.querySelector('.not-support').removeAttribute('hidden')
}

window.onload = function () {
  // hmm what happens when "battery" variable stays undefined?

  // show updated status when the battery changes
  battery.addEventListener('chargingchange', function () {
    readBattery()
  })

  battery.addEventListener('levelchange', function () {
    readBattery()
  })
}

Our end-to-end test in spec file no-battery.js removes both navigator.battery and navigator.getBattery properties and reaches the final - else path, causing the window.onload to crash - the variable battery remains undefined. If only we could see right away before writing no-battery.js that this code path is a problem!

Instrumenting the code as a pre-processor step

Cypress users who think collecting application code coverage during end-to-end tests proposed several solutions. All solutions instrument the web application code during the build step, then save the collected code coverage data after the tests. Finally, yet another step generates coverage report from the saved data. Let's see how it can be done for our battery API demo application.

You can find this code in branch coverage-step in the demo-battery-api repository.

First, we will instrument our application code as a pre-processing step using IstanbulJS library and its command line wrapper nyc. I will install the following two libraries

1
2
3
npm install --save-dev nyc@14 istanbul-lib-coverage@2
+ [email protected]
+ [email protected]

I have added the command to instrument application code in src folder and place the instrumented code into build/src. The cp command copies the rest of the application files: HTML and styles.

package.json
1
2
3
4
5
6
7
8
9
10
{
  "scripts": {
    "build": "npm run instrument && npm run cp",
    "preinstrument": "npm run clean",
    "instrument": "nyc instrument --compact false src build/src",
    "cp": "cp src/*.css build/src && cp src/*.png build/src && cp index.html build",
    "clean": "rm -rf build .nyc_output || true",
    "report:coverage": "nyc report --reporter=html"
  }
}

If we execute npm run build we will get the instrumented application in the build folder. For demo purposes I pass --compact false during instrument step to avoid minification. The output code looks like this (this is only a small part of the instrumented code):

build/src/index.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
if (navigator.battery) {
  cov_w7e0b4vcv.b[11][0]++;
  cov_w7e0b4vcv.s[31]++;
  readBattery(navigator.battery);
} else {
  cov_w7e0b4vcv.b[11][1]++;
  cov_w7e0b4vcv.s[32]++;

  if (navigator.getBattery) {
    cov_w7e0b4vcv.b[12][0]++;
    cov_w7e0b4vcv.s[33]++;
    navigator.getBattery().then(readBattery);
  } else {
    cov_w7e0b4vcv.b[12][1]++;
    cov_w7e0b4vcv.s[34]++;
    document.querySelector('.not-support').removeAttribute('hidden');
  }
}

The instrumentation just inserts counters into the code, incrementing them for each statement (the cov_w7e0b4vcv.s[31]++ line) and if - else branch (the cov_w7e0b4vcv.b[11][0]++ line). After the code finishes running, we can map the numbers from the object window.cov_w7e0b4vcv back to the original source code to see which lines were covered.

During the tests

The application code has been instrumented, and we serve the build folder during end-to-end tests, rather than the original src folder. Here is what we need to do in the browser during each test:

  • reset the collected coverage before tests begin
  • merge the code coverage collected during each test with previously collected data
  • generate a coverage report in desired format after all tests have finished

The above 3 things are controlled by the cypress/support/index.js file that gets bundled with each spec file automatically.

cypress/support/index.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
/// <reference types="cypress" />
before(() => {
  cy.task('resetCoverage')
})

afterEach(() => {
  // save coverage after each test
  // because the entire "window" object is about
  // to be recycled by Cypress before next test
  cy.window().then(win => {
    if (win.__coverage__) {
      cy.task('combineCoverage', win.__coverage__)
    }
  })
})

after(() => {
  cy.task('coverageReport')
})

Each cy.task jumps from the browser context to the backend context in Cypress where we have the full access to the file system. Thus we can save the code coverage JSON file and execute nyc to produce the report. Here is the cypress/plugins/index.js that keeps the code coverage object and generates the report:

cypress/plugins/index.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
const istanbul = require('istanbul-lib-coverage')
const { join } = require('path')
const { existsSync, mkdirSync, writeFileSync } = require('fs')
const execa = require('execa')

module.exports = (on, config) => {
  let coverageMap = istanbul.createCoverageMap({})
  const outputFolder = '.nyc_output'
  const nycFilename = join(outputFolder, 'out.json')

  if (!existsSync(outputFolder)) {
    mkdirSync(outputFolder)
    console.log('created folder %s for output coverage', outputFolder)
  }

  on('task', {
    /**
     * Clears accumulated code coverage information
     */
    resetCoverage () {
      coverageMap = istanbul.createCoverageMap({})
      console.log('reset code coverage')
      return null
    },

    /**
     * Combines coverage information from single test
     * with previously collected coverage.
     */
    combineCoverage (coverage) {
      coverageMap.merge(coverage)
      return null
    },

    /**
     * Saves coverage information as a JSON file and calls
     * NPM script to generate HTML report
     */
    coverageReport () {
      writeFileSync(nycFilename, JSON.stringify(coverageMap, null, 2))
      console.log('wrote coverage file %s', nycFilename)
      console.log('saving coverage report')
      return execa('npm', ['run', 'report:coverage'], { stdio: 'inherit' })
    }
  })
}

Let's run the instrumented tests. I will run just a single test that removes navigator.battery property.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
context('navigator.getBattery', () => {
  it.only('shows battery status of 75%', function () {
    cy.visit('/', {
      onBeforeLoad (win) {
        delete win.navigator.battery
        win.navigator.getBattery = cy
          .stub()
          .resolves({
            level: 0.75,
            charging: false,
            addEventListener: () => {}
          })
          .as('getBattery')
      }
    })
    cy.contains('.battery-percentage', '75%')
    cy.get('@getBattery').should('have.been.calledOnce')
  })
})

The Cypress Test Runner shows the code coverage related commands.

Single test with code coverage

Great, let us see the covered lines.

The coverage report

The saved code coverage object is just a large JSON file .nyc_output/out.json like this

JSON coverage object example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
  "/demo-battery-api/src/index.js": {
    "path": "/demo-battery-api/src/index.js",
    "statementMap": {
      "0": {
        "start": {
          "line": 1,
          "column": 1
        },
        "end": {
          "line": 76,
          "column": 4
        }
      },
    },
    "s": {
      "0": 10,
      "1": 14,
      "2": 14,
      "3": 14,
      "4": 14,
      "5": 14,
      ...
    }
  ...

The object contains location of all statements, functions and branches in the original code, and the counters for how many times each item has been executed (for example the s counters are for statements). The human coverage report is generated from the .nyc_output/out.json file using npm script "report:coverage": "nyc report --reporter=html" is placed in coverage folder. It is a folder with a static site.

coverage folder
1
2
3
4
5
6
7
8
9
coverage/
  base.css
  block-navigation.js
  index.html
  index.js.html
  prettify.css
  prettify.js
  sort-arrow-sprite.png
  sorter.js

We can open the coverage/index.html file and see that our test really has hit just a single if - else line. Lines in red show the code NOT covered by the tests.

The getBattery path hit by the test

We can enable all tests and see more lines covered - and the only uncovered line is the third code path - else when there is neither navigator.battery nor navigator.getBattery methods.

Missed line

Tip: I prefer having a separate npm script to generate the code coverage report, because we may want to generate reports in different formats. For example we can generate a summary report and show it in the terminal: "report:coverage": "nyc report --compact"

1
2
3
4
5
6
7
$ npm run report:coverage
----------|----------|----------|----------|----------|-------------------|
File      |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
----------|----------|----------|----------|----------|-------------------|
All files |       95 |    92.31 |      100 |       95 |                   |
 index.js |       95 |    92.31 |      100 |       95 |             31,64 |
----------|----------|----------|----------|----------|-------------------|

We can even fail the build if the code coverage (lines, statements, branches or per file) is below given threshold:

1
2
3
4
5
6
7
8
9
10
$ npx nyc report --check-coverage --statements 100
----------|----------|----------|----------|----------|-------------------|
File      |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
----------|----------|----------|----------|----------|-------------------|
All files |     97.5 |    96.15 |      100 |     97.5 |                   |
 index.js |     97.5 |    96.15 |      100 |     97.5 |                64 |
----------|----------|----------|----------|----------|-------------------|
ERROR: Coverage for statements (97.5%) does not meet global threshold (100%)
$ echo $?
1

Better write more tests.

Code coverage from multiple spec files

If we have more than a single test (spec) file, we need to be careful when we reset the coverage information. There are two different cases, depending on how the test runner is running:

  • if the test runner is in the interactive mode using cypress open then we can reset the coverage before the tests. This works for a single spec file, or when running all specs using "Run all specs" button.
  • if the test runner is in the headless mode using cypress run, then each spec is processed separately. It is almost like executing a series of separate commands: cypress run --spec test1.js, cypress run --spec test2.js, etc. We cannot reset the coverage in that case - we risk destroying the information collected from the previous spec file. We need to reset the coverage before running Cypress. Luckily this is simple to do, here I am using npm precy:run script that automatically runs before cy:run script.
package.json
1
2
3
4
5
6
7
{
  "scripts": {
    "cy:open": "cypress open",
    "cy:run": "cypress run",
    "precy:run": "rm -rf .nyc_output || true"
  }
}

From the tests, we can pass isInteractive flag to the task, and only reset the coverage file when isInteractive is true.

cypress/support/index.js
1
2
3
4
5
6
before(() => {
  // we need to reset the coverage when running
  // in the interactive mode, otherwise the counters will
  // keep increasing every time we rerun the tests
  cy.task('resetCoverage', { isInteractive: Cypress.config('isInteractive') })
})
cypress/plugins/index.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
module.exports = (on, config) => {
  on('task', {
    /**
     * Clears accumulated code coverage information.
     *
     * Interactive mode with "cypress open"
     *    - running a single spec or "Run all specs" needs to reset coverage
     * Headless mode with "cypress run"
     *    - runs EACH spec separately, so we cannot reset the coverage
     *      or we will lose the coverage from previous specs.
     */
    resetCoverage ({ isInteractive }) {
      if (isInteractive) {
        console.log('reset code coverage in interactive mode')
        const coverageMap = istanbul.createCoverageMap({})
        writeFileSync(nycFilename, JSON.stringify(coverageMap, null, 2))
      }
      /*
        Else:
          in headless mode, assume the coverage file was deleted
          before the `cypress run` command was called
          example: rm -rf .nyc_output || true
      */
      return null
    }
    ...
  }
}

Perfect.

Code coverage on CI

The generated HTML code coverage report can be stored on continuous integration server as a static artifact. I am running these builds on CircleCI at https://circleci.com/gh/bahmutov/demo-battery-api/tree/coverage-step using Cypress Circle Orb. After the test run, I store the coverage folder as a test artifact. Here is the entire CI config file circle.yml

circle.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# see https://github.com/cypress-io/circleci-orb
version: 2.1
orbs:
  cypress: cypress-io/cypress@1
workflows:
  build:
    jobs:
      - cypress/run:
          # we need to start the web application
          start: npm start
          # there are no jobs to follow this one
          # so no need to save the workspace files (saves time)
          no-workspace: true
          # store the created coverage report folder
          # you can click on it in the CircleCI UI
          # to see live static HTML site
          post-steps:
            - store_artifacts:
                path: coverage

Note that on CI we do not need to remove the code coverage folder .nyc_output before starting cypress run - because CI server automatically gets a fresh workspace folder.

You can see the artifact with each build. For example you can browse to the artifacts in the build #28

Code coverage folder as a build artifact

Click on index.html to see the JavaScript coverage report

Code coverage report

We have only a single JavaScript application file, open the report for index.js by clicking on it.

Code coverage for index.js file

100% code coverage

Hitting a 100% code coverage is a hard and thankless task. Code coverage is not the goal in itself, rather the gap in the code coverage helps me identify a missing test and an overlooked edge condition. In our case, the uncovered logical branch shows that we did not test what happens when the browser navigator object does not have the battery API. Let's fix the issue and enable the previously crashing test.

src/index.js
1
2
3
4
5
6
7
8
9
10
11
12
window.onload = function () {
  // add a guard condition to prevent crashing
  if (battery) {
    battery.addEventListener('chargingchange', function () {
      readBattery()
    })

    battery.addEventListener('levelchange', function () {
      readBattery()
    })
  }
}

You need to rerun the instrumentation step again (npm run build in my case). Enable the previously crashing test - and it is passing now!

No battery API support - no problem

Run all tests again - interactively using "Run all tests" button, or using npm run cy:run and get the 100% code covered.

100% code covered

See also