products

Combined End-to-end and Unit Test Coverage

How to achieve 100% code coverage by combining end-to-end and unit tests without losing your sanity.

E2E code coverage overview

In my previous blog post "Stub navigator API in end-to-end tests" I have introduced an application that uses navigator browser API to show the battery charge percentage. While exploring the ways to test this application using Cypress.io we have discovered an edge case that caused that web application to crash when running in any browser but Chrome. The edge case is shown below - the application crashes if the battery variable remains undefined by the time window.load tries to attach an event listener.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
var battery
if (navigator.battery) {
  readBattery(navigator.battery)
} else if (navigator.getBattery) {
  navigator.getBattery().then(readBattery)
} else {
  // edge case!
  document.querySelector('.not-support').removeAttribute('hidden')
}
window.onload = function () {
  // WHAT HAPPENS WHEN "battery" IS undefined?

  // show updated status when the battery changes
  battery.addEventListener('chargingchange', function () {
    ...
  })
}

While Cypress allows writing end-to-end tests with ease, it does not tell us what tests to write. It is up to the person who is developing the web application to know what use cases to cover with end-to-end tests. But code coverage could be a good metric that highlights the untested logical edge cases. In the blog post "Code Coverage for End-to-end Tests" I have shown how to instrument application JavaScript code as a build step. The collected code coverage saved after the end-to-end tests passed makes the missed line really visible.

Missed line

Once we add a test to hit this line, we discover the problem in the application code, add a missed guard and make application robust.

1
2
3
4
5
6
7
8
window.onload = function () {
  // add a guard condition to prevent crashing
  if (battery) {
    battery.addEventListener('chargingchange', function () {
      ...
    })
  }
}

In another blog post "Code Coverage by Parcel Bundler" I have set up on-the-fly code coverage using the excellent Parcel Bundler. Instead of generating and saving an intermediate instrumented application source, we can instrument the application while serving it. The details will vary depending on the source bundler used, but most bundlers can use babel-plugin-istanbul to instrument ES6 on the fly. In our case, the application's code will be instrumented because I have added a tiny .babelrc file:

.babelrc
1
2
3
{
  "plugins": ["istanbul"]
}

To save the code coverage results correctly, my end-to-end tests are using cypress-istanbul plugin - and the tests reach 96% code coverage.

The missing 4%

Hmm, great, but we are still 4% short of the perfect 100%. If we look at each source file, we can see that missing lines and logical branches are from this utility function toTime that converts number of seconds to hours:minutes string label.

toTime function not covered by e2e tests

We have missed logical branches when the number of hours has double digits, and we have missed a branch of code when the number of minutes has a single digit. The function toTime is called from another function batteryStats with battery.chargingTime.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
export function batteryStats (battery) {
  const percentage = parseFloat((battery.level * 100).toFixed(2)) + '%'
  const charging = battery.charging
  let fully
  let remaining

  if (charging && battery.chargingTime === Infinity) {
    fully = 'Calculating...'
  } else if (battery.chargingTime !== Infinity) {
    fully = toTime(battery.chargingTime)
  } else {
    fully = '---'
  }
  ...
}

We could add end-to-end tests passing battery object that has just the right number in battery.chargingTime property to hit both missed hours and minutes cases. Hmm, but that is so weird - trying to essentially unit test an internal function via end-to-end tests! Instead, let's cover toTime function with unit tests. Luckily, Cypress can execute unit tests for us.

I have written cypress/integration/utils-spec.js that imports the function toTime directly and hits all logical branches and statements.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import { toTime } from '../../src/utils'

describe('toTime', () => {
  // hit all if - else branches in the "toTime" function

  it('handles single digit units', () => {
    const hhmm = toTime(0)
    expect(hhmm).to.equal('00:00')
  })

  it('handles double digit units', () => {
    expect(toTime(36001)).to.equal('10:00')
    expect(toTime(601)).to.equal('00:10')
    // there are no seconds in the returned string
    // but we can still cover the logical branches
    expect(toTime(20)).to.equal('00:00')
  })
})

Combined coverage

Just like that the coverage in this function is complete, but hmm, the new coverage is NOT reflected in the generated code coverage reports - it is as if the unit tests did not contribute anything to the coverage numbers. This is to be expected - because we only collected coverage from the application code loaded by the page during cy.visit command. We never instrumented the unit test code, the code loaded from the spec file.

Our .babelrc only told our application bundler to instrument the source code, Cypress Test Runner has no idea that it should instrument the spec code - instead Cypress uses its own bundler to process and load code.

Luckily, we can easily use the same babel-plugin-istanbul instrumentation to bundle our spec files (which includes code loaded directly from the spec files) and save the code coverage. Just tell Cypress to use ".babelrc" during bundling. We need to add the following to the cypress/plugins/index.js file.

cypress/plugins/index.js
1
2
3
4
5
6
7
8
const browserify = require('@cypress/browserify-preprocessor')

module.exports = (on, config) => {
  // tell Cypress to use .babelrc when bundling spec code
  const options = browserify.defaultOptions
  options.browserifyOptions.transform[1][1].babelrc = true
  on('file:preprocessor', browserify(options))
}

Of course, we also need to install @cypress/browserify-preprocessor npm package for this to work

1
$ npm install --save-dev @cypress/browserify-preprocessor

We don't have to change anything else - cypress-istanbul v1.1.0 already knows how to correctly merge code coverage from the application and from unit tests - and saves the combined code coverage report. The spec files themselves are NOT in the report, only the application source files are. The above unit tests cover the toTime function pretty well:

toTime covered by unit tests

What about the rest of the code? Everything gets covered - the DOM updates and battery API code get covered by end-to-end tests, while individual little functions are covered by the unit tests. When cypress-istanbul combines the coverage, it saves the report showing full 100% code coverage.

Full coverage

You can also see this coverage report at coveralls.io/github/bahmutov/demo-battery-api?branch=cover-unit-tests-3, the coverage information is set by the CI build job. You can find the source code at bahmutov/demo-battery-api repo in branch "cover-unit-tests-3".

See also