products

Code Coverage On The Fly

Use the "cy.intercept" command to instrument the application code and produce code coverage.

If you want to see which parts of your source code are covered by E2E tests, you can use code coverage. The problem is often instrumenting the code. Sometimes you don't have the control over the source code to insert the babel-plugin-istanbul. Sometimes you can only test the deployed uninstrumented application.

I have used a code coverage proxy before. It sits between the application and the E2E tests and instruments the returned application scripts on the fly. In this blog post, I will show how you can use the cy.intercept command to instrument JavaScript bundles on the fly. You can produce useful code coverage reports for any web application!

🎓 The example application for this blog post is in the repo bahmutov/tdd-calc. It is a demo app used in my TDD Calculator Cypress online course.

The example application is a standalone HTML page that downloads 2 scripts

public/index.html
1
<script type="module" src="./app.js"></script>
public/app.js
1
2
3
// calculator logic

import { appendDot } from './utils.js'
public/utils.js
1
2
3
4
5
// pure functions used during computation

export function appendDot(expression) {
  ...
}

Here is the spec that simply loads the app

cypress/e2e/code-coverage.cy.ts
1
2
3
4
5
import { CalculatorPage } from './calculator-po'

it('collects code coverage on the fly', () => {
  cy.visit('public/index.html')
})

The calculator app

We can see both scripts loaded

Scripts shown in the Network tab

Let's instrument those scripts

Custom intercept

Let's spy on the scripts loaded by the application. We can modify the outgoing browser request and the incoming response. I will remove the caching headers to always receive the script source.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
it('collects code coverage on the fly', () => {
  cy.intercept(
    {
      method: 'GET',
      resourceType: 'script',
    },
    (req) => {
      // delete the caching headers
      // to always get the full script source code
      delete req.headers['if-none-match']
      delete req.headers['if-modified-since']

      req.continue((res) => {
        res.body = '// instrumented ' + req.url + '\n' + res.body
        return res
      })
    },
  )
  cy.visit('public/index.html')
})

Great, we just modified each source file by inserting the script URL comment.

Modified source code

The URL looks random since Cypress serves a static file using a random local port. Let's remove the random part, we are only interested in the public/... filename.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
const baseUrl = Cypress.config('baseUrl')
const proxyServer = Cypress.config('proxyServer') + '/'

it('collects code coverage on the fly', () => {
  cy.intercept(
    {
      method: 'GET',
      resourceType: 'script',
    },
    (req) => {
      // delete the caching headers
      // to always get the full script source code
      delete req.headers['if-none-match']
      delete req.headers['if-modified-since']

      const relativeUrl = req.url
        .replace(baseUrl, '')
        .replace(proxyServer, '')

      req.continue((res) => {
        res.body = '// instrumented ' + relativeUrl + '\n' + res.body
        return res
      })
    },
  )
})

Cleaned up script URL

Sometimes the page loads its own scripts that we want instrumented plus 3rd-party scripts we want to skip. Let's use a wildcard pattern or a regular expression to pick the scripts to instrument.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
cy.intercept(
  {
    method: 'GET',
    resourceType: 'script',
    url: '**/public/**.js',
  },
  (req) => {
    // delete the caching headers
    // to always get the full script source code
    delete req.headers['if-none-match']
    delete req.headers['if-modified-since']

    const relativeUrl = req.url
      .replace(baseUrl, '')
      .replace(proxyServer, '')

    req.continue((res) => {
      res.body = '// instrumented ' + relativeUrl + '\n' + res.body
      return res
    })
  },
)

Only some scripts will be processed

Great.

Instrument the response

Now that we have the application's source code, let's instrument it. I will install the nyc module that includes the babel-plugin-istanbul dependency.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
import { CalculatorPage } from './calculator-po'
import { createInstrumenter } from 'istanbul-lib-instrument'

const instrumenter = createInstrumenter({
  esModules: true,
  compact: false,
  preserveComments: true,
})

const baseUrl = Cypress.config('baseUrl')
const proxyServer = Cypress.config('proxyServer') + '/'

it('collects code coverage on the fly', () => {
  cy.intercept(
    {
      method: 'GET',
      resourceType: 'script',
      url: '**/public/**.js',
    },
    (req) => {
      // delete the caching headers
      // to always get the full script source code
      delete req.headers['if-none-match']
      delete req.headers['if-modified-since']

      const relativeUrl = req.url
        .replace(baseUrl, '')
        .replace(proxyServer, '')

      req.continue((res) => {
        const instrumented = instrumenter.instrumentSync(
          res.body,
          relativeUrl,
        )
        res.body = instrumented
        return res
      })
    },
  )

  cy.visit('public/index.html')
})

Hmm, is it working? Let's look at the scripts as received by the browser.

Instrumented utils script

We can also check if the instrumentation succeeded by looking at the global window.__coverage__ object that keeps all the collected information.

The global coverage object

We can even confirm the code coverage is being collected from the test itself

1
2
3
4
5
6
7
cy.visit('public/index.html')
// after instrumenting the coverage should be collected in the window object
// under window.__coverage__ key
// There should be two keys: one for the main script and one for the spec
cy.window()
  .should('have.property', '__coverage__')
  .should('have.keys', ['public/app.js', 'public/utils.js'])

Checking the coverage object

Coverage report

The global window.__coverage__ object has all the function, branch, and statement counters. If we interact with our application, those counters get incremented. Let's produce the report after the tests are finished. I will install my @bahmutov/cypress-code-coverage plugin.

cypress.config.js
1
2
3
4
5
6
7
8
9
10
11
// in the E2E config
setupNodeEvents(on, config) {
  // implement node event listeners here
  // and load any plugins that require the Node environment

  // https://github.com/bahmutov/cypress-code-coverage
  require('@bahmutov/cypress-code-coverage/plugin')(on, config)

  // IMPORTANT to return the config object
  return config
}
cypress/support/e2e.js
1
2
// https://github.com/bahmutov/cypress-code-coverage
require('@bahmutov/cypress-code-coverage/support')

Finally, let's produce HTML and text summary report when the tests finish.

package.json
1
2
3
4
5
6
7
8
{
  "nyc": {
    "reporter": [
      "text",
      "html"
    ]
  }
}

When we run the test now, we can see the plugin logging its steps. Everything seems to be working.

Coverage plugin log message

Hmm, the coverage is pretty low: 15% of statements

Coverage is low

The coverage report

Open the HTML report located in the coverage folder

1
$ open coverage/index.html

The coverage report index page

We have some coverage for the public/app.js source script, but nothing for the public/utils.js script. Let's open the app.js report. It shows line by line coverage.

The code coverage for the app.js script

Of course: we simply visited the page, we never interacted with our calculator. Let's increase the code coverage by computing an expression.

1
2
3
4
5
6
7
8
9
10
11
// intercept code
cy.visit('public/index.html')
// after instrumenting the coverage should be collected in the window object
// under window.__coverage__ key
// There should be two keys: one for the main script and one for the spec
cy.window()
  .should('have.property', '__coverage__')
  .should('have.keys', ['public/app.js', 'public/utils.js'])

// compute an expression and see the increased code coverage
CalculatorPage.compute('1+2', '3')

Compute 1+2

The code coverage jumps to 45%

Updated code coverage report

Tip: if you use bundle splitting, the page might load only some scripts! The code coverage report shows the results for the loaded scripts, thus you might miss the source files that were never loaded.

Hmm, our utils.js bundle is still at zero. It has a single appendDot function

Zero coverage for the utils.js code

Our expression only has whole numbers, so the appendDot was never called! Let's update our test to "hit" the appendDot code

1
2
// compute an expression and see the increased code coverage
CalculatorPage.compute('1+2.1', '3.1')

The E2E test adds a floating-point number

Nice, the coverage shot up!

The utils coverage when using a floating-point number

We can look at the code coverage report as a treasure hunt map and keep adding tests until we go through all implemented code features.

The code coverage plugin

Finally, instead of implementing the instrumentation ourselves, we can use the @bahmutov/cypress-code-coverage plugin to instrument and report. Simply tell the plugin the wildcard pattern / regular expression to instrument.

cypress.config.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
const { defineConfig } = require('cypress')

module.exports = defineConfig({
  e2e: {
    // baseUrl, etc
    // configure files that cypress-watch-and-reload will watch
    env: {
      coverage: {
        // intercept and instrument scripts matching these URLs
        instrument: '**/public/*.js',
      },
    },
    // enable running all specs together
    // https://on.cypress.io/experiments
    experimentalRunAllSpecs: true,
    setupNodeEvents(on, config) {
      // implement node event listeners here
      // and load any plugins that require the Node environment
      // https://github.com/bahmutov/cypress-code-coverage
      require('@bahmutov/cypress-code-coverage/plugin')(on, config)

      // IMPORTANT to return the config object
      return config
    },
  },
})
cypress/e2e/code-coverage.cy.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
import { CalculatorPage } from './calculator-po'

it('collects code coverage on the fly', () => {
  cy.visit('public/index.html')
  // after instrumenting the coverage should be collected in the window object
  // under window.__coverage__ key
  // There should be two keys: one for the main script and one for the spec
  cy.window()
    .should('have.property', '__coverage__')
    .should('have.keys', ['public/app.js', 'public/utils.js'])

  // compute an expression and see the increased code coverage
  CalculatorPage.compute('1+2.1', '3.1')
})

The test produces the same report. Let's now run all tests using npx cypress run and see what the full coverage shows

The full code coverage after running all specs

Beautiful, we got all 100% statements covered and just a few "ELSE" branches were not tested (edge cases).

Note: the plugin does not use source map transformations to map the intercepted JS bundle back to individual files.