Full Code Coverage For Free

In the previous blog post Run Cypress Specs In Parallel For Free I introduced my plugin cypress-split that lets you run Cypress specs in parallel without using any paid external service. In this blog post I will show my solution to easily run Cypress end-to-end and component tests with code coverage in parallel and then combine the produced code coverage reports into a single one. With this approach, getting 100% code coverage is easy:

• use end-to-end tests to quickly cover lots of web application code
• use component tests to test edge cases that are hard to reach through normal end-to-end flows

🎓 This blog post summarizes what I cover in my course Testing The Swag Store. You can take that course to practice each step yourself. The example application I am using is in the repo bahmutov/taste-the-sauce.

The plugins

First, these are the plugins I am using to split the specs and collect code coverage

• cypress-split to run specs in parallel.
• @bahmutov/cypress-code-coverage to generate code coverage reports.

For E2E tests, I am instrumenting the application using @cypress/instrument-cra, but any Istanbul.js-compatible code coverage tool could be used. For component tests I am inserting babel-plugin-istanbul into the transpile pipeline by adding it to the Webpack options.

Because I need to split both E2E and component Cypress specs, my Cypress config file looks like you are seeing double.

cypress.config.js

const { defineConfig } = require('cypress')
// https://github.com/bahmutov/cypress-on-fix
const cypressOnFix = require('cypress-on-fix')
// https://github.com/bahmutov/cypress-split
const cypressSplit = require('cypress-split')
const path = require('path')

module.exports = defineConfig({
  e2e: {
    // baseUrl, etc
    baseUrl: 'http://localhost:3000',
    setupNodeEvents(cypressOn, config) {
      // implement node event listeners here
      // and load any plugins that require the Node environment
      // fix https://github.com/cypress-io/cypress/issues/22428
      const on = cypressOnFix(cypressOn)
      cypressSplit(on, config)
      // https://github.com/bahmutov/cypress-watch-and-reload
      require('cypress-watch-and-reload/plugins')(on, config)
      // https://github.com/bahmutov/cypress-code-coverage
      require('@bahmutov/cypress-code-coverage/plugin')(on, config)
      // IMPORTANT to return the config object
      // with the any changed environment variables
      return config
    },
  },

  component: {
    setupNodeEvents(cypressOn, config) {
      const on = cypressOnFix(cypressOn)
      cypressSplit(on, config)
      // https://github.com/bahmutov/cypress-code-coverage
      require('@bahmutov/cypress-code-coverage/plugin')(on, config)
      // IMPORTANT to return the config object
      // with the any changed environment variables
      return config
    },
    devServer: {
      framework: 'create-react-app',
      bundler: 'webpack',
      webpackConfig: {
        mode: 'development',
        devtool: false,
        module: {
          rules: [
            // application and Cypress files are bundled like React components
            // and instrumented using the babel-plugin-istanbul
            {
              test: /\.jsx?$/,
              // do not instrument node_modules
              // or Cypress component specs
              exclude: /node_modules|\.cy\.js/,
              use: {
                loader: 'babel-loader',
                options: {
                  presets: ['@babel/preset-env', '@babel/preset-react'],
                  plugins: ['istanbul'],
                },
              },
            },
          ],
        },
      },
    },
  },
})

Tip: I am using cypress-on-fix because sometimes we have to work around bugs on our own.

The GitHub Actions workflow

I love using GitHub Actions and often ask myself "Is there anything GH Actions cannot do?". Usually, the answer is No. In my case, I want to install all NPM dependencies, run E2E tests in parallel, run component tests in parallel, and then combine the code coverage reports into a single report. Here is the GH Actions workflow file that calls my reusable cypress-workflows.

name: ci
on: [push]
jobs:
  # run end-to-end and component tests in parallel
  tests:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      nE2E: 4 # use 4 containers for E2E tests
      nComponent: 2 # use 2 containers for component tests
      # for E2E tests need to start the web app
      start: npm run start-18
      wait-on: 'http://127.0.0.1:3000'
      # merge E2E and component code coverage into a single report
      coverage: true

That is it, just a single reusable workflow with input parameters. We allocate 4 containers to E2E specs and 2 for component specs. The workflow spawns these containers at the same time, cutting the total testing time. Want to run tests faster? Bump the nE2E and nComponent inputs.

Tip: you might be wondering what the "merge-reports" job is in the above workflow. Read the blog post Cypress Mochawesome HTML Reports.

All code coverage and other testing artifacts, like screenshots and videos are uploaded to the GitHub Action page.

You can simply look at the code coverage summary written as a table. We are doing pretty well there, fully testing an online web store.

Or you can download the merged code coverage Zip file and open the included HTML report

Using the cypress-split and @bahmutov/cypress-code-coverage plugins plus cypress-workflows split v2 on GitHub Actions is like a breath of fresh air. A few lines of code and the full testing cycle completes in a few minutes.

I am happily testing 🎉

Update 1: Performance

Instrumenting source code and running end-to-end tests probably adds a little bit of the overhead and increases the total test run time. Here is my advice about the test run performance: it does not matter. The code coverage overheads makes the tests take maybe 25% longer to finish. It does not matter, because you can simply increase the nE2E parameter to keep the overall testing time the same!

tests:
  uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
  with:
    # nE2E: 4 # use 4 containers for E2E tests
    # use more containers to keep the test run short
    nE2E: 5

Problem solved. Remember: the human developer's time is very expensive. Allowing untested code that might have a bug into the production is very expensive. One more CI container to run your tests is very very very cheap.