products

Backend Code Coverage from Cypress API tests

How to cover the entire backend code using Cypress API tests.

📦 you can find the source code for this blog post in the repository bahmutov/todomvc-express-api-test-coverage

Best practices

  • Instrument the backend JavaScript or TypeScript server using nyc and create coverage reports using @cypress/code-coverage plugin
  • Hit the backend from Cypress tests using cy.request
  • Set up the continuous integration and code coverage services early
  • Start by adding tests for the major features, this is will quickly increase the coverage percentage
  • Later concentrate on the edge cases using the code coverage report to find the missing source lines

The example application

Let's test a TodoMVC application - except our application will be completely rendered server-side. Here is how the application looks to the user

TodoMVC application UI

If we look at the source code we can see the sources disabled using Content-Security-Policy and the entire application just using forms.

TodoMVC application source code

Note: for demo purpose this application has artificial delays added to its code.

Cypress API tests

We could write end-to-end tests to operate on the app via its user interface, as I have done in the bahmutov/todomvc-express repo. But let's have some fun. Let's test our application by executing just API tests. We can use cy.request commands and assert the results. If we wanted we could even use cy.api custom command to show each request in the empty iframe, as I described in Black box API testing with server logs.

For example, let's validate that we can add a todo by executing a POST request.

cypress/integration/api-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
describe('TodoMVC API', () => {
  beforeEach(() => {
    cy.request('POST', '/reset')
  })

  it('adds a todo', () => {
    cy.request('/todos').its('body').should('have.length', 0)
    cy.request('POST', '/', {
      what: 'new todo',
    })
    cy.request('/todos')
      .its('body')
      .should('have.length', 1)
      .its('0')
      .should('include', {
        what: 'new todo',
        done: false,
      })
      .and('have.property', 'id')
      // our uuid is lowercase
      .should(
        'match',
        /^[0-9a-f]{8}\b-[0-9a-f]{4}\b-[0-9a-f]{4}\b-[0-9a-f]{4}\b-[0-9a-f]{12}$/,
      )
  })
})

The test passes - we can add a todo using API requests

The first API test

Show the result

We have an empty iframe where the application would normally be. Let's use it. We can simply load the base url using cy.visit('/') command at the end of the test.

1
2
3
4
5
6
7
8
9
10
// the same API test
cy.visit('/')

// now that the server response is in the test runner
// let's query it like a normal site
cy.get('.todo-list li')
  .should('have.length', 1)
  .first()
  .find('label')
  .should('have.text', 'new todo')

We have confirmed the todo has the expected text

Confirm the app shows the added todo

We can also go about this the other way - the GET / should return the rendered HTML page. Let's write this HTML into the document directly, without using cy.visit

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
// the same API test
cy.log('**render HTML**')
cy.request('/')
  .its('body')
  .then((html) => {
    cy.document().then((doc) => {
      doc.write(html)
    })
  })

// now that the server response is in the test runner
// let's query it like a normal site
cy.get('.todo-list li')
  .should('have.length', 1)
  .first()
  .find('label')
  .should('have.text', 'new todo')

The tested application looks the same way.

Writing the HTML into the document

Tip: a good way to always replace the document's HTML is to use a helper method

1
2
3
4
5
6
7
8
9
10
11
// replaces any current HTML in the document with new html
const writeHtml = (html) => {
  cy.document().then((doc) => {
    doc.open()
    doc.write(html)
    doc.close()
  })
}

cy.log('**render HTML**')
cy.request('/').its('body').then(writeHtml)

Code coverage

Let's measure the backend code in the "src" folder we exercise with our test. We can install cypress-io/code-coverage to generate the reports and nyc to instrument our Node.js server.

1
2
3
$ npm i -D nyc @cypress/code-coverage
+ [email protected]
+ @cypress/[email protected]

To instrument the server prepend the node command with nyc in the package.json start script

1
2
- "start": "node src/start.js"
+ "start": "nyc --silent node src/start.js"

And add an endpoint returning the code coverage object when needed. Since our application uses Express.js we can use the Express helper code included with @cypress/code-coverage

src/app.js
1
2
3
4
5
6
7
8
const express = require('express')
const app = express()
// returns code coverage information if available
// https://github.com/cypress-io/code-coverage
/* istanbul ignore next */
if (global.__coverage__) {
  require('@cypress/code-coverage/middleware/express')(app)
}

Our Cypress config file tells the plugin where to find the backend code coverage. It also tells the plugin not to expect any frontend code coverage - since the app is purely HTML.

cypress.json
1
2
3
4
5
6
7
8
9
{
  "baseUrl": "http://localhost:3000",
  "env": {
    "codeCoverage": {
      "url": "http://localhost:3000/__coverage__",
      "expectBackendCoverageOnly": true
    }
  }
}

The application runs and shows messages from the code coverage plugin.

Messages from the code coverage plugin

If you click on the last message "Generating report" you can find the output folder printed to the DevTools console

The output code coverage report folder

The folder coverage contains reports in multiple formats: html, clover, JSON, lcov. I mostly look at the html report

1
$ open coverage/lcov-report/index.html

Not bad - our single test covers 72% of the backend source code

Code coverage after one test

Continuous integration service

Before we write more tests, I want to run E2E tests on CI. I will pick CircleCI because I can use Cypress orb to run tests, and because I can store the code coverage folder as a test artifact there.

.circleci/config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
version: 2.1
orbs:
  cypress: cypress-io/cypress@1

commands:
  report-coverage:
    description: Store coverage report as an artifact
    steps:
      - store_artifacts:
          path: coverage
      - run: npx nyc report --reporter=text || true

workflows:
  build:
    jobs:
      - cypress/run:
          start: npm start
          wait-on: 'http://localhost:3000'
          no-workspace: true
          post-steps:
            - report-coverage

Out tests pass and we can see the coverage report - it is a static HTML file served directly by CircleCI. Just click on the "Test Artifacts" tab in the job and select the "index.html" link.

Viewing the code coverage report as CircleCI test artifact

Code coverage service

We want to ensure that our app works. Thus we want to test every commit and every pull request using the CI service. Similarly, we want to ensure that every feature we add to the web application is tested at the same time. The simplest way to ensure our code coverage increases with new features, and does not increase with code refactorings is to use code coverage as a service. For this example, I picked codecov.io and you can see the latest results for this blog post's example repository at todomvc-express-api-test-coverage URL.

After every CI run, we need to upload the code coverage report (the JSON file in this case) to Codecov service. The simplest way to do this is to use their orb

.circleci/config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
version: 2.1
orbs:
  cypress: cypress-io/cypress@1
  codecov: codecov/codecov@1

commands:
  report-coverage:
    description: Store coverage report as an artifact
    steps:
      - store_artifacts:
          path: coverage
      - run: npx nyc report --reporter=text || true
      - codecov/upload:
          file: coverage/coverage-final.json

workflows:
  build:
    jobs:
      - cypress/run:
          start: npm start
          wait-on: 'http://localhost:3000'
          no-workspace: true
          post-steps:
            - report-coverage

We literally added 3 lines to our YML file: Codecov figures out the commit, the branch, the pull request automatically using the environment variables.

1
2
3
codecov: codecov/codecov@1
- codecov/upload:
    file: coverage/coverage-final.json

The dashboard shows approximately the same numbers (Codecov uses line coverage by default)

Codecov report

Codecov GitHub app

We will write more tests, but first let's ensure that every pull request increases the code coverage or at least keeps it the same. The simplest way to check the pull request against the current baseline coverage number is by installing a Codecov GitHub application.

After the application has been installed and configured to work on [bahmutov/todomvc-express-api-test-coverage] todomvc-express-api-test-coverage let's open a pull request. Let's inspect the source files with low code coverage to find a feature we are not testing yet.

Inspecting the code coverage to find untested code

By inspecting the coverage report we see that deleting todos has not been tested yet. Let's write a test and open a pull request.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
it('deletes todo', () => {
  cy.request('POST', '/', {
    what: 'new todo',
  })
  cy.request('/todos')
    .its('body')
    .should('have.length', 1)
    .its('0.id')
    .then((id) => {
      cy.request('DELETE', '/', { id })
    })

  // after deleting the todo, we should get back to zero todos
  cy.request('/todos').its('body').should('have.length', 0)
})

Delete todo test

Let's open the pull request #1. The CI runs and the code coverage is sent to Codecov, which posts status checks on the pull request.

CI and code coverage status checks

The status checks tell us the code coverage has increased by almost 6%. The more details are available in the PR comment posted by Codecov.

Code coverage details comment posted by Codecov

We can click on the comment to open the Codecov PR page where we can find every source file with changed code coverage. For example the src/app.js shows the following lines are now covered.

Several source lines previously not covered are now being tested

We can safely merge this pull request. The tests pass and the code coverage increases.

Increasing code coverage

We can repeat the process several times:

  1. inspect the code coverage report to find major features without tests
  2. write an API test to exercise the feature
  3. open a pull request and merge if tests pass and code coverage increases

Mark todo completed

For example, marking a todo completed needs a test.

Mark todo source lines are not covered

Let's write a test

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
it('completes todo', () => {
  cy.request('POST', '/', {
    what: 'new todo',
  })
  cy.request('/todos')
    .its('body')
    .should('have.length', 1)
    .its('0.id')
    .then((id) => {
      cy.request('PATCH', '/', { id, done: 'true' })

      // confirm the todo was marked
      cy.request('/todos')
        .its('body')
        .should('deep.equal', [
          {
            id,
            what: 'new todo',
            done: true,
          },
        ])
    })
})

The test passes locally

Mark todo test

The pull request#2 increases the code coverage by another 6%.

Mark todo pull request code coverage comment

We merge it.

Clear completed todos

Next we can add a test for uncovered lines for clearCompleted function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
it('clears completed todo', () => {
  // let's add two todos and mark one of them completed
  // after clearing the completed todos we can check
  // that only 1 item remains
  cy.request('POST', '/', {
    what: 'first todo',
  })
  cy.request('POST', '/', {
    what: 'second todo',
  })
  cy.request('/todos')
    .its('body')
    .should('have.length', 2)
    .then((todos) => {
      // confirm the order of returned todos
      // our application returns the last added todo first
      expect(todos[0], 'first item').to.contain({
        what: 'second todo',
        done: false,
      })
      expect(todos[1], 'second item').to.contain({
        what: 'first todo',
        done: false,
      })

      cy.request('PATCH', '/', { id: todos[1].id, done: 'true' })
      cy.request('POST', '/clear-completed')
      // confirm a single item remains
      cy.request('/todos').its('body').should('deep.equal', [todos[0]])
    })
})

Clear completed todos test

The pull request #3 increases the code coverage by 4%

Clear completed todos pull request status checks

Individual pages

Inspecting the lines not covered by tests we can find handlers for the individual pages

Page handlers were not covered by the tests

The above functions are called in response to the individual page routes

src/app.js
1
2
3
4
app.get('/todo/:id', sendTodoPage)
app.get('/active', activeTodosPage)
app.get('/completed', completedTodosPage)
app.get('/todos', sendTodos)

Let's test these pages. For example to test the individual todo item page, we could do

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
it('has todo page', () => {
  cy.request('POST', '/', {
    what: 'new todo',
  })

  cy.request('/todos')
    .its('body')
    .should('have.length', 1)
    .its('0.id')
    .then((id) => {
      cy.log('**render todo page HTML**')
      const url = `/todo/${id}`
      cy.request(url)
        .its('body')
        .then((html) => {
          cy.document().then((doc) => {
            doc.write(html)
          })
        })
      // and confirm a single todo is shown and it is a link to itself
      cy.get('.todo-list li')
        .should('have.length', 1)
        .first()
        .find('label')
        .should('have.text', 'new todo')
        .find('a')
        .should('have.attr', 'href', url)
    })
})

Single todo page test

We can create a couple todos, mark one completed, the other should be displayed on the active page

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
it('has active page', () => {
  cy.request('POST', '/', {
    what: 'first todo',
  })
  cy.request('POST', '/', {
    what: 'second todo',
  })
  cy.request('/todos')
    .its('body')
    .should('have.length', 2)
    .then((todos) => {
      cy.request('PATCH', '/', { id: todos[1].id, done: 'true' })

      cy.log('**the first todo is active**')
      cy.visit('/active')
      cy.get('.todo-list li')
        .should('have.length', 1)
        .first()
        .find('label')
        .should('have.text', todos[0].what)
    })
})

Similarly, we can confirm the /completed page shows only the completed items, and every item has class "completed"

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
it('has completed page', () => {
  cy.request('POST', '/', {
    what: 'first todo',
  })
  cy.request('POST', '/', {
    what: 'second todo',
  })
  cy.request('/todos')
    .its('body')
    .should('have.length', 2)
    .then((todos) => {
      cy.request('PATCH', '/', { id: todos[1].id, done: 'true' })

      cy.log('**the second todo is completed**')
      cy.visit('/completed')
      cy.get('.todo-list li')
        .should('have.length', 1)
        .first()
        .should('have.class', 'completed')
        .find('label')
        .should('have.text', todos[1].what)
    })
})

Completed todos page test

The pull request #4 increases the code coverage by another 5% to 96%

Pull request to test special pages

Covering the edge cases

So far we have looked at the code coverage and added the tests for the features, like adding todos, deleting them, marking them completed. We have reached 96% code coverage this way. What remains now are the individual uncovered lines that really are edge cases in our code. For example, trying to delete a non-existent todo item is an edge case:

An edge case not covered by our delete test

Usually such edge cases are hard to reach through the typical UI - after all, if the application has been wired correctly it should not try deleting a non-existent todo. The API tests on the other hand are ideal - we can just execute a DELETE action passing a random ID to reach these lines.

When adding an edge case test to the "delete" feature, I like to organize such tests under a suite of tests.

1
2
3
4
5
6
7
8
// instead of a single "deletes todo" test
describe('TodoMVC API', () => {
  context('delete', () => {
    it('deletes todo', () => { ... })

    it('handles non-existent ID', () => { ... })
  })
})

💡 As the suites of tests grow you can move them into own spec file, and even split into a group of specs kept in a subfolder, see Make Cypress Run Faster by Splitting Specs.

The test its the delete endpoint

1
2
3
4
5
6
7
8
it('handles non-existent ID', () => {
  cy.request('POST', '/', {
    what: 'new todo',
  })
  cy.request('/todos').its('body').should('have.length', 1)
  cy.request('DELETE', '/', { id: 'does-not-exist' })
  cy.request('/todos').its('body').should('have.length', 1)
})

It passes

The delete non-existent ID test passes

When working on the edge cases, I like running the test by itself and then checking the coverage report saved locally after the test. I see the above test precisely hits the lines we wanted to hit.

The test reaches the line we wanted to cover

You can find this and other edge case tests in the pull request #5. Covering the edge cases pushes the total code coverage to 99.5%

Edge cases pull request code coverage comment

The last step to reach 100

If we look at the report and drill into the missing coverage, we see that our tests missed just a single source line

Single missing line

This line is not needed, the "clear completed" button works even without a JavaScript handler, since it is part of the form. In fact, this code line is a leftover and should not remain in our server-rendered app. We can confirm this by removing the source line and adding a test that clicks on the "Clear completed" button. Because our target is the "render" UI code, we want to test this code by visiting the page and interacting with the UI elements, rather than by simply hitting the API endpoints

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
it('clears completed todos by clicking the button', () => {
  cy.request('POST', '/', {
    what: 'first todo',
  })
  cy.request('POST', '/', {
    what: 'second todo',
  })
  cy.visit('/')
  cy.get('.todo-list li')
    .should('have.length', 2)
    .first()
    .find('button[name=done]')
    .click()
  // after the page reloads 1 item should have class completed
  cy.get('.todo-list li.completed').should('have.length', 1)
  cy.contains('button', 'Clear completed').click()
  cy.get('.todo-list li')
    .should('have.length', 1)
    .and('not.have.class', 'completed')
})

The test passes

Clear completed test passes

The pull request #6 shows the result

We have reached 100% code coverage for our backend code

100%

See also