Element coverage for end-to-end tests

Let us take a simple TodoMVC application, from cypress-example-todomvc for example, and add a single Cypress end-to-end test like this one

beforeEach(() => {
  // baseUrl in cypress.json is set to http://localhost:8888
  cy.visit('/')
})
it('works', function () {
  cy.get('.new-todo').type('first todo{enter}')
  cy.get('.new-todo').type('second todo{enter}')
  cy.get('.todo-list li').should('have.length', 2)
})

The test passes.

But what did this test cover? Code coverage is only useful for unit tests. In the final web application, the code reachable from the user interface is probably a very small part of the total bundle; plus there is a lot of vendor code and polyfill libraries - all rendering 100% code coverage a mirage.

What can we do instead of code coverage? Well, looking at the application the page elements are a natural candidate, isn't it? Did our test type into the input field? Yes, it did! Did the test click on the check box to mark a todo item completed? No, our test did not do that. Our tests also did not click on the filters "All, Active, Completed" at the bottom.

It would be nice to show the elements covered by the tests, so we can easily see how to extend a test, or maybe write another one, in order to cover the entire application's user interface.

If you are using Cypress like I do here, this is a cool experiment to try. You can find the code I am about to write in a branch mark-touched-dom-elements of cypress-example-todomvc.

First, let us collect all elements the test types into using cy.type command. We can detect the type command by overwriting it.

cypress/support/commands.js

Cypress.Commands.overwrite('type', function (type, $el, text, options) {
  // collect $el reference? or its selector
  console.log($el.selector)
  return type($el, text, options)
})

The same test now prints the selector .new-todo twice in the DevTools console.

The jQuery property .selector has been deprecated, and is not very useful. Instead I found NPM package @medv/finder that is well-tested and produces quite nice selectors even for deeply nested elements. Let's use this module, and also let us collect all selectors we see.

cypress/support/commands.js

const finder = require('@medv/finder').default

before(() => {
  window.testedSelectors = []
})

const getSelector = ($el) => {
  if ($el.attr('data-cy')) {
    // prefer data-cy="..." attribute
    return `[data-cy=${$el.attr('data-cy')}]`
  }

  // or use finder module
  return finder($el[0], {
    // a trick to point "finder" at the application's iframe
    root: cy.state('window').document.body,
  })
}

Cypress.Commands.overwrite('type', function (type, $el, text, options) {
  const selector = getSelector($el)

  window.testedSelectors.push(selector)

  return type($el, text, options)
})

Now that we have collected a list of selectors, we should do something after all tests finish. For example, we could highlight all those elements with a nice magenta border.

after(() => {
  const selectors = Cypress._.uniq(window.testedSelectors)

  // eslint-disable-next-line no-console
  console.log('tested the following selectors:', selectors)

  // shortcut to get application's window context
  // without going through cy.window() command
  const win = cy.state('window')

  selectors.forEach((selector) => {
    const el = win.document.querySelector(selector)

    if (el) {
      el.style.opacity = 1
      el.style.border = '1px solid magenta'
    }
  })
})

The test runs again, and we see the input element highlighted.

Let us mark one item completed and see it. We are going to use cy.check command in the test, and we will overwrite it in our cypress/support/commands.js file

// cypress/integration/add_spec.js
it('works', function () {
  cy.get('.new-todo').type('first todo{enter}')
  cy.get('.new-todo').type('second todo{enter}')
  cy.get('.todo-list li').should('have.length', 2)
  .first()
  .find(':checkbox').check()
})
// cypress/support/commands.js
Cypress.Commands.overwrite('check', function (check, $el, options) {
  const selector = getSelector($el)

  window.testedSelectors.push(selector)

  return check($el, options)
})

The found selector "li:nth-child(1) .toggle" is not the most attractive one, but if we want a better selector we should set a data attribute on the element in our render function, right?

Finally, let us cover the filters at the bottom, since we now have both completed and unfinished items in the list.

// cypress/integration/add_spec.js
it('works', function () {
  cy.get('.new-todo').type('first todo{enter}')
  cy.get('.new-todo').type('second todo{enter}')
  cy.get('.todo-list li').should('have.length', 2)
  .first()
  .find(':checkbox').check()

  cy.contains('.filters a', 'Active').click()
  cy.url().should('include', 'active')

  cy.contains('.filters a', 'Completed').click()
  cy.url().should('include', 'completed')

  cy.contains('.filters a', 'All').click()
  cy.url().should('include', '#/')
})
// cypress/support/commands.js
Cypress.Commands.overwrite('click', function (click, $el, options) {
  const selector = getSelector($el)

  window.testedSelectors.push(selector)

  return click($el, options)
})

Now a lot of visual elements are covered by the test

Even selectors look nice, because I have added attributes in footer.jsx

<a
  href="#/"
  data-cy="show-all"
  className={cx({selected: nowShowing === app.ALL_TODOS})}>
    All
</a>

Using data-* attributes for testing is our recommended best practice when selecting elements.

Limitations

While seeing the elements covered by the test is nice, and you can even take a screenshot at the end using cy.screenshot, it is of limited utility. Because while it marks an element the test has acted upon (typed into, checked, clicked), it does not show what kind of interaction it was, or even if the interaction has covered all possible interactions. For example, our test checked an checkbox element, and we see this, our test has NOT unchecked it. To me, this screams to require state coverage, rather than element coverage.