Element coverage for end-to-end tests

How to see which UI elements the end-to-end tests have interacted with.

Let us take a simple TodoMVC application, from cypress-example-todomvc for example, and add a single Cypress end-to-end test like this one

1
2
3
4
5
6
7
8
9
beforeEach(() => {
// baseUrl in cypress.json is set to http://localhost:8888
cy.visit('/')
})
it('works', function () {
cy.get('.new-todo').type('first todo{enter}')
cy.get('.new-todo').type('second todo{enter}')
cy.get('.todo-list li').should('have.length', 2)
})

The test passes.

The above test passes, there are two items in the list

But what did this test cover? Code coverage is only useful for unit tests. In the final web application, the code reachable from the user interface is probably a very small part of the total bundle; plus there is a lot of vendor code and polyfill libraries - all rendering 100% code coverage a mirage.

What can we do instead of code coverage? Well, looking at the application the page elements are a natural candidate, isn't it? Did our test type into the input field? Yes, it did! Did the test click on the check box to mark a todo item completed? No, our test did not do that. Our tests also did not click on the filters "All, Active, Completed" at the bottom.

It would be nice to show the elements covered by the tests, so we can easily see how to extend a test, or maybe write another one, in order to cover the entire application's user interface.

If you are using Cypress like I do here, this is a cool experiment to try. You can find the code I am about to write in a branch mark-touched-dom-elements of cypress-example-todomvc.

First, let us collect all elements the test types into using cy.type command. We can detect the type command by overwriting it.

cypress/support/commands.js
1
2
3
4
5
Cypress.Commands.overwrite('type', function (type, $el, text, options) {
// collect $el reference? or its selector
console.log($el.selector)
return type($el, text, options)
})

The same test now prints the selector .new-todo twice in the DevTools console.

We have typed into .new-todo element

The jQuery property .selector has been deprecated, and is not very useful. Instead I found NPM package @medv/finder that is well-tested and produces quite nice selectors even for deeply nested elements. Let's use this module, and also let us collect all selectors we see.

cypress/support/commands.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
const finder = require('@medv/finder').default

before(() => {
window.testedSelectors = []
})

const getSelector = ($el) => {
if ($el.attr('data-cy')) {
// prefer data-cy="..." attribute
return `[data-cy=${$el.attr('data-cy')}]`
}

// or use finder module
return finder($el[0], {
// a trick to point "finder" at the application's iframe
root: cy.state('window').document.body,
})
}

Cypress.Commands.overwrite('type', function (type, $el, text, options) {
const selector = getSelector($el)

window.testedSelectors.push(selector)

return type($el, text, options)
})

Now that we have collected a list of selectors, we should do something after all tests finish. For example, we could highlight all those elements with a nice magenta border.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
after(() => {
const selectors = Cypress._.uniq(window.testedSelectors)

// eslint-disable-next-line no-console
console.log('tested the following selectors:', selectors)

// shortcut to get application's window context
// without going through cy.window() command
const win = cy.state('window')

selectors.forEach((selector) => {
const el = win.document.querySelector(selector)

if (el) {
el.style.opacity = 1
el.style.border = '1px solid magenta'
}
})
})

The test runs again, and we see the input element highlighted.

See all page elements our test typed into

Let us mark one item completed and see it. We are going to use cy.check command in the test, and we will overwrite it in our cypress/support/commands.js file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// cypress/integration/add_spec.js
it('works', function () {
cy.get('.new-todo').type('first todo{enter}')
cy.get('.new-todo').type('second todo{enter}')
cy.get('.todo-list li').should('have.length', 2)
.first()
.find(':checkbox').check()
})
// cypress/support/commands.js
Cypress.Commands.overwrite('check', function (check, $el, options) {
const selector = getSelector($el)

window.testedSelectors.push(selector)

return check($el, options)
})

The found selector "li:nth-child(1) .toggle" is not the most attractive one, but if we want a better selector we should set a data attribute on the element in our render function, right?

Elements the test typed into and checked

Finally, let us cover the filters at the bottom, since we now have both completed and unfinished items in the list.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// cypress/integration/add_spec.js
it('works', function () {
cy.get('.new-todo').type('first todo{enter}')
cy.get('.new-todo').type('second todo{enter}')
cy.get('.todo-list li').should('have.length', 2)
.first()
.find(':checkbox').check()

cy.contains('.filters a', 'Active').click()
cy.url().should('include', 'active')

cy.contains('.filters a', 'Completed').click()
cy.url().should('include', 'completed')

cy.contains('.filters a', 'All').click()
cy.url().should('include', '#/')
})
// cypress/support/commands.js
Cypress.Commands.overwrite('click', function (click, $el, options) {
const selector = getSelector($el)

window.testedSelectors.push(selector)

return click($el, options)
})

Now a lot of visual elements are covered by the test

Filter links are now covered by the test

Even selectors look nice, because I have added attributes in footer.jsx

1
2
3
4
5
6
<a
href="#/"
data-cy="show-all"
className={cx({selected: nowShowing === app.ALL_TODOS})}>
All
</a>

Using data-* attributes for testing is our recommended best practice when selecting elements.

Limitations

While seeing the elements covered by the test is nice, and you can even take a screenshot at the end using cy.screenshot, it is of limited utility. Because while it marks an element the test has acted upon (typed into, checked, clicked), it does not show what kind of interaction it was, or even if the interaction has covered all possible interactions. For example, our test checked an checkbox element, and we see this, our test has NOT unchecked it. To me, this screams to require state coverage, rather than element coverage.