Save The Page On Test Failure

Sometimes a Cypress test fails on CI, and you cannot determine why the test has failed from the screenshot and video alone. For example, if a cy.get('[data-cy=...]') command fails to find an element, but you see the element in the screenshot and are wondering "what has happened...". Maybe the data attribute was missing due to some feature flag? Maybe it is a bug in Cypress? How can you look at the DOM of the page itself?

Here is where my cyclope comes in handy. It has a feature for saving the full page whenever you want, including automatically saving the page on test failure.

The app

Let's take an example application in the repo bahmutov/todo-app-for-cyclope and write a test. Since we don't want to spend a lot of time, we will write a single test to add and complete items.

cypress/integration/spec.js

/// <reference types="cypress" />

it('adds and removes todos', () => {
  cy.visit('/')
  cy.get('[data-cy="add-todo"]').clear()
  cy.get('[data-cy="add-todo"]').type(
    'record the test{enter}',
  )
  cy.get('[data-cy="add-todo"]').clear()
  cy.get('[data-cy="add-todo"]').type(
    'find the secret{enter}',
  )
  cy.get('[data-cy="add-todo"]').clear()
  cy.get('[data-cy="add-todo"]').type('use Studio{enter}')
  cy.get('[data-cy=todo]')
    .should('have.length', 3)
    .eq(1)
    .find('.cb-input')
    .check()
    .should('be.checked')
  cy.get('#items-left').should('have.text', '2')
  cy.get('#completed').click()
  cy.get('#completed').should('have.class', 'on')
})

The test passes successfully

Super, now let's run this test on CI. I picked CircleCI using the Cypress CircleCI Orb.

.circleci/config.yml

# https://github.com/cypress-io/circleci-orb
version: 2.1
orbs:
  cypress: cypress-io/cypress@1
workflows:
  build:
    jobs:
      - cypress/run:
          start: npm start
          wait-on: 'http://localhost:3777'
          config: 'baseUrl=http://localhost:3777'
          # no need to store the workspace, as there are no jobs after this one
          no-workspace: true

The test passes successfully on CI.

The failed test

Now I am getting too excited and I add one more assertion to verify the "Completed" view shows a single Todo.

cypress/integration/spec.js

  cy.get('#completed').click()
  cy.get('#completed').should('have.class', 'on')
+ cy.get('[data-cy=todo]:visible')
+   .should('have.length', 1)
+   .contains('find the secret')

I push the code without checking it first, and ... the test fails.

To debug the failure I could do three things:

1. Run the test locally, hoping it shows the same problem
2. Record the test to Cypress Dashboard
3. Store images and video automatically generated by Cypress on CircleCI as a test artifact

Let's store the test images and video on CI by adding the following to the Cypress Run job

.circleci/config.yml

  - cypress/run:
      ...
      no-workspace: true
+     store_artifacts: true

We can click on the MP4 file to watch the video, and click on the image filename to see the screenshot - CircleCI acts as a static HTML server by default. Let's look at the screenshot.

Notice something suspicious - cy.get('[data-cy=todo]') command has found 3 elements, but some of them were invisible, as indicated by the eye badge.

📺 Watch my video "Visibility Of Multiple Elements Explained"

So some elements were invisible, some visible, and that is why the test has failed. What are those three elements? This is where an image screenshot is not enough to say. We need to look at the page DOM elements to figure it out.

The cyclope plugin

We can install the cyclope using NPM or Yarn

$ npm i -D cyclope
# or using Yarn
$ yarn add -D cyclope

+ [email protected]

From the plugins file initialize the plugin and return the config object.

cypress/plugins/index.js

module.exports = (on, config) => {
  // https://github.com/bahmutov/cyclope
  require('cyclope/plugin')(on, config)

  // IMPORTANT to return the config object
  // with the any changed environment variables
  return config
}

In the support file register the afterEach hook to save the full page if a test fails

cypress/support/index.js

// https://github.com/bahmutov/cyclope
import { savePageIfTestFailed } from 'cyclope'
afterEach(savePageIfTestFailed)

You can see the plugin in action locally - it will save the full page in a subfolder of cypress/failed. Let's store this folder on CircleCI as an artifact.

.circleci/config.yml

  - cypress/run:
      ...
      no-workspace: true
      store_artifacts: true
+     post-steps:
+       - store_artifacts:
+           path: cypress/failed

Push the code changes and see the messages in the terminal telling the full page was saved.

Let's look at the "Artifacts" tab on CircleCI where the screenshots, videos, and the full pages are listed.

Click on the "index.html" and see the static page open right from CircleCI, which acts as a static web server. There is no JavaScript running - it is just a DOM snapshot at the moment of failure. Open the DevTools and inspect the todo area - and see that there are invisible Todo elements!

Ok, so the application just hides the Todo items when viewing Completed items. Thus our assertion has to limit itself to the visible items using the jQuery :visible pseudo selector.

- cy.get('[data-cy=todo]')
+ cy.get('[data-cy=todo]:visible')
    .should('have.length', 1)
    .contains('find the secret')

The test passes. We can see the video of the successful test stored on CircleCI. There are no screenshots and no full page screenshots since no tests have failed. Tip: you can store a full page HTML using cy.close(...) command provided by the cyclope plugin at any point during the test, similar to calling the cy.screenshot(...) command.