Connecting crash reporting with end to end tests

I love love love crash reporting services like Sentry. Nothing allows me to quickly fix a bug that causes a crash like knowing about it the instant it happens.

Recently I started using Cypress to perform end to end testing of my web applications. This tool is amazing in its simplicity and power; it quickly replaced PhantomJS / CasperJS / Nightmare / Selenium / Protractor tools I have used before.

Once the number of tests has grown we hit a problem. During automated testing there might be crashes. These crashes are reported to Sentry (as they should!) When a developer looks in Sentry at the exception's information, there is very little that can tie a particular crash to one of the E2E tests that caused it. In some cases the crash might be severe enough to fail the test. Often the crash happens, but the E2E test still passes because it looks at a feature unaffected by this particular error.

Finding and fixing the root cause of these crashes becomes a guessing game; looking at bread crumbs and trying to figure out which tests might leave them is time-consuming. If only we could tell right away from the exception tags which particular E2E test caused it!

This is why I wrote send-test-info function - a tiny utility that grabs the name of each test (and optionally the spec filename) and attaches it to the Raven client inside the window object.

Using send-test-info is very simple. Install from NPM and include from your Cypress spec file:

const sendTestInfo = require('../..')
sendTestInfo(__filename)

describe('Test page', function () {
  // unit tests
})

Under the hood sendTestInfo uses beforeEach BDD hook to grab the full test title.

function sendTestInfo (spec) {
  beforeEach(function () {
    const testName = this &&
      this.currentTest && this.currentTest.fullTitle() || 'anonymous'
    const info = {testName}
    // set the info object as context in Raven
  })
}

To actually access Raven object we need a reference to the "window" object and wait until Raven becomes available (which might be after going to another page for example after login). There are several ways to do this, I picked an interval timer.

function sendTestInfo (spec) {
  beforeEach(function () {
    // grabbed test title
    cy.window()
      .then(w => {
        // we are not clearing the interval because the page
        // might be reloaded and the first Raven replaced with another one
        setInterval(() => {
          if (hasRaven(w)) {
            w.Raven
              .setExtraContext(info)
              .setTagsContext(info)
          }
        }, 5000)
      })
  })
}

The result

A simple E2E test included with the project shows that an exception sent over the wire includes additional fields.

The Sentry issue view can now immediately tell us which spec file should be executed to debug the error, and which particular test caused it.

Alternatives

There could be another way to achieve the same result without setting intervals - one could inject a script into each page and use the Raven's "on captured exception" callback to attach additional test information. In the future I hope to use this alternative approach to send more detailed intelligent test steps collected by Cypress to Sentry, allowing me to quickly determine when the crash happened. I see it being very close to the list shown by the Cypress UI in the left column

Visit <url>
Assert 'title' equals 'title'
Click button 'Run'
Crash!

This information would make debugging crashes that happened during E2E tests a pleasure rather than a chore.

Related

• There more run time checks you use the better. I use lazy assertions liberally throughout my code because I do not trust the server or the user. You can even verify the data schema by building the runtime type check right into the constructors using Archetype.
• I have written many blog posts about crash reporting
• Web testing nirvana with Cypress

Update 1

I added a listener that grabs Cypress events (try using Cypress.on('all') to start) and adds them to the Raven breadcrumbs list. The Sentry then shows a very useful list of test steps before the crash happens.

Update 2

In the newer versions of Sentry for React, I am setting the test information using the scope

import * as Sentry from '@sentry/browser';
// in the constructor
if (typeof window !== 'undefined' && window.Cypress) {
  const tags = {
    spec: window.Cypress.spec.relative,
    title: window.Cypress.currentTest.titlePath.join(' / '),
  };
  Sentry.configureScope(scope => {
    scope.setTags(tags);
  });
}