Testing Sentry Call with Cypress

I love using Sentry for tracking errors in my web applications. Recently I have added crash reporting to our open source 350-mass-cambridge-somerville/350-actions-client project in pull request #24 - see the app at https://www.maclimateactions.com/.

The entire web application is tested using Cypress - let's validate that the app is sending a crash report to Sentry service using an end-to-end test too. Let's start a placeholder test that opens the app and does nothing.

describe('current action', () => {
  beforeEach(() => {
    cy.server()
    cy.route('/actioncards/latest/', 'fixture:latest')
    cy.visit('/')
    // check the page - should have info from the stubbed response
    // loaded from cypress/fixtures/latest.json
    cy.contains('Action Card 23').should('be.visible')
    cy.get('[data-cy=action-check-display]').should('have.length', 6)
  })

  it.only('sends an error to Sentry', () => {
    // TODO write test commands
  })
})

Once the test runs and stops, open the DevTools inside Cypress. Switch to the application's context using a drop down. A nice trick to throw an unhandled exception in the application, and not in the DevTools it to make it asynchronous - for example by using setTimeout function. You can do setTimeout(() => { throw new Error('test error') }, 1000) and see Sentry handler executing an XHR call.

Note: by default, Sentry uses fetch protocol to send the error object to its service API. Since Cypress does not support stubbing fetch requests yet, we force every client-side library to drop down to XHR using the example from "Stubbing window.fetch" recipe. We simply delete the window.fetch property when every window is created.

cypress/support/index.js

let polyfill

// grab fetch polyfill from remote URL, could be also from a local package
before(() => {
  const polyfillUrl = 'https://unpkg.com/unfetch/dist/unfetch.umd.js'

  cy.request(polyfillUrl).then(response => {
    polyfill = response.body
  })
})

Cypress.on('window:before:load', win => {
  delete win.fetch
  // since the application code does not ship with a polyfill
  // load a polyfilled "fetch" from the test
  win.eval(polyfill)
  win.fetch = win.unfetch
})

Let's confirm this call happens. We need to prepare to intercept the call from the test. Look at the call's details, especially at the method and the destination URL.

Also look at the response object the Sentry service sends back to the caller.

Let's stub this route using cy.route command and response with similar object.

it('sends an error to Sentry', () => {
  cy.route('POST', 'https://sentry.io/api/*/store/*', {
    id: 'abc123',
  }).as('sentry')
  // create an error, as if application has thrown it
  cy.window().invoke(
    'setTimeout',
    () => {
      throw new Error('test error')
    },
    1000,
  )
  // confirm the call has happened
  cy.wait('@sentry')
})

The test runs - and we can see the stubbed XHR call to Sentry correctly detected (and stopped)

Great, but - a thrown error is still a thrown error, and Cypress treats is as a test failure. Thus we need to handle it as described in the blog post Testing Edge Data Cases with Network Stubbing and App Actions. We will expect uncaught:exception event and will confirm it has our test error message and not some other actual application error.

it('sends an error to Sentry', () => {
  cy.route('POST', 'https://sentry.io/api/*/store/*', {
    id: 'abc123',
  }).as('sentry')

  cy.on('uncaught:exception', e => {
    // only ignore OUR test error message
    return e.message === 'test error'
  })

  // create an error, as if application has thrown it
  cy.window().invoke(
    'setTimeout',
    () => {
      throw new Error('test error')
    },
    1000,
  )
  // confirm the call has happened
  cy.wait('@sentry')
})

The test passes. Let's confirm some of the fields the Sentry browser client sends to the service - it should have at least the error message.

// confirm the call has happened
cy.wait('@sentry')
  .its('requestBody')
  .should(body => {
    expect(body.level).to.equal('error')
    expect(body.exception.values).to.have.length(1)
    expect(body.exception.values[0]).to.deep.contain({
      type: 'Error',
      value: 'test error',
    })
  })

Tip: writing XHR assertions this way is cumbersome, I suggest using helper library cy-spok in the blog post Asserting Network Calls from Cypress Tests.

See also

• Connecting crash reporting with end to end tests shows how to add additional test information to Sentry crashes happening during end-to-end tests
• Simple Ajax testing shows how to observe calls to Sentry from Node tests
• Know unknown unknowns with Sentry is probably my first blog post praising Sentry

Update 1: use cy.intercept

After updating to cy.intercept command, my Sentry test is much simpler:

// intercept all network requests going to sentry.io
// using https://on.cypress.io/intercept
// and do not let it proceed to the actual server
cy.intercept('POST', /sentry\.io\/api\//, {}).as('sentry')

const message = `Cypress test error ${Cypress._.random(1e4)}`
// Cypress will fail the test if the web application
// throws an error. Thus we need to prepare to ignore our error
cy.on('uncaught:exception', (e) => {
  // only ignore OUR test error message
  return e.message === message
})

cy.visit('/')
cy.window().invoke(
  'setTimeout',
  () => {
    throw new Error(message)
  },
  100,
)
cy.wait('@sentry')
  .its('request.body')
  // the POST body is a JSON string
  // let's convert it into an actual object
  .then(JSON.parse)
  .should('be.an', 'object')
  .then((errorDetails) => {
    // the object is large, just confirm the main properties are there
    expect(errorDetails).to.have.property('breadcrumbs').to.be.an('array')
    expect(errorDetails)
      // drill down to the error and confirm its message
      .to.have.property('exception')
      .to.have.property('values')
      .to.have.property('0')
      .to.have.property('value', message)
    expect(errorDetails).to.have.property('release').to.be.a('string')
  })