What Needs To Retry

Imagine a typical application that loads some data. The app starts in the initial state, starts loading, then one of 3 possible states are possible

1. The data loads fine
2. The data loads an empty list of items
3. The app fails to load due to an error

To avoid flake, we need to think about these 3 states. If we simply check the first case, the test might be both slow and a failed test would not tell us why it has failed. Here is the typical test that simply confirms that several items load.

it('loads items', { defaultCommandTimeout: 5000 }, () => {
  cy.visit('cypress/items-load-with-error.html')
  cy.contains('h1', 'Items load')
  cy.get('#items li').should('have.length.greaterThan', 1)
})

Everything is great. The items might take a while to load, no big deal - the cy.get command retries until the assertion .should('have.length.greaterThan', 1) passes or the 5 second timeout elapses.

Zero state

What happens if there are no items? The test waits for 5 seconds and fails.

Ughh, the wait is unnecessary - the app shows "No items" pretty quickly. Yet, the test keeps retrying the cy.get command in vain. The error message is not that useful - it tells us that it could not find "#items li" elements, it does not tell us that "zero items" component was shown instead. A developer has to look (or ask AI to check the error screenshot).

Error state

What if the backend fails to return data? Maybe there is an API hiccup and it returns 5xx error, maybe something else.

Again, the test waits for too long and does not tell us why it failed.

Consider all states

So we know what the app can potentially do and how it might fail. We can consider app states explicitly in our end-to-end test. Here is one way to do it using the cy.depends command from cypress-if plugin.

it('loads items or shows error', { defaultCommandTimeout: 5000 }, () => {
  cy.visit('cypress/items-load-with-error.html')
  cy.contains('h1', 'Items load')

  cy.depends({
    '#items #error': new Error('Items failed to load'),
    '#items #empty': new Error('Items are empty'),
    '#items li': 'Items are loaded',
  })

  cy.get('#items li').should('have.length.greaterThan', 1)
})

We consider #error and #empty elements to be errors and fail the test immediately when they are found.

Look at the failed test above. It fails quickly in 122ms without waiting full 5 seconds. It also tells you exactly what went wrong: "Items failed to load". What happens if the items load? They load as before.

What needs to retry

Now let's go back to the test and decide what do we want to do when the app shows an error or fails to load items. Do we want to fail the test and be done? Or do we retry the test using the built-in retry mechanism?

it(
  'loads items or shows error with retries',
  { defaultCommandTimeout: 5000, retries: 2 },
  () => {
    cy.visit('cypress/items-load-with-error.html')
    cy.contains('h1', 'Items load')

    cy.depends({
      '#items #error': new Error('Items failed to load'),
      '#items #empty': new Error('Items are empty'),
      '#items li': 'Items are loaded',
    })

    cy.get('#items li').should('have.length.greaterThan', 1)
  },
)

In the run below the test retried twice before succeeding

In my real-world end-to-end tests I find test retries a blunt instrument. If a test takes 2 minutes to run, and needs to retry on every small hiccup, it will still be flaky or likely to fail - because it retries from the very beginning. Instead I prefer to handle common situations using "in place" retries using cypress-recurse plugin. In our test, we want to do the following:

• if we find items, all good, we continue
• if we find zero items, we reload the page, hoping to see items
• if we find the error message, we retry the entire test

Here is how it might look

import { recurse } from 'cypress-recurse'

it(
  'loads items or shows error with local and test retries',
  { defaultCommandTimeout: 5000, retries: 2 },
  () => {
    cy.visit('cypress/items-load-with-error.html')
    cy.contains('h1', 'Items load')

    recurse(
      () =>
        cy
          .depends({
            '#items #error': new Error('Items failed to load'),
            '#items #empty': 'Items are empty',
            '#items li': 'Items are loaded',
          })
          .its('selector'),
      (selector) => selector === '#items li',
      {
        log: true,
        timeout: 30_000,
        limit: 5,
        post() {
          cy.log('Empty items, reloading the page')
          cy.reload()
        },
      },
    )

    cy.get('#items li').should('have.length.greaterThan', 1)
  },
)

We no longer throw an error if we find an empty list of items. Instead we simply log "Items are empty" message. The cy.depends command yields an object and we can grab the matched selector string. The recurse(fn, predicate) can look at the selector to see if we matched "#items li", in which case we are done, anything else means we run the post() callback where we reload the page, which tries to load the data again. Here is a typical run - it reloads the page a couple of times, then finds the data and the test passes.

We still terminate the test if we find the error component, letting test retries re-run the entire test. But for zero data, we use local "retry" using the recurse plugin. This should make the test run faster, while still giving us an accurate log message.

🎁 This blog post uses example tests from bahmutov/cypress-if repo specs.