Check Every Box In Cypress Tests Without Flake

Imagine you are testing a TodoMVC application, and you need to complete all items. You simply click every checkbox and confirm the application preserves the "0 todos left" state. Normally everything goes well:

But sometimes a weird thing happens: one or more checkboxes remain unchecked!

You start looking at the test code. Looks ok, right?

cypress/e2e/todos.cy.js

it('are completed by checking the boxes', function () {
  const todos = '.todo-list li'
  cy.visit('/')
  cy.get(todos).should('have.length', this.n)
  // complete all todos by clicking the checkboxes
  cy.get('.todo-list li .toggle').click({ multiple: true })
  // confirm all todos are marked as completed
  // after reloading the page
  cy.reload()
  cy.get('.loaded')
  cy.get('[data-cy="remaining-count"]').should('have.text', '0')
})

This test starts by creating a random number of Todo items, something I covered in the "Cypress Vs Playwright" online course available at cypress.tips/courses.

Ok, there seems to be some flake, and we know how to solve end-to-end testing flake pretty well. We start adding assertions, trying to confirm that our commands have finished successfully before reloading the page.

it('are completed by checking the boxes', function () {
  const todos = '.todo-list li'
  cy.visit('/')
  cy.get(todos).should('have.length', this.n)
  // complete all todos by clicking the checkboxes
  cy.get('.todo-list li .toggle').click({ multiple: true })

  // confirm all todos are done
  cy.get('[data-cy="remaining-count"]').should('have.text', '0')

  // confirm all todos are marked as completed
  // after reloading the page
  cy.reload()
  cy.get('.loaded')
  cy.get('[data-cy="remaining-count"]').should('have.text', '0')
})

We inserted the "0 todos" check before the cy.reload command, and ... it did not solve the problem!

What is happening, how can the same assertion pass once, but then immediately fail? The clue is in the number of network calls shown int the Command Log. In this instance, we have 4 todos to complete. Yet, there are only 3 "PATCH" network calls!

This is typical web app behavior: update the local state immediately, and send the update to the backend. But what happens if the app does not have time to send the network call before the page reloads? The network call does not happen, and the backend does not "see" one or more "PATCH" network calls. Checking just the page UI using cy.get('[data-cy="remaining-count"]').should('have.text', '0') does not solve the problem: the problem is that the backend still has not been updated.

We can solve this issue in several ways.

Check the network call count

Using the incredibly powerful cy.intercept command, we can "continue" the test after N network calls happen.

it('are completed by checking the boxes (N network calls)', function () {
  const todos = '.todo-list li'
  cy.visit('/')
  cy.get(todos).should('have.length', this.n)

  cy.intercept('PATCH', '/todos/*').as('updateTodo')

  // complete all todos by clicking the checkboxes
  cy.get('.todo-list li .toggle').click({ multiple: true })

  // confirm all network calls have finished
  cy.get('@updateTodo.all').should('have.length', this.n)

  // confirm all todos are marked as completed
  // after reloading the page
  cy.reload()
  cy.get('.loaded')
  cy.get('[data-cy="remaining-count"]').should('have.text', '0')
})

We spy on the PATCH /todos/* calls using the cy.intercept command before we start clicking. Then we check the count of intercepts calls using the cy.get('@updateTodo.all').should('have.length', this.n) command and assertion combo, which retries. Even if the app takes a few seconds to fire the network call, the test will be flake-free.

Observe each network call

Instead of using .click({ multiple: true }), we could click each box individually and confirm its network calls finished. I like this approach even more than the "click multiple elements" option, since it works for any action, not just cy.click, For example, let's use cy.check command, which does not have multiple: true option.

it('are completed by checking the boxes (one at a time)', function () {
  const todos = '.todo-list li'
  cy.visit('/')
  cy.get(todos).should('have.length', this.n)

  cy.intercept('PATCH', '/todos/*').as('updateTodo')

  cy.get('.todo-list li .toggle').each(($el) => {
    cy.wrap($el, { log: false }).check()
    // confirm a single network call has finished
    cy.wait('@updateTodo')
  })

  // confirm all todos are marked as completed
  // after reloading the page
  cy.reload()
  cy.get('.loaded')
  cy.get('[data-cy="remaining-count"]').should('have.text', '0')
})

We are using cy.each command. It gives us each element as a jQuery object. To properly click it using Cypress, simply cy.wrap silently to avoid Command Log noise and run the cy.check, followed by cy.wait command with a network alias - it waits for 1 network call. If you want to confirm the call was successful, add an assertion.

cy.get('.todo-list li .toggle').each(($el) => {
  cy.wrap($el, { log: false }).check()
  // confirm a single network call has finished successfully
  cy.wait('@updateTodo')
    .its('response')
    .should('have.property', 'statusCode', 200)
})

Ask the server

If we are not sure when the application has finished updating the backend, why not ask the backend directly? We can ping the backend ourselves, just like the app does using the cy.request command. To ping the server multiple times until all todos are completed, we can use my plugin cypress-recurse:

import { recurse } from 'cypress-recurse'

it('are completed by checking the boxes (check the backend)', function () {
  const todos = '.todo-list li'
  cy.visit('/')
  cy.get(todos).should('have.length', this.n)

  cy.get('.todo-list li .toggle').each(($el) => {
    cy.wrap($el, { log: false }).check()
  })

  // confirm the backend has only completed todos
  recurse(
    () => cy.request('/todos').its('body'),
    (todos) => todos.every((todo) => todo.completed),
    {
      log: 'All todos are completed on the server',
      timeout: 30_000,
      delay: 1000,
    },
  )

  // confirm all todos are marked as completed
  // after reloading the page
  cy.reload()
  cy.get('.loaded')
  cy.get('[data-cy="remaining-count"]').should('have.text', '0')
})

The important part is the recurse call with two functions: fetching the list of todos and the predicate to know when to stop pinging:

recurse(
  () => cy.request('/todos').its('body'), // produce the values
  (todos) => todos.every((todo) => todo.completed), // check the value
  // options: how many times to check, how long to wait between the checks, etc
  {
    log: 'All todos are completed on the server',
    timeout: 30_000,
    delay: 1000,
  },
)

The recurse calls the first function repeatedly, passes the yielded value into the predicate, and stops the iteration when the predicate returns a truthy value. We use 1 second delays for clarity, and I slowed down the web app to space out network calls. You can see several REQUEST /todos commands - this is our check working

Map chain

Finally, what if you want to check the list of updated todos? We need their ids, so our iteration must produce a list of numbers. But cy.each yields the original list of elements, not a custom value.

Plugin cypress-map to the rescue! It has cy.mapChain command where you can do something with each element, but then produce new values for the next command in the chain. Once we get all completed ids, we can compare it with the list from the beforeEach where we saved the created todo ids

import 'cypress-map'

beforeEach(function createRandomTodos() {
  const n = Cypress._.random(1, 3)
  ...
  // save created ids for later
  cy.wrap(todos.map((t) => t.id)).as('ids')
})

it('are completed by checking the boxes (collect their ids)', function () {
  const todos = '.todo-list li'
  cy.visit('/')
  cy.get(todos).should('have.length', this.n)

  cy.intercept('PATCH', '/todos/*').as('updateTodo')
  cy.get('.todo-list li .toggle')
    .mapChain(($el) => {
      cy.wrap($el, { log: false }).check()
      // from each network call, grab the id of the updated todo
      cy.wait('@updateTodo').its('response.body.id')
    })
    .should('deep.equal', this.ids)

  // confirm all todos are marked as completed
  // after reloading the page
  cy.reload()
  cy.get('.loaded')
  cy.get('[data-cy="remaining-count"]').should('have.text', '0')
})

The command .mapChain(fn) runs the commands inside the function and collects all yielded values into an array that it will yield to the next assertion. Thus our code works, here is the relevant part:

.mapChain(($el) => {
  cy.wrap($el, { log: false }).check()
  // from each network call, grab the id of the updated todo
  cy.wait('@updateTodo').its('response.body.id')
})
.should('deep.equal', this.ids)

You can see the original ids in the "BEFORE EACH" hook, and you can see the array assertion inside the test; these are the same ids, and the order is correct.

Nice. I must say these tests and how they interact with this example application are better shown than explained in a blog post. I will record a video showing these tests in action and will post on my youtube.com/@gleb channel, stay tuned.