products

Deal With Flaky APIs From Cypress Tests

Speed up your tests and make them flake-free by isolating problematic API requests.

Imagine you are testing an item's HTML page. You need an item, and to remove any dependencies, you create the item from the test itself. To create an item, you need to make a network request to the API endpoint. You can make the request using the standard cy.request command:

1
2
3
4
5
6
7
8
9
10
11
12
13
// make the item name unique
// to avoid finding previously created items
const name = `apple-${Cypress._.random(1_000)}`
cy.log(name)
const item = {
  'item-name': name,
  price: 10,
}
// create the item by making a POST request
cy.request('POST', '/add-item', item)
cy.visit(`/items/${name}`)
// confirm the item was found
cy.contains('h3', name)

If the item is created during the network call, everything is peachy. Cypress test continues after the cy.request('POST', '/add-item', item) finishes, thus by the tim we visit the item's page, it is ready.

What happens if the item is created asynchronously? What happens if the cy.request('POST', '/add-item', item) call simply starts the item creation and returns? How do we know when we can visit the item's page?

🎓 This blog post shows how to:

  • ping an API endpoint instead of hard-coded wait
  • cache the item to avoid re-creating it
  • deal with a flaky testing API endpoint

Slow item creation

Imagine the item is created after some delay. Normally, an item is created within a second, but can take up to a minute. If we try visiting the item's page with cy.visit('/items/${name}'), the test fails with a 404 status code.

The test fails because the item is not ready yet

We need to "wait" somehow in our test for the item to be ready. We can hard-code the maximum sleep period of 1 minute:

1
2
3
4
cy.request('POST', '/add-item', item)
// 🚨 ANTI-PATTERN: hard-coded waits
cy.wait(60_000)
cy.visit(`/items/${name}`)

Oops, the server logs show that the item was ready after 16 seconds, yet the test ran for 1 minute

1
will add { name: 'apple-518', price: 10 } to the database after 16 seconds

The test took way to long to visit the item page

We need a more intelligent way of know when the item is ready. We can ping the item's page every couple of seconds. Once the ping returns 200, we know the page is ready and we can use cy.visit('/items/${name}') to load the page. The best way to do something periodically in Cypress is via my plugin cypress-recurse. Here is the test pinging the page every 4 seconds and immediately continuing when the page responds:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
// https://github.com/bahmutov/cypress-recurse
import { recurse } from 'cypress-recurse'

it('waits for the item to be created', () => {
  // make the item name unique
  // to avoid finding previously created items
  const name = `apple-${Cypress._.random(1_000)}`
  cy.log(name)

  const item = {
    'item-name': name,
    price: 10,
  }
  // create the item by making a POST request
  cy.request('POST', '/add-item', item)
  // it might take a while to create an item
  recurse(
    () =>
      cy.request({
        url: `/items/${name}`,
        failOnStatusCode: false,
      }),
    (resp) => resp.status === 200,
    {
      log: 'found the new item ✅',
      delay: 4_000,
      timeout: 60_000,
    },
  )
  cy.visit(`/items/${name}`)
  // confirm the item was found
  cy.contains('h3', name)
})

The item was created after 4 seconds, according to the server logs

1
will add { name: 'apple-622', price: 10 } to the database after 4 seconds

The test took 8 seconds total (since we have 4-second ping interval)

A test that pings the server to know when the item is ready

Ok, the test is much faster. But what if we can eliminate waiting completely?

🎁 This blog post is based on a series of hands-on lessons in my online advanced course Cypress Network Testing Exercises. In particular, the blog sections are based on the lessons in the "Bonus 14x" series.

Skip item creation using data caching

If we create an item in one test, it is probably likely that we create a similar item in another test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
// https://github.com/bahmutov/cypress-recurse
import { recurse } from 'cypress-recurse'

it('checks the created item`s name', () => {
  // make the item name unique
  // to avoid finding previously created items
  const name = `apple-${Cypress._.random(1_000)}`
  cy.log(name)

  const item = {
    'item-name': name,
    price: 10,
  }
  // create the item by making a POST request
  cy.request('POST', '/add-item', item)
  // it might take a while to create an item
  recurse(
    () =>
      cy.request({
        url: `/items/${name}`,
        failOnStatusCode: false,
      }),
    (resp) => resp.status === 200,
    {
      log: 'found the new item ✅',
      delay: 4_000,
      timeout: 70_000,
    },
  )
  cy.visit(`/items/${name}`)
  // confirm the item was found
  cy.contains('h3', name)
})

it('checks the created item`s price', () => {
  // make the item name unique
  // to avoid finding previously created items
  const name = `apple-${Cypress._.random(1_000)}`
  // use random price for the item
  const price = Cypress._.random(1, 100)
  cy.log(`${name} price is ${price}`)

  const item = {
    'item-name': name,
    price,
  }
  // create the item by making a POST request
  cy.request('POST', '/add-item', item)
  // it might take a while to create an item
  recurse(
    () =>
      cy.request({
        url: `/items/${name}`,
        failOnStatusCode: false,
      }),
    (resp) => resp.status === 200,
    {
      log: 'found the new item ✅',
      delay: 4_000,
      timeout: 70_000,
    },
  )
  cy.visit(`/items/${name}`)
  // confirm the correct price is shown
  cy.contains('p', `Price ${price}`)
})

Each test creates its own item, which might be slow.

Both tests are slowed down by creating separate items

Do we need a completely separate item for the second test? Probably not. We could give the item a unique name AND price to avoid accidentally checking some matching data, but we could reuse the same item.

1
2
3
4
5
6
// make the item name unique
// to avoid finding previously created items
const name = `apple-${Cypress._.random(1_000)}`
// use random price for the item
const price = Cypress._.random(1, 100)
cy.log(`${name} price is ${price}`)

The challenge is to write the independent tests while caching the single item:

  • we should create the item if it does not exist yet
  • if the item exists, we simply visit it
  • it should work not matter if we run the first test only, the second test only, or both tests

I wrote plugin cypress-data-session specifically to solve this problem. In our case, we can do the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
// https://github.com/bahmutov/cypress-recurse
import { recurse } from 'cypress-recurse'
// https://github.com/bahmutov/cypress-data-session
import 'cypress-data-session'

// each test creates a new item which is slow
// we could reuse the same created item for both tests

beforeEach(() => {
  // Please create the random data item just once in
  // a before each hook and cache it using cypress-data-session
  cy.dataSession({
    name: 'item',
    setup() {
      // make the item name unique
      // to avoid finding previously created items
      const name = `apple-${Cypress._.random(1_000)}`
      // use random price for the item
      const price = Cypress._.random(1, 100)
      cy.log(`${name} price is ${price}`)

      const item = {
        'item-name': name,
        price,
      }
      // create the item by making a POST request
      cy.request('POST', '/add-item', item)
      // it might take a while to create an item
      recurse(
        () =>
          cy.request({
            url: `/items/${name}`,
            failOnStatusCode: false,
          }),
        (resp) => resp.status === 200,
        {
          log: 'found the new item ✅',
          delay: 4_000,
          timeout: 60_000,
        },
      )

      // yield the item's information
      cy.wrap({ name, price })
    },
  })
})

it('checks the created item`s name', function () {
  const { name, price } = this.item
  cy.visit(`/items/${name}`)
  // confirm the item was found
  cy.contains('h3', name)
})

it('checks the created item`s price', function () {
  const { name, price } = this.item
  cy.visit(`/items/${name}`)
  // confirm the correct price is shown
  cy.contains('p', `Price ${price}`)
})

Our data session setup function yields an object with the unique name and price. This object is stored in the Cypress alias "item" (the data session name) and available this this.item property when we use the it(..., function () { ... }) tests.

The first time we run this spec, we can see a much faster second test, since it simply reuses the cached in memory data item.

The first test creates the item but the second test simply reuses it

But the biggest payoff happens when we press the "R" button or click the test re-run button: the spec is blazingly fast, finishing in just 81ms in this case; all data is already been created.

Both tests reuse the item created in the previous runs

Cypress recurse and data session plugins are pretty powerful for real-world testing.

Flaky test creation

Now let's consider the last problem. What if the item creation API is flaky and can simply fail? We don't want to fail the test for the item's page; we want to overcome the flake. We want to retry creating the item. I recommend a two-level recursion for this:

  • use the code we already wrote to ping the item to know when it is ready
  • disable failing on timeout when checking the item
  • retry creating the item if pinging fails

I typically have two utility functions checkItem and createItem to hide the messy details:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
// https://github.com/bahmutov/cypress-recurse
import { recurse } from 'cypress-recurse'

/**
 * A function that checks if the item with the given name exists.
 * Uses cypress-recurse to ping the item again and again
 * until it is found or the timeout is reached.
 * @param {string} name - the item name
 * @returns {Cypress.Chainable<boolean>} true if the item was found
 */
function checkItem(name) {
  cy.log(`**checking for the new item ${name}**`)
  return (
    recurse(
      () =>
        cy.request({
          url: `/items/${name}`,
          failOnStatusCode: false,
        }),
      (resp) => resp.status === 200,
      {
        log: 'found the new item ✅',
        delay: 4_000,
        timeout: 30_000,
        // IMPORTANT:
        // if we keep pinging and the item is not found
        // we yield the last cy.request response
        doNotFail: true,
        yield: 'value',
      },
    )
      // the above "recurse" call always yields the lat
      // cy.request response, which we can check
      // and transform the response into a boolean
      // showing the success of the operation
      .then((resp) => resp.status === 200)
  )
}

function createItem(item) {
  recurse(
    () => {
      cy.log('**creating the new item (might be flaky)**')
      cy.request('POST', '/add-item-flaky', item)
      return checkItem(item['item-name'])
    },
    (foundItem) => foundItem === true,
    {
      log: 'created the new item ✅',
      limit: 3,
      timeout: 300_000,
    },
  )
}

Look inside the createItem function: it makes the request to the POST /add-item-flaky endpoint and yields the result of checking the item's existence via checkItem. That function simply pings the item periodically, yielding true or false, but without failing if the item is not found. If the result is true, we are good, the item i there. Else we try creating the item using the flaky API endpoint again.

The code work great with our test

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
it('adds an item via a flaky api endpoint', function () {
  // make the item name unique
  // to avoid finding previously created items
  const name = `apple-${Cypress._.random(1_000)}`
  // use random price for the item
  const price = Cypress._.random(1, 100)
  cy.log(`${name} price is ${price}`)

  const item = {
    'item-name': name,
    price,
  }

  // the top-level function creates the item
  // and checks that it can be found
  // and retries the creation up to 3 times
  createItem(item)

  cy.log('**visiting the item**')
  cy.visit(`/items/${name}`)
  // confirm the item was found
  cy.contains('h3', name)
})

Here is our test during a run when the item was created on the very first attempt

The item was found on the first creation attempt

And here is the same test on a run when the item creation timed out and the logic in the createItem function had to create it again

The test code retried creating an item and was successful on the second attempt

Like my girl Aaliyah said: "If at first you don't succeed, dust yourself off and try again"