products

You Should Test More Using APIs

How to verify that 3rd party services work using the cy.request API calls and cypress-recurse plugin.

If you are always using the page UI during end-to-end tests, your tests might be slower than needed. If such test fails, it might not tell you much about the root cause of the error. In this blog post, I will show a particular example of how to improve such tests using our Mercari US online marketplace tests as an example.

🎁 This blog post uses an example application from the repo bahmutov/fastify-example and tests to write in bahmutov/fastify-example-tests, see the "bonus33.js" spec file. These exercises and their solutions constitute the "Bonus 33" lesson in my Cypress Network Testing Exercises advanced course. Purchase the full course if you are interested in improving your understanding of advanced Cypress testing topics.

The application

Imagine a typical situation (at least for us at Mercari US):

  1. the user enters a new item via the web form
  2. the new item is added to the internal database. It might take a few seconds to prepare the item's page
  3. the new item's page is then scraped by the 3rd party search service. It might take up to a minute for the search results to include the new item.

Here are the screenshots showing the stages of the user journey.

The user enters the item details

The user submits the form

The system adds the item page. It can take up to a minute for the page to be ready

The user can try searching for the item. In this case, the item has not been scraped yet

The item has been scraped and is returned as the search result

Here is diagram of what happens to the item as it is added

The added item makes its way through the services before the user can find it

How would you write the search test to confirm that the new items are scraped correctly? Would you add the item, wait several minutes, and then use the search page? The static wait would have to be long enough to guarantee that even if the 3rd party services are at their slowest rate, by the time you search, the item has been scraped already. Alternatively, let's add the item, and then retry the search. If the item is not found, not big deal - we will wait N seconds, then try again. But what if the search returns nothing? Where is the problem? Was the item NOT added to our database? Was the item NOT scraped? A good test points at the root cause when it fails. Thus we will solve the same problem one more time by "tracing" the item's progress via cy.request calls.

beforeEach hook

Each test enters a new item using the following beforeEach hook

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
beforeEach(function enterItem() {
  cy.visit('/items.html')
  const name = `Item ${Cypress._.random(1e6)}`
  const price = Cypress._.random(1e3)
  // save the item name under an alias
  cy.wrap(name).as('name')
  // save the price under an alias
  cy.wrap(price).as('price')

  cy.get('#item-name').type(name)
  cy.get('#price').type(String(price))
  cy.get('[value=Submit]').click()
  cy.location('pathname').should('equal', '/add-item')
  cy.contains('h3', name)
})

Let's see how we can write the test itself to confirm the number is scraped and can be found by the user.

Solution 1: static wait

Let's use only the user interface to test how the new item can be added.

Cypress test can enter the item, then find it using the item search page

If it can take up to one minute for the item to be added to the database, and up to another minute for the search service to scrape it, then we can simply wait 2 minutes.

1
2
3
4
5
6
7
8
9
10
11
it('adds a new item and then finds it (static wait)', function () {
  // wait for the item to be added to the database (one minute)
  // plus for the search service to scrape it (another minute)
  cy.wait(120_000)
  // find the item
  cy.visit('/find-item.html')
  cy.get('#item-text').type(this.name + '{enter}')
  cy.contains('#output', this.name)
    .contains('.price', this.price)
    .should('be.visible')
})

The test finds the item after waiting 2 minutes

In the fastify-example server logs I can see the following messages while the test is running

1
2
3
4
5
adding item request { name: 'Item 625873', price: 533 }
will add { name: 'Item 625873', price: 533 } to the database after 36 seconds
...
adding the item { name: 'Item 625873', price: 533 } to the database
will scrape the item { name: 'Item 625873', price: 533 } after 31 seconds so it can be found

On average, adding the new item to the database takes 30 seconds, and scraping it takes on average 30 seconds. One minute is the worst case scenario, thus our test spent unnecessarily waiting an entire minute...

Solution 2: retry the search

Let's add the item and immediately go to the search page. We can try finding the item, and if it is not yet ready, we can try again after some delay. I will use the cypress-recurse plugin to retry Cypress commands until a condition becomes true.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
it.only('adds a new item and then finds it (retries the search)', function () {
  cy.visit('/find-item.html')
  recurse(
    () => {
      cy.get('#item-text').clear().blur()
      cy.get('#item-text').type(this.name + '{enter}')
      return cy
        .contains('#output', this.name)
        .should(Cypress._.noop)
    },
    ($el) => $el.length,
    {
      log: 'found the item',
      delay: 10_000,
      timeout: 120_000,
    },
  )
  cy.contains('#output', this.name)
    .contains('.price', this.price)
    .should('be.visible')
})

The test retried the item search until it found the new item

The server logs for this test show the following numbers:

1
2
3
4
5
adding item request { name: 'Item 740551', price: 561 }
will add { name: 'Item 740551', price: 561 } to the database after 40 seconds
...
adding the item { name: 'Item 740551', price: 561 } to the database
will scrape the item { name: 'Item 740551', price: 561 } after 29 seconds so it can be found

The item was ready to be found after 70 seconds, and the test took 75 seconds, not bad!

Solution 3: pinging APIs to trace the item's progress

The previous test is ok, yet it has a flaw. If the test fails, we have no idea why the item has not been found. Was it NOT added to our internal database? Was it NOT scraped correctly?

The test that fails at the end does not tell us which service failed

Instead of retrying the search through the user interface, let's us poll the APIs to trace the item added to the database, and then ping the search API to check when the item has been scraped. Once we know the item has been scraped successfully, we can go to the search page and find it.

The test can ping the API for each service to confirm it has processed the item successfully

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
it('adds a new item and then finds it (retries the API calls)', function () {
  cy.log('**call the API until the item is returned**')
  recurse(
    () =>
      cy.request({
        url: '/items/' + encodeURIComponent(this.name),
        failOnStatusCode: false,
      }),
    (response) => response.isOkStatusCode,
    {
      log: '✅ item is in our database',
      delay: 10_000,
      timeout: 60_000,
    },
  )
  // call the search API until it finds the item
  cy.log('**call the search API**')
  recurse(
    () =>
      cy.request({
        url: '/find-item/' + encodeURIComponent(this.name),
        failOnStatusCode: false,
      }),
    (response) => response.isOkStatusCode,
    {
      log: '✅ item has been scraped',
      delay: 10_000,
      timeout: 60_000,
    },
  )

  cy.log('**use the UI to find the scraped item**')
  cy.visit('/find-item.html')
  cy.get('#item-text').type(this.name + '{enter}')
  // the item _must_ be found now
  // https://on.cypress.io/contains
  cy.contains('#output', this.name)
    .contains('.price', this.price)
    .should('be.visible')
})

This solution confirms the item is added to our internal application database by pinging the item's page (or some other API endpoint). Then it pings the search service to check when the item has been scraped. Only after we know the search service returns it, we visit the page and use the UI to confirm the search feature is working.

Test test using API calls to confirm each step of the item processing

The server logs show that this aligns with our operations

1
2
3
4
5
adding item request { name: 'Item 905905', price: 682 }
will add { name: 'Item 905905', price: 682 } to the database after 35 seconds
...
adding the item { name: 'Item 905905', price: 682 } to the database
will scrape the item { name: 'Item 905905', price: 682 } after 23 seconds so it can be found

If the test had failed, we would know precisely which of the 3 parts was not working: the item addition, the search scraping, or the search UI.

If we run all three tests together using cypress run, we can see that test timing improvements.

1
2
3
4
5
✓ adds a new item and then finds it (static wait) (122050ms)
✓ adds a new item and then finds it (retries the search) (64312ms)
✓ adds a new item and then finds it (retries the API calls) (42017ms)

3 passing (4m)

The 3rd test faster than the 2nd in this particular run, but on average, it should take almost the same time. The 3rd solution has a big advantage other the other two solutions: if the test fails, it would immediately point at the culprit service.

See also