products

Solve Tough Pagination Cases Using Cypress

A few edge test cases that might trip your Cypress end-to-end tests.

In my recent blog post Cypress Pagination Challenge I have shown several solutions to a common testing problem: flip through the rows of data, looking for something to be there (or to not be there). In this blog post, I will take it up a notch. I will show several pagination edge cases that make writing a good flake-free end-to-end test difficult. I have seen these errors in my tests and in the testing code written by others. It is time to solve it for good. Let's go.

🎁 You can find the full source code in the repo bahmutov/cypress-pagination-tough-case.

The easy case

Let us start with a nice happy path. We see a list with a few items. We can click on the "Next" button and go to the second page. For simplicity, the list ends on the second page, but our solution does not know that. The list has only the words first, second, third, fourth, fifth, and sixth. The test should stop checking the list when the "Next" button becomes disabled.

Here is the code for the page. As you can see, the app is very fast. It instantly renders the first list, then instantly renders the second page after clicking the "Next" button.

cypress/fixtures/spec1.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<body>
  <main>
    <ul id="list">
      <li>first</li>
      <li>second</li>
      <li>third</li>
    </ul>
    <button id="next">Next ▶</button>
  </main>
  <script>
    // the page is _very fast_
    // and the new page loads synchronously
    const list = document.getElementById('list')
    const next = document.getElementById('next')
    next.addEventListener('click', () => {
      list.innerHTML = `
        <li>fourth</li>
        <li>fifth</li>
        <li>sixth</li>
      `
      next.setAttribute('disabled', true)
    })
  </script>
</body>

How would you write this test?

cypress/e2e/spec1.cy.js
1
2
3
4
5
6
7
8
9
beforeEach(() => {
  cy.visit('cypress/fixtures/spec1.html')
})

it('confirms the list has no such item', () => {
  // text of an item that should NOT be in the list
  const item = 'apples'
  // ???
})

We are doing an example of negative testing, we are trying to confirm that something is NOT there. This is always more complex than confirming the presence, since an item can missing for many reasons. In any case, let's write a recursive algorithm, I will put it in the cypress/e2e/utils.js file so we can use it from other specs. In the code below we are doing a conditional command depending on the "Next" button state. For more examples of conditional clicking see the blog post Click Button If Enabled.

cypress/e2e/utils.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
/**
 * Checks if the given item is not present on the page and
 * recursively clicks the "Next" button until the item is found
 * or the button is disabled.
 * @param {string} item - The item to check for on the page.
 * @returns {void}
 */
export const checkPage1 = (item) => {
  cy.contains('li', item).should('not.exist')
  cy.get('button#next')
    .invoke('is', ':enabled')
    .then((enabled) => {
      if (enabled) {
        cy.get('button#next').click()
        checkPage1(item)
      } else {
        cy.log('**end of the list**')
      }
    })
}

Our test can call the checkPage1 with the a string argument. Note: for this blog post, I slow down every Cypress commands by 200ms using my plugin cypress-slow-down.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// https://github.com/bahmutov/cypress-slow-down
import { slowCypressDown } from 'cypress-slow-down'
import { checkPage1 as checkPage } from './utils'

slowCypressDown(200)

beforeEach(() => {
  cy.visit('cypress/fixtures/spec1.html')
})

it('confirms the list has no such item', () => {
  // text of an item that should NOT be in the list
  const item = 'apples'
  checkPage(item)
})

The test passes. If we inspect the contains ... commands for both pages, we see that the test did check the list at the right moment. In the first instance it checked the list when the list had values first, second, and third. And in the second instance, the test checked the list when it was on the second page showing the values fourth, fifth, and sixth.

The two places where the test checked the list

If you look at the video again, I am showing the page at the moment the cy.contains command ran twice.

Unhappy paths

Let's break the test on purpose to make sure it works correctly. Let's give it a string value that is present on the page. The value is second so it should be found on the first page.

1
2
3
4
5
6
it('fails if the item is on the first page', () => {
  // test with an item that is on the first page
  // => the test should fail
  const item = 'second'
  checkPage(item)
})

The test fails if the item is found on the first page

What if the item is on the second page? Let's test it.

1
2
3
4
it('fails if the item is on the second page', () => {
  const item = 'fifth'
  checkPage(item)
})

The test fails if the item is found on the second page

Beautiful, our checkPage1 recursive test function is correct. Or is it?

Slow application load

A common problem in the SPA is the slow initial data load. I simulate it in the spec2.html page

cypress/fixtures/spec2.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
<body>
  <main>
    <ul id="list">
      loading...
    </ul>
    <button id="next">Next ▶</button>
  </main>
  <script>
    // the initial list loads slowly,
    // but then the page is very fast
    // and the new page loads synchronously
    const list = document.getElementById('list')
    const next = document.getElementById('next')

    setTimeout(() => {
      list.innerHTML = `
        <li>first</li>
        <li>second</li>
        <li>third</li>
      `
    }, 1000)
    next.addEventListener('click', () => {
      list.innerHTML = `
        <li>fourth</li>
        <li>fifth</li>
        <li>sixth</li>
      `
      next.setAttribute('disabled', true)
    })
  </script>
</body>

The initial list loads after 1 second, meanwhile the app is showing the loading... text. Let's run the same 3 tests again and see if we have the same outcome:

1
2
3
4
// EXPECTED
✅ confirms the list has no such item
🚨 fails if the item is on the first page
🚨 fails if the item is on the second page
cypress/e2e/spec2.cy.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// https://github.com/bahmutov/cypress-slow-down
import { slowCypressDown } from 'cypress-slow-down'
import { checkPage1 as checkPage } from './utils'

slowCypressDown(200)

beforeEach(() => {
  cy.visit('cypress/fixtures/spec2.html')
})

it('confirms the list has no such item', () => {
  // text of an item that should NOT be in the list
  const item = 'apples'
  checkPage(item)
})

it('fails if the item is on the first page', () => {
  // test with an item that is on the first page
  // => the test should fail
  const item = 'second'
  checkPage(item)
})

it('fails if the item is on the second page', () => {
  const item = 'fifth'
  checkPage(item)
})

Hmm. We are getting one test "flipped"

1
2
3
4
// ACTUAL
✅ confirms the list has no such item
✅ fails if the item is on the first page
🚨 fails if the item is on the second page

The second test somehow passes

The second test that checks the item that should be found on the first page is green for some reason. Let's debug it. We can expand the test commands and click on the "contains 'second'" command to see how the page looked when the test checked the list.

The app page when the second test checked it

Ooops, the test looked for the item with the test "second", the page was showing "loading...", the test happily continued on its way.

Remember: a negative assertion can pass for many reasons

The reason for the test passing while it should have been failing is confusion to what state the application is in. The test thinks the app is showing the list. The app is still showing the loading element. The solution is to "sync" the application and the test states. For example, the test can wait for the li elements to appear before running a negative assertion. This will ensure the app is in the right state showing the items.

cypress/e2e/utils.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
/**
 * Checks if the given item is not present on the page and
 * recursively clicks the "Next" button until the item is found
 * or the button is disabled.
 * Waits for the items to load first.
 * @param {string} item - The item to check for on the page.
 * @returns {void}
 */
export const checkPage2 = (item) => {
  cy.get('li').log('**has list items**')
  cy.contains('li', item).should('not.exist')
  cy.get('button#next')
    .invoke('is', ':enabled')
    .then((enabled) => {
      if (enabled) {
        cy.get('button#next').click()
        checkPage2(item)
      } else {
        cy.log('**end of the list**')
      }
    })
}

In a TodoMVC application, the code might set a loaded class to signal that the page has the list, and the test could do something like this:

1
2
cy.visit('/')
cy.get('.loaded')

In our code, the checkPage2 function confirms the list items are present before checking them: cy.get('li').log('**has list items**'). Let's see if the checkPage2 test utility leads to the correct test outcome.

cypress/e2e/spec2.cy.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// https://github.com/bahmutov/cypress-slow-down
import { slowCypressDown } from 'cypress-slow-down'
import { checkPage2 as checkPage } from './utils'

slowCypressDown(200)

beforeEach(() => {
  cy.visit('cypress/fixtures/spec2.html')
})

it('confirms the list has no such item', () => {
  // text of an item that should NOT be in the list
  const item = 'apples'
  checkPage(item)
})

it('fails if the item is on the first page', () => {
  // test with an item that is on the first page
  // => the test should fail
  const item = 'second'
  checkPage(item)
})

it('fails if the item is on the second page', () => {
  const item = 'fifth'
  checkPage(item)
})

One passing, two failing tests as expected

If we debug the second test, we see that it fails for the right reason. The application is showing the actual list items first, second, and third when the first "contains" assertion checks it.

The second test fails for the right reason

Good. But this is not the entire story.

Slow page transition

Let's introduce another challenge. Our application might be slow to update the list after the user clicks the "Next" button. I have seen applications where clicking the "Next" button updated the button itself, yet the list was still showing the old items for X milliseconds. Here is how I simulate it in the spec3.html code:

cypress/fixtures/spec3.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
<body>
  <main>
    <ul id="list">
      loading...
    </ul>
    <button id="next">Next ▶</button>
  </main>
  <script>
    // the initial list loads slowly,
    // and when clicking on the next button
    // there is a delay before the new list is shown
    const list = document.getElementById('list')
    const next = document.getElementById('next')

    setTimeout(() => {
      list.innerHTML = `
        <li>first</li>
        <li>second</li>
        <li>third</li>
      `
    }, 1000)
    next.addEventListener('click', () => {
      setTimeout(() => {
        list.innerHTML = `
          <li>fourth</li>
          <li>fifth</li>
          <li>sixth</li>
        `
      }, 1000)
      next.setAttribute('disabled', true)
    })
  </script>
</body>

Let's use our checkPage2 function to run against this application.

cypress/e2e/spec3.cy.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// https://github.com/bahmutov/cypress-slow-down
import { slowCypressDown } from 'cypress-slow-down'
import { checkPage2 as checkPage } from './utils'

slowCypressDown(200)

beforeEach(() => {
  cy.visit('cypress/fixtures/spec3.html')
})

it('confirms the list has no such item', () => {
  // text of an item that should NOT be in the list
  const item = 'apples'
  checkPage(item)
})

it('fails if the item is on the first page', () => {
  // test with an item that is on the first page
  // => the test should fail
  const item = 'second'
  checkPage(item)
})

it('fails if the item is on the second page', () => {
  const item = 'fifth'
  checkPage(item)
})

We are expecting again the first test to pass and the last two tests to fail. But we are getting something else:

1
2
3
4
5
6
7
8
// EXPECTED
✅ confirms the list has no such item
🚨 fails if the item is on the first page
🚨 fails if the item is on the second page
// ACTUAL
✅ confirms the list has no such item
🚨 fails if the item is on the first page
✅ fails if the item is on the second page

Why is the third test passing?!

Let's debug the third test. Hover or click over the second "contains li, fifth" command. Why is it still showing the first page?!!

We expected to see the second page, but the first page was visible still

Again, this is the confusion between the test and the application state. The test assumed that after clicking on the "Next" button and seeing list items, the app would be on the second page. But the application still showed the first page. In other circumstances we could have looked at the URL to confirm the page number

1
2
3
4
// if the app rendered the page in the URL
cy.get('button#next').click()
cy.location('pathname').should('equal', '/pages/2')
checkPage2(item)

But we don't have it. So somehow we need to check that new list item elements are there on the page. People often use the text of an element to detect when the new list was rendered:

1
2
3
4
5
6
7
8
// 🚨 INCORRECT, LEADS TO FLAKE
cy.get('li:first')
  .invoke('text')
  .then(text => {
    cy.get('button#next').click()
    cy.contains('li:first', text).should('not.exist')
    checkPage2(item)
  })

The above code might work in some cases, but if the list has duplicates, it would fail. Imagine our list has items A, A, and A on the first page, and the items A, B, and C on the second. It would not be able to tell the two A strings apart. A better way would be to check the element references, since the application replaces <LI> elements on click:

1
2
3
4
5
6
7
8
9
10
11
// ✅ check element references
cy.get('li:first')
  .then($el => {
    cy.get('button#next').click()
    cy.get('li:first').should($li => {
      // compare two element references
      expect($el[0]).to.not.equal($li[0])
    })
    // the list has new <LI> elements
    checkPage2(item)
  })

To simplify the above code we can use a query cy.stable from my cypress-map plugin. The query retries until an element's reference remains constant for N milliseconds. Thus, if we know that the list switch takes at most 1 second, we can wait for the reference to remain stable for slightly longer period.

cypress/e2e/utils.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// https://github.com/bahmutov/cypress-map
import 'cypress-map'

/**
 * Checks if the given item is not present on the page and
 * recursively clicks the "Next" button until the item is found
 * or the button is disabled.
 * Waits for the items to load first.
 * When going to the next page, waits for the current list
 * to not change its text for 1500ms.
 * @param {string} item - The item to check for on the page.
 * @returns {void}
 */
export const checkPage3 = (item) => {
  cy.get('li').log('**has list items**')
  cy.contains('li', item).should('not.exist')
  cy.get('button#next')
    .invoke('is', ':enabled')
    .then((enabled) => {
      if (enabled) {
        cy.get('button#next').click()
        cy.get('li').first().stable('element', 1500, { timeout: 3000 })
        checkPage3(item)
      } else {
        cy.log('**end of the list**')
      }
    })
}

Let's see our spec now.

The tests work exactly as expected. The third test gets to the second page, waits for the <LI> elements to be stable, and then correctly finds the item with text "fifth"..

The stable list on the second page

Speed

A huge downside to the cy.stable command is that it must wait for N milliseconds. In our case, it waits for 1500ms. If the <LI> elements switch after 300ms, then it would wait for 300 + 1500ms. We can make our test better. Remember: if the test "knows" what state the applicatin is in, then it can precisely wait for it, before running a negative assertion. Here is how I would modify my application to make it testable.

cypress/fixtures/spec4.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
// the list sets the correct data attribute
// with the rendered page number
const list = document.getElementById('list')
const next = document.getElementById('next')

setTimeout(() => {
  list.setAttribute('data-page', 1)
  list.innerHTML = `
    <li>first</li>
    <li>second</li>
    <li>third</li>
  `
}, 1000)
next.addEventListener('click', () => {
  setTimeout(() => {
    list.setAttribute('data-page', 2)
    list.innerHTML = `
      <li>fourth</li>
      <li>fifth</li>
      <li>sixth</li>
    `
  }, 1000)
  next.setAttribute('disabled', true)
})

The only thing this code does it sets the data-page attribute when it renders the page:

1
2
3
4
5
6
7
8
9
10
11
12
13
list.setAttribute('data-page', 1)
list.innerHTML = `
  <li>first</li>
  <li>second</li>
  <li>third</li>
`
// and later
list.setAttribute('data-page', 2)
list.innerHTML = `
  <li>fourth</li>
  <li>fifth</li>
  <li>sixth</li>
`

Our test can take the advantage of the data-page=... attribute to be much simpler

cypress/e2e/utils.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
/**
 * Checks if the given item is not present on the page and
 * recursively clicks the "Next" button until the item is found
 * or the button is disabled.
 * Confirm the list page has rendered before checking
 * by checking a data attribute.
 * @param {string} item - The item to check for on the page.
 * @param {number} page - The current page number (1-based), default 1
 * @returns {void}
 */
export const checkPage4 = (item, page = 1) => {
  cy.get(`ul[data-page=${page}]`)
  cy.get('li').log(`**page ${page} has list items**`)
  // we can use zero timeout since we are on the right page
  // and we know the LI elements have finished loading
  cy.contains('li', item, { timeout: 0 }).should('not.exist')
  cy.get('button#next')
    .invoke('is', ':enabled')
    .then((enabled) => {
      if (enabled) {
        cy.get('button#next').click()
        checkPage4(item, page + 1)
      } else {
        cy.log('**end of the list**')
      }
    })
}

Once the command cy.get("ul[data-page=${page}]") passes, the test is good to check the items. We can even use a very short timeout limit to run our negative assertion, since we know the items are already there and won't change.

The best test is quick and deterministic

Nice.