products

Do Not Use SHA To Compare HTML During E2E Tests

Use positive assertions rather than SHA changes in your tests.

Recently a user posted an example in one of Cypress GitHub repo issues showing a "problem" computing asynchronously the SHA-256 code from the element's HTML. The original test as I understand it tries to confirm that the table's HTML changes when the user types a search string, and the table updates in response. Here is the relevant code snippet as posted by the user:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// 🚨 DOES NOT WORK, JUST A QUESTION
describe('Filter textbox', () => {
  beforeEach(() => {
    cy.get(test('suplovaniTable')).as('suplovaniTable')
    cy.get(test('filterTextbox')).as('filterTextbox')
  })

  it('changes data on filter text change', async () => {
    const table = await cy.get('@suplovaniTable')
    const hash = sha256(table[0].innerHTML)

    const filterCell = await cy.get(
      '[data-test=suplovaniTable] > tbody > :nth-child(1) > :nth-child(2)',
    )
    await cy.get('@filterTextbox').type(filterCell[0].innerText)

    const newTable = await cy.get('@suplovaniTable')
    const newHash = sha256(newTable[0].innerHTML)
    expect(hash).to.not.equal(newHash)
  })
})

Do not use SHA

Ok, let's put the entire async / await vs reactive streams for writing tests question aside. Does using SHA256 to confirm the table changes a good idea? No. It is a terrible idea. Here is a typical example. I will grab the SHA implementation from Mozilla docs:

1
2
3
4
5
6
7
8
9
10
11
12
13
async function digestMessage(message) {
  // encode as (utf-8) Uint8Array
  const msgUint8 = new TextEncoder().encode(message)
  const hashBuffer = await crypto.subtle.digest(
    'SHA-256',
    msgUint8,
  ) // hash the message
  const hashArray = Array.from(new Uint8Array(hashBuffer))
  const hashHex = hashArray
    .map((b) => b.toString(16).padStart(2, '0'))
    .join('') // convert bytes to hex string
  return hashHex
}

The SHA256 can change for any reason and you have no way to say why it did change.

1
2
3
4
5
6
7
cy.wrap(digestMessage('<div>Hello</div>')).then((sha1) => {
  return digestMessage('<div>Hello</div> ').then((sha2) => {
    expect(sha1, 'extra space at the end').to.not.equal(sha2)
  })
})
// expected 55486289576c734c46fe5bfe662f51a4c31c1ffba3ebe0e497f263d1af299cc1
// to not equal d68ab47032d3f15c89b39c79e9e19c3f65be54a589979f3f057b572e6f9dd7c1

Even worse, the HTML of the page can change for any reason. The assertion "SHA should be different" does not see why the change has happened. Let's take a page that suddenly shows an error:

1
2
3
4
5
6
7
8
9
10
11
<button id="load">Load data</button>
<div id="output" />
<script>
  document
    .getElementById('load')
    .addEventListener('click', () => {
      document.getElementById('output').innerHTML = `
        <div>Error loading...</div>
      `
    })
</script>

Our test can easily incorporate asynchronous functions into the Cypress chain. This code comes from cypress-examples site.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
async function digestMessage(message) {
  // encode as (utf-8) Uint8Array
  const msgUint8 = new TextEncoder().encode(message)
  const hashBuffer = await crypto.subtle.digest(
    'SHA-256',
    msgUint8,
  ) // hash the message
  const hashArray = Array.from(new Uint8Array(hashBuffer))
  const hashHex = hashArray
    .map((b) => b.toString(16).padStart(2, '0'))
    .join('') // convert bytes to hex string
  return hashHex
}

Cypress.Commands.add(
  'sha',
  { prevSubject: 'element' },
  ($el) => {
    // put a message to Command Log
    Cypress.log({ name: 'sha' })
    const html = $el.html().trim()
    return digestMessage(html)
  },
)
1
2
3
4
5
6
cy.get('#output')
  .sha()
  .then((sha) => {
    cy.contains('button', 'Load data').click()
    cy.get('#output').sha().should('not.equal', sha)
  })

Unfortunately, the test passes, yet the app is obviously broken.

The passing test does not notice the error

Again, the SHA of the HTML can change for a billion reason. The assertion "SHA should change" accepts any of them. Your tests should instead verify the single expected behavior using positive assertions.

Use positive assertions

What can we use instead? Let's get back to the filtered table. I have prepared this example in the repo bahmutov/sorted-table-example.

1
2
3
4
5
// cypress/e2e/filter.cy.js

it('filters rows by name', () => {
  cy.visit('app/filter.html')
})

If you type in part of the first name, it filters the table - and there is a small delay between typing and filtering to simulate realistic application.

The filtered table page

Ok, so what do we want to confirm in this case? If you write down instructions for a human tester, I think they would look something like this:

  • visit the page
  • there should be several rows in the table
  • type "Jo" into the input box
  • the table should filter rows
    • there should be fewer rows
    • each row's first cell should include string "Jo"

Let's write this test using Cypress and cypress-map plugin.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// https://github.com/bahmutov/cypress-map
import 'cypress-map'

it('filters rows by name', () => {
  // visit the page
  cy.visit('app/filter.html')
  // there should be several rows in the table
  cy.get('#people tbody tr')
    .its('length')
    .should('be.gt', 2)
    .then((n) => {
      // type "Jo" into the input box
      cy.get('input#by-name').type('Jo')
      // there should be fewer rows
      cy.get('#people tbody tr').should('have.length.lessThan', n)
    })
  // every row's first cell should include the string "Jo"
  cy.get('#people tbody td:nth-child(1)')
    // cy.map and cy.print come from cypress-map
    .map('innerText')
    .print('names %o')
    .should((names) => {
      names.forEach((name) => {
        expect(name).to.include('Jo')
      })
    })
})

The table filters

A better test

Even the above test is far from perfect. We do not control the data, so we cannot guarantee several key moments:

  • there might be zero rows in the table, so our assertion might fail
1
2
3
cy.get('#people tbody tr')
  .its('length')
  .should('be.gt', 2)
  • every first name in the table might include "Jo", thus the middle assertion might fail
1
2
// there should be fewer rows
cy.get('#people tbody tr').should('have.length.lessThan', n)

To really confirm the application works we need to control the data. Notice the application is making a network call to fetch the data using the GET /people call. The call fails, so the app shows its local data. Let's change this. Let's stub the network call using cy.intercept command and then we will know exactly the data on the page.

cypress/fixtures/people.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[
  {
    "name": "Peter",
    "date": "2000-01-01",
    "age": 23
  },
  {
    "name": "Pete",
    "date": "2000-01-01",
    "age": 23
  },
  {
    "name": "Mary",
    "date": "2000-01-01",
    "age": 23
  },
  {
    "name": "Mary-Ann",
    "date": "2000-01-01",
    "age": 23
  }
]
1
2
3
4
5
6
7
it('controls the network data', () => {
  cy.intercept('/people', { fixture: 'people.json' }).as('people')
  cy.visit('app/filter.html')
  cy.get('#people tbody tr').should('have.length', 4)
  cy.get('input#by-name').type('Mary')
  cy.get('#people tbody tr').should('have.length', 2)
})

This is a much better and simpler test.

Stub the network call and confirm the table works

Of course, you can make the above test ever stricter by confirming the table cell contents, see Cypress Plugins course lessons, but you get my point:

  • do NOT use "SHA should change" assertion
  • use positive meaningful assertions to verify the application's behavior
  • if you control the data the application receives, the test becomes clear and simple.

Happy testing!