products

Be Careful With Negative Assertions

Because negative assertions can pass for the wrong reason.

I like to think of some assertions as "positive" and other assertions as "negative". Examples are

1
2
3
4
5
6
7
// positive
cy.get('.todo-item')
  .should('have.length', 2)
  .and('have.class', 'completed')
// negative
cy.contains('first todo').should('not.have.class', 'completed')
cy.get('#loading').should('not.be.visible')

From my experience, it is easy to have the negative assertions pass for the wrong reason. This blog shows a common mistake I see when testing if the loading element goes away.

Loading element

Imagine our TodoMVC application loads the Todo items from the backend on startup. The loading element markup changes the visibility based on the app's data:

1
<div class="loading" v-show="loading">Loading data ...</div>

🎁 Find this example in the repository cypress-io/testing-workshop-cypress.

So we can write a test that asserts the loading indicator goes away when we visit the page.

1
2
3
4
it('hides the loading element', () => {
  cy.visit('/')
  cy.get('.loading').should('not.be.visible')
})

The test is very quick - our server runs locally, there is no data, thus the entire call happens very quickly.

Test asserts the loading element is invisible

Notice the Command Log on the left side shows the XHR request happening after the negative assertion passes. This is a red flag - did the assertion pass before the application actually started loading?

Our application can take a short period to bootstrap or scaffold itself before it makes an Ajax call to the server. Can the assertion cy.get('.loading').should('not.be.visible') pass before the Ajax call starts? The test application I use can be "slowed" down by passing a search parameter. Let's start the Ajax call three seconds after the page load to see what happens:

1
2
3
4
it('uses negative assertion and passes for the wrong reason', () => {
  cy.visit('/?delay=3000')
  cy.get('.loading').should('not.be.visible')
})

Test finishes even before the Ajax call starts

Hmm, our negative assertion passes for the wrong reasons. We naively expected the loading element to be invisible after the data has been loaded. But it passed because the loading element is invisible at the start.

The solution

To avoid the above situation I suggest always following the negative assertion after a positive assertion. The positive assertion should be tied to the application effect in response to the test's command. For example, we first want to see the loading element (application effect), then assert that it hides (the negative assertion).

1
2
3
4
5
6
7
it('use positive then negative assertion (flakey)', () => {
  cy.visit('/?delay=3000')
  // first, make sure the loading indicator shows up (positive assertion)
  cy.get('.loading').should('be.visible')
  // then assert it goes away (negative assertion)
  cy.get('.loading').should('not.be.visible')
})

The above test passes for the right reasons.

Positive then negative assertions

Flakiness

Note that I called the above test flakey - because the loading element flashes so quickly that sometimes Cypress does not "see" it via its command retries. This is when test retries come in handy

1
2
3
4
5
6
7
8
9
describe('Careful with negative assertions', { retries: 2 }, () => {
  it('use positive then negative assertion (flakey)', () => {
    cy.visit('/?delay=3000')
    // first, make sure the loading indicator shows up (positive assertion)
    cy.get('.loading').should('be.visible')
    // then assert it goes away (negative assertion)
    cy.get('.loading').should('not.be.visible')
  })
})

Here is a screenshot of the first attempt failing to "catch" the loading element, and the second attempt catching it.

Test retries to the rescue

We can make the test better in other ways. For example, we can stub the call to the server and delay it to make sure the loading element clearly stays visible. This has additional benefit - the loading element state will be clearly visible in the spec's video captured by Cypress on the continuous integration server.

1
2
3
4
5
6
7
8
9
10
11
it('slows down the network response (works)', () => {
  cy.intercept('/todos', {
    body: [],
    delayMs: 5000
  })
  cy.visit('/?delay=3000')
  // first, make sure the loading indicator shows up (positive assertion)
  cy.get('.loading').should('be.visible')
  // then assert it goes away (negative assertion)
  cy.get('.loading').should('not.be.visible')
})

Delaying the network request to make the loading indicator clearly visible

🤷‍♂️ Wondering why I used delayMs: 5000 and not simply delayMs: 2000 for a 2 second delay in the test above? Because while writing this test I have discovered that cy.intercept sets the time NOT from the request but from its own execution timestamp, see #14511. In the test the page load takes 3 seconds, thus the response with 5 second delay from the start of the test would have the server "respond" after 2 seconds.

Update 1: RealWorld App Example

Here is another example of negative assertions leading to flaky tests. In our Cypress RealWorld App the following test was constantly generating visual differences and failing the Percy check:

1
2
3
4
5
6
7
8
9
it("rejects a transaction request", function () {
  cy.visit(`/transaction/${ctx.transactionRequest!.id}`);
  cy.wait("@getTransaction");

  cy.getBySelLike("reject-request").click();
  cy.wait("@updateTransaction").should("have.property", "status", 204);
  cy.getBySelLike("reject-request").should("not.exist");
  cy.visualSnapshot("Transaction Rejected");
});

Here is the visual difference between the good brunch "develop" and every PR

Visual diff showing blank area during the tests for PR

Our test uses negative assertion before taking the visual snapshot test.

1
2
cy.getBySelLike("reject-request").should("not.exist");
cy.visualSnapshot("Transaction Rejected");

The good snapshot from the develop branch shows non-blank main area. What is going on? Let's run the test using cypress open and use the time-traveling debugger to inspect the DOM snapshot at the moment the negative assertion passes.

Inspecting the state of the app at the time the negative assertion passes

Notice the problem? When the negative assertion passes the reject DOM element does not exist. The application area is blank. The test proceeds to take the visual snapshot. What does the application do meanwhile? It renders the default transaction view! So our visual snapshot command is now racing against the application - will we take the snapshot of the blank area or the updated default view? Who knows! Locally it seems to be fast, on CI the test runner might be taking the snapshot faster than the application updates. This will become the flaky test.

Let's tighten it. Once we reject the transaction we want to wait for the application to render the default view and then take the snapshot.

1
2
3
cy.getBySelLike("reject-request").should("not.exist");
cy.getBySel("transaction-detail").should("be.visible");
cy.visualSnapshot("Transaction Rejected");

The pull request #736 shows no visual flake.

Once we fix this one test, let's search the spec file for other negative assertions. As you can see there are five tests that use negative assertion before taking a visual snapshot. These tests are also flaky and needs to be strengthen against the race condition.

Finding other tests using negative assertion

Following the above example I add a positive assertion before the visual snapshot to tighten these tests against the flake.

Fun fact: adding positive assertion found the incorrect visual snapshot in the opposite direction 😀

The test shows incorrect baseline image in the develop branch

Of course we can approve this visual change to have the correct baseline images from now on.

Read more