products

Negative Assertions And Missing States

Negative applications do not eliminate enough states to accurately tell what the application is doing right now.

Imagine you got lost in a foreign town. You see someone and ask them a question in whatever little foreign language you have: "what street is this?" When they tell you the name of the street, you are no longer lost, assuming you know every street or have a map with you. But what if you ask, and the stranger instead of naming the street simply states a negative "you are NOT on Main street", then what? If the town only has two streets: Main and Second, then you are fine too. If you are not on Main street, then you are on Second street. Of course, if there are more than two streets, the answer "you are NOT on Main street" does not help you at all. You could be on any of the town's other N - 1 streets.

In the same town, now imagine you were walking with some directions. From the ocean shore, you were going down the Ocean ave, then you were taking the Main street to your destination. But the town has a fork in the road, the Ocean ave can lead to Main street and to Second street. At the end of Second street there is a fire-breathing dragon, and he is hungry. You do not want to go there.

The town you are walking through

Let's say at some point in your journey you ask someone "This is NOT Second street, is this?" You only hope to eliminate the dangerous section. But that is a wrong question to ask (props to Lemony Snicket). It only eliminates one street, and our town has three. Thus we could be on two streets, including Ocean avenue. We could still be approaching the dangerous fork and not even know it!

Application states

Often when walking through a new city, it helps to separate the journey into logical sections. In our case, the two legs of the journey are "before the fork" and "after the fork".

The journey has two sections

If we get disoriented, but know the section we are on, we can safely use a "This is NOT Second street, is this?" Each section cuts the number of possible choices to two.

  • when walking before the fork, the only answer possible is "No, this is not Second street" (we can only be on Ocean ave)
  • when walking after the fork, the only two choices are Second and Main. If we hear "Yes, you are on Second street", we turn around and run back to the fork in the road.

Here is how this translates to web applications. Let's model our application's state using xstate.js library. In my case, we will start with the simple Promise machine example from the Xstate documentation.

machine.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import { createMachine } from 'xstate'

export const promiseMachine = createMachine({
  id: 'promise',
  initial: 'pending',
  states: {
    pending: {
      on: {
        RESOLVE: { target: 'resolved' },
        REJECT: { target: 'rejected' },
      },
    },
    resolved: {
      type: 'final',
    },
    rejected: {
      type: 'final',
    },
  },
})

The machine only has two 3 states, almost like the 3 streets in our town. We can copy / paste the above code into the Xstate visualizer to see the states.

The Promise state machine

If we emit the "REJECT" event, the machine transitions to the "rejected" state. If we emit the "RESOLVE" event, the machine goes to the "resolved" state.

Let's implement a web interface.

App.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
import { useMachine } from '@xstate/react'
import { promiseMachine } from './machine'

const Component = () => {
  const [state, send] = useMachine(promiseMachine)

  return (
    <div className="flex flex-col">
      <div className="mb-5 self-center">
        {/** You can listen to what state the service is in */}
        {state.matches('pending') && <p>Loading it...</p>}
        {state.matches('rejected') && (
          <p className="settled">Promise Rejected</p>
        )}
        {state.matches('resolved') && (
          <p className="settled">Promise Resolved</p>
        )}
      </div>
      <div>
        {/** You can send events to the running service */}
        <button
          onClick={() => send('RESOLVE')}
          type="button"
          disabled={!state.matches('pending')}
        >
          Resolve
        </button>
        <button
          onClick={() => send('REJECT')}
          type="button"
          disabled={!state.matches('pending')}
        >
          Reject
        </button>
      </div>
    </div>
  )
}

export default function App() {
  return (
    <div className="flex flex-col justify-center items-center h-screen">
      <Component />
    </div>
  )
}

The application reacts to your button clicks. Each button emits its own event, which the machine uses as input. The machine the transitions to the new state, and the web app re-renders.

The app in action

🎁 You can find the source code for this blog post in the repo bahmutov/xstate-load-state.

How can we verify that application is working correctly? We can write a Cypress test to click the button and check the page.

1
2
3
4
5
6
it('resolves the promise (wrong question)', () => {
  cy.visit('/')
  cy.contains('Loading...').should('be.visible')
  cy.contains('button', 'Resolve').click()
  cy.contains('Loading...').should('not.exist')
})

The above test uses a negative assertion cy.contains('Loading...').should('not.exist') and while it passes, it does not tell us if the state machine is working correctly.

The test passes

For example, the application might wire the state machine incorrectly. On clicking the "Resolve" button it might go to the "Rejected" state, yet our test will still pass.

1
2
3
4
5
6
7
  pending: {
    on: {
-     RESOLVE: { target: 'resolved' },
+     RESOLVE: { target: 'rejected' },
      REJECT: { target: 'rejected' },
    },
  },

The test passes but the application is incorrect

Ughh, our test gives us false confidence. The application is broken, yet the test is green. It asked a wrong question. Remember that there 3 three streets, and your negative assertion only eliminates one of them.

What if we ask a negative assertion about the "Rejected" message? That still does not eliminate all the possible problem states.

1
2
3
4
5
6
it('resolves the promise (more wrong questions)', () => {
  cy.visit('/')
  cy.contains('Loading it...').should('be.visible')
  cy.contains('button', 'Resolve').click()
  cy.contains('Promise Rejected').should('not.exist')
})

The negative assertion cy.contains('Promise Rejected').should('not.exist') only helps us IF we are in the binary situation "Promise Rejected | Promise Resolved". If the application wires the states incorrectly, we could still be at the loading state! Let's break the app by removing the "onClick" handler from the "Resolve" button.

App.js
1
2
3
4
5
6
7
  <button
-   onClick={() => send('RESOLVE')}
    type="button"
    disabled={!state.matches('pending')}
  >
    Resolve
  </button>

The app is still loading, yet we asked about the rejected state

Negative assertions only work if we are for sure in a binary situation. The best way in our situation to ensure this is to eliminate states, even with negative assertions.

1
2
3
4
5
6
7
it('resolves the promise after no longer loading', () => {
  cy.visit('/')
  cy.contains('Loading it...').should('be.visible')
  cy.contains('button', 'Resolve').click()
  cy.contains('Loading it...').should('not.exist')
  cy.contains('Promise Rejected').should('not.exist')
})

Now the test correctly detects the broken application.

The test uses negative assertion correctly

Test the state

While checking the UI is nice, we can go beyond it. We can expose the current state from the application during end-to-end tests by checking the presense of window.Cypress property. Then our test assertions can be a lot more powerful.

App.js
1
2
3
4
5
6
7
8
  const Component = () => {
    const [state, send] = useMachine(promiseMachine)

+   if (window.Cypress) {
+     window.state = state
+   }
    ...
  }

From the test we can get to the window.state object and directly ask it.

spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
it('moves away from the pending state', () => {
  cy.visit('/')
  cy.contains('button', 'Resolve').click()
  // observe the state machine
  // Tip: have to use a should(callback)
  // because window.state is replaced when the state changes
  // see https://on.cypress.io/retry-ability
  cy.window().should((win) => {
    expect(win.state.matches('pending')).to.be.false
  })
  // now check if the promise is resolved or rejected
  // even better, check for the correct state
  cy.window().should((win) => {
    expect(win.state.matches('resolved')).to.be.true
  })
})

The test checks the state machine

So remember, instead of checking if you no longer can see the ocean, check if you are on the right street. And maybe then check if you for real cannot see the ocean any more.

So by using positive assertions, or by using negative assertions in binary situations, we can correctly test everything? Almost - as long as the state machine is correct.

The missing state

When someone gives you directions, they might misremember, or assume you know the details. A fork in the road might be more like a T intersection. Ocean ave might become Main street before Second street branches out.

Ocean ave becomes Main ave slightly before Second street fork

The above situation is dangerous to the traveler. They might think they are safe as they see "Main street" signs, and then accidentally continue down Second street. Asking "This is NOT Second street, is it?" gives them a false sense of security, as they do not know that the answer might change as they keep walking - the fork is still coming up!

The states "Ocean ave = before the road fork", "Main ave = after the road fork" do not cover the real world. The part between the street changing the name from Ocean ave to Main st, but before Second st branches out is ... weird. It is not covered by our description at the start of the walk. Asking questions (even good ones, like "Are we on Main street?") does not gives us enough information. Even when the answer is "Yes", we are still in danger, and we can turn to Second street accidentally.

This part of the road is missing its own state. Our journey is really three states as shown in the next diagram: before the street changes its name, after the name change, but before the fork, and after the fork.

We need to use three states to show the journey

Testing an app with a missing state

Here is how a missing state can happen in your web application. We will take our "Promise" state machine which works very well, and then we will try to implement "application loading" state machine. We will start by what we think is a reasonable first step - emit the event "Reject" after a short delay.

App.js
1
2
3
4
5
6
7
8
  <button
-   onClick={() => send('RESOLVE')}
+   onClick={() => setTimeout(() => send('REJECT'), 70)}
    type="button"
    disabled={!state.matches('pending')}
  >
    Resolve
  </button>

Adding a 70ms delay should not change the app too much, right? I hope our tests catch the error though - we are sending the "REJECT" event to the app instead of "RESOLVE". Let's see our test.

1
2
3
4
5
6
it('A negative assertion and app delay', () => {
  cy.visit('/')
  cy.contains('Loading it...').should('be.visible')
  cy.contains('button', 'Reject').click()
  cy.contains('Promise Resolved').should('not.exist')
})

We assume that clicking the button "Reject" moves the app out of "pending" state to the "Rejected | Resolved" state. Now that we have binary option to check, we can use the negative assertion cy.contains('Promise Resolved').should('not.exist') to correctly test the app.

The test passes even though it should have failed

The test does not catch the problem. The assumption that we are no longer in the "pending" state immediately after clicking the button "Reject" is false - the application stays in that state during onClick={() => setTimeout(() => send('RESOLVE'), 70)} duration.

I see this error a lot. A widget is loading, and the test checks if an error message is shown. There is no error message, so the test continues. But that is wrong, since the app is still loading. Then the app finishes the loading and displays an error, but it is too late for the test to detect.

So what's a better way?

  1. Explicitly model the "loading" state here. Let's change our state machine a little.
loadingMachine.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import { createMachine } from 'xstate'

export const loadingMachine = createMachine({
  id: 'loading',
  initial: 'pending',
  states: {
    pending: {
      on: {
        LOAD: {
          target: 'loading',
        },
      },
    },
    loading: {
      on: {
        RESOLVE: { target: 'resolved' },
        REJECT: { target: 'rejected' },
      },
    },
    resolved: {
      type: 'final',
    },
    rejected: {
      type: 'final',
    },
  },
})

Let's visualize this tate machine

The above state machine visualized

  1. Have an explicit way to check the current state from the test. This way the test can check if the application is still loading the data.

For example, you could set class or data attribute

1
2
3
4
5
6
7
8
9
10
11
<div className="mb-5 self-center">
  {/** You can listen to what state the service is in */}
  {state.matches('pending') && <p data-state="initial">Initial</p>}
  {state.matches('loading') && <p data-state="loading">Loading it...</p>}
  {state.matches('rejected') && (
    <p data-state="loaded">Promise Rejected</p>
  )}
  {state.matches('resolved') && (
    <p data-state="loaded">Promise Resolved</p>
  )}
</div>

Then the test can explicitly check how the state moved along the expected transitions.

1
2
3
4
5
6
7
it('Checks the state', () => {
  cy.visit('/')
  cy.get('[data-state="initial"]')
  cy.contains('button', 'Reject').click()
  cy.get('[data-state="loaded"]')
  cy.contains('Promise Resolved').should('not.exist')
})

Another alternative to checking the page, is to check the underlying state of the state machine. We can access the state machine from the test, just like before; the application code needs to put the state object on the window object. I suggest adding a custom command for checking the expected state.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Cypress.Commands.add('reachedState', (state) => {
  cy.log(`expecting **${state}** state`)
  cy.window({ log: false }).should((win) => {
    if (win.state.matches(state) !== true) {
      throw new Error(`Expected state "${state}", got "${win.state.value}"`)
    }
  })
})

it('Checks the state machine', () => {
  cy.visit('/')
  cy.reachedState('pending')
  cy.contains('button', 'Reject').click()
  // optional: check the intermediate state
  cy.reachedState('loading')
  cy.reachedState('rejected')
  cy.contains('Promise Resolved').should('not.exist')
})

For example, in our case it catches the transition to the wrong state.

The application gets into an unexpected state

Finally, after fixing the application code, we have a passing test

1
2
3
4
5
6
7
8
9
it('Checks the state machine', () => {
  cy.visit('/')
  cy.reachedState('pending')
  cy.contains('button', 'Reject').click()
  // optional: check the intermediate state
  cy.reachedState('loading')
  cy.reachedState('rejected')
  cy.contains('Promise Rejected').should('be.visible')
})

The passing test for the fixed application

The test is tight - it will be hard for the application to be broken with this kind of test that confirms the app's state transitions step by step using positive assertions.

Use better questions

  • when getting lost in a city, ask which street intersection you are on. That gives you a unique precise position, eliminating all other possibilities.
  • when testing a web application and the user flow, use properties that eliminate as many other states as possible. For example, I love using cy.location to know when the application has transitioned to the new state. A positive assertion eliminates a lot more possible states than a negative assertion.

See also