process

Cypress Test Statuses

How to track the written tests using the Cypress test statuses

After the Cypress spec completes every test has one of the 4 statuses: passing, failing, pending, or skipped. Let's look into each status.

Passing

Passing tests are the best, aren't they? The have successfully completed all their commands, never failing any assertions. Our smoke test below is a passing test:

Passing test

Note that a test can pass after several test retries. In that case the Command Log shows the attempts, but ultimately the entire test has finished successfully.

Failing

Good news - the failing test has found a problem. Could be much worse - it could be a user hitting this bug.

The test failed because there are two items and not 100

After the test fails, use the screenshot and the video to find the problem and fix it.

Pending tests

The placeholder tests we have written were marked pending. They did not execute because the user did not want them to run - the user did not write them. Slightly confusingly, if the user does not want an existing test to run, they can use the it.skip or xit syntax to skip the test or a block of tests. Cypress marks all these tests pending.

1
2
3
4
5
6
7
8
9
10
11
12
13
describe('TodoMVC', () => {
  it('is not written yet')

  it.skip('adds 2 todos', function () {
    cy.visit('/')
    cy.get('.new-todo').type('learn testing{enter}').type('be cool{enter}')
    cy.get('.todo-list li').should('have.length', 100)
  })

  xit('another test', () => {
    expect(false).to.true
  })
})

All 3 tests above are marked pending when Cypress finishes running the spec file.

Different ways to get a pending test

So remember - if you (the test writer) knowingly skip a test using one of the above three ways, Cypress counts it as a pending test. The result will be known in the future when you enable the test again.

The status and the name "pending" comes from Mocha.js test engine that Cypress uses internally. See the pending tests section of the Mocha's documentation captured below.

The definition of the pending test from Mocha

Skipped tests

The last test status is for tests that the user meant to run, but these tests were skipped due to some run-time error. For example, imagine a group of tests sharing the same beforeEach hook. The hook executes the common commands. For example, we could visit the page in the beforeEach hook because every test needs to start at the page.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
/// <reference types="cypress" />

describe('TodoMVC', () => {
  beforeEach(() => {
    cy.visit('/')
  })

  it('hides footer initially', () => {
    cy.get('.filters').should('not.exist')
  })

  it('adds 2 todos', () => {
    cy.get('.new-todo').type('learn testing{enter}').type('be cool{enter}')
    cy.get('.todo-list li').should('have.length', 2)
  })
})

If the beforeEach hook completes and both tests finish, two tests are passing.

Two passing tests

But what happens if a command inside the beforeEach hook fails? For example, let's pretend we want to visit a non-existent page /hmm instead of the /. If we change our beforeEach to fail:

1
2
3
beforeEach(() => {
  cy.visit('/hmm')
})

When Cypres starts executing the first test, the beforeEach hook fails. Now the first test is marked "failing". BUT if the beforeEach hook failed once, why would we execute it again before the second test? It would just fail the same way! So Cypress skips the remaining tests in that block, because they would also fail for sure due to the bad beforeEach hook.

One failed test and the rest of tests were skipped

If we collapse the test commands, we can see the empty box marking the skipped test "adds 2 todos".

Skipped test

The tests that were meant to be executed, but were skipped due to some run-time problem are marked "skipped" by Cypress.

The same situation happens if instead of the beforeEach hook, the before hook fails.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
/// <reference types="cypress" />

describe('TodoMVC', () => {
  before(() => {
    cy.visit('/hmm')
  })

  it('hides footer initially', () => {
    cy.get('.filters').should('not.exist')
  })

  it('adds 2 todos', () => {
    cy.get('.new-todo').type('learn testing{enter}').type('be cool{enter}')
    cy.get('.todo-list li').should('have.length', 2)
  })
})

The Cypress Test Runner fails the first test (the one that really executes the before hook), and skips the rest of the scheduled tests.

The rest of the tests are skipped when the before hook fails

Hope this explanation clarifies the Cypress test statuses.

Bonus: track the test statuses over time

I have described in the blog post Writing Tests Progress how to start testing the project by writing a lot of pending tests and then fill the tests with actual implementations.