Cypress Metaprogramming

Take a Cypress test

it('works', () => {
  expect(true).to.be.true
})

🧭 You can find the source code for this post in the repository bahmutov/cy-metaprogramming-example

The test passes.

What if the test fails? What if we want the test to fail, but for the right reason? We need a secondary level of test criteria, something to tell Cypress "expect this error in this test, the test should fail, but that's ok".

Let's write a failing test

it('fails', () => {
  expect(5).to.equal(42)
})

The test fails, as expected

Ignoring failed assertion

How do we catch the failure and ignore it? When the test fails, Cypress emits 'fail' event. You can register a listener and ignore the failing test error

it('fails', () => {
  cy.on('fail', (e) => {
    console.error(e)
  })

  expect(5).to.equal(42)
})

By the way, the first failed assertion stops the test execution. Thus if you have multiple assertions like this test does, you only will get a single 'fail' event from the first assertion.

it('fails', () => {
  cy.on('fail', (e) => {
    console.error(e)
  })

  expect(5).to.equal(42)
  expect(true).to.be.false
})

The event listener cy.on(...) only makes sense inside a test. If you place it outside the test, it will be ignored

cy.on('fail', (e) => {
  console.error(e)
})

it('fails', () => {
  expect(5).to.equal(42)
})

The above code happily fails the test. If you want to register a global event handler to ignore all failed assertions, you should add the event listener using Cypress.on(...) method.

Cypress.on('fail', (e) => {
  console.error(e)
})

it('fails', () => {
  expect(5).to.equal(42)
})

Checking the assertion

Let's make sure our test fails for the right reason. We can inspect the assertion error in the cy.on('fail', ...) callback. If the assertion error is "right", we can pass the test. If the failure is for some other reason, we can fail the test by throwing the error.

it('fails', () => {
  cy.on('fail', (e) => {
    if (e.message !== 'expected 5 to equal 42') {
      throw e
    }
  })

  expect(5).to.equal(42)
})

I suggest encoding that the test is expected to fail and the the expected error message in the test title. We have access to the test title in the second parameter passed to the cy.on('fail', cb) callback function.

it('fails with expected 5 to equal 42', () => {
  cy.on('fail', (e, test) => {
    if (test.title.startsWith('fails with') &&
      test.title.endsWith(e.message)) {
      console.log('test "%s" is allowed to fail with error "%s"',
        test.title, e.message)
    } else {
      throw e
    }
  })

  expect(5).to.equal(42)
})

What about a test that should have fail yet never did fail? We could always end such test with a "wrong" assertion. If the test ever gets there, then the expected failure never occurred and the test unexpectedly passed.

const emergency = () => {
  throw new Error('Should never happen')
}

it('fails with expected 5 to equal 42', () => {
  cy.on('fail', (e, test) => {
    if (test.title.startsWith('fails with') &&
      test.title.endsWith(e.message)) {
      console.log('test "%s" is allowed to fail with error "%s"',
        test.title, e.message)
    } else {
      throw e
    }
  })
  // expect(5).to.equal(42)
  emergency()
})

Failed tests should really fail

I don't like the above approach, it is easy to forget to include the emergency brake. Instead I prefer to keep track of the failed tests and throw an error if a test that should have failed somehow passes. To do this, we can move the cy.on('fail', ...) logic into common place using Cypress.on('fail', ...) and afterEach hook.

const failedTests = {}
const isFailingTest = (test) =>
  test.title.startsWith('fails with')

afterEach(function () {
  console.log(this.currentTest)
  console.log(failedTests)
  if (!isFailingTest(this.currentTest)) {
    return // all good
  }
  if (!failedTests[this.currentTest.title]) {
    throw new Error(`Test "${this.currentTest.title}" somehow passed`)
  }
})

Cypress.on('fail', (e, test) => {
  if (isFailingTest(test) &&
    test.title.endsWith(e.message)) {
    console.log('test "%s" is allowed to fail with error "%s"',
      test.title, e.message)
    failedTests[test.title] = test
  } else {
    throw e
  }
})

The above code keeps track of all seen failed tests, and then after each test, if the test title starts with fails with ... and has not been seen - that test passed accidentally. Let's take an example where one test should be failing, but does not.

it('works', () => {
  expect(true).to.be.true
})

it('fails with expected 5 to equal 42', () => {
  expect(5).to.equal(42)
})

it('fails with some error', () => {
  // notice that this test should have failed,
  // judging by the title, but does not
  expect(5).to.equal(5)
})

The last test gets "caught".

Meta-programming by counting

The above approach is "low-level". What if we just want to confirm that in the given project 10 tests are passing, while 2 tests are failing? In our example, let's have 1 passing test, 1 failing, and 1 skipped test (and we will disable all Cypress.on('fail', ...) logic)

it('works', () => { ... })

it('fails with expected 5 to equal 42', () => { ... })

it.skip('fails with some error', () => { ... })

If we use cypress run, the Test Runner prints the numbers at the end

Let's get the same numbers and check if there is a single passing test and a single failing test. We can get to these numbers using Cypress NPM module API as I have described in the blog post Wrap Cypress Using NPM Module API.

First, let's install cypress-expect.

$ npm i -D cypress-expect
+ [email protected]

Now replace the command npx cypress run with npx cypress-expect run --passing 1 --failing 1 --pending 1, for example in package.json

package.json

{
  "scripts": {
    "test": "cypress-expect run --passing 1 --failing 1 --pending 1"
  }
}

Tip: if your original cypress run command had other CLI arguments, just use them with cypress-expect run. For example:

npx cypress run --config baseUrl=http://localhost:3000 ...
npx cypress-expect run --passing 4 --config baseUrl=http://localhost:3000 ...

Let's run the tests using npm t command.

Notice the failing test, yet the exit code is zero. The wrapper of cypress-expect lets cypress run finish, then looks at the numbers. If they match the expected counts, the process exits with code zero.