process

Use Required Test Tags Instead Of Skipping Tests

Use "@bahmutov/cy-grep" plugin to automatically recheck failing tests instead of skipping them.

Imagine the scenario: one of your Cypress tests fails. You are not sure why yet, so to avoid blocking the pipeline, you skip the test using it.skip. You even put a comment above the test with the GitHub issue link while you investigate.

1
2
3
4
5
// SKIP https://github.com/acme/project/issues/1
it.skip('deletes an item', () => {
  ...
  expect(1).to.equal(2)
})

What happens next? The number of skipped tests slowly grows. Unless you periodically come back, modify the spec file to unskip the test and try running it locally or via CI, the skipped tests stay ignored. There is a danger here. The skipped tests might get out of sync with the application and would have to be rewritten again.

🎁 You can find the full source code for this blog post in the repo bahmutov/skip-vs-required-tag.

A better way

Here is my solution to this problem. Use @bahmutov/cy-grep plugin and put those tests under required tag rather than skipping them.

1
2
$ npm i -D @bahmutov/cy-grep
+ @bahmutov/[email protected]

Configure the plugin following its README instructions

cypress.config.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
const { defineConfig } = require('cypress')

module.exports = defineConfig({
  env: { grepFilterSpecs: true, grepOmitFiltered: true },
  e2e: {
    // baseUrl, etc
    supportFile: false,
    fixturesFolder: false,
    setupNodeEvents(on, config) {
      require('@bahmutov/cy-grep/src/plugin')(config)
      // IMPORTANT: return the config object
      return config
    },
  },
})

Add to your support file

cypress/support/e2e.js
1
2
import registerCypressGrep from '@bahmutov/cy-grep/src/support'
registerCypressGrep()

Now mark each test that you want to skip by default with the required tag @skip

cypress/e2e/spec.cy.js
1
2
3
4
// SKIP https://github.com/acme/project/issues/1
it('deletes an item', { requiredTags: '@skip' }, () => {
  expect(1).to.equal(2)
})

Now let's run all tests skipping the one above.

1
2
3
4
5
6
7
8
9
10
11
12
$ npx cypress run
cy-grep: will omit filtered tests

...
  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  spec2.cy.js                               97ms        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                         97ms        1        1        -        -        -

The entire spec.cy.js was skipped because it only had it('deletes an item', { requiredTags: '@skip' } test. Now let's run just the tests with the tag @skip

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
$ npx cypress run --env grepTags=@skip
cy-grep: filtering using tag(s) "@skip"
cy-grep: will omit filtered tests

Running:  spec.cy.js                                  (1 of 1)


  1) deletes an item

  0 passing (575ms)
  1 failing

  1) deletes an item:

      AssertionError: expected 1 to equal 2
      + expected - actual

      -1
      +2

      at Context.eval (webpack:///./cypress/e2e/spec.cy.js:7:15)

We verified that the test "deletes an item" still fails. So it should keep its required tag @skip for now.

CI workflows

Let's add two GitHub Actions workflows to our repo. One runs all tests and another runs only the tests with the tag @skip

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
name: ci
on: [push]
jobs:
  test:
    # use the reusable workflow to check out the code, install dependencies
    # and run the Cypress tests
    # https://github.com/bahmutov/cypress-workflows
    uses: bahmutov/cypress-workflows/.github/workflows/standard.yml@v1
.github/workflows/skipped.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# a workflow to run all skipped tests nightly
name: skipped
on:
  schedule:
    - cron: '0 3 * * *'
  # also allow running this workflow by manually starting it
  # from the GitHub actions page
  workflow_dispatch:
jobs:
  test:
    # use the reusable workflow to check out the code, install dependencies
    # and run the Cypress tests
    # https://github.com/bahmutov/cypress-workflows
    uses: bahmutov/cypress-workflows/.github/workflows/standard.yml@v1
    with:
      # pass the required tag, just like "--env grepTags=@skip" from command line
      env: grepTags=@skip

When the "skipped" workflow finishes, we simply see that it failed.

Skipped tests workflow

Hmm, we need a better workflow summary to see which tests are now passing and which tests are still failing.

GitHub Actions test summary

We can use another plugin I wrote bahmutov/cypress-json-results to output GitHub Actions summary to show the individual test results.

1
2
$ npm i -D cypress-json-results
+ [email protected]

Include the plugin from the cypress.config.js file

cypress.config.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
const { defineConfig } = require('cypress')

module.exports = defineConfig({
  env: { grepFilterSpecs: true, grepOmitFiltered: true },
  e2e: {
    // baseUrl, etc
    fixturesFolder: false,
    setupNodeEvents(on, config) {
      require('@bahmutov/cy-grep/src/plugin')(config)
      // bahmutov/cypress-json-results
      require('cypress-json-results')({
        on,
        filename: false, // do not write JSON file
        githubActionsSummary: 'test',
      })

      // IMPORTANT: return the config object
      return config
    },
  },
})

Run the tests on GitHub Actions. The main workflow runs passing tests and reports success.

Passing tests results for the ci workflow

Meanwhile we have fixed a mistake in one of the "skipped" tests and fixed it (maybe). These tests are still ignored when the main workflow runs.

1
2
3
4
5
6
7
8
9
// SKIP https://github.com/acme/project/issues/1
it('deletes an item', { requiredTags: '@skip' }, () => {
  expect(1).to.equal(2)
})

// SKIP https://github.com/acme/project/issues/2
it('updates an item', { requiredTags: '@skip' }, () => {
  expect(1).to.equal(1)
})

Let's trigger the "skipped" workflow to verify our fix.

One of the skipped tests is working now

We verified that the test "updates an item" from the spec file "cypress/e2e/spec.cy.js" is working and we can remove the requiredTags: '@skip' from its settings.

Tip: we could probably check if any tests we expect to be failing are passing using my plugin cypress-expect when running the skipped workflow.

Links