products

Wrap Cypress Using NPM Module API

How to wrap Cypress commands using its NPM module API to customize the test runner behavior.

Cypress NPM module API

Cypress test runner has NPM module API that lets you call Cypress programmatically from a Node script. For example, you might get the number of passing and failed tests from the test results.

1
2
3
4
5
const cypress = require('cypress')
cypress.run().then(testResults => {
  console.log('passed %d tests, failed %d tests',
    testResults.totalPassed, testResults.totalFailed)
})

When cypress.run executes, it resolves with an object that contains detailed information about all tests that Cypress executed or skipped. But before we can check the test results, we must check if Cypress ran at all - maybe the Cypress binary was not installed? Thus we check using status property:

1
2
3
4
5
6
7
8
9
10
11
12
13
cypress.run(options)
  .then((runResults) => {
    if (runResults.status === 'failed') {
      // Cypress could not run, something is terrible wrong
      console.error(runResults.message)
      return process.exit(1)
    }

    // otherwise the should be test results, but let's check first
    if (runResults.status === 'finished') {
      // drill into runResults object
    }
  })

Thus we check runResults.status property that can be either failed or finished to tell if the Test Runner executed before drilling into the test results.

How do we get the right options object to pass into cypress.run call? You do not want to implement your own CLI parsing logic - because it might parse arguments differently from Cypress' CLI. Recently we have introduced an utility to parse the cypress run ... command line arguments to make creating such Node wrappers very easy.

Parsing the CLI arguments

When running Cypress, you might be passing cypress run CLI parameters to use a specific the browser, record the test results on the Cypress Dashboard:

1
$ npx cypress run --browser chrome --record

How would you pass the same CLI arguments to cypress.run({...}) Node call? You need to parse the CLI arguments and convert to the equivalent options object. Recently we have exposed the CLI parsing logic to allow you to do exactly this:

src/wrap.js
1
2
3
4
const cypress = require('cypress')
const runOptions = await cypress.cli.parseRunArguments(process.argv.slice(2))
console.log(runOptions)
const testResults = await cypress.run(runOptions)

If we run this wrap script, it will accurately transform the CLI arguments to an options object

1
2
3
4
5
$ node ./src/wrap run --browser chrome --record
{
  browser: 'chrome',
  record: true
}

The same code is used internally by npx cypress run code, thus the Cypress behavior should be the same.

Meta testing

Now that we parse the Cypress arguments using the its own logic, we can create useful cypress run wrappers. For example, I might want to validate the number of passing tests in a project to ensure that accidentally we don't skip bunch of tests and get a green build only because the tests were not run.

Tip: you can find this wrapper in bahmutov/cypress-expect

The wrapper needs its own command line parameters, for example we want to specify the number of tests, and the rest of the arguments should go into cypress.cli.parseRunArguments like this:

1
2
3
# run Cypress tests in Chrome browser, record on the Dashboard
# and check if there are 5 passing tests
npx cypress-expect run --expect 5 --browser chrome --record

The cypress-expect needs to grab its own parameter --expect and then let Cypress' logic to parse the rest of the arguments. We can use the parsing module arg for this:

cypress-expect/index.js
1
2
3
4
5
6
7
#!/usr/bin/env node
const arg = require('arg')
const args = arg({
  '--passing': Number, // number of total passing tests to expect
})
// let Cypress parse the CLI arguments unclaimed by `arg`
const cypressOptions = await cypress.cli.parseRunArguments(args._)

Once Cypress runs, we can check if the two numbers match - and if they don't we can fail the program.

1
2
3
4
5
6
7
8
9
10
11
const testResults = await cypress.run(cypressOptions)
if (testResults.status === 'finished') {
  if (testResults.totalPassed !== args['--passing']) {
    console.error(
      'ERROR: expected %d passing tests, got %d',
      args['--passing'],
      testResults.totalPassed,
    )
    process.exit(1)
  }
}

On top of the exact match, we can specify the minimum expected number of passing tests. We can even extend the program in the future to do meta-testing - allow specifying tests we expect to fail or to skip.

Repeat tests

If our tests suffer from flake, we can run the tests multiple times to catch the flaky behavior. The project cypress-repeat does exactly this - it runs the same project the specified number of times to flush out conditions leading to flaky tests.

1
$ npx cypress-repeat run -n <N> ... rest of "cypress run" arguments

The arguments are parsed the same way as in cypress-expect

cypress-repeat/index.js
1
2
3
4
5
6
#!/usr/bin/env node
const arg = require('arg')
const args = arg({
  '-n': Number,
})
const cypressOptions = await cypress.cli.parseRunArguments(args._)

Then we iterate and run the same test N times - and we can adjust the options to avoid clashing with previously recorded groups.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
const Bluebird = require('bluebird')
const cypressOptions = await cypress.cli.parseRunArguments(args._)

const allRunOptions = []

for (let k = 0; k < repeatNtimes; k += 1) {
  const runOptions = clone(cypressOptions)

  if (options.record && options.group) {
    runOptions.group = options.group

    if (runOptions.group && repeatNtimes > 1) {
      // make sure if we are repeating this example
      // then the recording has group names on the Dashboard
      // like "example-1-of-20", "example-2-of-20", ...
      runOptions.group += `-${k + 1}-of-${repeatNtimes}`
    }
  }

  allRunOptions.push(runOptions)
}

// now can iterate over all run options and run Cypress for every object
Bluebird.mapSeries(allRunOptions, (runOptions, k, n) => {
  return cypress.run(ruOptions)
})

By using cypress-repeat and running projects five, even ten times in a row on CI we are able to flush out all the weird test situations that cause flake and finally fix them.

Link to Cypress Dashboard

If you record a Cypress run with cypress run --record you will see the Dashboard URL shown in the terminal output.

1
2
3
4
5
6
7
8
9
(Run Starting)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress:    5.2.0                                                                              │
│ Browser:    Electron 83 (headless)                                                             │
│ Specs:      1 found (app_spec.js)                                                              │
│ Params:     Tag: false, Group: false, Parallel: false                                          │
│ Run URL:    https://dashboard.cypress.io/projects/abc123/runs/1321                             │
└────────────────────────────────────────────────────────────────────────────────────────────────┘

If you have your own CI scripts you might want to grab this URL to send in a build notification or add to a test status page, linking back to the recorded run. Now it is simple to do by writing own wrapper

1
2
3
4
5
6
cypress.run(...)
  .then(testResults => {
    if (testResults.status === 'finished') {
      console.log('Dashboard URL', testResults.runUrl)
    }
  })

Links