products

Cypress Mochawesome HTML Reports

Generate and store test reports and artifacts on GitHub Actions.

Yesterday I have shown how to run Cypress specs in parallel for free using some of the most popular CI providers: GitHub Actions, CircleCI, and GitLab CI. Today, I will show you how to store test artifacts to help you understand the test failures on CI. We will generate Mochawesome HTML reports, and I will show a single CI job and a parallel one using cypress-split.

The test artifacts

I will start this blog post with the results achieved. On every commit, GitHub Actions test my application using cypress run command. Cypress saves screenshots and videos plus I have configured Mochawesome reporter to save the HTML reports. I have introduced a failed test on purpose to make the test artifacts useful:

GitHub Actions workflow

🎁 You can find the full source code and the test artifacts in the repo bahmutov/cy-report-example.

Under the workflow summary, you can see all test artifacts. We have a single "standard" job with all specs, plus 3 parallel test jobs running the same specs.

Stored test artifacts

Let's download the artifact "standard-cypress-screenshots" to see how our Test Runner looked when it failed:

The failed test screenshot

We can see that the spec viewports.cy.ts has a failure. Let's look at the Mochawesome HTML report in the test artifact "standard-mochawesome". When we download the Zip archive from GitHub, we get a folder with a separate HTML report for each spec

1
2
3
4
5
standard-mochawesome/
  assets/
  spec1.html
  spec2.html
  viewports.html

Let's open the viewports.html report.

Mochawesome report for viewports spec file

In the failed test, we have its source code plus the error message. Taken together with the Test Runner screenshot and the "standard-cypress-videos" artifact with the run MP4 videos, we have enough information to determine the cause of the failure.

Cypress configuration

The Cypress configuration is very straightforward. We will add the Mochawesome plugins (plus cypress-split if you want)

1
2
3
4
5
$ npm i -D mochawesome mochawesome-merge mochawesome-report-generator

[email protected]
[email protected]
[email protected]

In the cypress.config.ts add the reporter settings

cypress.config.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import { defineConfig } from 'cypress'
// https://github.com/bahmutov/cypress-split
// @ts-ignore
const cypressSplit = require('cypress-split')

export default defineConfig({
  // https://github.com/adamgruber/mochawesome
  reporter: 'mochawesome',
  reporterOptions: {
    reportDir: 'cypress/results',
    reportFilename: '[name].html',
    overwrite: true,
    html: true,
    json: false,
  },
  e2e: {
    baseUrl: 'http://localhost:3000',
    supportFile: false,
    setupNodeEvents(on, config) {
      cypressSplit(on, config)
      // IMPORTANT: return the config object
      return config
    },
  },
})

When you use cypress run command you should see messages after each spec file.

Mochawesome report generated message for spec1 file

All reports are saved in cypress/results folder. You can open them locally - these are just static files.

GitHub Actions workflow

Let's configure the workflow. I will use my cypress-workflows reusable workflows to run Cypress tests.

1
2
3
4
5
6
7
8
name: ci
on: [push]
jobs:
  single:
    # https://github.com/bahmutov/cypress-workflows
    uses: bahmutov/cypress-workflows/.github/workflows/[email protected]
    with:
      start: npm start

The workflow standard.yml has a parameter store-artifacts that is on by default:

1
2
3
4
5
store-artifacts:
  description: 'Store screenshots and videos from the cypress folder'
  type: boolean
  required: false
  default: true

If you look at the output of the workflow when it runs, you will see how it stores cypress/screenshots, cypress/videos, and cypress/results folders (if they exist).

The reusable workflow standard saves the test artifacts including the Mochawesome reports

Split GitHub Actions workflow

Let's speed the tests by running them in parallel. I have made split.yml reusable workflow and it includes the parameter store-artifacts=true by default. Thus our CI workflow file .github/workflows/ci.yml is a one-liner still:

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
name: ci
on: [push]
jobs:
  tests:
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v1
    with:
      n: 3              # run the tests on 3 CI machines
      start: npm start  # start the app before running the tests

This workflow adds an index to each saved artifact to avoid overwriting them. In the workflow summary, we see the failed viewport.cy.ts spec was included in the 3rd CI job.

The specs were split across 3 CI machines

The 3rd machine means its index is 2, and we should inspect the artifacts "split-2-..." to see the test results.

The test artifacts for the 3rd machine

Nice.

Future work

Currently we have generated a separate Mochawesome HTML report for each spec file. In the next installment, I will combine the separate reports into a single report with all tests.

See also