products

Repeating Tests

How to run the same test or spec files again and again by launching it from GitHub Actions page.

Imagine you are doubting if a test works consistently. You could run it using cypress-grep locally. But what if someone working on the backend API comes in and asks if they could launch the tests too? You could give them the command line utility run-cy-on-ci to launch the tests from the terminal. Or you could let them launch the tests from the GitHub Actions page and it is much much simpler procedure.

🎁 You can find these examples in the repo bahmutov/test-todomvc-using-app-actions.

Run a test N times

Using the plugin cypress-grep you can select the test to run using a part of its title and skip all other tests and specs. In our GitHub Actions workflow repeat-test.yml we can pass the grep and burn parameters connecting the user interface to the test run.

.github/workflows/repeat-test.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
name: repeat-test
on:
  workflow_dispatch:
    description: Lets the user run a test by title N times
    inputs:
      title:
        description: Full or part of the test title
        required: true
        type: string
      n:
        description: Number of times to repeat the test
        required: false
        default: 1
        type: integer

jobs:
  repeat-test:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v2

      # https://github.com/cypress-io/github-action
      - name: Run the test N times
        uses: cypress-io/github-action@v2
        with:
          start: npm run start-quiet
          env: 'grep=${{ github.event.inputs.title }},burn=${{ github.event.inputs.n }}'

All the user needs to do is to enter a part of the test's title and pick the number of times to run the test.

Start the workflow to run a test five times in a row

The same test selected by its title is repeated five times in a row

The same test has been repeated fives times

Run all tests tagged X N times

In my repo, I organize the tests using tags, see How To Tag And Run End-to-End Tests. If I want to run all tests tagged @editing, I need a different workflow repeat-tag.yml to pass the tag value using grepTag variable to the cypress-grep plugin.

.github/workflows/repeat-tag.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
name: repeat-tag
on:
  workflow_dispatch:
    description: Lets the user run all the tests with specific tag N times
    inputs:
      tag:
        description: Tag to run
        required: true
        type: choice
        options:
          - '@adding'
          - '@complete'
          - '@editing'
          - '@item'
          - '@persistence'
          - '@regression'
          - '@routing'
          - '@sanity'
      n:
        description: Number of times to repeat the tests
        required: false
        default: 1
        type: integer

jobs:
  repeat-tag:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v2

      # https://github.com/cypress-io/github-action
      - name: Run the tagged tests N times
        uses: cypress-io/github-action@v2
        with:
          start: npm run start-quiet
          env: 'grepTags=${{ github.event.inputs.tag }},burn=${{ github.event.inputs.n }}'

Tip: use find-cypress-specs utility to print all available test tags.

1
2
3
4
5
6
7
8
9
10
11
12
$ npx find-cypress-specs --tags

Tag           Tests
------------  -----
@adding       6
@complete     6
@editing      5
@item         3
@persistence  1
@regression   17
@routing      5
@sanity       8

Let's launch the @editing tests and run them 3 times each.

Launching workflow to run all tests tagged @editing three times in a row

Each tagged test ran three times

Run a test file N times

Finally, let's imagine we want to run a particular test file (spec) several times. We can install the plugin cypress-repeat that uses Cypress NPM module API to run the test runner N times. For example, we can run the spec cypress/integration/item-spec.js twice using the terminal command

1
$ npx cypress-repeat run --spec cypress/integration/item-spec.js -n 2 ...

In the workflow repeat-spec.yml we will use cypress-repeat as a custom command option:

.github/workflows/repeat-spec.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
name: repeat-spec
on:
  workflow_dispatch:
    description: Lets the user run a spec file N times in a row
    inputs:
      spec:
        description: Part or full path to the spec file like "editing-spec.js"
        required: true
        type: string
      n:
        description: Number of times to repeat the test
        required: false
        default: 1
        type: integer

jobs:
  repeat-spec:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout 🛎
        uses: actions/checkout@v2

      # https://github.com/cypress-io/github-action
      - name: Run the spec N times
        uses: cypress-io/github-action@v2
        with:
          start: npm run start-quiet
          # use https://github.com/bahmutov/cypress-repeat to repeat the spec
          command: npx cypress-repeat -n ${{ github.event.inputs.n }} --spec '**/${{ github.event.inputs.spec }}'

Note that we add the wildcard to the spec filename to let the user simple specify the file name without the cypress/integration/ prefix.

Launching workflow to run the item-spec.js test file

In the action output, you should see the cypress-repeat messages like

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
cypress-repeat: will repeat Cypress command 2 time(s)
***** cypress-repeat: 1 of 2 *****
====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        9.3.1                                                                          │
  │ Browser:        Electron 94 (headless)                                                         │
  │ Node Version:   v16.13.2 (/usr/local/bin/node)                                                 │
  │ Specs:          1 found (item-spec.js)                                                         │
  │ Searched:       **/item-spec.js                                                                │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
...

***** cypress-repeat: 2 of 2 *****
====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        9.3.1                                                                          │
  │ Browser:        Electron 94 (headless)                                                         │
  │ Node Version:   v16.13.2 (/usr/local/bin/node)                                                 │
  │ Specs:          1 found (item-spec.js)                                                         │
  │ Searched:       **/item-spec.js                                                                │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
...
***** finished 2 run(s) successfully *****

Tip: you can run multiple spec files by giving a common parent folder name. For example, to run all integration specs in the folder cypress/integration enter the "integration" as the spec name. It will invoke Cypress with parameter --spec **/integration which will match all specs.

See more