products

Cypress GitHub Actions Slash Command

How to run Cypress tests by making a pull request comment.

Imagine you have a lot of Cypress tests. Which ones would you run when opening a GitHub pull request? Of course, if you can, you should run all of them. But what if there are too many specs to run, even in parallel using cypress-split? You would probably run tests filtered by the test tags using my plugin @bahmutov/cy-grep. But how would you specify which tests to run for a pull request? In my previous blog post I showed how to Pick Tests To Run Using The Pull Request Text. In this blog post I will show a more flexible approach that uses GitHub pull request comments to trigger test run.

🎁 You can find the example repository with the source code I am using as my example in the repo bahmutov/cypress-gha-slash-command-example.

The tests

In my example repo, I have several specs, which I can print using find-cypress-specs tool

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
$ npx find-cypress-specs --names
cypress/e2e/spec-b.cy.js (2 tests)
└─ Feature B [@featureB]
  ├─ works 1
  └─ works 2

cypress/e2e/spec-c.cy.js (2 tests)
└─ Feature C [@featureC]
  ├─ works 1 [@sanity]
  └─ works 2

cypress/e2e/spec-d.cy.js (2 tests)
└─ Misc
  ├─ works 1 [@sanity]
  └─ works 2

cypress/e2e/feature-a/spec-a1.cy.js (2 tests)
└─ Feature A [@featureA]
  ├─ works 1 [@sanity]
  └─ works 2

cypress/e2e/feature-a/spec-a2.cy.js (2 tests)
└─ Feature A [@featureA]
  ├─ works 3
  └─ works 4

found 5 specs (10 tests)

A typical spec looks like this:

cypress/e2e/feature-a/spec-a1.cy.js
1
2
3
4
5
6
7
8
9
describe('Feature A', { tags: '@featureA' }, () => {
  it('works 1', { tags: '@sanity' }, () => {
    cy.wait(15_000)
  })

  it('works 2', () => {
    cy.wait(15_000)
  })
})

Print tests and tags

Let's look at the test tags across all specs

1
2
3
4
5
6
7
$ npx find-cypress-specs --tags
Tag        Tests
---------  -----
@featureA  4
@featureB  2
@featureC  2
@sanity    3

Tip: I usually print the test names and tags using NPM script commands:

package.json
1
2
3
4
5
6
7
{
  "scripts": {
    "test": "cypress run",
    "names": "find-cypress-specs --names",
    "tags": "find-cypress-specs --tags"
  }
}

After configuring the @bahmutov/cy-grep plugin, we can execute all feature A specs like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
$ npx cypress run --env grepTags=@featureA

cy-grep: filtering using tag(s) "@featureA"
cy-grep: will omit filtered tests

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        12.10.0                                                                        │
  │ Browser:        Electron 106 (headless)                                                        │
  │ Node Version:   v16.17.0                                                                       │
  │ Specs:          2 found (spec-a1.cy.js, spec-a2.cy.js)                                         │
  │ Searched:       cypress/e2e/feature-a/spec-a1.cy.js, cypress/e2e/feature-a/spec-a2.cy.js       │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

...
  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  spec-a1.cy.js                            00:30        2        2        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  spec-a2.cy.js                            00:30        2        2        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        01:00        4        4        -        -        -

We can run all @sanity tests similarly

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
$ npx cypress run --env grepTags=@sanity
Couldn't find tsconfig.json. tsconfig-paths will be skipped
cy-grep: filtering using tag(s) "@sanity"
cy-grep: will omit filtered tests

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        12.10.0                                                                        │
  │ Browser:        Electron 106 (headless)                                                        │
  │ Node Version:   v16.17.0                                                                       │
  │ Specs:          3 found (spec-c.cy.js, spec-d.cy.js, feature-a/spec-a1.cy.js)                  │
  │ Searched:       cypress/e2e/spec-c.cy.js, cypress/e2e/spec-d.cy.js, cypress/e2e/feature-a/spec │
  │                 -a1.cy.js                                                                      │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

...
  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  spec-c.cy.js                             00:15        1        1        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  spec-d.cy.js                             00:15        1        1        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  feature-a/spec-a1.cy.js                  00:15        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:45        3        3        -        -        -

The GitHub Actions workflow

I will bring a GitHub Actions from the blog post Run And Trigger GitHub Workflow, but I will only trigger it manually or via repository dispatch events. There are two jobs: one to collect and merge the input parameters, and the second to actually run the tests using my reusable workflow from bahmutov/cypress-workflows.

.github/workflows/ci.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
name: ci

# default values to use
env:
  tags: '' # grep tags
  machines: 1 # split run across N machines

# run the workflow for different events
# - when we trigger the workflow manually
# - when we dispatch an event
on:
  workflow_dispatch:
    inputs:
      machines:
        description: Number of machines
        type: number
        default: 1
        required: false
      tags:
        description: Test tags to filter
        type: string
        default: ''
        required: false
  repository_dispatch:
    types: [on-ci]
jobs:
  # collect the variables and parameters into
  # tags and number of machines
  prepare:
    runs-on: ubuntu-20.04
    outputs:
      tags: ${{ steps.variables.outputs.tags }}
      machines: ${{ steps.variables.outputs.machines }}
    steps:
      - name: Merge variables
        id: variables
        run: |
          echo "tags=${{ github.event.inputs.tags || github.event.client_payload.tags || env.tags }}" >> "$GITHUB_OUTPUT"
          echo "machines=${{ github.event.inputs.machines || github.event.client_payload.machines || env.machines }}" >> "$GITHUB_OUTPUT"

      - name: Print the merged variables
        run: |
          echo "test tags ${{ steps.variables.outputs.tags }}"
          echo "number of machines ${{ steps.variables.outputs.machines }}"

  tests:
    needs: prepare
    # https://github.com/bahmutov/cypress-workflows
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      # tip: need to pass the number of machines as a number, and not as a string
      nE2E: ${{ fromJson(needs.prepare.outputs.machines) }}
      env: grepTags=${{ needs.prepare.outputs.tags }}

Trigger workflow from GitHub

Let's run the sanity tests. We can trigger the tests manually

Trigger the sanity tests

We see the entire workflow created automatically by the reusable workflow "split"

The finished workflow with several jobs

Let's inspect the merged variables. We entered the @sanity tag, and the default number of machines is 1

The merged input variables

The tag is passed to Cypress and only 3 tests are set to run

Only the sanity tests will execute

Perfect.

Trigger workflow using curl

Let's run all specs by splitting them across 3 machines. We can trigger the run by dispatching an event using curl.

1
2
3
4
5
6
7
curl -L \
  -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <my github token>"\
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/repos/bahmutov/cypress-gha-slash-command-example/dispatches \
  -d '{"event_type":"on-ci","client_payload":{"machines":3}}'

The workflow shows 3 containers running in parallel

The workflow runs across 3 machines

The plugin cypress-split outputs GitHub Actions summary for each machine. You can see how different specs ran across each machine.

Cypress-split summary for each machine

The slash command

Ok, now let's have some fun. I will use peter-evans/slash-command-dispatch to implement listening to the pull request comments and automatically running E2E tests. I will add two workflow files: one to listen to the new comments and dispatch command events, and another to actually run the tests.

.github/workflows/slash-command-dispatch.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
name: Slash Command Dispatch
on:
  issue_comment:
    types: [created]
jobs:
  slash_command_dispatch:
    runs-on: ubuntu-20.04
    steps:
      # https://github.com/peter-evans/slash-command-dispatch
      - name: Slash Command Dispatch
        uses: peter-evans/slash-command-dispatch@v3
        with:
          token: ${{ secrets.GH_PERSONAL_TOKEN }}
          reaction-token: ${{ secrets.GH_PERSONAL_TOKEN }}
          dispatch-type: workflow
          static-args: |
            repository=${{ github.repository }}
            comment-id=${{ github.event.comment.id }}
          # we only have a single "/cypress" command in this repo
          commands: |
            cypress

I picked the dispatch-type: workflow to make it simpler and look similar to the existing cy.yml workflow we have seen before. All we need are two more parameters to be able to "report" back on the comment. Since our command is cypress in the above list, I name the actual test workflow cypress-command.yml

.github/workflows/cypress-command.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
name: Cypress command

# default values to use
env:
  tags: '' # grep tags
  machines: 1 # split run across N machines

# run the workflow for different events
# - when we trigger the workflow manually
# - when we dispatch an event
on:
  workflow_dispatch:
    inputs:
      machines:
        description: Number of machines
        type: number
        default: 1
        required: false
      tags:
        description: Test tags to filter
        type: string
        default: ''
        required: false
      repository:
        description: 'The repository from which the slash command was dispatched'
        required: true
      comment-id:
        description: 'The comment-id of the slash command'
        required: true
jobs:
  # collect the variables and parameters into
  # tags and number of machines
  prepare:
    runs-on: ubuntu-20.04
    outputs:
      tags: ${{ steps.variables.outputs.tags }}
      machines: ${{ steps.variables.outputs.machines }}
    steps:
      - name: Merge variables
        id: variables
        run: |
          echo "tags=${{ github.event.inputs.tags || env.tags }}" >> "$GITHUB_OUTPUT"
          echo "machines=${{ github.event.inputs.machines || env.machines }}" >> "$GITHUB_OUTPUT"

      - name: Print the merged variables
        run: |
          echo "test tags ${{ steps.variables.outputs.tags }}"
          echo "number of machines ${{ steps.variables.outputs.machines }}"

  tests:
    needs: prepare
    # https://github.com/bahmutov/cypress-workflows
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      # tip: need to pass the number of machines as a number, and not as a string
      nE2E: ${{ fromJson(needs.prepare.outputs.machines) }}
      env: grepTags=${{ needs.prepare.outputs.tags }}

Run feature tests

Let's open a pull request PR 7. We think we have changed something related to the feature B. Let's run the tests for that feature using 1 machine. I will comment /cypress tags=@featureB

Add a comment to trigger the tests

Two workflows run in response to the comment. The dispatch looks at the /cypress part of the comment and triggers the cypress-command.yml workflow, passing the parameter tags=...

The triggered slash command workflows

The workflows even post a reaction emoji on my comment!

Run sanity tests across N machines

Maybe we want to run the sanity tests, we know there are 3 of them. Let's run them across 3 machines.

Running the sanity tests across 3 machines

The specs ran across 3 machines

The sanity tests ran across 3 machines

Avoid using @ tags in the comment

We are using test tags that start with the character @ - it is purely a convention. There is no special meaning to this character in the cy-grep plugin. But there is a special meaning for GitHub, which is why it highlights the tags in bold. Let's update our @bahmutov/cy-grep settings in cypress.config.js to automatically enforce @ at the start of the tag. This way we don't have to use @ character in our GitHub comment.

cypress.config.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
const { defineConfig } = require('cypress')
// https://github.com/bahmutov/cypress-split
const cypressSplit = require('cypress-split')

module.exports = defineConfig({
  e2e: {
    // baseUrl, etc
    fixturesFolder: false,
    video: false,
    env: { grepFilterSpecs: true, grepOmitFiltered: true, grepPrefixAt: true },
    setupNodeEvents(on, config) {
      // https://github.com/bahmutov/cy-grep
      require('@bahmutov/cy-grep/src/plugin')(config)
      // after filtering by tags, split the remaining specs
      cypressSplit(on, config)

      // IMPORTANT: return the config object
      return config
    },
  },
})

Now we can type tags without @ and they would work just as well. Let's run tests for features B and C across two machines.

Using tags without the prefix character @

The tags and the machines are passed to the dispatched workflow

Features B and C will be selected

The two specs were executed in parallel on two machines

Two specs and two machines

Comment back

We can post meaningful content back to update the original /cypress ... comment with results of the test run. Using peter-evans/create-or-update-comment lets simply post back the received parameters. See the last job comment in the workflow below

.github/workflows/cypress-command.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
name: Cypress command

# default values to use
env:
  tags: '' # grep tags
  machines: 1 # split run across N machines

# run the workflow for different events
# - when we trigger the workflow manually
# - when we dispatch an event
on:
  workflow_dispatch:
    inputs:
      machines:
        description: Number of machines
        type: number
        default: 1
        required: false
      tags:
        description: Test tags to filter
        type: string
        default: ''
        required: false
      # information about the comment that triggered this workflow
      repository:
        description: 'The repository from which the slash command was dispatched'
        required: true
      comment-id:
        description: 'The comment-id of the slash command'
        required: true
jobs:
  # collect the variables and parameters into
  # tags and number of machines
  prepare:
    runs-on: ubuntu-20.04
    outputs:
      tags: ${{ steps.variables.outputs.tags }}
      machines: ${{ steps.variables.outputs.machines }}
    steps:
      - name: Merge variables
        id: variables
        run: |
          echo "tags=${{ github.event.inputs.tags || env.tags }}" >> "$GITHUB_OUTPUT"
          echo "machines=${{ github.event.inputs.machines || env.machines }}" >> "$GITHUB_OUTPUT"

      - name: Print the merged variables
        run: |
          echo "test tags ${{ steps.variables.outputs.tags }}"
          echo "number of machines ${{ steps.variables.outputs.machines }}"

  tests:
    needs: prepare
    # https://github.com/bahmutov/cypress-workflows
    uses: bahmutov/cypress-workflows/.github/workflows/split.yml@v2
    with:
      # tip: need to pass the number of machines as a number, and not as a string
      nE2E: ${{ fromJson(needs.prepare.outputs.machines) }}
      env: grepTags=${{ needs.prepare.outputs.tags }}

  comment:
    needs: [tests, prepare]
    runs-on: ubuntu-20.04
    steps:
      - name: Post comment
        # https://github.com/peter-evans/create-or-update-comment
        uses: peter-evans/create-or-update-comment@v2
        with:
          token: ${{ secrets.GH_PERSONAL_TOKEN }}
          repository: ${{ github.event.inputs.repository }}
          comment-id: ${{ github.event.inputs.comment-id }}
          body: |
            Cypress tests called with:
            tags "${{ needs.prepare.outputs.tags }}"
            machines ${{ needs.prepare.outputs.machines }}

Updated comment

It would be cool if the split workflow produced the total number of passing / failed / skipped tests and a summary were posted into the comment.

Run workflow on the right branch

There is one important detail that we skipped. Let's say we created a new branch example-branch-1 and opened a pull request to merge our code into the main branch. We comment in the pull request (in this case it is pull request number 15). Let's say we want to run sanity tests for the feature A

Cypress slash command to run tests for this PR

The only information GitHub passes in the comment event to our dispatch code is the pull request number 15. Luckily we can determine the branch name by calling GitHub API, see my action bahmutov/get-branch-name-by-pr that I forked and updated from andrevalentin/get-branch-name-by-pr.

.github/workflows/slash-command-dispatch.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
name: Slash Command Dispatch
on:
  issue_comment:
    types: [created]
jobs:
  slash_command_dispatch:
    runs-on: ubuntu-20.04
    steps:
      - name: Print inputs 🖨️
        run: echo "${{ toJson(github.event.issue) }}"

      # we only know the pull request number, like 12, 20, etc
      # but to trigger the workflow we need the branch name
      # https://github.com/bahmutov/get-branch-name-by-pr
      - name: Find the PR branch name 🔎
        uses: bahmutov/get-branch-name-by-pr@v1
        id: pr
        with:
          repo-token: ${{ secrets.GH_PERSONAL_TOKEN }}
          pr-id: ${{ github.event.issue.number }}

      # https://github.com/peter-evans/slash-command-dispatch
      - name: Slash Command Dispatch
        uses: peter-evans/slash-command-dispatch@v3
        with:
          token: ${{ secrets.GH_PERSONAL_TOKEN }}
          reaction-token: ${{ secrets.GH_PERSONAL_TOKEN }}
          dispatch-type: workflow
          static-args: |
            ref=${{ steps.pr.outputs.branch }}
            repository=${{ github.repository }}
            comment-id=${{ github.event.comment.id }}
          # we only have a single "/cypress" command in this repo
          commands: |
            cypress

We see the correct branch name when calling the manual workflow via ref parameter.

Calling Cypress workflow on the right branch

Because we use the dispatch-type: workflow option, use the workflow runs on branch example-branch-1 and tests the latest commit for that branch

The tests run on the right branch

Super.

Update 1: Switch branch using repository_dispatch

If you want to use repository_dispatch event from the slash command, you can pass the name of the current branch. Here is the slash command workflow:

.github/workflows/slash-command-dispatch.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
name: Slash Command Dispatch
on:
  issue_comment:
    types: [created]
jobs:
  slash_command_dispatch:
    runs-on: ubuntu-20.04
    steps:
      - name: Print inputs 🖨️
        run: echo "${{ toJson(github.event.issue) }}"

      # note: the token SLASH_COMMAND_GH_TOKEN
      # is classic GH token because could not get
      # the workflows dispatched using the new ones

      # we only know the pull request number, like 12, 20, etc
      # but to trigger the workflow we need the branch name
      # https://github.com/bahmutov/get-branch-name-by-pr
      - name: Find the PR branch name 🔎
        uses: bahmutov/get-branch-name-by-pr@v1
        id: pr
        with:
          repo-token: ${{ secrets.SLASH_COMMAND_GH_TOKEN }}
          pr-id: ${{ github.event.issue.number }}

      # https://github.com/peter-evans/slash-command-dispatch
      - name: Slash Command Dispatch
        uses: peter-evans/slash-command-dispatch@v3
        with:
          token: ${{ secrets.SLASH_COMMAND_GH_TOKEN }}
          reaction-token: ${{ secrets.SLASH_COMMAND_GH_TOKEN }}
          static-args: |
            ref=${{ steps.pr.outputs.branch }}
            repository=${{ github.repository }}
            comment-id=${{ github.event.comment.id }}
          # we only have a single "/cypress" command in this repo
          commands: |
            cypress

Here is the workflow file that prints the slash inputs and switches to the right branch to show differences between the main and the current branches.

.github/workflows/cypress-command.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
name: Cypress command

on:
  repository_dispatch:
    types: ['cypress-command']

jobs:
  test-comment:
    runs-on: ubuntu-22.04
    steps:
      - name: Print event 🖨️
        run: echo "${{ toJson(github.event.client_payload.slash_command) }}"

      - name: Checkout repo 🛎️
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Print branches 🕊️
        run: git branch -a -v

      - name: Check out the reference branch ${{ github.event.client_payload.slash_command.args.named.ref }}
        run: git checkout ${{ github.event.client_payload.slash_command.args.named.ref }}

      - name: Print branches again 🕊️
        run: git branch -a -v

      - name: Changed files in this branch
        run: git diff --name-only --diff-filter=AMR origin/main

You can find the above workflows in the repo bahmutov/swag-store.