products

Separate Application And Tests Repos GitHub Actions Setup

How to trigger tests in a separate repo using slash command and pass the results back.

In my previous blog post How to Keep Cypress Tests in Another Repo While Using GitHub Actions I have described how the end-to-end tests can live in a separate repository from the web application. Keeping a separate tests repo comes with its advantages and challenges. I like using a separate tests repo when we want to iterate over the tests really quickly. In this blog post I will describe how I set up two repos using GitHub Actions. To summarize: every time we open a web app pull request, we can enter a new comment with the text /cypress and it will trigger a test run in the tests repo. The results of the test run will be posted back to the comment.

The overall diagram

🎁 You can find the web application repo at bahmutov/todo-app, and the tests repo at bahmutov/todo-app-tests.

Todo-app: Cypress slash command

To trigger the test run, we will trust the developer who opened a pull request or is reviewing it to type a new comment /cypress. This is our "slash" command, and there is a reusable Github Action peter-evans/slash-command-dispatch that can handle it for us beautifully.

.github/workflows/slash-command-dispatch.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# a workflow that runs when a user enters a pull request comment
# if the user enters "/cypress" we trigger the workflow "slash-command-cypress"
name: Slash Command Dispatch

on:
  issue_comment:
    types: [created]

jobs:
  slash_command_dispatch:
    runs-on: ubuntu-latest
    steps:
      - name: Print event 🖨️
        run: |
          echo off
          echo '${{ toJson(github.event) }}'

      # we only know the pull request number, like 12, 20, etc
      # but to trigger the workflow we need the branch name
      # use personal token to be able to query the branch by PR number
      # https://github.com/bahmutov/get-branch-name-by-pr
      - name: Find the PR branch name 🔎
        uses: bahmutov/get-branch-name-by-pr@v1
        id: pr
        with:
          repo-token: ${{ secrets.PERSONAL_TOKEN }}
          pr-id: ${{ github.event.issue.number }}

      - name: Slash Command Dispatch
        # https://github.com/peter-evans/slash-command-dispatch
        uses: peter-evans/slash-command-dispatch@v3
        with:
          # use personal token to be able to trigger more workflows
          token: ${{ secrets.PERSONAL_TOKEN }}
          # the personal token to post the comment emoji
          reaction-token: ${{ secrets.PERSONAL_TOKEN }}
          permission: none
          static-args: |
            repository=${{ github.repository }}
            comment-id=${{ github.event.comment.id }}
            ref=${{ steps.pr.outputs.branch }}
          commands: |
            cypress

A couple of notes about the above workflow that you can find at slash-command-dispatch.yml:

It runs on every new issue and pull request comment

1
2
3
on:
  issue_comment:
    types: [created]

It grabs the branch name using my GitHub Action bahmutov/get-branch-name-by-pr because we will need to check out the code when running the test. We need to use a GitHub personal access token set as a repo secret.

1
2
3
4
5
6
7
8
9
10
# we only know the pull request number, like 12, 20, etc
# but to trigger the workflow we need the branch name
# use personal token to be able to query the branch by PR number
# https://github.com/bahmutov/get-branch-name-by-pr
- name: Find the PR branch name 🔎
  uses: bahmutov/get-branch-name-by-pr@v1
  id: pr
  with:
    repo-token: ${{ secrets.PERSONAL_TOKEN }}
    pr-id: ${{ github.event.issue.number }}

Similarly, we use the personal token to trigger the specific workflow for the "cypress" command

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
- name: Slash Command Dispatch
  # https://github.com/peter-evans/slash-command-dispatch
  uses: peter-evans/slash-command-dispatch@v3
  with:
    # use personal token to be able to trigger more workflows
    token: ${{ secrets.PERSONAL_TOKEN }}
    # the personal token to post the comment emoji
    reaction-token: ${{ secrets.PERSONAL_TOKEN }}
    permission: none
    static-args: |
      repository=${{ github.repository }}
      comment-id=${{ github.event.comment.id }}
      ref=${{ steps.pr.outputs.branch }}
    commands: |
      cypress

You can have multiple commands dispatched by this workflow, and they can even parse arguments. For example, I could trigger Cypress tests run against a new pull request, or measure its performance using /lighthouse command, following the blog post Trying Lighthouse.

1
2
3
commands: |
  cypress
  lighthouse

Ok, so let's trigger the Cypress command workflow. The dispatch workflow adds the comment id and triggers the actual slash-command-cypress.yml workflow

The dispatch workflow execution

If the workflow is found, the dispatch immediately adds two emoji reactions to the comment.

This reactions mean the command workflow was found and started

Todo-app: Cypress workflow

We are still in the "todo-app" repository. We triggered the slash-command-cypress.yml workflow. You can find the current YAML code in slash-command-cypress.yml and below:

.github/workflows/slash-command-cypress.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# this workflow runs when the user comments "/cypress" on a pull request
name: Slash Command (Cypress)

on:
  repository_dispatch:
    types: [cypress-command]

permissions:
  contents: read
  pull-requests: write

jobs:
  launc-tests:
    name: Launch API tests
    runs-on: ubuntu-latest
    steps:
      - name: Print event 🖨️
        run: |
          echo off
          echo '${{ toJson(github.event.client_payload.slash_command) }}'

      - name: Check out the repo 🛎️
        # https://github.com/actions/checkout
        uses: actions/checkout@v4
        with:
          # only check out one JS file
          # that we use to trigger workflow in the tests repo
          sparse-checkout: |
            .github/workflows/trigger-cypress.js
          sparse-checkout-cone-mode: false

      - name: Trigger Cypress run
        # https://github.com/actions/github-script
        uses: actions/github-script@v6
        id: trigger
        # pass this repo's name and PR number
        # plus the comment ID so the tests workflow can post results back
        env:
          REPO_NAME: ${{ github.event.client_payload.slash_command.args.named.repository }}
          PR_NUMBER: ${{ github.event.client_payload.pull_request.number }}
          REF: ${{ github.event.client_payload.slash_command.args.named.ref }}
          FEEDBACK_COMMENT_ID: ${{ github.event.client_payload.github.payload.comment.id }}
        with:
          # unfortunately we need to use a personal token to trigger
          # a workflow in another repo, cannot use just GITHUB_TOKEN
          github-token: ${{ secrets.PERSONAL_TOKEN }}
          script: |
            const result = await require('./.github/workflows/trigger-cypress.js')({ github, core });
            return result;

Let me explain what the workflow is doing step by step.

The workflow runs when triggered by the GitHub API with repository_dispatch event and type=cypress-command

1
2
3
on:
  repository_dispatch:
    types: [cypress-command]

This workflow needs to trigger a new run in another repo "bahmutov/todo-app-tests". A simple way to call GitHub API is to use the GitHub's own action actions/github-script. Since I put some logic into a Node.js script, which I will show in a second, I need to check out just the file .github/workflows/trigger-cypress.js. This is the sparse checkout step:

1
2
3
4
5
6
7
8
9
- name: Check out the repo 🛎️
  # https://github.com/actions/checkout
  uses: actions/checkout@v4
  with:
    # only check out one JS file
    # that we use to trigger workflow in the tests repo
    sparse-checkout: |
      .github/workflows/trigger-cypress.js
    sparse-checkout-cone-mode: false

Now let's see what the dispatch workflow sent us in the payload. I am printing the object first

1
2
3
4
- name: Print event 🖨️
  run: |
    echo off
    echo '${{ toJson(github.event.client_payload.slash_command) }}'

For our run, it shows:

Cypress todo-app workflow was given parsed arguments by the dispatch workflow

Great, so if we want to grab individual arguments, like the name of the branch, we can use expression to get the nested property ${{ github.event.client_payload.slash_command.args.named.ref }}. We now call the script trigger-cypress.js and pass the individual values as environment variables:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
- name: Trigger Cypress run
  # https://github.com/actions/github-script
  uses: actions/github-script@v6
  id: trigger
  # pass this repo's name and PR number
  # plus the comment ID so the tests workflow can post results back
  env:
    REPO_NAME: ${{ github.event.client_payload.slash_command.args.named.repository }}
    PR_NUMBER: ${{ github.event.client_payload.pull_request.number }}
    REF: ${{ github.event.client_payload.slash_command.args.named.ref }}
    FEEDBACK_COMMENT_ID: ${{ github.event.client_payload.github.payload.comment.id }}
  with:
    # unfortunately we need to use a personal token to trigger
    # a workflow in another repo, cannot use just GITHUB_TOKEN
    github-token: ${{ secrets.PERSONAL_TOKEN }}
    script: |
      const result = await require('./.github/workflows/trigger-cypress.js')({ github, core });
      return result;

Inside the script we can use GitHub wrapper objects that call the API methods using the personal token we passed.

.github/workflows/trigger-cypress.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
module.exports = ({ github, core }) => {
  const payload = {
    // the name of this repo
    repo: process.env.REPO_NAME,
    pullRequestNumber: process.env.PR_NUMBER,
    ref: process.env.REF,
    feedbackCommentId: process.env.FEEDBACK_COMMENT_ID,
  }

  core.info(JSON.stringify(payload))

  return new Promise((resolve, reject) => {
    // https://octokit.github.io/rest.js/v19#repos-create-dispatch-event
    github.rest.repos
      .createDispatchEvent({
        owner: 'bahmutov',
        repo: 'todo-app-tests',
        event_type: 'trigger-tests',
        client_payload: payload,
      })
      .then(() => {
        resolve({
          status: 'success',
          message: '✅ Success',
          payload,
        })
      })
      .catch((err) => {
        core.error(`Failed to dispatch event:${err}`)
        // In order to send error message in the following steps, do not raise an error
        resolve({
          status: 'fail',
          message: `❌ Error: ${err.message || 'Unknown error'}`,
          payload,
        })
      })
  })
}

The action shows the parameters being passed to the repo bahmutov/todo-app-tests correctly

Trigger the workflow run in the tests repo

Now we can shift our attention to the repo bahmutov/todo-app-tests that runs the E2E tests.

Todo-app-tests: Trigger workflow

Any external caller can trigger the tests run in our todo-app-tests repo by calling GitHub API and sending the repository_dispatch event with type trigger-tests. For details, see Run And Trigger GitHub Workflow blog post. Here is the full workflow .github/workflows/trigger.yml that receives this event.

.github/workflows/trigger.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
name: trigger
# run this workflow on trigger or manually
on:
  # trigger this workflow by calling GitHub API
  repository_dispatch:
    types: [trigger-tests]

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - name: Print variables 🖨️
        run: |
          echo '### Workflow info 🖨️' >> $GITHUB_STEP_SUMMARY
          echo 'PR number ${{ github.event.client_payload.pullRequestNumber }}' >> $GITHUB_STEP_SUMMARY
          echo 'original repo ${{ github.event.client_payload.repo }}' >> $GITHUB_STEP_SUMMARY
          echo 'original repo reference ${{ github.event.client_payload.ref }}' >> $GITHUB_STEP_SUMMARY

      # quickly post the workflow URL back in the original repo PR
      - name: Post workflow URL 🔗
        if: ${{ github.event.client_payload.repo && github.event.client_payload.feedbackCommentId }}
        # https://github.com/peter-evans/create-or-update-comment
        uses: peter-evans/create-or-update-comment@v3
        with:
          # need a personal token to be able to post a comment back
          # in the original repo
          token: ${{ secrets.PERSONAL_TOKEN }}
          comment-id: ${{ github.event.client_payload.feedbackCommentId }}
          issue-number: ${{ github.event.client_payload.pullRequestNumber }}
          repository: ${{ github.event.client_payload.repo }}
          body: |
            Tests workflow at ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

      # check out both the app and the tests
      - name: Checkout this repo 🛎
        uses: actions/checkout@v4

      - name: Checkout the application repo 🛎
        uses: actions/checkout@v4
        with:
          repository: ${{ github.event.client_payload.repo }}
          ref: ${{ github.event.client_payload.ref }}
          path: app

      - name: Install app dependencies 📦
        uses: bahmutov/npm-install@v1
        with:
          working-directory: app

      - name: Start the application 🎬
        run: |
          cd app
          npm run start &

      - name: Run E2E tests 🏃🏻‍♂️
        id: tests
        uses: cypress-io/github-action@v6

      - name: Post results 📨
        if: ${{ always() && github.event.client_payload.repo && github.event.client_payload.feedbackCommentId }}
        uses: peter-evans/create-or-update-comment@v3
        with:
          # need a personal token to be able to post a comment back
          # in the original repo
          token: ${{ secrets.PERSONAL_TOKEN }}
          comment-id: ${{ github.event.client_payload.feedbackCommentId }}
          issue-number: ${{ github.event.client_payload.pullRequestNumber }}
          repository: ${{ github.event.client_payload.repo }}
          body: |
            Tests result: ${{ steps.tests.outcome }}
          reactions: |
            ${{ steps.tests.outcome == 'success' && 'hooray' || '-1' }}

When we triggered the workflow using our /cypress comment, it finished successfully.

The tests workflow has finished

There are several things this workflow does to make it developer-friendly in two repos.

It posts the main information with the parameters it has received:

1
2
3
4
5
6
- name: Print variables 🖨️
  run: |
    echo '### Workflow info 🖨️' >> $GITHUB_STEP_SUMMARY
    echo 'PR number ${{ github.event.client_payload.pullRequestNumber }}' >> $GITHUB_STEP_SUMMARY
    echo 'original repo ${{ github.event.client_payload.repo }}' >> $GITHUB_STEP_SUMMARY
    echo 'original repo reference ${{ github.event.client_payload.ref }}' >> $GITHUB_STEP_SUMMARY

This is what you see right away in the workflow summary

Client payload params in the job summary

Then it forms the browser URL to the workflow run that you see in the browser screenshot and appends it back to the original comment using the repo name and the comment id. It uses GitHub Action peter-evans/create-or-update-comment by the same person Peter Evans that wrote the slash comment dispatch action.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# quickly post the workflow URL back in the original repo PR
- name: Post workflow URL 🔗
  if: ${{ github.event.client_payload.repo && github.event.client_payload.feedbackCommentId }}
  # https://github.com/peter-evans/create-or-update-comment
  uses: peter-evans/create-or-update-comment@v3
  with:
    # need a personal token to be able to post a comment back
    # in the original repo
    token: ${{ secrets.PERSONAL_TOKEN }}
    comment-id: ${{ github.event.client_payload.feedbackCommentId }}
    issue-number: ${{ github.event.client_payload.pullRequestNumber }}
    repository: ${{ github.event.client_payload.repo }}
    body: |
      Tests workflow at ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

If you are looking at the pull request in the "todo-app" repo, you will see the tests workflow that you can click.

The tests workflow URL is appended to the original slash comment

Then we need to check out the tests, the application code, and start the application. We will check out the application code using the branch reference name passed to us using ref: ${{ github.event.client_payload.ref }} parameter.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# check out both the app and the tests
- name: Checkout this repo 🛎
  uses: actions/checkout@v4

- name: Checkout the application repo 🛎
  uses: actions/checkout@v4
  with:
    repository: ${{ github.event.client_payload.repo }}
    ref: ${{ github.event.client_payload.ref }}
    path: app

- name: Install app dependencies 📦
  uses: bahmutov/npm-install@v1
  with:
    working-directory: app

- name: Start the application 🎬
  run: |
    cd app
    npm run start &

Now that the application is running in the background, let's run Cypress tests

1
2
3
- name: Run E2E tests 🏃🏻‍♂️
  id: tests
  uses: cypress-io/github-action@v6

I am assigning this step an id tests so that later we can refer to the outcome of the step. We want to post the result back in the original comment in the "todo-app" repository.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
- name: Post results 📨
  if: ${{ always() && github.event.client_payload.repo && github.event.client_payload.feedbackCommentId }}
  uses: peter-evans/create-or-update-comment@v3
  with:
    # need a personal token to be able to post a comment back
    # in the original repo
    token: ${{ secrets.PERSONAL_TOKEN }}
    comment-id: ${{ github.event.client_payload.feedbackCommentId }}
    issue-number: ${{ github.event.client_payload.pullRequestNumber }}
    repository: ${{ github.event.client_payload.repo }}
    body: |
      Tests result: ${{ steps.tests.outcome }}
    reactions: |
      ${{ steps.tests.outcome == 'success' && 'hooray' || '-1' }}

The steps.tests.outcome can be "success", "failure", "cancelled", or "skipped". We are only interested in the "success" vs the others. In our case, the step succeeded, and thus we see in the comment the final status

The tests outcome posted as text and reaction

Nice, this pull request is passing its tests.

In the next blog post I will describe how you can post a commit status back in the original repo. This way you can protected the branch and require the tests to pass before merging pull requests. Read the blog post Set Commit Status In Another Repo.