process

Pick Tests Using Test Ids From Another Source Repo

Select tests in another repo based on the source file changes.

This blog post continues the work shown in the post Using Test Ids To Pick Cypress Specs To Run. In the current example, the source and the tests live in two separate repositories. Both repos are using GitHub Actions to build and test the code. We will be using plugin changed-test-ids to pick the test ids from the source code changes, then pick the tests to run based on those ids.

🎁 You can find the example application in the repo bahmutov/swag-store and the test in the repo bahmutov/swag-store-tests. I am using v1 of the plugin changed-test-ids.

The application

When we open pull requests in the source repo, we want to build the application, maybe run some linting tools, and then trigger the end-to-end tests. Here is my example workflow, for simplicity I am not passing the build artifacts to the test workflow, just the current Git reference SHA.

.github/workflows/pr.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
name: pr
on: [pull_request]
jobs:
  targeted-specs:
    runs-on: ubuntu-22.04
    steps:
      - name: Checkout 🛎️
        uses: actions/checkout@v4
        with:
          # check out all branches so we can get
          # the list of changed source files
          fetch-depth: 0

      - name: Install dependencies
        uses: bahmutov/npm-install@v1

      - name: Find test ids used in the changed source files
        id: find-ids
        run: |
          npx find-ids --sources 'src/**/*.jsx' \
            --branch main --parent --set-gha-outputs

      - name: Trigger E2E tests based on test IDs
        # the tests reside in the repo bahmutov/swag-store-tests
        run: |
          curl -L \
            -X POST \
            -H "Accept: application/vnd.github+json" \
            -H "Authorization: Bearer ${{ secrets.TRIGGER_TESTS_TOKEN }}"\
            -H "X-GitHub-Api-Version: 2022-11-28" \
            https://api.github.com/repos/bahmutov/swag-store-tests/dispatches \
            -d '{"event_type":"specs-by-test-ids","client_payload":{"ref":"${{ github.event.ref }}","testIds":"${{ steps.find-ids.outputs.changedTestIds }}"}}'

The step find-ids looks at the changed source files src/**/*.js and finds all testId=... JSX attributes. The npx find-ids command comes from changed-test-ids NPM package alias.

1
2
3
4
5
- name: Find test ids used in the changed source files
  id: find-ids
  run: |
    npx find-ids --sources 'src/**/*.jsx' \
      --branch main --parent --set-gha-outputs

These test ids are set as the outputs of the step and can be used in the next step to trigger workflow run using curl command.

1
2
3
-d '{"event_type":"specs-by-test-ids","client_payload":
  {"ref":"${{ github.event.ref }}",
  "testIds":"${{ steps.find-ids.outputs.changedTestIds }}"}}'

The changed-test-ids even writes its summary, so you can see it in the GitHub Actions UI. For example, let's look at the pull request #1. The changed source file only touched a last name component.

The source file changed an attribute

Only this JSX file changed in this branch versus its parent branch main, and the find-ids tool detected the following testId=... values: cancel,continue,firstName,lastName,postalCode

The detected test id attributes in the changed source files

These test ids are passed in the trigger CI workflow event in the repo bahmutov/swag-store-tests

Ids are included in the CI trigger API call to the tests repo

Finally, the user sees the detected ids in the action summary panel.

The found test ids are written to the action summary

The tests repo

Let's run the right specs based on the test ids in the source files. In the repo bahmutov/swag-store-tests I use the same workflow to run the tests on push event, when manually starting the workflow (and we allow passing a list of test ids to us via UI), plus when triggering the workflow using an API call.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
name: ci
on:
  push:
  # trigger this workflow from GitHub UI
  workflow_dispatch:
    inputs:
      testIds:
        description: |
          Comma-separated list of data test ids in the source files
        required: false
        type: string
  # trigger this workflow by calling GitHub API
  repository_dispatch:
    types: [specs-by-test-ids]

jobs:
  e2e:
    runs-on: ubuntu-22.04
    steps:
      - name: Checkout 🛎️
        # https://github.com/actions/checkout
        uses: actions/checkout@v4

      - name: Check out the app 🛎️
        uses: actions/checkout@v4
        with:
          repository: bahmutov/swag-store
          path: swag-store
          # use the given Git reference to check out the app source
          ref: ${{ github.event.client_payload.ref }}
          token: ${{ secrets.SWAG_REPO_GH_TOKEN }}

      # inside the cloned repo with tests
      # the application is in its own "swag-store" subfolder

      # https://github.com/cypress-io/github-action
      - name: Install dependencies in this repo 📦
        uses: cypress-io/github-action@v6
        with:
          runTests: false

      - name: Install app dependencies 📦
        # https://github.com/bahmutov/npm-install
        uses: bahmutov/npm-install@v1
        with:
          working-directory: swag-store

      - name: Start the app
        run: npm start &
        working-directory: swag-store

      - name: Find specs for test ids
        # if the workflow is triggered via repository_dispatch
        # then the list of test ids will be in the "client_payload" context object
        # if the workflow is started via workflow_dispatch
        # then the list of test ids will be in the "inputs" object
        if: ${{ github.event.client_payload.testIds || github.event.inputs.testIds }}
        id: find-specs
        run: |
          npx find-ids --test-ids ${{ github.event.client_payload.testIds || github.event.inputs.testIds }} \
            --specs 'cypress/e2e/**/*.cy.js' --command getByTestId,containsTestId \
            --set-gha-outputs

      - name: Run found E2E tests
        if: ${{ steps.find-specs.outputs.specsToRunN }}
        timeout-minutes: 3
        uses: cypress-io/github-action@v6
        with:
          install: false
          wait-on: 'http://localhost:3000'
          record: true
          spec: '${{ steps.find-specs.outputs.specsToRun }}'
          tag: '${{ github.event.client_payload.testIds || github.event.inputs.testIds }}'
        env:
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

      - name: Run all E2E tests
        if: ${{ !steps.find-specs.outputs.specsToRunN }}
        timeout-minutes: 3
        uses: cypress-io/github-action@v6
        with:
          install: false
          wait-on: 'http://localhost:3000'
          record: true
        env:
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

I check out the application source code, install dependencies, and install the test repo's own dependencies. Now we can find the specs to run, depending on the test ids, if any.

1
2
3
4
5
6
7
8
9
10
11
- name: Find specs for test ids
  # if the workflow is triggered via repository_dispatch
  # then the list of test ids will be in the "client_payload" context object
  # if the workflow is started via workflow_dispatch
  # then the list of test ids will be in the "inputs" object
  if: ${{ github.event.client_payload.testIds || github.event.inputs.testIds }}
  id: find-specs
  run: |
    npx find-ids --test-ids ${{ github.event.client_payload.testIds || github.event.inputs.testIds }} \
      --specs 'cypress/e2e/**/*.cy.js' --command getByTestId,containsTestId \
      --set-gha-outputs

In the workflow triggered from the swag-store pull request we looked above, it finds the following specs and sets the list as the output:

Only one E2E Cypress spec uses any of the given test ids

If we find any specific specs, we run only those specs using the spec: ... argument

1
2
3
4
5
6
7
8
9
10
11
12
- name: Run found E2E tests
  if: ${{ steps.find-specs.outputs.specsToRunN }}
  timeout-minutes: 3
  uses: cypress-io/github-action@v6
  with:
    install: false
    wait-on: 'http://localhost:3000'
    record: true
    spec: '${{ steps.find-specs.outputs.specsToRun }}'
    tag: '${{ github.event.client_payload.testIds || github.event.inputs.testIds }}'
  env:
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

A single Cypress spec checkout.cy.js was executed

The test finishes and we can see the output of the changed-test-ids command npx find-ids --test-ids ... --specs ... --set-gha-outputs

changed-test-ids find specs command summary

I also pass the test ids as the Cypress Cloud run tag list, which makes finding the run in the list easy:

Test ids from changed source files used as Cloud run tags

What if there are no detected test ids, or we simply want to run the tests? No problem, the "if - else" testing step runs all the specs if there are no steps.find-specs.outputs.specsToRunN.

1
2
3
4
5
6
7
8
9
10
- name: Run all E2E tests
  if: ${{ !steps.find-specs.outputs.specsToRunN }}
  timeout-minutes: 3
  uses: cypress-io/github-action@v6
  with:
    install: false
    wait-on: 'http://localhost:3000'
    record: true
  env:
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

Picking specs to run by the test ids used in the changed source files might be useful when you have a lot of tests to pick from.