process

Using Test Ids To Pick Cypress Specs To Run

Pick Cypress specs based on test attributes in the changed source files.

Imagine you have 100s of Cypress specs with end-to-end tests. You have modified a React component source file "Hello.jsx". Which E2E tests should you run when you open a pull request? Ideally, you would run all of them to prevent accidentally breaking some tests. Running all specs might take a while, even if you parallelize Cypress specs for free. In this post I will show a better way. We will pick specs that use the the test IDs in the changed source file "Hello.jsx". We can run these specs first and then run all specs later.

🎁 You can find the example application shown in this blog post in the repo bahmutov/taste-the-sauce-test-ids. We will use plugin bahmutov/changed-test-ids to determine which specs to run.

The application

Our application is a typical React web app. We have pages and components, and we use different test IDs to select elements on the page during testing.

src/pages/CheckOutStepOne.jsx
1
2
3
4
5
6
7
8
9
10
11
12
13
...
<InputError
  isError={Boolean(error)}
  type={INPUT_TYPES.TEXT}
  value={lastName}
  onChange={handleLastNameChange}
  testId="lastName"
  placeholder="Last name"
  // Custom
  id="last-name"
  autoCorrect="off"
  autoCapitalize="none"
/>

The above CheckOutStepOne component has testId attribute "lastName". When the application runs, this becomes the HTML attribute data-test="lastName".

The specs

We use Cypress specs to go through the application. Right now we have only a few high level specs

1
2
3
4
5
6
7
cypress/
  e2e/
    checkout/
      checkout.cy.js
    login/
      login-form.cy.js
      logout.cy.js

In the specs, I used two Cypress custom commands to find elements by data-test attribute. These two commands are similar to cy.get and cy.contains commands. These commands select HTML elements on the page using the attribute or attribute plus element's text.

cypress/support/commands.ts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// find element(s) by data-test attribute
Cypress.Commands.add('getByTestId', (testId: string) => {
  const log = Cypress.log({ name: 'getByTestId', message: testId })
  // query the elements by the "data-test=..." attribute
  const selector = `[data-test="${testId}"]`
  cy.get(selector)
})

// find elements by data-test attribute with text inside
Cypress.Commands.add('containsTestId', (testId: string, text: string) => {
  const log = Cypress.log({ name: 'containsTestId', message: testId })
  // query the elements by the "data-test=..." attribute
  const selector = `[data-test="${testId}"]`
  cy.contains(selector, text)
})

Here is a typical checkout test using these custom commands cy.getByTestId and cy.containsTestId

cypress/e2e/checkout/checkout.cy.js
1
2
3
4
5
6
7
8
9
10
11
// part of the checkout spec
cy.visit('/checkout-step-one')
cy.get('input[type=submit]').click()
cy.containsTestId('error', 'Error: First Name is required').should(
  'be.visible',
)
cy.getByTestId('firstName').type('Joe')
cy.get('input[type=submit]').click()
cy.containsTestId('error', 'Error: Last Name is required').should(
  'be.visible',
)

One of the steps of the above Checkout test

Right now all tests are passing.

A small source file change

Let's say we open a pull request where we modify the CheckOutStepOne.jsx source file just a bit.

CheckOutStepOne.jsx
1
2
- placeholder="Last name"
+ placeholder="Last Name"

You can find this change in the pull request #3. Which specs should we run? Do we need to run the login-form.cy.js and logout.cy.js specs?

This is where the small utility changed-test-ids comes handy. I have installed it as a dev dependency

1
2
$ npm i -D changed-test-ids
+ [email protected]

We can use this utility to parse both JSX and Cypress specs to find data test attributes used in both files. For example, let's see all test IDs in the source files:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ npx find-ids --sources 'src/**/*.jsx'

back-to-products
cancel
checkout
continue
continue-shopping
error
finish
firstName
lastName
login-button
password
postalCode
product_sort_container
username

The changed-test-ids installs NPM bin script find-ids. The above command parses all src/**/*.jsx source files to find all testId="..." props and then outputs them.

What about our Cypress specs? We use custom commands cy.getByTestId and cy.containsTestId, so let's parse the specs looking for those commands and report all collected ids.

1
2
3
4
5
6
7
8
9
10
11
$ npx find-ids --specs 'cypress/e2e/**/*.cy.js' \
  --command getByTestId,containsTestId

cancel
error
firstName
lastName
login-button
password
postalCode
username

So our specs matching the glob pattern cypress/e2e/**/*.cy.js use 8 data test attributes. There are more test ids in the source files than in the specs, so some elements are not directly selected by our E2E specs. We can warn the user about this by using both --sources and --specs parameters and combining the above two commands:

1
2
3
4
5
6
7
8
9
10
11
$ npx find-ids --sources 'src/**/*.jsx' \
  --specs 'cypress/e2e/**/*.cy.js' \
  --command getByTestId,containsTestId

⚠️ found 6 test id(s) not covered by any specs
back-to-products
checkout
continue
continue-shopping
finish
product_sort_container

Pull request workflow

Cool, let's use this test ID information to pick Cypress specs to run based on the test IDs in the changed source files in the pull request. I am using GitHub Actions, and I can extract the test IDs from the sources changed between the current branch and the main branch.

.github/workflows/pr.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
name: pr
on: [pull_request]
jobs:
  targeted-specs:
    runs-on: ubuntu-22.04
    steps:
      - name: Checkout 🛎️
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      # https://github.com/cypress-io/github-action
      - name: Install 📦
        uses: cypress-io/github-action@v6
        with:
          runTests: false

      - name: Find Cypress specs for changed source files
        id: find-ids
        run: |
          npx find-ids --sources 'src/**/*.jsx' \
            --specs 'cypress/e2e/**/*.cy.js' --command getByTestId,containsTestId \
            --branch main --parent --set-gha-outputs

      - name: Run any detected Cypress specs
        if: ${{ steps.find-ids.outputs.specsToRunN }}
        uses: cypress-io/github-action@v6
        with:
          start: npm start
          spec: ${{ steps.find-ids.outputs.specsToRun }}

The step Find Cypress specs for changed source files is crucial.

1
2
3
4
5
6
- name: Find Cypress specs for changed source files
  id: find-ids
  run: |
    npx find-ids --sources 'src/**/*.jsx' \
      --specs 'cypress/e2e/**/*.cy.js' --command getByTestId,containsTestId \
      --branch main --parent --set-gha-outputs

It looks at the Git information, finds the changed source files, finds the test IDs used in those source files, and then picks only the specs that use these test IDs.

Detected specs based on test IDs in the changed source files

In the changed source file CheckOutStepOne.jsx there are multiple testId attributes: cancel, continue, firstName, lastName, etc. We have detected only one spec that covers at least some of them: checkout.cy.js. This is the spec we should run. By using --set-gha-outputs the output list of specs and the number of specs is saved in the GitHub Actions outputs values. We can then pass it to the cypress-io/github-action step to run those specs only:

1
2
3
4
5
6
- name: Run any detected Cypress specs
  if: ${{ steps.find-ids.outputs.specsToRunN }}
  uses: cypress-io/github-action@v6
  with:
    start: npm start
    spec: ${{ steps.find-ids.outputs.specsToRun }}

Cypress ran only the single checkout spec

Compared to running all specs, we saved some time, even on this tiny project :)

Running 1 specs is faster than running all 3 specs

Beautiful.

Update 1: tests in another repo

I have described how the same plugin changed-test-ids can be used to pass the detected test ids to pick tests to run, even if the tests reside in another repo. Read the blog post Pick Tests Using Test Ids From Another Source Repo.

Update 2: improve checking out the repo

To compute the changes between the current code and the default repository branch, we can only check out the current code and the default branch main

1
2
3
4
5
- name: Checkout the current merge commit 🛎️
  uses: actions/checkout@v4

- name: Pull the default main branch
  run: git fetch origin main:main

To compute the difference, remove the --parent parameter

1
2
3
4
5
- name: Find test ids used in the changed source files
  id: find-ids
  run: |
    npx find-ids --sources 'src/**/*.jsx' \
      --branch main --set-gha-outputs

If you want to see the verbose debug logs, add DEBUG: changed-test-ids environment variable tothe find-ids step

1
2
3
4
5
6
7
- name: Find test ids used in the changed source files
  id: find-ids
  env:
    DEBUG: changed-test-ids
  run: |
    npx find-ids --sources 'src/**/*.jsx' \
      --branch main --set-gha-outputs