Pick Tests To Run Using The Pull Request Text

I have been looking closely at keeping the tests in a separate repo from the application. At Mercari US our frontend lives in its own repository, and triggers the Cypress tests in another repository using trigger-circleci-pipeline utility. We even report the test statuses back in the original repo using GitHub statuses via cypress-set-github-status plugin. But there is a disconnect between opening a feature pull request and picking the tests to run. For speed, we run any changed specs in the test repo branch with the matching name, and then run all tests tagged @sanity (read how we tag the tests in the blog post How To Tag And Run End-to-End Tests).

If the developer wants to run more tests, they can trigger the tests from the command line using the run-cy-on-ci utility. For convenience, we also allow you to start a new test run via GitHub Actions manual workflow where you can pick the test tags to run via a web form.

While the above approaches are ok, they are inconvenient. You have to trigger a separate test run, not tied to the original feature pull request. What if you could specify the tests to run when opening your pull request? This is what this blog post will show you.

The application

For this blog post, I will use the bahmutov/todomvc-no-tests-vercel repo. On every pull request, a preview environment is deployed. The tests live in their own repo bahmutov/todomvc-tests-circleci. The tests are tagged, and let's use the utility find-cypress-specs to learn about them.

$ npm i -D find-cypress-specs
+ [email protected]

package.json

{
  "scripts": {
    "tags": "find-cypress-specs --tags",
    "names": "find-cypress-specs --names"
  }
}

$ npm run names

> [email protected] names
> find-cypress-specs --names


cypress/integration/last-spec.js (1 test)
└─ last test [@sanity]

cypress/integration/log-spec.js (2 tests)
└─ Log [@log]
  ├─ logs message on startup [@sanity]
  └─ logs message when adding a todo

cypress/integration/second-spec.js (1 test)
└─ completes second item [@sanity, @user]

cypress/integration/spec.js (2 tests)
├─ works
└─ has no visit logic

cypress/integration/third-spec.js (1 test)
└─ third test

found 5 specs (7 tests)

Let's look at the tags applied to the tests

$ npm run tags

> [email protected] tags
> find-cypress-specs --tags


Tag      Tests
-------  -----
@log     2
@sanity  3
@user    1

These are the tags we want to the user to pick when they make a new pull request to change the TodoMVC application behavior. Plus of course, the user might want to run all the tests without any filtering.

Application pull request

Let's create a pull request template in the application repo. It should be named .github/PULL_REQUEST_TEMPLATE.md for GitHub to automatically use it when opening a new PR.

.github/PULL_REQUEST_TEMPLATE.md

# Summary

## Tests to run

Please pick all tests you would like to run against this pull request

- [ ] all tests
- [ ] tests tagged `@log`
- [ ] tests tagged `@sanity`
- [ ] tests tagged `@user`

When a pull request is opened, the application gets deployed, triggering the deployment_status event. If the deployment has been successful, we trigger the tests pipeline in the tests repo. See the deploy.yml workflow for details, but we pass the URL to test, and the current commit SHA as test pipeline parameters to test the right thing

export TEST_URL=${{ github.event.deployment_status.target_url }}

npx trigger-circleci-pipeline \
  --org bahmutov --project todomvc-tests-circleci --branch ${BRANCH_NAME} \
  --parameters TEST_URL=${TEST_URL},TEST_BRANCH=${BRANCH_NAME},TEST_COMMIT=${GITHUB_SHA}

Sometimes we know the pull request number right away. In my situation, that number of unavailable. At least the head commit SHA was known, thus I could pass it to the test pipeline via TEST_COMMIT=${GITHUB_SHA} parameter.

The tests

In the bahmutov/todomvc-tests-circleci CircleCI workflow we receive the TEST_ parameters and run Cypress tests against them. But we also install the grep-tests-from-pull-requests plugin and configure it.

$ npm i -D grep-tests-from-pull-requests
+ [email protected]

In the plugin file we put the grep-tests-from-pull-requests first, before the cypress-grep plugin.

cypress/plugins/index.js

module.exports = async (on, config) => {
  await require('grep-tests-from-pull-requests')(on, config, {
    // try to find checkbox lines in the pull request body with these tags
    tags: ['@log', '@sanity', '@user'],
    // repo with the pull request text to read
    owner: 'bahmutov',
    repo: 'todomvc-no-tests-vercel',
    // pass the pull request number in the above repo
    // we will grab the tests to run from the body of the pull request (if the number is known)
    pull: config.env.pullRequest,
    // if the pull request number is unknown, pass the commit SHA
    // as a fallback. The plugin will try to find the PR with this head commit
    commit: config.env.testCommit,
    // to get a private repo above, you might need a personal token
    token: process.env.PERSONAL_GH_TOKEN || process.env.GITHUB_TOKEN,
  })

  // https://github.com/bahmutov/cypress-grep
  require('cypress-grep/src/plugin')(config)

  return config
}

In my situation the pull request number is unknown, thus the plugin will try to list all pulls requests in the repo bahmutov/todomvc-no-tests-vercel and find the pull request with the head commit equal to the given SHA. See the full CircleCI workflow file and the cypress/plugins/index.js.

A pull request

Let's open a new pull request in the application repo.

When opening the pull request, I checked the @user line. The triggered CircleCI pipeline in the todomvc-tests-circleci shows the correct tests to run by tag. It has found the pull request number 11, read the Markdown text, and found the line with checked box for tag @user

Only a single spec file with a single test has executed. Now let's say we want to execute all tests. We check the box all tests and push an empty commit.

The option "all: true" overwrites all other options and all tests will execute

Trigger workflow by clicking on the checkbox

Tip: for "normal" GitHub workflows, you can re-trigger the workflow when clicking or clearing the checkboxes using the pull request event with custom activation types:

name: pr
on:
  # run tests when the user opened / reopened a PR
  # or pushed a new commit (synchronize type)
  # and also run the selected tests when the user possible
  # checked a box with the type of the tests to run
  # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request
  pull_request:
    types:
      - opened
      - synchronize
      - reopened
      - edited

Nice!

Since each checkbox change produces "edited" event and triggers the workflow, I suggest if you want to check and uncheck multiple boxes to edit the pull request text then click "Save" instead of clicking the individual checkboxes.

Bonus 1: status checks

We want to see the feedback on each application pull request right there in the application pull request. Using cypress-set-github-status plugin, you can report the Cypress test statuses back on the original commit. The plugin is set up very similarly to the grep-tests-from-pull-requests plugin. In the Cypress plugins file, point at the original repository and pass the commit and the token to set the status.

cypress/plugins/index.js

// when we are done, post the status to GitHub
// application repo, using the passed commit SHA
// https://github.com/bahmutov/cypress-set-github-status
require('cypress-set-github-status')(on, config, {
  owner: 'bahmutov',
  repo: 'todomvc-no-tests-vercel',
  commit: config.env.testCommit || process.env.TEST_COMMIT,
  token: process.env.GITHUB_TOKEN || process.env.PERSONAL_GH_TOKEN,
})

In the pull request I have used, the statuses are displayed using my GitHub profile, since I am using my personal GH token.

You can see the status checks for yourself in the PR #11.

Bonus 2: baseUrl

You can also specify the baseUrl to run the tests against using the pull request text, or even find it from one of the comments (like some automated deployment systems do). If the pull request has a line like baseUrl <...> or TestURL: <...> it will be automatically picked up and used:

// pull request text
some test tags

These tests should be run against this URL
baseUrl https://preview-1.acme.co

Bonus 3: extra variables

Finally, we have added extracting additional user variables to be passed via Cypress.env() from the text lines that start with CYPRESS_... prefix

CYPRESS_num=1
CYPRESS_correct=true
CYPRESS_FRIENDLY_GREETING=Hello

The above PR text will automatically be converted into Cypress.env() object

{
  "num": 1,
  "correct": true,
  "FRIENDLY_GREETING": "Hello"
}

See bahmutov/grep-tests-from-pull-requests for details.