products

How to Keep Cypress Tests in Another Repo While Using GitHub Actions

How to keep the tests in sync with the features, while keeping them in separate repositories.

Imagine you want to start writing end-to-end tests for your web application, but it is hard to convince everyone on the team to include Cypress in the repository. Maybe you want to show the tests in action first. Maybe you want to solve technical blockers. You have decided to keep the E2E tests in a separate repository, at least at first. How would it work?

🎁 You can find the example application in the repository bahmutov/todomvc-no-tests and its end-to-end tests in the repository bahmutov/todomvc-tests.

The app

Let's say we are writing an app and every pull request is automatically deployed to a preview environment. In this blog post, my application from repo bahmutov/todomvc-no-tests is deployed to Netlify https://todomvc-no-tests.netlify.app/ after build. Every pull request is also deployed to its own preview environment.

The deployed Todo application

The tests

The tests live in another repo bahmutov/todomvc-tests and by default assume the application is running locally at port 3000.

cypress.json
1
2
3
4
5
6
7
{
  "baseUrl": "http://localhost:3000",
  "fixturesFolder": false,
  "supportFile": false,
  "pluginsFile": false,
  "video": false
}

We can start with a single sanity test that goes through the main features of the application.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
/// <reference types="cypress" />

it('works', () => {
  cy.visit('/')
  // application starts with 3 todos
  cy.get('.todo').should('have.length', 3)
  cy.get('input[type=text]').type('Add tests!{enter}')
  cy.get('.todo')
    .should('have.length', 4)
    .eq(3)
    .should('include.text', 'Add tests!')

  cy.contains('.todo', 'Learn about React')
    .contains('button', 'Complete')
    .click()
  cy.contains('.todo', 'Learn about React').find('[data-cy=remove]').click()
  cy.get('.todo').should('have.length', 3)
  cy.contains('.todo', 'Learn about React').should('not.exist')
})

The test passes

Testing the Todo application

Netlify setup

The application itself is built and deployed to Netlify. I will use GitHub Actions to run E2E tests after each deploy. In the todomvc-tests repository I will configure a GitHub workflow that only runs on the workflow dispatch event.

To trigger the GitHub workflow after Netlify has finished the deploy, I will use my plugin netlify-plugin-github-dispatch. I will need to create a personal GitHub token with "repo" permission.

Creating a new personal GitHub token

Then I set this token as an environment variable during the "build" step on Netlify.

Set the created GitHub token as Netlify environment variable in the Build step

Tip: make sure to keep Netlify to NOT allow the forked pull requests to run without review (which is the default), otherwise someone might steal your personal GitHub token.

Now create a new file netlify.toml in the application's repository to invoke the GitHub workflow after the successful deploy.

1
2
3
4
5
6
7
# https://github.com/bahmutov/netlify-plugin-github-dispatch
[[plugins]]
  package = "netlify-plugin-github-dispatch"
  [plugins.inputs]
    owner = "bahmutov" # use the target organization name
    repo = "todomvc-tests" # use the target repo name
    workflow = ".github/workflows/e2e.yml" # use workflow relative path

We need to install the plugin netlify-plugin-github-dispatch

1
2
$ npm install -D netlify-plugin-github-dispatch
+ [email protected]

When we first push the above code, the Netlify reports a plugin error after successful deployment - we have not created the e2e.yml yet!

The Netlify plugin needs the repository to have the workflow ready

Let's create our workflow - we need to check out code and run Cypress tests. We will NOT run the tests by default, only when someone dispatches the workflow_dispatch event. Note: this event can be triggered via API from Netlify or manually from the GitHub web UI.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# .github/workflows/e2e.yml
# test the deployed Netlify site
name: e2e
on:
  workflow_dispatch:
    inputs:
      siteName:
        description: Netlify Site Name
        required: false
      deployPrimeUrl:
        description: Deployed URL
        required: true
        default: 'https://todomvc-no-tests.netlify.app/'
jobs:
  # example job showing the Netlify information
  show-event:
    runs-on: ubuntu-20.04
    steps:
      - run: echo "Testing url ${{ github.event.inputs.deployPrimeUrl }}"
      - run: echo "Site name ${{ github.event.inputs.siteName }}"

  tests:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      # https://github.com/cypress-io/github-action
      # Installs and caches dependencies, runs all Cypress tests
      - name: Cypress run
        uses: cypress-io/github-action@v2
        with:
          # we want to test the URL passed by Netlify
          config: baseUrl=${{ github.event.inputs.deployPrimeUrl }}
          # store video and screenshots on Cypress Dashboard
          record: true
        env:
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

The above workflow is using cypress-io/github-action to abstract installing dependencies and running Cypress tests. We point Cypress at the deployed URL using baseUrl=${{ github.event.inputs.deployPrimeUrl }} syntax.

Tip: we could store the captured test run videos as GitHub test artifacts, or record the test results on Cypress Dashboard. I prefer the Dashboard since it provides a lot more information and is easier to use.

How to work in two repos

Now that we have the Netlify deploys and GitHub workflows configured, let's see how we can work day to day. I assume you are using feature branches for deployment. Then you open a pull request to merge the new feature to the main branch. If all tests pass, and if the reviewers agree, the feature is merged into the main branch. Here is how to use E2E tests together with the feature work.

  1. Pull both repos to the local machine.
  2. Open a branch in the application repo and a branch with the same name in the tests repo.

For example, let's improve the selectors in our application so our tests can find DOM elements following the Cypress' best practices. I will name the branches better-selectors.

1
2
3
4
5
6
7
~/git/todomvc-tests on main
$ git checkout -b better-selectors
Switched to a new branch 'better-selectors'

~/git/todomvc-no-tests on main
$ git checkout -b better-selectors
Switched to a new branch 'better-selectors'

I will update the application and the spec while running Cypress to make sure the tests pass locally. The test has changed some of the selectors

1
2
- cy.get('input[type=text]').type('Add tests!{enter}')
+ cy.get('[data-cy=new-todo]').type('Add tests!{enter}')
  1. Commit and push the tests first. You can even open a pull request in the tests repository. Remember, these tests do not run on commit, and they do not run on pull request. The tests must be triggered in order to run.

Pushed the "better-selectors" branch to the tests repo

  1. Commit and push the application branch better-selectors. Because this is not the main repository branch, Netlify does nothing.
  2. Open a new pull request and Netlify will trigger the preview deploy.

Netlify deploys the preview site for the pull request from the "better-selectors" branch

If you look at the Netlify deploy logs, the plugin netlify-plugin-github-dispatch triggers the workflow E2E using ref: <branch name>. Thus GitHub runs the workflow in the same branch name as the first repo branch.

The dispatch triggers workflow using "better-selectors" branch

Note: if there is no branch with the same name, the dispatch will fail.

  1. The GitHub Actions tab shows the triggered workflow.

The triggered E2E workflow

We can drill into the E2E job to see Cypress output.

The Cypress tests finish successfully

We can go to the shown Cypress Dashboard URL to watch the video of the run or see the captured screenshot. Notice the updated data-cy selectors in the Command Log and the preview URL.

The Cypress test screenshot

Tip: we can post the status check from the tests job back to the app repo, see the section at the end of this blog post.

If the tests are passing or not, I would put the link to the tests PR in the description of the application PR to let the reviewers see the updated tests together with the application code change.

Put the link to the tests PR in the body of the application PR

Now that the tests are passing, let's merge the tests and the code change.

  1. First merge the updated tests. Remember - the merged tests are not going to run, unless the workflow is triggered.
  2. Merge the updated application. The deploy to the main branch will trigger the tests already merged to the main branch (I assume both repos use a matching main branch name).

Manual trigger

If for some reason you want to re-run the tests, you can trigger a re-deploy on Netlify. Or you can trigger the workflow manually using the GitHub UI. Go to the Actions tab, pick the E2E workflow and click the "Run workflow" button. Change the inputs to what you desire and start the workflow.

Manually trigger the E2E workflow

Status checks

We have two separate repositories, thus we need to manage status checks ourselves. The application repo as soon as there is a pull request can post a pending status check on the merge.

.github/workflows/pr.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# in repo bahmutov/todomvc-no-tests
name: pull
on: pull_request
jobs:
  set-status:
    runs-on: ubuntu-20.04
    steps:
      # https://github.com/marketplace/actions/github-status-action
      - name: set pending status
        uses: Sibz/github-status-action@v1
        with:
          authToken: ${{secrets.GITHUB_TOKEN}}
          context: 'E2E tests'
          description: 'Tests pending deploy'
          # success, error, failure, or pending
          state: 'pending'
          sha: ${{github.event.pull_request.head.sha}}

This check will be "pending" thus the reviewer knows that the tests have not finished.

In the tests repository we can use similar code to set the status check in the original repo. We will get the merge commit SHA from the optional workflow input parameter commit - this parameter will be set by the netlify-plugin-github-dispatch code when it calls the workflow. Here is the entire workflow file from the test repo:

.github/workflows/e2e.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
# todomvc-tests repo
# test the deployed Netlify site
name: e2e
on:
  workflow_dispatch:
    inputs:
      siteName:
        description: Netlify Site Name
        required: false
      deployPrimeUrl:
        description: Deployed URL
        required: true
        default: 'https://todomvc-no-tests.netlify.app/'
      commit:
        description: Original repo commit SHA
        required: false
jobs:
  # example job showing the Netlify information
  show-event:
    runs-on: ubuntu-20.04
    steps:
      - run: echo "Testing url ${{ github.event.inputs.deployPrimeUrl }}"
      - run: echo "Site name ${{ github.event.inputs.siteName }}"
      - run: echo "App commit SHA ${{ github.event.inputs.commit }}"

  tests:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      # https://github.com/cypress-io/github-action
      # Installs and caches dependencies, runs all Cypress tests
      - name: Cypress run
        uses: cypress-io/github-action@v2
        # let's give this action an ID so we can refer
        # to its output values later
        id: cypress
        # Continue the build in case of an error, as we need to set the
        # commit status in the next step, both in case of success and failure
        continue-on-error: true
        with:
          # we want to test the URL passed by Netlify
          config: baseUrl=${{ github.event.inputs.deployPrimeUrl }}
          # store video and screenshots on Cypress Dashboard
          record: true
        env:
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

      # after e2e tests finish, set the status back in the original repo
      # https://github.com/marketplace/actions/github-status-action
      - name: Set commit status
        if: ${{ github.event.inputs.commit }}
        uses: Sibz/github-status-action@v1
        with:
          # create personal GitHub token to be able to
          # set status in other repositories
          # https://github.com/settings/tokens/new
          authToken: ${{secrets.PERSONAL_GITHUB_TOKEN}}
          context: 'E2E tests'
          description: 'Cypress ran the tests'
          # state can be success, error, failure, or pending
          # let's grab it from the Cypress step outcomes
          # https://github.com/cypress-io/github-action#outputs
          state: ${{ steps.cypress.outcome }}
          owner: 'bahmutov'
          repository: 'todomvc-no-tests'
          sha: ${{github.event.inputs.commit}}
          target_url: ${{ steps.cypress.outputs.dashboardUrl }}

We need to use our personal GitHub token to set the status in the first repository. The status check step runs every time. Here is how the pull request in the first repo looks when the Cypress tests have failed.

Cypress tests failed status check

The "details" link opens the Cypress Dashboard URL.

Check out both repos

Let's say you want to upgrade your test dependencies. You might need to check out the application too, install its dependencies, and run the app locally while running the tests. This example shows how do so by checking out the application repo into the subfolder "app".

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
name: ci
on: push
jobs:
  test:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout this repo 🛎
        uses: actions/checkout@v3

      - name: Checkout the application repo 🛎
        uses: actions/checkout@v3
        with:
          repository: bahmutov/fastify-example
          path: app

      - name: Install app dependencies 📦
        uses: bahmutov/npm-install@v1
        with:
          working-directory: app

      - name: Start the application 🎬
        run: |
          cd app
          npm run start &

      - name: Run tests 🧪
        # https://github.com/cypress-io/github-action
        uses: cypress-io/github-action@v3

Related