products

Run and group tests the way you want to

How to load balance multiple Cypress end-to-end tests in parallel mode, or group different tests together and in general control test runs in any way you want.

This blog post will show how to get a "typical" CI/CD pipeline set up that is fast yet powerful. The final pipeline will:

  • runs all tests quickly on CI server using load balancing with --parallel option
  • deploys app to the production environment
  • runs just a few smoke tests against the production url
  • groups all tests and smoke tests under a single run in Cypress Dashboard for clarity

The app

Here is an example TodoMVC test project - bahmutov/todomvc which is copied from cypress-io/todomvc. A typical Cypress test that adds two items and verifies that there are two items in the list looks like this:

1
2
3
4
5
6
7
8
9
// cypress/integration/first.js
it('adds 2 todos', () => {
  cy.visit('/')
  cy.get('.new-todo')
    .type('learn testing{enter}')
    .type('be cool{enter}')
  cy.get('.todo-list li')
    .should('have.length', 2)
})

When running Cypress in the interactive mode (cypress open) we can see each command and how the DOM looked during that moment.

First test

Writing tests with Cypress is easy - and a typical TodoMVC app needs them! Pretty soon I can write tests that exercise all aspects of a typical TodoMVC app - adding and editing items, routing, etc, putting 30 tests into cypress/integration/app.js

All tests

Running tests locally

In order to run the tests we need to start the local server. There is NPM script that starts the server - we can call it from one terminal npm start and the server runs at localhost:8888. Cypress knows about this url because I put it in the cypress.json file:

cypress.json
1
2
3
{
  "baseUrl": "http://localhost:8888"
}

Great, but I don't want to remember to start a server just to run the tests, and I always forget to shut it down after the tests finish. So I use a utility I wrote called start-server-and-test. It executes "npm start", waits until port 8888 responds, then runs the "npm test" command - which runs the headless tests. Here are the scripts

package.json
1
2
3
4
5
6
7
{
  "scripts": {
    "start": "http-server -p 8888 -c-1 --silent",
    "test": "cypress run",
    "test:ci": "start-test 8888"
  }
}

Whenever I want to run all tests headlessly I can execute npm run test:ci.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
$ npm run test:ci

> [email protected] test:ci /Users/gleb/git/todomvc
> start-test 8888

starting server using command "npm run start"
and when url "http://localhost:8888" is responding
running tests using command "test"

> [email protected] start /Users/gleb/git/todomvc
> http-server -p 8888 -c-1 --silent


> [email protected] test /Users/gleb/git/todomvc
> cypress run


==============================================================================

  (Run Starting)

  ┌──────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    3.1.0                                                        │
  │ Browser:    Electron 59 (headless)                                       │
  │ Specs:      2 found (app.js, first.js)                                   │
  └──────────────────────────────────────────────────────────────────────────┘


...

  (Run Finished)


      Spec                         Tests  Passing  Failing  Pending  Skipped
  ┌──────────────────────────────────────────────────────────────────────────┐
  │ ✔ app.js              00:31       28       28        -        -        - │
  ├──────────────────────────────────────────────────────────────────────────┤
  │ ✔ first.js            00:01        1        1        -        -        - │
  └──────────────────────────────────────────────────────────────────────────┘
    All specs passed!

That is how I run the tests locally

Running tests on CI

Cypress works great on any CI. I like CircleCI for its simplicity and flexibility, so I set up Circle run for this open source projects and wrote circle.yml file. For now it is just a workflow with a single job that installs dependencies including Cypress binary, then runs the tests. We need to cache ~/.npm and ~/.cache folders for each build to start quickly. The build job passes all installed files to the test-locally job via CircleCI workspaces.

circle.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
defaults: &defaults
  working_directory: ~/app
  docker:
    - image: cypress/base:10

version: 2
jobs:
  build:
    <<: *defaults
    steps:
      - checkout
      # find compatible cache from previous build,
      # it should have same dependencies installed from package.json checksum
      - restore_cache:
          keys:
            - cache-{{ .Branch }}-{{ checksum "package.json" }}
      - run:
          name: Install Dependencies
          command: npm ci
      # run verify and then save cache.
      # this ensures that the Cypress verified status is cached too
      - run: npm run cy:verify
      # save new cache folder if needed
      - save_cache:
          key: cache-{{ .Branch }}-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      # all other test jobs will run AFTER this build job finishes
      # to avoid reinstalling dependencies, we persist the source folder "app"
      # and the Cypress binary to workspace, which is the fastest way
      # for Circle jobs to pass files
      - persist_to_workspace:
          root: ~/
          paths:
            - app
            - .cache/Cypress

  test-locally:
    <<: *defaults
    steps:
      # restore application and Cypress binary before running the test command
      - attach_workspace:
          at: ~/
      - run: npm run test:ci

workflows:
  version: 2
  build_and_test:
    jobs:
      - build
      - test-locally:
          requires:
            - build

Great, Circle runs the tests, and they pass ... and I don't see videos or error screenshots. I need to set up test recording on Cypress Dashboard. Once I do this, and set CYPRESS_RECORD_KEY environment variable on Circle, I need to change my commands to execute cypress run --record. My full set of scripts becomes larger

package.json
1
2
3
4
5
6
7
8
9
10
11
12
{
  "scripts": {
    "start": "http-server -p 8888 -c-1 --silent",
    "test": "cypress run",
    "test:ci": "start-test 8888",
    "test:ci:record": "start-test 8888 cy:record",
    "cy:run": "cypress run",
    "cy:record": "cypress run --record",
    "cy:open": "cypress open",
    "cy:verify": "cypress verify"
  }
}

And the Circle script command becomes npm run test:ci:record. Great, I can see the video of the run and CLI output at https://dashboard.cypress.io/#/projects/r9294v/runs/1/specs. Hmm, interesting, both tests ran on a single Circle machine.

Two tests, one machine

Can we run 2 tests on 2 machines in parallel?

Parallelization

Recently we have added test parallelization to Cypress tests. To load balance all our specs across 2 machines, we need:

  • tell Circle to give us 2 machines. Luckily we just need to set parallelization: 2 in circle.yml file like this when we define a job:
circle.yml
1
2
3
4
5
6
7
8
9
test-locally:
  <<: *defaults
  # run this job on 2 machines at once
  parallelism: 2 # <===== add this line!
  steps:
    # restore application and Cypress binary before running the test command
    - attach_workspace:
        at: ~/
    - run: npm run test:ci:record
  • we also need to add flag --parallel to cypress run --record command, so the script command becomes "cy:record": "cypress run --record --parallel".

You can see the test run at https://dashboard.cypress.io/#/projects/r9294v/runs/4/specs and here is a totally expected thing - the total run is completely dominated by the app.js spec file!

Two tests on two machines

The app.js took 34 seconds, while first.js took 1 second. So if we want to load balance these specs, we better split the longer one into smaller spec files, preferably by feature. I split app.js into 6 spec files, each with a few tests. You can find the split in this commit. Let's push the commit and run the CI again. The Cypress Dashboard shows a much better "balance" of specs!

Split specs load balanced

Much better machine utilization. We don't win any time though, because of the overhead of handling each spec - the test runner needs to contact the Dashboard service, upload video file and other artifacts after each spec and ask for the next spec. When specs finish as quickly as my short example specs, in the order of below 5 seconds, the overhead matters a LOT. In more realistic situations, load balancing across 2, 3, 10 machines is absolutely crucial. And the command cypress run --record --parallel does not care how many machines will be joining - they all will be load balanced automatically.

Deployment

For this static application I picked the simplest deployment - the TodoMVC app is sent to GitHub pages using gh-pages with NPM script command "deploy": "gh-pages -d dist". You can find the deployed version at https://glebbahmutov.com/todomvc/.

Ok, the deployment is simple to do from the local terminal. What about deploying from CircleCI? Well, when you use workflows, it might be tricky. Here is how to do this, and you can always consult circle.yml.

I will add another job to run after local tests pass.

circle.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
deploy:
  <<: *defaults
  steps:
    - attach_workspace:
        at: ~/
    - run: npm run deploy

workflows:
  version: 2
  build_and_test:
    jobs:
      - build
      # runs a couple of machines in parallel
      # with load balanced all tests against a local server
      - test-locally:
          requires:
            - build
      # pushes app to https://glebbahmutov.com/todomvc
      - deploy:
          requires:
            - test-locally

It is also a good idea to only deploy from master branch, which we can control using from the workflow

circle.yml
1
2
3
4
5
6
7
8
9
# the rest of the file
# add "filters + branches" to "deploy" job
- deploy:
    filters:
      branches:
        only:
          - master
  requires:
    - test-locally

Great, all set ...

Except the deployment is NOT happening due to a weird problem 😖

1
2
3
4
5
6
7
8
npm run deploy

> [email protected] deploy /root/app
> gh-pages -d dist

The authenticity of host 'github.com (192.30.253.113)' can't be established.
RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? Step was canceled

The Circle job is hanging, and has to be killed manually. You can tell SSH to trust github.com using a command ssh-keyscan -H github.com >> ~/.ssh/known_hosts except if you add this command to the deploy job it is NOT working.

circle.yml
1
2
3
4
5
6
7
8
deploy:
  <<: *defaults
  steps:
    # restore application and Cypress binary before running the test command
    - attach_workspace:
        at: ~/
    - run: ssh-keyscan -H github.com >> ~/.ssh/known_hosts
    - run: npm run deploy
1
2
ssh-keyscan -H github.com >> ~/.ssh/known_hosts
/bin/bash: /root/.ssh/known_hosts: No such file or directory

Here is the trick - the SSH setup on Circle happens only if the job has checkout step. Thus we need to change the first job in the workflow, the one that checks out source code from GitHub. We also should store ~/.ssh folder in the workspace passed from the first job to other jobs in the workflow.

circle.yml
1
2
3
4
5
6
7
8
9
10
11
12
build:
  <<: *defaults
  steps:
    - checkout
    # other NPM commands ...
    - run: ssh-keyscan -H github.com >> ~/.ssh/known_hosts
    - persist_to_workspace:
        root: ~/
        paths:
          - app
          - .cache/Cypress
          - .ssh

Great, the pipeline goes through and deploys the dist folder to the GitHub pages.

Deployment

Tesing in production

So now that our application is deploying to "production" environment, we should ... test it again. Because who knows - the production application might be misconfigured, missing files, assume a different base url, or something else. We want to make sure the deployment went smoothly. We don't have to run all end-to-end tests, but we can run just a few sanity tests.

So we want to do two things:

  • run just a single spec file as a smoke test. We can do it using cypress run --spec ... option
  • run tests against production baseUrl, and not against localhost:8888

I have added a new script name

package.json
1
2
3
4
5
{
  "scripts: {
    "test:smoke": "CYPRESS_baseUrl=https://glebbahmutov.com/todomvc cypress run --spec cypress/integration/first.js",
  }
}

And one last thing - we want to record this test on Cypress dashboard and even add it to the same run as our load balanced job did. Because really, this is part of the same CI workflow execution, so it makes sense to show them together as a single logical run. Except we do NOT want to mix it up with all the tests 2 machines executed in parallel. This is possible - just mark this test as a different group with cypress run --group <name> option.

circle.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
post-deploy-test:
  <<: *defaults
  steps:
    # restore application and Cypress binary before running the test command
    - attach_workspace:
        at: ~/
    - run: npm run test:smoke -- --record --group "Smoke test"
workflows:
  version: 2
  build_and_test:
    jobs:
      # build, and local tests jobs
      - deploy:
          requires:
            - test-locally
      - post-deploy-test:
          requires:
            - deploy

The new pipeline finishes, and the Cypress Dashboard run shows two groups of tests - the smoke test with a single spec, and "unnamed" group with all specs (load balanced). You can see this run at https://dashboard.cypress.io/#/projects/r9294v/runs/18/specs

All tests and smoke test groups

Of course we could have given that group a name, because you can combine the two options, like cypress run --parallel --group "all tests". Read more about options how to group and parallelize test runs in Cypress parallelize docs. For now here is the overview of the final CI workflow.

CI workflow

Wait for me

Our pipeline runs a deploy job between running all tests and running smoke tests. Sometimes the deployment takes a long time. Which means that by the time the smoke tests start, Cypress Dashboard thinks the run has already finished and no new groups should be added.

Smoke tests cannot be added to the run

Every time a group of tests finishes, the Cypress Dashboard starts a countdown, waiting for any new groups to join. Once the countdown gets to zero, the run completes, and no new groups are allowed to join; the run is finished.

Time limit

Here are the good news: you can configure the time limit on per project basis. Go the project's settings in the Dashboard and set a longer time limit.

Time limit field under project's settings

By picking a longer time limit, you can get any pipelines passing, like Netlify + Cypress or Zeit + Cypress and see all tests together.

Conclusions

  • Running E2E tests should be quick and easy. Cypress test runner solved the easy part from its very beginning. Now, with the help of the dashboard service, the quick is really true too.
  • You can run different tests in groups, and load balance each group separately if needed. All groups are still added to the same logical "run" on the Cypress Dashboard

Happy testing, and of course keep reading my Cypress blog posts here and at the Cypress official blog!