products

Caching Cypress On CircleCI

How to cache Cypress when running app and the tests from subfolders of the repo.

Imagine a situation where you have a monorepo with the web application and the Cypress tests living in their separate subfolders. You might want to keep the tests slightly separate to avoid clashing dependencies and typings. In the example repo bahmutov/todo-app-subfolders I have put the frontend, the api, and the Cypress tests in their own 3 subfolders

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
repo/
  api/
    REST API running on port 3000
    - package.json
    - package-lock.json
  frontend/
    static site running on port 5555
    - package.json
    - package-lock.json
  e2e/
    Cypress tests for localhost:5555
    - package.json
    - package-lock.json

  root has its own package.json
  with basic scripts and Prettier
  - package.json
  - package-lock.json

We need to run npm install in each subfolder when running locally. To test the app, we need to start the api, start the frontend, and then open Cypress. Tip: the project uses start-server-and-test utility to do it all with a single npm run dev command.

How do we install the dependencies and run the tests on CircleCI? Normally, I would use Cypress CircleCI Orb, but in this case, it is simpler to do the caching on our own, since everything resides in different subfolders. Fear not, we can write a config file without a problem.

🎁 If you want to see the completed config file, visit the bahmutov/todo-app-subfolders and check out its .circleci/config.yml file.

Note: if Cypress folder is outside the frontend application folder, it might be hard to set up the component testing, as Cypress won't be able to find the webpack / application settings to bundle the tests correctly. In my example, I only use the end-to-end tests, so that is not a concern.

Install the dependencies

First, things first. We need to install the dependencies in each subfolder. Let's write a simple "install" job

.circleci/config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
version: 2.1

# https://circleci.com/docs/2.0/executor-intro/
executors:
  ci-image:
    docker:
      # https://github.com/cypress-io/cypress-docker-images
      - image: cypress/base:16.14.2

commands:
  list-files:
    steps:
      - run:
          name: List files
          command: |
            echo "current folder $(pwd)"
            echo ""
            echo "current folder contents"
            ls -l
            echo ""
            echo "api folder"
            ls -l api
            echo ""
            echo "frontend folder"
            ls -l frontend
            echo ""
            echo "e2e folder"
            ls -l e2e

jobs:
  install-cypress:
    executor: ci-image
    steps:
      - checkout
      - list-files
      - run:
          name: Install API dependencies
          command: npm ci
          working_directory: api
      - run:
          name: Install frontend dependencies
          command: npm ci
          working_directory: frontend
      - run:
          name: Install E2E dependencies
          command: npm ci
          working_directory: e2e
      - run:
          name: Check Cypress
          command: npx cypress verify
          working_directory: e2e
workflows:
  version: 2.1
  build-and-test:
    jobs:
      - install-cypress

Tip: I love printing the local files using my custom list-files command, since you never know for sure how a workspace or a cache is restored and what files you get. In our case, it looks correct after checking out the files from the repo:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
current folder /root/project

current folder contents
total 40
-rw-r--r-- 1 root root    96 Aug 19 19:57 README.md
drwxr-xr-x 4 root root   129 Aug 19 19:57 api
drwxr-xr-x 3 root root   112 Aug 19 19:57 e2e
drwxr-xr-x 3 root root    65 Aug 19 19:57 frontend
-rw-r--r-- 1 root root 31689 Aug 19 19:57 package-lock.json
-rw-r--r-- 1 root root   587 Aug 19 19:57 package.json

api folder
total 128
-rw-r--r-- 1 root root    434 Aug 19 19:57 README.md
-rw-r--r-- 1 root root     18 Aug 19 19:57 data.json
drwxr-xr-x 2 root root     23 Aug 19 19:57 img
-rw-r--r-- 1 root root 117455 Aug 19 19:57 package-lock.json
-rw-r--r-- 1 root root    962 Aug 19 19:57 package.json
drwxr-xr-x 2 root root     31 Aug 19 19:57 scripts

frontend folder
total 68
-rw-r--r-- 1 root root 64596 Aug 19 19:57 package-lock.json
-rw-r--r-- 1 root root   349 Aug 19 19:57 package.json
drwxr-xr-x 3 root root    72 Aug 19 19:57 public

e2e folder
total 128
drwxr-xr-x 3 root root     17 Aug 19 19:57 cypress
-rw-r--r-- 1 root root    250 Aug 19 19:57 cypress.config.js
-rw-r--r-- 1 root root     90 Aug 19 19:57 jsconfig.json
-rw-r--r-- 1 root root 118740 Aug 19 19:57 package-lock.json
-rw-r--r-- 1 root root    305 Aug 19 19:57 package.json

The job runs through its commands and installs the dependencies one by one.

Installing all dependencies one by one on CI

Nice, it works. If we print the files again, we will see api/node_modules, frontend/node_modules, and e2e/node_modules folders. Running the next CI workflow has a problem; it has to reinstall everything from scratch, making it slow. Can we speed things up? What folders should we cache and restore to avoid reinstalling? Every time we run the npm ci command, it removes the local node_modules folder. Thus we cannot cache api/node_modules, frontend/node_modules, etc. We need to cache the downloaded NPM modules stored by default in the ~/.npm folder. Plus we need to cache Cypress binary, which normally is in the ~/.cache/Cypress folder.

Tip: you can see where Cypress keeps its downloaded binary files using the npx cypress cache path command. For example, on a Mac laptop:

1
2
$ npx cypress cache path
/Users/glebbahmutov/Library/Caches/Cypress

Tip 2: you can show the basic Cypress information after installing it using the npx cypress info:

1
2
3
4
- run:
    name: Cypress info
    command: npx cypress info
    working_directory: e2e

Printing Cypress information on CircleCI using the cypress info command

The npx cypress info shows that Cypress binaries are cached inside /root/.cache/Cypress folder, which is the user home folder that can be accessed via ~/.cache/Cypress in general. We need to cache this folder to skip re-downloading the Cypress binary file on each test flow run.

Thus we will cache the following folders:

  • ~/.npm where NPM downloads modules during npm i and npm ci command before placing them into node_modules
  • ~/.cache/Cypress where Cypress downloads and stores its binary file

Tip: if we are using Yarn, we could simply cache ~/.cache/yarn and ~/.cache/Cypress folders, or even ~/.cache.

We will throw away the cached files if any of our lock files change.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
install-cypress:
  executor: ci-image
  steps:
    - checkout
    - list-files
    - restore_cache:
        keys:
          - deps-v1-{{ checksum "api/package-lock.json" }}-{{ checksum "e2e/package-lock.json" }}-{{ checksum "frontend/package-lock.json" }}
    # a separate cache for Cypress binary
    - restore_cache:
        keys:
          - cypress-v2-{{ checksum "e2e/package-lock.json" }}
    - run:
        name: Install API dependencies
        command: npm ci
        working_directory: api
    - run:
        name: Install frontend dependencies
        command: npm ci
        working_directory: frontend
    - run:
        name: Install E2E dependencies
        command: npm ci
        working_directory: e2e
    - run:
        name: Check Cypress
        command: npx cypress verify
        working_directory: e2e
    # save the downloaded NPM modules
    - save_cache:
        key: deps-v1-{{ checksum "api/package-lock.json" }}-{{ checksum "e2e/package-lock.json" }}-{{ checksum "frontend/package-lock.json" }}
        paths:
          - ~/.npm
    # save the installed Cypress binary
    - save_cache:
        key: cypress-v2-{{ checksum "e2e/package-lock.json" }}
        paths:
          - ~/.cache/Cypress

Tip: you might want to split single NPM cache into 3 separate ones for each project. This way if one of the lock files changes, then no need to create a new cache. But in practice, that is not that important.

Pass the installed files

Great, we have one CI job that installs the dependencies very quickly, because it restores the dependencies very quickly unless the package lock files have changed. How about running Cypress tests?

We could have run the end-to-end tests right there in the install-cypress job:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# install-cypress job
# after saving the caches
 - run:
    name: Start API 🎬
    command: npm start
    working_directory: api
    background: true
- run:
    name: Start frontend 🎬
    command: npm start
    working_directory: frontend
    background: true
- run:
    name: Run tests 🚀
    command: npx cypress run
    working_directory: e2e

But as the number of tests grows, it makes more sense to run the tests in parallel using several test jobs. Those jobs should grab all files installed by the install-cypress job. Here is where we need to play a trick and combine CircleCI workspaces with restoring a cache. Let's revisit the end of the install-cypress job and extend it with persist_to_workspace CircleCI command:

1
2
3
4
5
6
7
# install-cypress job
# after saving the caches
# pass all the files (including the local node_modules)
# to the next job in the workflow
- persist_to_workspace:
    root: .
    paths: '*'

Any job that requires the install-cypress job will get its workspace, which is all files from the project's repo. The command persist_to_workspace runs inside the current project working directory /root/project and saves all repo files plus the installed node_modules subfolders already present. It won't save any outside files like ~/.cache/Cypress though, so keep this in mind when restoring (attaching) the workspace in the job

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
test-cypress:
  executor: ci-image
  steps:
    - attach_workspace:
        at: '.'
    - list-files
    - restore_cache:
        keys:
          - cypress-v2-{{ checksum "e2e/package-lock.json" }}
    - run:
        name: Check Cypress
        command: npx cypress verify
        working_directory: e2e
    - run:
        name: Start API 🎬
        command: npm start
        working_directory: api
        background: true
    - run:
        name: Start frontend 🎬
        command: npm start
        working_directory: frontend
        background: true
    - run:
        name: Run tests 🚀
        command: npx cypress run
        working_directory: e2e

workflows:
  version: 2.1
  build-and-test:
    jobs:
      - install-cypress
      - test-cypress:
          requires:
            - install-cypress

CircleCI workflow with install and test jobs

We are passing all /root/project files by attaching the previously saved workspace using the attach_workspace command. This restores the files in the /root/project folder (notice we don't have to use the checkout command at all)

The workspace already includes installed NPM dependencies

Then we restore the Cypress binary cache folder ~/.cache/Cypress and we are good to go - all dependencies are there. The entire workspace and how it passes files from one job to another can be seen in the diagram below

How the files are passes from the install to the test job

We start both services as background processes, run the tests, and CircleCI shuts down the running background processes when the job finishes.

CircleCI completes the test-cypress job

Note: the command persist_to_workspace can only pass local files, and our ~/.cache folder is outside. We could install Cypress in a different local subfolder using CYPRESS_CACHE_FOLDER, see Cypress caching guide. Then we could skip the restore_cache command.

Parallel testing

Ok, all is going well. Until we get more tests, and suddenly a single test job running the tests one after another takes too long.

The workflow takes 13 minutes because of the tests

Each test file takes about 3 minutes and we have 4 of them

We need to turn on Cypress spec parallelization, thus running specs in parallel in different test containers. No problem.

  • start recording the tests on Cypress dashboard by creating a new project from Cypress Test Runner

Create a new project for recording our tests

  • set the record key as CircleCI environment variable CYPRESS_RECORD_KEY

Keep the record key a secret on CircleCI

  • use N CircleCI test containers in parallel and run the tests in parallel
.circleci/config.yml
1
2
3
4
5
6
7
8
9
10
11
test-cypress:
  executor: ci-image
  parallelism: 5
  steps:
    - attach_workspace:
        at: '.'
    ...
    - run:
        name: Run tests 🚀
        command: npx cypress run --record --parallel
        working_directory: e2e

That's it - you only need the parallelism parameter and pass the cypress run --record --parallel arguments, and Cypress does the rest. Let's look at the new timings.

The workflow duration dropped from 14 to 4 and a half minutes

CircleCI ran 5 test jobs in parallel

Each test job grabbed the workspace created by the single install job and quickly restored the Cypress binary from cache. Then the tests were split by the Cypress Dashboard across the machines that joined the test run.

All tests were executed in parallel

Note: CircleCI has tested using 5 jobs in parallel, but the Cypress Dashboard is showing specs split across 4 machines. Yup, by the time the 5th machine has started, the specs have already been allocated and the last machine had nothing to do, so it has finished quickly.

5 CircleCI machines executed the tests in parallel in test-cypress job

The last machine joined late and had nothing to test

Bonus 1: Wait for the site before starting the tests

Sometimes our frontend takes a while to start. We want to wait and check for the site to be up before running the tests. I always use the utility wait-on for this (and sometimes I combine starting the server with waiting for it to start using start-server-and-test CLI module). Let's add the wait-on as a dev dependency to the frontend package

1
2
$ npm i -D wait-on
+ [email protected]

I will add a script to wait for the local port 5555 to respond to HTTP GET requests (by default wait-on sends HTTP HEAD command, which many bundlers ignore)

frontend/package.json
1
2
3
4
5
6
{
  "scripts": {
    "start": "serve -p 5555 public",
    "wait-for-app": "wait-on http-get://localhost:5555 --log --timeout 60000"
  }
}

On CircleCI we will run the wait-for-app command after starting the server but before running Cypress tests

.circleci/config.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
- run:
    name: Start API 🎬
    command: npm start
    working_directory: api
    background: true
- run:
    name: Start frontend 🎬
    command: npm start -- --no-clipboard
    working_directory: frontend
    background: true
- run:
    name: Wait for frontend ⏳
    command: npm run wait-for-app
    working_directory: frontend
- run:
    name: Run tests 🚀
    command: npx cypress run --record --parallel
    working_directory: e2e

That's it - even if the server takes up to a minute to start responding to the external requests, it is ok. End-to-end tests will run only after the web app starts.

See also

Bonus 1: set Cypress Dashboard tag for the main branch

Let's say we want to tag the test runs on Cypress Dashboard but only the ones made from the main branch. We can write a conditional command when running the tests

1
2
3
4
5
6
7
8
9
10
11
12
13
- run:
    name: Run tests 🚀
    # if the current branch is "main",
    # tag this Cypress Dashboard recording with "main"
    # otherwise do not add any tags
    # https://circleci.com/docs/env-vars#built-in-environment-variables
    command: |
      if [ "${CIRCLE_BRANCH}" == "main" ]; then
        npx cypress run --record --parallel --tag "main"
      else
        npx cypress run --record --parallel
      fi
    working_directory: e2e