products

Parallel end to end testing with Cypress, Docker and GitLab

How to build multiple test bundles and run E2E test jobs in parallel.

Update: November 2018 - this way of manual balancing for Cypress tests is obsolete. There is a much faster and simpler way to run multiple specs in parallel using Cypress parallelization flag. See Run and group tests the way you want to.

We have been enjoying end to end testing with Cypress a lot, and this blog post describes our test build system. We just open sourced the build tool multi-cypress that allows everyone to

  • reuse parts of the testing code using ES6 import syntax
  • build smaller bundles without dead code using tree-shaking
  • generate a GitLab CI project file with multiple test projects
  • use off the shelf base Docker with Cypress and its dependencies installed
  • run multiple test tasks in parallel on GitLab instance

This will be useful to anyone who is using Cypress, and possibly is looking to run the tests on the private hardware, instead of the cloud solution from the Cypress team.

Rolling the bundles

Imagine a small test spec file to be executed by Cypress. It opens a page and confirms that the loader element is invisible.

1
2
3
4
5
6
7
describe('page', () => {
  it('loads', () => {
    cy.visit(...)
    cy.get('.loader')
      .should('not.be.visible')
  })
})

What if we add more tests and want to reuse the "the loader goes away" logic? It is simple if all the tests go into the same file, but realistically they should not - we do want to split tests into separate spec files to allow running them in parallel. So we need a way to load JavaScript from external files in our spec. The Cypress framework does not support this natively - its runner just loads the given spec file and assumes it has everything it needs. One can understand why - there are different ways to bundle JavaScript for the browser, and Cypress team felt picking any particular solution would not solve this problem for everyone.

Luckily, there are good solutions for bundling separate files into a single file, and my favorite one right now is Rollup. It has a great "bonus" feature - it only includes into each output bundle pieces that were actually used. Thus we can factor out the common test utilities into a separate file without paying a penalty

src/utils.js
1
2
3
4
5
6
7
8
export function noLoader () {
  cy.get('.loader')
    .should('not.be.visible')
}
export function successMessage () {
  cy.get('.success')
    .should('be.visible')
}
src/page-spec.js
1
2
3
4
5
6
7
import { noLoader } from './utils'
describe('page', () => {
  it('loads', () => {
    cy.visit(...)
    noLoader()
  })
})

We could build the complete bundle using Rollup's CLI tool

1
rollup -o cypress/integration/page-spec.js src/page-spec.js

Note that the bundle only includes noLoader function from src/utils.js - the other functions were "shaken off" by the Rollup, since they were never imported.

cypress/integration/page-spec.js
1
2
3
4
5
6
7
8
9
10
function noLoader () {
  cy.get('.loader')
    .should('not.be.visible')
}
describe('page', () => {
  it('loads', () => {
    cy.visit(...)
    noLoader()
  })
})

Even if you stop reading right now, you can benefit from using Rollup to split your testing code into simpler individual files.

Rolling multiple bundles

Rollup is great when one produces a single output bundle. We can use either CLI options or create a rollup.config.js that describes the input and output paths.

rollup.config.js
1
2
3
4
export default {
  entry: 'src/page-spec.js',
  dest: 'cypress/integration/page-spec.js'
}

But this does not scale up - we want to have multiple output spec bundles, because we do plan to run multiple Cypress instances on CI, each instance executing its own spec file. Rollup only supports a single entry, and as our tests grow we have multiple files to bundle instead.

1
2
3
4
5
6
7
8
9
10
11
/
  src/
    page-spec.js
    api-spec.js
    search-spec.js
    utils.js
  cypress/
    integration/
      page-spec.js
      api-spec.js
      search-spec.js

Luckily, Rollup provides a convenient module API, and I wrote a small wrapper that "rolls" multiple files. Check it out at bahmutov/rollem. You can hardcode the spec names, or find them automatically using a file mask

rollem.config.js
1
2
3
4
5
6
7
8
function toConfig(filename) {
  return {
    entry: filename,
    dest: 'dist/' + path.basename(filename)
  }
}
const configs = glob.sync('src/**/*-spec.js').map(toConfig)
module.exports = configs

Rollem even supports watching source files for any changes, and rebuilding the bundles, shortening each "code - run - test" iteration.

The base Cypress Docker image

Running Cypress on your own CI server requires preparing the environment; most importantly it requires xvfb to run. The simplest way to prepare and isolate the environment is to make a Docker image with all dependencies pre-installed. We have prepared an image and publicly released it under a personal name; the Cypress team will probably release an official image when they have some free time. The Docker file we have released uses Node 6 base image and currently includes the Cypress executable v0.16.4; the version can be easily customized via an argument.

1
2
3
4
5
6
7
8
9
10
11
12
13
FROM node:6
# Need Xvfb
RUN apt-get update --yes
RUN apt-get install --yes libgtk2.0-0 libnotify4 libgconf2-4 libnss3
RUN apt-get install --yes xvfb
# Need Cypress itself
RUN npm set progress=false
RUN npm i -g [email protected]
ARG CYPRESS_VERSION
ENV CYPRESS_VERSION ${CYPRESS_VERSION:-0.16.4}
RUN echo Cypress version to install $CYPRESS_VERSION
RUN cypress install
RUN cypress verify

The built Docker image is hosted at the public hub, and is built automatically using the Automated build feature. Every time we push a commit to the GitHub repo bahmutov/cypress-image, the new Docker image is created.

Parallel GitLab builds

We plan to generate multiple test "spec" files, and then execute them using our own GitLab CI server. To isolate the individual tests, we will use Docker containers, each based on the base Cypress image. When we roll multiple separate test bundles, we also generate a single .gitlab-ci.yml file. It has a single "build" job (that uses rollem to build the bundles) and multiple "test" jobs. The "test" jobs will only execute after the successful "build" job finishes; the "test" jobs can execute in parallel. By adding more individual workers to the project, we can potentially execute all N "test" jobs in parallel on N workers, cutting the end to end testing time to 1-2 minutes.

Here is a part of the typical .gitlab-ci.yml file generated by the multi-cypress tool (built on top of Rollem).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
image: bahmutov/cypress-image
# caching node_modules folder
# https://about.gitlab.com/2016/03/01/gitlab-runner-with-docker/
cache:
  paths:
  - node_modules/
stages:
  - build
  - test
build-specs:
  stage: build
  script:
    - npm install
    - npm test
    - npm run build
  artifacts:
    paths:
      - cypress/integration

The "build" step only generates the bundles spec files (and passes them to the "test" step as artifacts). All "test" jobs are kind of the same, only the spec filename differs, thus we use a special template feature in the GitLab CI file. It allows us to reuse the common test step commands without repeating same commands.

1
2
3
4
5
6
7
8
9
10
11
12
# Hidden job that defines an anchor named 'e2e_test_definition'
# This job will be automatically assigned "test" phase
.job_template: &e2e_test_definition
  script:
    - cypress ci --spec "cypress/integration/$CI_BUILD_NAME.js"
# as many job definitions as spec files
# runs cypress ci --spec cypress/integration/one_spec.js
one_spec:
  <<: *e2e_test_definition
# runs cypress ci --spec cypress/integration/two_spec.js
two_spec:
  <<: *e2e_test_definition

The result, even on the public GitLab shared runner shows 4-5 test jobs executing in parallel. The below image shows the typical pipeline of the cloned kitchen sink Cypress example project. I have not switched everything in the specs, thus two specs failed

tests

The shared GitLab runner does not reflect the accurate performance advantages; it spends 90-150 seconds per test, but about 90% of the time is spent downloading the base Cypress Docker image - the shared worker does NOT cache the bahmutov/cypress-image Docker image! On our private runners, the image is cached, thus each test job starts almost immediately, cutting the time per spec job to just a few seconds.

Conclusions

We love using Cypress to test our web applications. It really shines when using locally, and we have been able to develop and open source the tools listed above to allow everyone to quickly run the same tests in house. I am sure that the Cypress team will invest heavily in its own hosted CI cloud solution; but for people like us who absolutely have to run tests in house, the multi-cypress fits the bill.

The described solution is a good reflection of our development philosophy. Each tool wraps another tool, extending its functionality. This keeps each tool simple and focused, yet the final solution accomplishes a lot via composition. In a sense, it is almost equivalent to functional composition:

1
parallel builds = multi-cypress ( rollem ( rollup ( src/**/*-spec.js )))

Hope your continuous testing solution can benefit from our tools, and if there are any questions or issues, let us know!