This blog post will show how to get a "typical" CI/CD pipeline set up that is fast yet powerful. The final pipeline will:
- runs all tests quickly on CI server using load balancing with
- deploys app to the production environment
- runs just a few smoke tests against the production url
- groups all tests and smoke tests under a single run in Cypress Dashboard for clarity
Here is an example TodoMVC test project - bahmutov/todomvc which is copied from cypress-io/todomvc. A typical Cypress test that adds two items and verifies that there are two items in the list looks like this:
When running Cypress in the interactive mode (
cypress open) we can see each command and how the DOM looked during that moment.
Writing tests with Cypress is easy - and a typical TodoMVC app needs them! Pretty soon I can write tests that exercise all aspects of a typical TodoMVC app - adding and editing items, routing, etc, putting 30 tests into
Running tests locally
In order to run the tests we need to start the local server. There is NPM script that starts the server - we can call it from one terminal
npm start and the server runs at
localhost:8888. Cypress knows about this url because I put it in the
Great, but I don't want to remember to start a server just to run the tests, and I always forget to shut it down after the tests finish. So I use a utility I wrote called start-server-and-test. It executes "npm start", waits until port 8888 responds, then runs the "npm test" command - which runs the headless tests. Here are the scripts
Whenever I want to run all tests headlessly I can execute
npm run test:ci.
$ npm run test:ci
That is how I run the tests locally
Running tests on CI
Cypress works great on any CI. I like CircleCI for its simplicity and flexibility, so I set up Circle run for this open source projects and wrote circle.yml file. For now it is just a workflow with a single job that installs dependencies including Cypress binary, then runs the tests. We need to cache
~/.cache folders for each build to start quickly. The
build job passes all installed files to the
test-locally job via CircleCI workspaces.
Great, Circle runs the tests, and they pass ... and I don't see videos or error screenshots. I need to set up test recording on Cypress Dashboard. Once I do this, and set
CYPRESS_RECORD_KEY environment variable on Circle, I need to change my commands to execute
cypress run --record. My full set of scripts becomes larger
And the Circle script command becomes
npm run test:ci:record. Great, I can see the video of the run and CLI output at https://dashboard.cypress.io/#/projects/r9294v/runs/1/specs. Hmm, interesting, both tests ran on a single Circle machine.
Can we run 2 tests on 2 machines in parallel?
Recently we have added test parallelization to Cypress tests. To load balance all our specs across 2 machines, we need:
- tell Circle to give us 2 machines. Luckily we just need to set
circle.ymlfile like this when we define a job:
- we also need to add flag
cypress run --recordcommand, so the script command becomes
"cy:record": "cypress run --record --parallel".
You can see the test run at https://dashboard.cypress.io/#/projects/r9294v/runs/4/specs and here is a totally expected thing - the total run is completely dominated by the
app.js spec file!
app.js took 34 seconds, while
first.js took 1 second. So if we want to load balance these specs, we better split the longer one into smaller spec files, preferably by feature. I split
app.js into 6 spec files, each with a few tests. You can find the split in this commit. Let's push the commit and run the CI again. The Cypress Dashboard shows a much better "balance" of specs!
Much better machine utilization. We don't win any time though, because of the overhead of handling each spec - the test runner needs to contact the Dashboard service, upload video file and other artifacts after each spec and ask for the next spec. When specs finish as quickly as my short example specs, in the order of below 5 seconds, the overhead matters a LOT. In more realistic situations, load balancing across 2, 3, 10 machines is absolutely crucial. And the command
cypress run --record --parallel does not care how many machines will be joining - they all will be load balanced automatically.
For this static application I picked the simplest deployment - the TodoMVC app is sent to GitHub pages using gh-pages with NPM script command
"deploy": "gh-pages -d dist". You can find the deployed version at https://glebbahmutov.com/todomvc/.
Ok, the deployment is simple to do from the local terminal. What about deploying from CircleCI? Well, when you use workflows, it might be tricky. Here is how to do this, and you can always consult circle.yml.
I will add another job to run after local tests pass.
It is also a good idea to only deploy from
master branch, which we can control using from the workflow
# the rest of the file
Great, all set ...
Except the deployment is NOT happening due to a weird problem 😖
npm run deploy
The Circle job is hanging, and has to be killed manually. You can tell SSH to trust
github.com using a command
ssh-keyscan -H github.com >> ~/.ssh/known_hosts except if you add this command to the
deploy job it is NOT working.
ssh-keyscan -H github.com >> ~/.ssh/known_hosts
Here is the trick - the SSH setup on Circle happens only if the job has
checkout step. Thus we need to change the first job in the workflow, the one that checks out source code from GitHub. We also should store
~/.ssh folder in the workspace passed from the first job to other jobs in the workflow.
Great, the pipeline goes through and deploys the
dist folder to the GitHub pages.
Tesing in production
So now that our application is deploying to "production" environment, we should ... test it again. Because who knows - the production application might be misconfigured, missing files, assume a different base url, or something else. We want to make sure the deployment went smoothly. We don't have to run all end-to-end tests, but we can run just a few sanity tests.
So we want to do two things:
- run just a single spec file as a smoke test. We can do it using
cypress run --spec ...option
- run tests against production
baseUrl, and not against
I have added a new script name
And one last thing - we want to record this test on Cypress dashboard and even add it to the same run as our load balanced job did. Because really, this is part of the same CI workflow execution, so it makes sense to show them together as a single logical run. Except we do NOT want to mix it up with all the tests 2 machines executed in parallel. This is possible - just mark this test as a different group with
cypress run --group <name> option.
The new pipeline finishes, and the Cypress Dashboard run shows two groups of tests - the smoke test with a single spec, and "unnamed" group with all specs (load balanced). You can see this run at https://dashboard.cypress.io/#/projects/r9294v/runs/18/specs
Of course we could have given that group a name, because you can combine the two options, like
cypress run --parallel --group "all tests". Read more about options how to group and parallelize test runs in Cypress
parallelize docs. For now here is the overview of the final CI workflow.
Wait for me
Our pipeline runs a deploy job between running all tests and running smoke tests. Sometimes the deployment takes a long time. Which means that by the time the smoke tests start, Cypress Dashboard thinks the run has already finished and no new groups should be added.
Every time a group of tests finishes, the Cypress Dashboard starts a countdown, waiting for any new groups to join. Once the countdown gets to zero, the run completes, and no new groups are allowed to join; the run is finished.
Here are the good news: you can configure the time limit on per project basis. Go the project's settings in the Dashboard and set a longer time limit.
- Running E2E tests should be quick and easy. Cypress test runner solved the
easypart from its very beginning. Now, with the help of the dashboard service, the
quickis really true too.
- You can run different tests in groups, and load balance each group separately if needed. All groups are still added to the same logical "run" on the Cypress Dashboard