Who loves waiting on CI to finish testing multiple files? Not me. Luckily modern CIs support isolation and running multiple test jobs in many containers, splitting single sequence of test jobs into parallel runs. My favorite system for doing this is GitLabCI, but the same can be done in many other popular CIs. This blog post shows how to set up parallel test runs in CircleCI (v2) and on TravisCI. If I do not write this down I will forget and will have to reinvent this in the future, I am sure!
Imagine you have a single repository with many folders with tests. The cypress/cypress-example-recipes is a good example. Different recipes are subfolders in the
examples folder - there are more than 20 folders!
exa -l examples
A single script going through every folder, running end-to-end tests takes about 15 minutes. There are two test services where we run tests: on Circle and on Travis. First, let us convert sequence to parallel jobs on Circle CI.
CircleCI v2 has introduces workflows that are extremely powerful. Think of a workflow as a series of jobs; for our purposes every job is independent. Here is how we can define a workflow - we will name each job after a folder.
Now we need to define each job. We need to specify the Docker container the job should run in, any environment variables to set and the actual job steps. While we could copy and paste these steps, in reality all our test job are the same - only the folder is different. So we can use YAML template feature to create common job definition and only substitute the job name. Here is how it looks:
Note that every job in the workflow automatically receives the name (as we defined it) in the environment variable
CIRCLE_JOB - which we used to change into the right folder. You can see the exact cicle.yml file at the moment of this writing.
TravisCI has recently introduced build stages. Personally I do not think they are as easy to use as CircleCI workflows, they are suitable for our task. Again we will put all common test steps into a template. Because there is no predefined environment variable with the folder name, I will pass it as an environment variable
# common build steps for each recipe
Now to the test job definitions. They all belong to the same stage named
test, thus they will run in parallel. Instead of job's name, I am setting environment variable
DIR to the folder name.
You can find the exact .travis.yml file here. Here is how TravisCI shows the progress of the test run - some jobs have finished, others are running and more jobs are queued up.