The source code can be found in the repo bahmutov/flaky-test-cypress. Each exercise is in its separate branch: level1
, level2
, etc. You can get the starting code by cloning the repo and checking out the right branch:
git checkout level1
npm install
npx cypress open
The application is a simply form with several input fields.
Run the app.cy.js
spec and see the flaky test. Can you fix it? You can only modify the spec source code, no touching the application source code allowed.
The original flaky test that sometimes passes and sometimes fails. Can you solve the problem before watching the video below?
🎓 Do you like practicing end-to-end test writing using hands-on exercises like this blog post shows? Check out my online courses at cypress.tips/courses that have hundreds of hands-on lessons!
How do you check if a test has flake? By running it multiple times. How would you run a test 10 or 50 times in a row?
The test should pass, yet somehow it fails to save the user record. Can you fix it?
The test types an email into the input box, but sometimes it loses the first couple of characters. Can you fix the test, even if you cannot fix the underlying problem?
The test clicks on the "Submit" button and occasionally the button is not there, even if you can see it. Why does that happen? How can you write this test better?
1 | // https://github.com/bahmutov/cypress-slow-down |
🎁 You can find the source code for this blog post and the test code in the branch
add-via-store
of my repo bahmutov/basta-spring-2024-cypress-and-playwright.
The test is passing. For clarity, I am using cypress-slow-down plugin to slow down each Cypress command by 100ms.
Nice.
Once we have tested adding a customer, we can test so many other things. For example, we can test deleting a customer. To delete a customer, we need to ... add a customer first. Will you write almost the same test as before?
1 | it('deletes a customer', () => { |
The test passes.
Hmm. If you look at the Command Log column, 24 out of 36 commands are the same as in the "adds a customer" test and are adding a new customer record using the page elements. It is slow too.
Will we write a page object to abstract adding a customer? We could.
1 | import { format } from 'date-fns'; |
We could use the above page object in our test.
1 | import { sidemenu } from '../pom/sidemenu'; |
I don't like such page objects. All they do is "hide" individual cy.contains
and cy.get
commands in their tiny methods. Of course, we could create an abstraction on top of these tiny methods.
1 | import { customer } from './customer'; |
Then the test could use customers.submitForm()
1 | it('adds a customer via UI (2nd version)', () => { |
Even with multiple levels of abstractions, we still have a test that repeats 2/3 of its commands between the test "adds a customer" and "deletes a customer". We also built up levels of testing code in the page objects ... that the actual user of our web page does not see or benefit from.
You are building levels of code on top of the elements on the page. The end user does not benefit from this code, and the tests simply repeat actions on the page from the other tests. By the time you get to the "deletes a customer" test, you know that adding a customer works. So repeating the same clicks and typing in the test simply spends time.
Inside the application's code, the event handlers take the input from the page and pass it on to the business logic. For example, when you add a customer, here is what the application code is doing:
1 | import { Component, inject, OnInit } from '@angular/core'; |
The most important lines are:
1 | const action = customerActions.add({ customer }); |
If you print the constructed action, it looks like this:
Ok, so all those UI clicks and typings lead to calling this.#store.dispatch(action)
. We can do it ourselves from the test. First, we need to expose the store object so that the test can access it. No biggie. We can expose the global store from any top-level component.
1 | export class CustomersComponent implements OnInit { |
Technical detail: in Angular applications we need to dispatch the actions using ngZone
wrapper. This will force all components to update their user interface after propagating our action. Thus we need to expose the ngZone
instance.
1 | export class AppComponent implements OnInit { |
Let's update our test.
1 | it('deletes a customer (app action)', () => { |
We build an object with the customer information, then dispatch an NgRx action. We wait for the window
object to have the store instance - then we know the app has finished loading.
1 | cy.window().should('have.property', 'store'); |
Because of the design of our application, we need to dispatch the load event too in order to update the entire list. Here is how the test looks.
The test loads the page and immediately creates a customer record, simply by calling win.store!.dispatch(addCustomer);
. Then we can use the page UI and delete that customer, just like a real user. We lost nothing in our testing by removing the duplicate commands between the two tests. The "adds a customer" still exercises the page object flow for adding a new record. The current test simply reaches into the app and calls the same production code as the regular page ui would. Want to have a better test experience? Improve the application code, don't create levels of "better" testing code.
We can still use some small Page Object utilities, like opening a specific menu
1 | class Sidemenu { |
1 | import { sidemenu } from '../pom/sidemenu'; |
But in general, we can do larger actions to add new data by calling the app's code. Cypress tests run in the same browser as the app code, so no problems with passing the real object references, circular objects, etc.
🎓 If you want to see what I think a Page Object should have, check out a free lesson "Lesson b5: Write a Page Object" in my course Write Cypress Tests Using GitHub Copilot.
A note about types: We need to "tell" the application and the testing code that the window
object might have properties Cypress
, ngZone
, and store
. We can use src/index.d.ts
file for this:
1 | import type { NgZone } from '@angular/core'; |
1 | <dl data-cy="price"> |
Can we verify the individual fees and confirm the total is correct?
🎁 You can find the source code for this blog post in the repo bahmutov/cypress-prices-check.
If you know the precise values, the test becomes easy.
1 | beforeEach(() => { |
We can refactor the test a little to avoid duplicated selectors
1 | const fee = (name, value) => { |
Tip: I usually refactor my tests with the help of Copilot, see my course Write Cypress Tests Using GitHub Copilot
Instead of checking each fee element, we can collect all fees and totals into a single array for checking. For simplicity and elegance, I will use custom queries from my cypress-map plugin.
1 | // https://github.com/bahmutov/cypress-map |
Long object and arrays are shortened by the Chai assertions, but we can increase the truncation threshold.
1 | chai.config.truncateThreshold = 300 |
Still, the array might be pretty long and have just a single item difference, yet we print it entirely.
Let's give each fee a name before comparing.
1 | it.only('shows the expected fees (one object)', () => { |
Again, cy.getInOrder
, cy.map
, and cy.apply
are queries from cypress-map.
Let's see how it looks when some of the properties are incorrect.
Which values in the large object are different?
We can use plugin cy-spok to better report the first different property.
1 | // https://github.com/bahmutov/cy-spok |
Unfortunately, cy-spok stops when it sees the first property with the different value. I would like to see all different properties.
We can compute the difference between the current subject and the expected object of values ourselves. Let's add a query command so it retries. If there are no differences, cy.difference
yields an empty object.
1 | Cypress.Commands.addQuery('difference', (expected) => { |
The error is now simply reports the properties with the different values.
Tip: you can use point-free programming by partially applying the first argument to the Cypress._.zipObject
method:
1 | cy.getInOrder(...selectors) |
Tip 2: there is cy.difference
query in the cypress-map
plugin, with the same logic.
📺 Watch the same test refactored from normal to
cy.difference
chain in the video Check Multiple Properties At Once Using cy.difference Query.
Tip 3: the command cy.difference
in cypress-map
even allows you to use predicates to check each property
1 | const tid = (id) => `[data-cy=${id}] dd` |
You can watch the above test explained in my video cy.difference Command With Predicates.
Now let's move away from confirming individual fees to computing the total. The final total
price on the page should equal to the sum of all fees. We must also consider negative fees and formatted currency.
Ok, let's convert each string to a number and confirm the sum is equal to the last number
1 | // https://github.com/bahmutov/cypress-map |
All commands in the above test are queries, thus they retry. Even if some fees load slowly, the test should pass. Imagine, the coupon's value is fetched from a remote service, thus there is a delay.
1 | <dl data-cy="total"> |
No problem, the test will retry. The invalid number is a null
, which causes the sum to be NaN
.
Just a precaution I like checking the total to be within some certain numerical range. This should avoid accidentally confirming 0 + 0 + ... + 0 = 0
.
1 | .should((numbers) => { |
We are using floats to compute the dollar sum. Floating-point numbers suffer from precision loss. Let's imagine we have two numbers that end in 4
and 3
. Their sum will have a floating-point error
1 | <dl data-cy="price"> |
We can deal with this problem in 3 different ways.
closeTo
assertion1 | .should((numbers) => { |
Instead of dollars, we can use cents to compute the sum and do the comparison.
1 | it('adds up all fees using cents', () => { |
In this blog post I will use Dinero.js v2. Let's install the necessary dev dependencies
1 | $ npm i -D dinero.js@alpha @dinero.js/currencies@alpha |
Here is out test that uses a few utility functions from Dinero library
1 | // https://github.com/bahmutov/cypress-map |
Nice.
]]>Let's see if can learn what SafeTest can and cannot do.
The first thing I have noticed was that SafeTest seems to be a production dependency. Yes, the docs say to install SafeTest using npm install --save-dev safetest
, but then you should include it in your src/index.tsx
file:
1 | import ReactDOM from "react-dom"; |
The example folders in kolodny/safetest all list safetest
as a prod dependency:
1 | { |
Call me old-fashioned, but I feel a testing library should be a dev dependency, just like Cypress.
Feature | SafeTest | Cypress Component Testing (CT) |
---|---|---|
Dependency | prod 👎 | dev 👍 |
With SafeTest you create a config file setup-safetest.tsx
, add package.json
scripts, and modify your src/index.tsx
file.
You also need to be running the application before you can start testing, just as if this was an end-to-end test against the local environment.
With Cypress, you just open it and click on the "Component Testing".
Cypress finds from the source code the framework your are using and creates appropriate files automatically.
The modifications to the cypress.config.ts
file:
1 | import { defineConfig } from "cypress"; |
That it is. You can start writing component tests.
Feature | SafeTest | Cypress Component Testing (CT) |
---|---|---|
Dependency | prod 👎 | dev 👍 |
Installation | manual 👎 | auto 👍 |
Start the app | needed 👎 | not needed 👍 |
Let's take a look at the example test shown in the SafeTest folder.
1 | import { describe, it, expect } from 'safetest/vitest'; |
We are rendering a simple React component with the text "Test1" and confirm it is visible. SafeTest is built on top of Playwright, thus the test runs in a real browser. The test confirms that the component is visible in the real browser. Nice. The same test can be written using Cypress:
1 | describe('simple', () => { |
Let's interact with a component. Here is the example test that clicks on the component 500 times.
1 | it('can do many interactions fast', async () => { |
And an equivalent Cypress component test
1 | it('can do many interactions fast', () => { |
Ok. How about using shortcut to change the state of the component to perform an "app action"? SafeTest can give you a "bridge" function, whatever it is:
1 | it('can use the bridge function', async () => { |
Seems we need the bridge to call the component code running in the browser from the Playwright test running in Node. Cypress can simply interact with the component, since it is the same browser code, so nothing special is needed.
1 | it('can use the bridge function', () => { |
People new to Cypress tip over its "schedule all commands, then run them with retries" execution model. This is why we use the following syntax to call the forceNumber
after we confirm the page confirms the component has text "Count is 1"
1 | cy.contains('Count is 1').then(() => { |
I always though the cy.then
command should have been called cy.later
, as I wrote in the blog post Replace The cy.then Command.
So let's add a few rows to our comparison:
Feature | SafeTest | Cypress Component Testing (CT) |
---|---|---|
Dependency | prod 👎 | dev 👍 |
Installation | manual 👎 | auto 👍 |
Start the app | needed 👎 | not needed 👍 |
Test syntax | easy 👍 | easy 👍 |
Execution environment | mix of Node and browser 👎 | browser 👍 |
Let's run the same test "can use the bridge function" by itself to see how long it takes in SafeTest vs Cypress
We start the SafeTest app with npm run dev
and run the single spec. The test is isolated to run by itself using it.only
Let's run the same test using Cypress component testing
Cypress component testing is very fast because it bundles only the component under the test plus the test itself. SafeTest loads the full E2E bundle and extracts the component to be tested. This is why you need to include it in the bundling entry and execute the application while running the component tests. In the CI mode, Cypress disables time-traveling debugger, thus alleviating the overhead. Thus Cypress CT can be faster than the SafeTest. Still, it does not matter. The component tests are fast enough in both cases.
Feature | SafeTest | Cypress Component Testing (CT) |
---|---|---|
Dependency | prod 👎 | dev 👍 |
Installation | manual 👎 | auto 👍 |
Start the app | needed 👎 | not needed 👍 |
Test syntax | easy 👍 | easy 👍 |
Execution environment | mix of Node and browser 👎 | browser 👍 |
Speed | fast 👍 | fast 👍 |
📝 SafeTest is extracting components / functions for testing from the production bundle. It is is a pretty cool idea. I have described doing it for Angular.JS (!) in the blog post Unit testing Angular from Node like a boss in 2015. Dmitriy Tishin described how to grab React components from Storybook bundles to use my Cypress component testing library
cypress-react-unit-test
in 2020.
SafeTest provides Jest mocks and spies
1 | import { describe, it, expect, browserMock } from 'safetest/jest'; |
Cypress includes Sinon.js mocks and spies.
1 | it('calls the passed logout handler when clicked', () => { |
Feature | SafeTest | Cypress Component Testing (CT) |
---|---|---|
Dependency | prod 👎 | dev 👍 |
Installation | manual 👎 | auto 👍 |
Start the app | needed 👎 | not needed 👍 |
Test syntax | easy 👍 | easy 👍 |
Execution environment | mix of Node and browser 👎 | browser 👍 |
Speed | fast 👍 | fast 👍 |
Spies and stubs | present 👍 | present 👍 |
SafeTest exposes createOverride
that let's you change the behavior of the component; one more proof that SafeTest is a production dependency. You can call use the override from the test
1 | // Records.tsx |
In Cypress, you could pass the overrides via the shared window
object to the component to implement the same substitution. I strongly discourage using the implementation-specific overrides. Test the interface of the component. You want to see how the component reacts to the loader error? How the loader shows? Stub the network call and confirm.
1 | import React from 'react' |
💻 The above example comes from my presentation Learn Cypress React component testing by playing Sudoku. The source code with all component tests is in the repo bahmutov/the-fuzzy-line.
Feature | SafeTest | Cypress Component Testing (CT) |
---|---|---|
Dependency | prod 👎 | dev 👍 |
Installation | manual 👎 | auto 👍 |
Start the app | needed 👎 | not needed 👍 |
Test syntax | easy 👍 | easy 👍 |
Execution environment | mix of Node and browser 👎 | browser 👍 |
Speed | fast 👍 | fast 👍 |
Spies and stubs | present 👍 | present 👍 |
Overrides | present ⚠️ | possible ⚠️ |
With Cypress you get an excellent interactive mode that shows the component and let's you write full tests quickly. It is no wonder that the SafeTest repo itself comes with lots of React components and none of them are tested.
But it is easy to write Cypress tests, here are a couple.
1 | import { Label } from './label.tsx' |
1 | import { Expandable } from './expandable' |
A good test example is a sanity test for the Accordion component
1 | import { Accordion } from './accordion.tsx' |
Feature | SafeTest | Cypress Component Testing (CT) |
---|---|---|
Dependency | prod 👎 | dev 👍 |
Installation | manual 👎 | auto 👍 |
Start the app | needed 👎 | not needed 👍 |
Test syntax | easy 👍 | easy 👍 |
Execution environment | mix of Node and browser 👎 | browser 👍 |
Speed | fast 👍 | fast 👍 |
Spies and stubs | present 👍 | present 👍 |
Overrides | present ⚠️ | possible ⚠️ |
Want to write tests | maybe 🤷♂️ | yes 👍 |
SafeTest was announced by Netflix. If they seriously use it to standardize on Playwright for the E2E tests and SafeTest for the component tests, it is going to be well-maintaned and supported. Cypress is backing its component testing. Judging from previous OSS library experience, I would say
Feature | SafeTest | Cypress Component Testing (CT) |
---|---|---|
Dependency | prod 👎 | dev 👍 |
Installation | manual 👎 | auto 👍 |
Start the app | needed 👎 | not needed 👍 |
Test syntax | easy 👍 | easy 👍 |
Execution environment | mix of Node and browser 👎 | browser 👍 |
Speed | fast 👍 | fast 👍 |
Spies and stubs | present 👍 | present 👍 |
Overrides | present ⚠️ | possible ⚠️ |
Want to write tests | maybe 🤷♂️ | yes 👍 |
Backed by | Netflix 👍 | Cypress.io 👎 |
SafeTest has other features that I like, for example the built-in visual comparisons
1 | const { page } = await render(<Header />); |
I do not count this feature towards SafeTest benefits, since it is built into Playwright.
Feature | SafeTest | Cypress Component Testing (CT) |
---|---|---|
Dependency | prod 👎 | dev 👍 |
Installation | manual 👎 | auto 👍 |
Start the app | needed 👎 | not needed 👍 |
Test syntax | easy 👍 | easy 👍 |
Execution environment | mix of Node and browser 👎 | browser 👍 |
Speed | fast 👍 | fast 👍 |
Spies and stubs | present 👍 | present 👍 |
Overrides | present ⚠️ | possible ⚠️ |
Want to write tests | maybe 🤷♂️ | yes 👍 |
Backed by | Netflix 👍 | Cypress.io 👎 |
Total | 4 👍 3 👎 | 8 👍 1 👎 |
Do you have a good SafeTest spec example and want to see how it looks in Cypress? Make the code public and send it my way.
Christian Bromann has published a nice comparison following the same metrics COMPONENT TESTING WITH SAFETEST VS. CYPRESS VS. WEBDRIVERIO.
]]>🎁 You can find the full source code in the repo bahmutov/cypress-pagination-tough-case.
Let us start with a nice happy path. We see a list with a few items. We can click on the "Next" button and go to the second page. For simplicity, the list ends on the second page, but our solution does not know that. The list has only the words first
, second
, third
, fourth
, fifth
, and sixth
. The test should stop checking the list when the "Next" button becomes disabled.
Here is the code for the page. As you can see, the app is very fast. It instantly renders the first list, then instantly renders the second page after clicking the "Next" button.
1 | <body> |
How would you write this test?
1 | beforeEach(() => { |
We are doing an example of negative testing, we are trying to confirm that something is NOT there. This is always more complex than confirming the presence, since an item can missing for many reasons. In any case, let's write a recursive algorithm, I will put it in the cypress/e2e/utils.js
file so we can use it from other specs. In the code below we are doing a conditional command depending on the "Next" button state. For more examples of conditional clicking see the blog post Click Button If Enabled.
1 | /** |
Our test can call the checkPage1
with the a string argument. Note: for this blog post, I slow down every Cypress commands by 200ms using my plugin cypress-slow-down.
1 | // https://github.com/bahmutov/cypress-slow-down |
The test passes. If we inspect the contains ...
commands for both pages, we see that the test did check the list at the right moment. In the first instance it checked the list when the list had values first
, second
, and third
. And in the second instance, the test checked the list when it was on the second page showing the values fourth
, fifth
, and sixth
.
If you look at the video again, I am showing the page at the moment the cy.contains
command ran twice.
Let's break the test on purpose to make sure it works correctly. Let's give it a string value that is present on the page. The value is second
so it should be found on the first page.
1 | it('fails if the item is on the first page', () => { |
What if the item is on the second page? Let's test it.
1 | it('fails if the item is on the second page', () => { |
Beautiful, our checkPage1
recursive test function is correct. Or is it?
A common problem in the SPA is the slow initial data load. I simulate it in the spec2.html
page
1 | <body> |
The initial list loads after 1 second, meanwhile the app is showing the loading...
text. Let's run the same 3 tests again and see if we have the same outcome:
1 | // EXPECTED |
1 | // https://github.com/bahmutov/cypress-slow-down |
Hmm. We are getting one test "flipped"
1 | // ACTUAL |
The second test that checks the item that should be found on the first page is green for some reason. Let's debug it. We can expand the test commands and click on the "contains 'second'" command to see how the page looked when the test checked the list.
Ooops, the test looked for the item with the test "second", the page was showing "loading...", the test happily continued on its way.
Remember: a negative assertion can pass for many reasons
The reason for the test passing while it should have been failing is confusion to what state the application is in. The test thinks the app is showing the list. The app is still showing the loading element. The solution is to "sync" the application and the test states. For example, the test can wait for the li
elements to appear before running a negative assertion. This will ensure the app is in the right state showing the items.
1 | /** |
In a TodoMVC application, the code might set a loaded
class to signal that the page has the list, and the test could do something like this:
1 | cy.visit('/') |
In our code, the checkPage2
function confirms the list items are present before checking them: cy.get('li').log('**has list items**')
. Let's see if the checkPage2
test utility leads to the correct test outcome.
1 | // https://github.com/bahmutov/cypress-slow-down |
If we debug the second test, we see that it fails for the right reason. The application is showing the actual list items first
, second
, and third
when the first "contains" assertion checks it.
Good. But this is not the entire story.
Let's introduce another challenge. Our application might be slow to update the list after the user clicks the "Next" button. I have seen applications where clicking the "Next" button updated the button itself, yet the list was still showing the old items for X milliseconds. Here is how I simulate it in the spec3.html
code:
1 | <body> |
Let's use our checkPage2
function to run against this application.
1 | // https://github.com/bahmutov/cypress-slow-down |
We are expecting again the first test to pass and the last two tests to fail. But we are getting something else:
1 | // EXPECTED |
Let's debug the third test. Hover or click over the second "contains li, fifth" command. Why is it still showing the first page?!!
Again, this is the confusion between the test and the application state. The test assumed that after clicking on the "Next" button and seeing list items, the app would be on the second page. But the application still showed the first page. In other circumstances we could have looked at the URL to confirm the page number
1 | // if the app rendered the page in the URL |
But we don't have it. So somehow we need to check that new list item elements are there on the page. People often use the text of an element to detect when the new list was rendered:
1 | // 🚨 INCORRECT, LEADS TO FLAKE |
The above code might work in some cases, but if the list has duplicates, it would fail. Imagine our list has items A
, A
, and A
on the first page, and the items A
, B
, and C
on the second. It would not be able to tell the two A
strings apart. A better way would be to check the element references, since the application replaces <LI>
elements on click:
1 | // ✅ check element references |
To simplify the above code we can use a query cy.stable
from my cypress-map plugin. The query retries until an element's reference remains constant for N milliseconds. Thus, if we know that the list switch takes at most 1 second, we can wait for the reference to remain stable for slightly longer period.
1 | // https://github.com/bahmutov/cypress-map |
Let's see our spec now.
The tests work exactly as expected. The third test gets to the second page, waits for the <LI>
elements to be stable, and then correctly finds the item with text "fifth"..
A huge downside to the cy.stable
command is that it must wait for N milliseconds. In our case, it waits for 1500ms. If the <LI>
elements switch after 300ms, then it would wait for 300 + 1500ms. We can make our test better. Remember: if the test "knows" what state the applicatin is in, then it can precisely wait for it, before running a negative assertion. Here is how I would modify my application to make it testable.
1 | // the list sets the correct data attribute |
The only thing this code does it sets the data-page
attribute when it renders the page:
1 | list.setAttribute('data-page', 1) |
Our test can take the advantage of the data-page=...
attribute to be much simpler
1 | /** |
Once the command cy.get("ul[data-page=${page}]")
passes, the test is good to check the items. We can even use a very short timeout limit to run our negative assertion, since we know the items are already there and won't change.
Nice.
]]>Conditional testing is an anti-pattern, but Cypress can do it. Here is how to solve this problem in several ways.
📺 Watch the first solution explained in the video Check If A Button Is Enabled Before Clicking.
Let's use plain Cypress command. We first get the button, then check its attribute inside cy.then(callback)
. We can schedule more commands in the callback
function.
1 | cy.contains('#btn', 'Click Me') |
Remember: any time you get something from the app, you must pass it forward to the next command, assertion, or cy.then(callback)
.
If the button is enabled, the test clicks and checks the updated text.
If the button happened to be disabled, the test simply logged a message and finished.
We can rewrite the test to use jQuery is
method to check if the button is enabled using the jQuery pseudo-selector :enabled
.
1 | cy.contains('#btn', 'Click Me') |
I like this syntax because it makes it clear what we are getting at; we are checking if the button is enabled. This is better than "hiding" the effects in the cy.then(callback)
via $btn.attr('disabled')
or even $btn.is(':enabled')
which are invisible to Cypress Command Log. If we are using cy.invoke command, we can click on it to see what it produced.
You can find the above examples under recipes on my Cypress examples site.
📺 Watch the next two solutions shown in the video Click Button If Enabled Using cypress-if And cypress-await Plugins.
I have introduced my plugin cypress-if in the blog post Conditional Commands For Cypress. It lets you do conditional "if / else" chains of commands based on the current subject. Here is how the above solution looks when using cypress-if
. The commands cy.if
and cy.else
are from the cypress-if
plugin
1 | // https://github.com/bahmutov/cypress-if |
When the current subject passes the jQuery assertion if('enabled')
, the logic takes the "IF" command chain path, executing all commands between cy.if
and cy.else
. Let's see how it behaves when the button is disabled.
So, so easy. Tip: want to run more commands? Just stick cy.then(callback)
inside the if / else
chain:
1 | cy.contains('button', 'Click Me') |
I really enjoy using cypress-if
, to be honest.
If you don't like working with the Cypress subjects and chains, I have cypress-await plugin for you. It sets up a spec file preprocessor that rewrites specs:
1 | // in your spec |
It looks synchronous, but under the hood all Cypress retries are still executing for each chain before assigning the value. Our conditional test can be rewritten like this:
1 | it( |
You can find more lessons on how to use cypress-if
and cypress-await
in my Cypress Plugins course.
While keeping the original contents in the center. You need to know the output width / height. For example to make the output image width 600 pixels with white border on both sides use the command
1 | convert input.png -background white -gravity center -extent 600 output.png |
Use +append
to create a single row. Use -append
to create a column.
1 | convert image1.png image2.png image3.png ... +append output.png |
See Give browser a chance blog post.
]]>🎁 You can find the source code for this blog post in the repo bahmutov/changed-specs-quickly-example.
If you have multiple specs plus utilities, it makes sense to precompute their import graph and store it in a JSON file in the repo. We will use my tool spec-change to compute dependencies among spec files.
1 | { |
We can run the npm run trace-deps
command locally. Here is its partial outut
1 | $ npm run trace-deps |
The file cypress/e2e/utils.ts
is popular. Several specs depend on it. If this file changes, we would consider e2e/adding-spec.ts
, e2e/clear-completed-spec.ts
and others changed too.
We can periodically update the deps.json
file on CI using a job like this:
1 | # this workflow computes the dependencies between Cypress spec |
The install step installs only the spec-change
NPM dependency and caches it by caching the ~/.npm
folder. We don't want to install all dependencies, and we don't want to download the Cypress binary.
1 | - name: Install spec-change 📦 |
When a user opens a pull request, they probably have modified some specs. We want to run the modified specs first, and if they pass, we would like to run all specs. We can find the changed between the current branch and the main branch using the find-cypress-specs utility. We can use either simple Git file change or trace dependencies.
1 | { |
I prefer using the trace so that any change to the utils.ts
forces the spec files that import it to run. Here is our workflow to determine the changed spec files.
1 | name: PR checks |
Notice the last job "Trace changed specs" has an id "trace". This job produces outputs, like the list of changed specs, the number of changed specs, and even the recommended number of machines to use to run the changed specs in parallel. We expose these outputs from the job find-changed-specs
:
1 | jobs: |
Let's pretend we modified the utils.ts
file and opened a pull request. The workflow runs and detects all the specs that rely on utils.ts
file
After finding the changed specs, we should quickly run them. First, we run all changed specs across the N recommended machines. Then we want ti run the other specs, also in parallel. Here is how we can do it using bahmutov/cypress-workflows reusable GitHub workflows. This is the rest of the PR workflow shown above; two jobs are after the "find-changed-specs" job.
1 | # run all changed specs in parallel |
We take the find-changed-specs
job's outputs and make two jobs, both using the split
workflow from bahmutov/cypress-workflows
. I used just two machines to run the rest of the specs. In the future I hope to compute a good number of machines based on the specs.
You can find the full workflow and source code in the repo bahmutov/changed-specs-quickly-example.
In this blog post I will give examples of Cypress api tests using the use-cypress-for-api-testing application as my example. The application is a simple TodoMVC web application on top of REST API. There are endpoints to create, delete, and modify Todo items, plus an endpoint to reset the data:
GET /todos
returns all todo itemsPOST /todos
adds a new todo itemPATCH /todos/:id
updates the given todo itemDELETE /todos/:id
deletes on todo itemPOST /reset
replaces the entire backend data with the given objectLet's write a few API tests.
Let's write a simple API test that resets all todos, adds a todo, then fetches all items. We will use the core Cypress command cy.request to make all HTTP calls
1 | describe('TodoMVC API', () => { |
The test passes. There is nothing to render in the Cypress application frame on the right, since we never did cy.visit
in the test. Still, the Command Log on the left shows every API test command.
If we are using the Cypress interactive mode cypress open
we can click on the REQUEST
command to see the request and the response. For example, let's examine the cy.request('POST', '/todos', todo)
command
When using the interactive cypress open
mode you can inspect each call by clicking on the command.
Once we know our backend API supports adding and fetching new items, let's confirm that we can complete an item.
1 | describe('TodoMVC API', () => { |
Let's confirm that we can delete an item using an API call. We can take a shortcut and set the backend data using POST /reset
instead of creating individual items via POST /todos
. We know that adding items is working from the first test.
1 | describe('TodoMVC API', () => { |
In our tests we used the built-in Chai assertions like deep.equal
. By default, these assertions truncate the data shown in the Command Log. Let's increase the truncation threshold to see more information.
1 | // show more information in each assertion |
Since we have an empty web application frame on the right and want to see more information for each request, let's redirect the request / response to the frame. I will use the cypress-plugin-api plugin to give my API tests a nice UI.
1 | // https://github.com/filiphric/cypress-plugin-api |
We simply imported the plugin and used cy.api
instead of cy.request
to make the calls. The Cypress UI is now much nicer.
We can even combine cy.request
and cy.api
to only show the relevant test information. For example, we might not want to see the POST /reset
calls while being interested in the other calls in the test:
1 | it('completes an item', () => { |
When creating a new item using POST /todos
call, the app sends the item, and the server responds with the same item. If the sent item does not include the id
property, the server will assign one. The server signals the success by returning status code 201. Let's confirm this.
1 | it('gets an id from the server', () => { |
The test is becoming verbose, and the Command Log is getting noisy. Let's use cy-spok plugin to make our assertions more powerful.
1 | // https://github.com/filiphric/cypress-plugin-api |
Using a single cy-spok
assertion we can match complex objects with nested properties and predicate checks.
Often API tests get something from the server and use that piece of data to make new calls. For example, if we want to get the ID of the created item to delete it, we could write the following test
1 | it('deletes the created item using its id', () => { |
Anytime the test gets something from the application, it needs to pass it forward to the cy.then(callback)
or to the next assertion. Some people are intimidated by such coding style; they miss async / await
syntax. For them I wrote cypress-await plugin that allows you to use the await
syntax with the Cypress command chains. Here is how we can rewrite the above test:
1 | // https://github.com/filiphric/cypress-plugin-api |
The test works the same, but it might be more intuitive to read.
Bonus: cypress-await plugin includes a sync mode which allows you to drop the verbose await
in front of every cy
command. The test runs the same and becomes simply:
1 | // sync mode from cypress-await |
We want to run API tests on CI from day one. Let's use GitHub Actions. We can use cypress-io/github-action to manage dependencies, caching, and running Cypress for us:
1 | name: ci |
Beautiful. Want even more power? Learn how to run tests in parallel using GitHub Actions in this video. In general, Cypress API tests running in the browser are slower than an equivalent Node-only API tests. But the debugging is faster, and on CI I can parallelize the tests using a one minute change; see cypress-split and cypress-workflows.
Because Cypress queues up its commands, it is easy to iterate over collections. For example, let's see how we can delete each item one by one
1 | // https://github.com/filiphric/cypress-plugin-api |
The expression to delete each Todo item simply queues up 4 Cypress commands:
1 | // delete each item one by one |
Since API tests do not use the browser to show the web page, there is less information to go on when the test fails. We can produce detailed terminal log (stdout and even JSON) using the plugin cypress-terminal-report. Here is an example terminal report output for the above "deletes all items one by one" test produced.
Finally, a nice feature of Cypress API tests is how easy it is to combine them with the UI web tests. Let's see one such example. The example application reloads all todos every minute:
1 | // how would you test the periodic loading of todos? |
Let's write a test that quickly tests this feature. We will create the initial data using API calls, then visit the site and confirm the initial list is visible. Then we can delete an item using an API call, fast-forward the app clock by 1 minute, and confirm the web view has updated and shows N - 1 todos.
1 | // show more information in each assertion |
I am using cy.clock and cy.tick commands to control the web application's clock. In between, we are deleting the data using the cy.request('DELETE', '/todos/101')
command.
For more clock examples, see Spies, Stubs & Clocks.
Cypress can be a powerful API testing tool. It has good API, assertions, and a visual interactive test runner great for seeing the test run. It also can be extended using open source plugins to provide even better API testing experience.
cy-api
, cy-spok
, cypress-plugin-api
, and lots of other plugins🎁 The source code for this blog post is in the public repo bahmutov/cypress-flakiness-debug-examples. I have grabbed the initial code from filiphric/cypress-flakiness-debug-examples branch
customer-subscriptions
. The examples in this blog post are using the application and the tests in the subfoldercustomer-subscriptions
.
The application displays a list of subscriptions. Some subscriptions are active and some are not.
The list is generated dynamically using @faker-js/faker module. The number of items can be from 1 to 6:
1 | import { faker } from '@faker-js/faker' |
There is also a loading "splash" screen before the items load.
1 | useEffect(() => { |
Let's see how Cypress end-to-end tests for this app can be flaky or solid.
If you just started the application, the webserver might take longer to bundle and return the homepage, especially in the dev mode. To better simulate the unpredictable initial load, I will add a random delay between 1 and 11 seconds to the fetch
call.
1 | useEffect(() => { |
Here is the first test as shown by Filip. It is flaky on purpose. Can you see at least two potential problems?
1 | it('Activates a subscription', () => { |
The first problem we see when running the test is the command cy.get('[data-cy=customer-item]')
failing.
The test runner simply did not "see" the subscriptions list in time. To determine it, click on the failed command GET
and look at the restored DOM snapshot: the app was still showing the "Loading" message.
But sometimes the test succeeds. If you inspect the same command GET
using the time-traveling debugger, instead of "Loading" you see the items.
This is test flake: depending on the application's speed the same command step fails or succeeds. We need to take the maximum loading time into the account. The error message "Timed out retrying after 4000ms: Expected to find element: [data-cy=customer-item]
, but never found it." tells us the solution: we must increase the timeout for the GET
command because the application might not show the items until after 11 seconds passed. Let's fix the test:
1 | cy.visit('/') |
The command now retries for 11 seconds instead of the default 4. You can see the slower progress bar.
Here is the second problem with the test. The application can show between 1 and 6 items. The test picks a random index between 1 and 6. If the test picks an index larger than the random number of items, the test will fail, since there is no such item. We must pick one of the existing items. Filip changed his test to do so:
1 | it('Activates a subscription', () => { |
💡 If you do need to pick a random number between min and max in your Cypress tests, please use the bundled Lodash function _.random:
1
2
3
4 // instead of this
let randomItem = Math.floor(Math.random() * 7)
// use Cypress._.random function
const randomItem = Cypress._.random(0, 6)
The above test is good. I would also print the picked item index to make it very clear which item we are subscribing to. It certainly removes the flake from trying to pick an item that is not there.
1 | .then((numberOfItems) => { |
The test is less flaky than before but occasionally it still fails. Here is a good example of the failing test:
We only have one item, so we pick it to activate the subscription. But the item is already active. We cannot click on it to activate again. When picking an item, we must only consider the items that are "trial" or "inactive".
We can look at the HTML markup to see if the "trial" and "inactive" items have any HTML attributes that we can use to easily query them while omitting the "active" subscriptions.
Hmm, nothing. No biggie, we can add data-status
attribute to our SubscriptionItem
component.
1 | const SubscriptionItem: React.FC<SubscriptionItemProps> = ({ |
Tip: I use separate data-
attribute to pass the status following the advice in my blog post Do Not Put Ids Into Test Ids.
Now our test can be very explicit and only consider the "trial" or "inactive" items by using the OR
CSS selector.
1 | cy.get( |
We can also go the other way and filter out all active subscriptions using the cy.not command.
1 | cy.get('[data-cy=customer-item]', { timeout: 11_000 }) |
We still have two problems with this test that will cause flake. Do you see them? One is caused by the random data, another by the test design.
Here is the problem caused by the test design. In the failure below we see that we are picking the item number one (zero-based index).
Hmm, let's hover over the items we picked initially using cy.get('[data-cy=customer-item]', { timeout: 11_000 }).not('[data-status=active]')
command. Seems we correctly picked 4 subscriptions.
Now hover over the EQ 1
command. Why it picking the already activated subscription that is NOT part of the original four items?
Ohhh, we picked the "trial" or "inactive" subscriptions initially to pick the random item index. But then we applied this index to all items
1 | cy.get('[data-cy=customer-item]', { timeout: 11_000 }) |
Let's fix it. Just apply the same logic to filter items.
1 | it('Activates a subscription', () => { |
The test works pretty well. But we can express our test even simpler by using my cypress-map plugin. Like Lodash, the cypress-map
provides a lot of "missing" queries and commands that make Cypress tests much simpler and stable. In our case, we need something like Lodash's _.sample
function.
1 | _.sample([1, 2, 3, 4]); |
1 | // https://github.com/bahmutov/cypress-map |
Boom. Simple and solid. Almost.
Here my #1 tip for you when writing Cypress tests. I have stated it many many years ago at a few conference presentations. When writing Cypress tests, write first "directions" to a human user. For example:
Write the steps as comments, telling the tester what to do, but not how to do it. Here are the steps I wrote as comments inside an empty Cypress test
1 | // https://github.com/bahmutov/cypress-map |
Now add an empty line in your VSCode and it should trigger GitHub Copilot. Here is what happens for me:
Nice! Our comments got "translated" into correct Cypress code that even uses the custom cy.sample
command from cypress-map
. Click "Tab" to accept the suggested code, and we have a passing test.
Let's continue generating our test. Write more user instructions.
Then we need to confirm the subscription was activated.
The test is complete.
Not bad, right?
Yet, there is one more source of test flake that we did not consider. Sometimes the test fails.
Again, by inspecting the Command Log column, you can see the problem. The test did find items. The test failed to find non-active items, because there all randomly generated subscriptions were "active" already. Our test always assumed there will be some inactive subscriptions, but that is an invalid assumption. We can do conditional testing in Cypress, even if it is an anti-pattern. The simplest way to run testing command only if there are items to be activated is by using my cypress-if plugin.
1 | // https://github.com/bahmutov/cypress-map |
If the NOT [data-status=active]
command yields no elements, we take the ELSE
branch where we simply print the info message.
Finally, let's confirm that our code works no matter what the backend sends us. We need to control the data, and the best way is to use cy.intercept command to stub the loading network call. We can use fixtures of different types: no inactive subscription, one item, a mixture of items to make sure our testing code can handle each possible situation. We can copy the starting response data straight from the browser.
The first JSON fixture will have a mixture of items, with "active" items first. This will test the case we discussed in the "Sample data" section above. The index of the inactive item should not accidentally apply to all items.
1 | [ |
Our next fixture will have just an active subscription to test the "ELSE" logic.
1 | [ |
Finally, we want to make sure we test the "trial" items and that they can be activated
1 | [ |
Let's write the tests. We can refactor the code to avoid the duplication.
1 | // https://github.com/bahmutov/cypress-map |
When using a network stub there is a danger that the server changes its response and our tests don't catch it. We can prevent this by adding a quick API test or a spy E2E test. Since we only want to ensure the properties / types of the objects in the response, I recommend using my cy-spok plugin.
In this test we will make a network call ourselves using the cy.request command and will validate the response. The response should be an array, and we can validate the first object using the built-in assertions plus spok
property predicates.
1 | import spok from 'cy-spok' |
There is no web page in this test, since we never called cy.visit
. Instead we simply see the assertions in the Command Log.
Sometimes it is hard to make a valid request from the test: the format of the call might be complicated, plus require authentication. It might be easier to just spy on the call made by the application. The same logic applies.
1 | import spok from 'cy-spok' |
Beautiful.
]]>🎓 In this blog post I used a lot of Cypress plugins. If you want to learn them better, I have an online hands-on course Cypress Plugins. You can also level up your network testing by taking my Cypress Network Testing Exercises course.
Can we catch broken images using Cypress tests?
🎁 The source code for this blog post can be found in my recipe "Image Has Loaded" at glebbahmutov.com/cypress-examples.
📺 Watch this example in the video Check Broken Images On The Page.
Imagine a single <img>
element. It loads the image using its src
attribute. The source could be a link URL, or a data:...
encoded image. Even if the URL is valid, the image itself could be corrupted and the browser might fail to decode it. Thus we want to check if the image loads, rather than simply checking the URL. A good way to check is to confirm the naturalWidth
or naturalHeight
of the img
element. For successfully loaded and decoded images it should be positive.
1 | <img |
1 | cy.get('#loads') |
If the src
URL points at a non-existent image, the test catches it.
📺 You can find this example explained in the video Check If An Image Loads.
Now let's check all images on the page. Rather than failing a test immediately, we will collect all broken image information before reporting it all at once. This allows us to judge how broken the page is and maybe even debug it faster. Imagine multiple images on the page. Some elements load the image from a remote URL, others use data:...
encoding, or even an SVG element.
1 | <img |
Let's check all images at once.
1 | const brokenImages = [] |
By reporting id
and alt
attribute we can point the team to the right direction. We could also use cyclope plugin to save the entire page as static HTML snapshot to debug the markup.
cy
command on my glebbahmutov.com/cypress-examples site. Tip: use the built-in search widget to quickly find the answers.Why learn just one web testing tool if you could learn both to double your chances of finding a better job? I have created Cypress Vs Playwright online course that trains you to solve practical testing problems using hands-on coding lessons. Each lesson poses a problem and gives you the starting code. You must solve the assignment using each test runner to appreciate the difference in syntax and capabilities between Cypress and Playwright.
]]>data-testid
or data-cy
to access elements in my end-to-end tests: use simple test ID values. Do not include randomly generated values. For example, when testing TodoMVC, each item could be:1 | <!-- 🚨 I do not recommend --> |
Separating the id part from the "test-id" will make writing tests much simpler.
I have created an example repo bahmutov/test-ids-blog-post to go with this blog post. The initial code uses data-testid
attributes for the TodoMVC fields.
The input element has data-testid=TodoInput
, the main list has data-testid=Todos
, and the two current todos have attributes data-testid="Todo-5589433909"
and data-testid="Todo-7392832596"
. The ids are coming from the application itself. If we look at the JSON data, we can see these values:
1 | { |
Great, so how does it affect writing the tests?
Tip: you can see which elements have data-testid
attribute by using my cypress-highlight plugin:
1 | import { highlight } from 'cypress-highlight' |
Let's select the important elements using the data-testid
attribute. This is what Cypress selector playground tool suggests using:
Plus having an explicit data-testid
signals everyone that this field is tested, makes finding tests easier, etc.
Let's start writing an end-to-end test
1 | it('adds 2 items', () => { |
Hmm, how would we confirm that 2 items are visible on the page? We could use LI
selector:
1 | it('adds 2 items', () => { |
Ok, it works, but what if we switch from using LI
elements to DIV
? Ok, let's use data-testid
attributes. Because we know the start of each attribute "Todo-" we could use a prefix attribute selector.
1 | it('adds 2 items', () => { |
Ughh, ok, it works.
Since we will use data-testid
in a lot of queries, let's make a tiny custom command. We want to have the equivalents to cy.get
parent and cy.find
child commands, so we add two commands.
1 | Cypress.Commands.add('getByTestId', (id) => { |
These commands make sense when checking the Todo items:
1 | cy.getByTestId('TodoInput').type('Write code{enter}').type('Test it{enter}') |
But when we check the number of remaining todos, we have to use the same prefix.
1 | cy.getByTestId('Footer') |
Because we need the prefix for Todo-
items, the cy.findByTestId
uses unnecessary prefix for [data-testid^="TodosRemaining"]
too. This might not be a problem, but let's try another situation. Let's say we want to confirm there are two Todo items in the <section class="todoapp">
top parent element.
1 | it('shows 2 items', () => { |
The test fails when it finds 5 items instead of 2.
Turns out, when looking for Todo
prefix we matched non-list items, like TodoInput
and TodosRemaining
. We could update the findByTestId
command to assume the -
separator:
1 | Cypress.Commands.add( |
But the larger question remains: why are we parsing the data attribute and encode values in our testing code? What if the application code changes?
Another situation shows how combining the element "type" with a random value in a single attribute is problematic. Let's say we spy on the added Todo items sent over the network. The network call contains the item's unique ID. Can we validate the ID against the element? Sure. Let's do this:
1 | it('has the correct item id', () => { |
The test passes and clearly shows the item's id, even if we had to concatenate Todo
and the id strings.
Great, but we do not want to do cy.getByTestId('Todo-${id}')
in our code. We want to get the Todos
component and find its child element with Todo-${id}
data test id attribute, just like we did before. Does it work?
1 | cy.wait('@newTodo') |
Oops, the fix the previous stray element problem broken finding the item when we know the entire attribute and do not need the trailing -
character.
Let's simplify our code. In the HTML template instead of concatenating Todo-<id>
we can put the id
value into its own data-...
attribute. Almost like normalizing database schema and splitting the first and last names into two columns, we can "normalize" data attributes to keep just a single type of value in each one.
1 | <!-- 🚨 BEFORE --> |
We can now remove test attribute prefix and simply confirm the Todo element.
1 | it('has the correct item id', () => { |
Let's create several todos and confirm the first item's id is unique on the page.
1 | it('has a unique item id', () => { |
Filtering all Todo elements by data-id
attribute is simple and uses the standard attribute selector.
1 | cy.getByTestId('Todo') |
The test passes - the first item has the unique id. We can even confirm it is the first one amongst its siblings.
1 | cy.wait('@newTodo') |
Beautiful.
]]>I noticed a few things about those tests that might be improved. Here is my refactoring of each of the three tests.
🎁 You can find the original source code in my repo bahmutov/webdriverSite. You can see the final solution in the branch solution. 📺 You can watch a recording of me refactoring all tests in the video Refactor Cypress Tests Checking Alerts And Modals.
The first test confirms the application calls the alert(...)
with the right text.
1 | describe('Lidando com Popups e Alerts', () => { |
Above the test I added my notes about what could be improved in this test. We never actually confirm that the application calls alert(...)
. You could comment out the .click()
and the test would still work. Here is the improved test that uses cy.stub command to confirm the app does call alert
with the right text argument.
1 | // need to ensure the alert actually happens |
For more stub examples and checking the alert
behavior, see my stubs examples page.
1 | it('Modal Popup', () => { |
We can take advantage of cy.contains command to find elements by text. We should also confirm the modal disappears after clicking the "Close" button.
1 | // refactor to use cy.contains command |
1 | it('Outra maneira para o Modal Popup', () => { |
In the test above we are spying on the network requests, but the spy is too permissive. We can limit it to only spy on the calls to the Google Analytics website. To learn more about network intercepts, see my course Cypress Network Testing Exercises
We can also fix several selectors to be precise. We also need to better check the behavior of the Loader element that first must appear and the go away.
1 | // use a better selector for to click the ajax loader |
For more details, read the blog posts Be Careful With Negative Assertions and Negative Assertions And Missing States. To learn more about testing different application states and confirming the application behaviors, see my course Testing The Swag Store
Tip: have a complex Cypress test you are unhappy with? Is it public? If yes, send it my way, and maybe I will record a video refactoring it to make the spec elegant, precise, and solid.
]]>🎁 You can find the web application repo at bahmutov/todo-app, and the tests repo at bahmutov/todo-app-tests.
When the user types /cypress
comment on a pull request in the first repo "test-app", we can get the commit SHA from the pull request itself.
1 | - name: Trigger Cypress run |
You can see the full workflow here. For example, in the PR 3 the last commit short SHA was 9053795
.
We can see this commit being sent to the test repo inside the payload body
When the tests workflow starts, it should update the commit status twice. First, before anything runs we set the commit status to "pending" to let the user know the tests are executing. After the tests finish, we can set the final result (success, failure, error) based on the outcome of the test job.
You can find the full testing workflow here. The relevant part is below
1 | # add a pending commit status check |
I am using my own utility cypress-set-github-status. Typically you would pass the commit status which is one of the values "pending", "success", "failure", or "error". But we can also use --status ${{ steps.tests.outcome }}
which will automatically convert job status string to the commit status: job outcome "success" is commit status "success", "failure" is "failure", and both job outcomes "canceled" and "skipped" are commit status "error".
Here is a passing test commit status set back in the "todo-app" pull request
Now we can configure the branch protection rule in the "todo-app" repo that requires the "todo-app-tests" status check to be green before merging.
]]>/cypress
and it will trigger a test run in the tests repo. The results of the test run will be posted back to the comment.🎁 You can find the web application repo at bahmutov/todo-app, and the tests repo at bahmutov/todo-app-tests.
To trigger the test run, we will trust the developer who opened a pull request or is reviewing it to type a new comment /cypress
. This is our "slash" command, and there is a reusable Github Action peter-evans/slash-command-dispatch that can handle it for us beautifully.
1 | # a workflow that runs when a user enters a pull request comment |
A couple of notes about the above workflow that you can find at slash-command-dispatch.yml:
It runs on every new issue and pull request comment
1 | on: |
It grabs the branch name using my GitHub Action bahmutov/get-branch-name-by-pr because we will need to check out the code when running the test. We need to use a GitHub personal access token set as a repo secret.
1 | # we only know the pull request number, like 12, 20, etc |
Similarly, we use the personal token to trigger the specific workflow for the "cypress" command
1 | - name: Slash Command Dispatch |
You can have multiple commands dispatched by this workflow, and they can even parse arguments. For example, I could trigger Cypress tests run against a new pull request, or measure its performance using /lighthouse
command, following the blog post Trying Lighthouse.
1 | commands: | |
Ok, so let's trigger the Cypress command workflow. The dispatch workflow adds the comment id and triggers the actual slash-command-cypress.yml
workflow
If the workflow is found, the dispatch immediately adds two emoji reactions to the comment.
We are still in the "todo-app" repository. We triggered the slash-command-cypress.yml
workflow. You can find the current YAML code in slash-command-cypress.yml and below:
1 | # this workflow runs when the user comments "/cypress" on a pull request |
Let me explain what the workflow is doing step by step.
The workflow runs when triggered by the GitHub API with repository_dispatch
event and type=cypress-command
1 | on: |
This workflow needs to trigger a new run in another repo "bahmutov/todo-app-tests". A simple way to call GitHub API is to use the GitHub's own action actions/github-script. Since I put some logic into a Node.js script, which I will show in a second, I need to check out just the file .github/workflows/trigger-cypress.js. This is the sparse checkout step:
1 | - name: Check out the repo 🛎️ |
Now let's see what the dispatch workflow sent us in the payload. I am printing the object first
1 | - name: Print event 🖨️ |
For our run, it shows:
Great, so if we want to grab individual arguments, like the name of the branch, we can use expression to get the nested property ${{ github.event.client_payload.slash_command.args.named.ref }}
. We now call the script trigger-cypress.js
and pass the individual values as environment variables:
1 | - name: Trigger Cypress run |
Inside the script we can use GitHub wrapper objects that call the API methods using the personal token we passed.
1 | module.exports = ({ github, core }) => { |
The action shows the parameters being passed to the repo bahmutov/todo-app-tests
correctly
Now we can shift our attention to the repo bahmutov/todo-app-tests that runs the E2E tests.
Any external caller can trigger the tests run in our todo-app-tests
repo by calling GitHub API and sending the repository_dispatch
event with type trigger-tests
. For details, see Run And Trigger GitHub Workflow blog post. Here is the full workflow .github/workflows/trigger.yml that receives this event.
1 | name: trigger |
When we triggered the workflow using our /cypress
comment, it finished successfully.
There are several things this workflow does to make it developer-friendly in two repos.
It posts the main information with the parameters it has received:
1 | - name: Print variables 🖨️ |
This is what you see right away in the workflow summary
Then it forms the browser URL to the workflow run that you see in the browser screenshot and appends it back to the original comment using the repo name and the comment id. It uses GitHub Action peter-evans/create-or-update-comment by the same person Peter Evans that wrote the slash comment dispatch action.
1 | # quickly post the workflow URL back in the original repo PR |
If you are looking at the pull request in the "todo-app" repo, you will see the tests workflow that you can click.
Then we need to check out the tests, the application code, and start the application. We will check out the application code using the branch reference name passed to us using ref: ${{ github.event.client_payload.ref }}
parameter.
1 | # check out both the app and the tests |
Now that the application is running in the background, let's run Cypress tests
1 | - name: Run E2E tests 🏃🏻♂️ |
I am assigning this step an id tests
so that later we can refer to the outcome of the step. We want to post the result back in the original comment in the "todo-app" repository.
1 | - name: Post results 📨 |
The steps.tests.outcome
can be "success", "failure", "cancelled", or "skipped". We are only interested in the "success" vs the others. In our case, the step succeeded, and thus we see in the comment the final status
Nice, this pull request is passing its tests.
In the next blog post I will describe how you can post a commit status back in the original repo. This way you can protected the branch and require the tests to pass before merging pull requests. Read the blog post Set Commit Status In Another Repo.
]]>In this blog post I will show how to refactor the above test to make it simpler and more accurate. There are hidden problems in the test code that might make it pass accidentally. We want to avoid such false positives, since they destroy the confidence in the ability of our automated tests to discover problems.
🎁 You can find the source code in the recipe "Check Cards" hosted on my Cypress examples site. You can watch me refactoring the code and explaining how I go about it in the video Check Cards Test Refactoring 📺.
Let's take the following HTML fragment to be our dummy "app".
1 | <main> |
We need the list of projects to be the source of truth. We will check the page against this array.
1 | const mockProjects = [ |
The initial code that closely resembles the code screenshot is below.
1 | cy.get('main') |
The test passes, all is green and good in the world.
Let's shorten the code. We don't have to use cy.wrap($el)
multiple times. We can use the expect($el).to.include.text(...)
assertion from Chai-jQuery library to verify the text is present.
1 | cy.get('main') |
Since the page is static by the time we start checking, the expect(...).to...
assertions work just fine. Alternatively, we can use cy.wrap($el).within(() => ...)
commands to limit the search to the element.
1 | cy.get('main') |
I really like using cy.within command. It shortens the code and makes it less likely to find a wrong element accidentally. Check out my video 📺 Find The Right Item Using The cy.within Command Or The Parent Selector for another cy.within
example.
There is a problem with out test. If you have over individual contains
commands in the left Command Log column, you can see the element if finds. Oops, seems we tried to find cy.contains('4')
and accidentally found 1 even in the last 24 hours
. Not what we expected to find.
I really dislike using cy.contains(partial text)
command, and instead prefer cy.contains(selector, partial text)
. Let's add data-cy
attributes to our HTML markup to be able to precisely find elements.
1 | <main> |
Now we can target the precise element when searching for each field.
1 | cy.get('main li.card').each(($el, index) => { |
Super. We now don't find stray elements with the same text.
Our test still does it backwards. It looks at the page and checks each li
item:
1 | cy.get('main li.card').each(($el, index) => { |
What if our page rendering is incorrect and it only renders the first card? No problem, the test passes!
Remember: you cannot trust the page, but you can trust the data. Thus you should iterate over the data list of projects and check each item against what is on the page.
1 | // confirm the correct number of items is shown |
You can get the data the page is derived from by observing network traffic using the cy.intercept
command or by accessing the data in the application.
1 | <!-- not nice --> |
Here is how I think about adding data test ids like data-testid
, or data-cy
to my HTML markup:
Tip: I suggest you read Cypress best practices for selecting elements
I like using explicit test ids to select elements for two reasons:
data-test
is modified or removed the developer is expected to update the tests.Nothing stops you from implementing a similar policy for aria attributes. You can also verify the a11y attributes from the test when needed:
1 | cy.getByTest('CloseButton').should('have.attr', 'aria-role', 'button') |
So how do I assign test ids to the elements on the page? Here are some practical examples from my Testing The Swag Store course.
Let's start with the first UI step: logging in. The container that has the input fields needs a test id, plus the input elements and the "Login" button.
Using a custom command cy.getByTest
in my project I can fill the form without relying on random HTML attributes like name
or password
. I will add data-test
attributes to:
Here is the test:
1 | it('fills the form and logs in', () => { |
Tip: I love using cy.within command to limit query commands to the parent element. But note that this command does not retry.
Selecting the login button by its data-test
attribute works better long-term than using text and cy.contains
command. The text might change, but the data-test
makes it clear: if you want to change this attribute, you better run the tests.
The inventory page has several buttons in the header, the list of products, plus the sort selection. Each inventory item has text that we might want to verify: the title, the price (maybe), plus the button to add the item to the cart.
I also add a data-test
attribute to the top level container element: it will help verify we are on the right page if the inventory has zero items.
Here is a typical test to verify the inventory page loads.
1 | beforeEach(() => { |
The two assertions make it easy to understand what is happening:
1 | // the inventory component loads |
Let's verify the information shown by the first item. Since we added test ids to the individual text fields, we can use the have.text
assertion:
1 | cy.getByTest('InventoryPageItem') |
Tip: Cypress comes with many Chai and Chai-jQuery and Sinon-Chai assertions, see my examples
Since checking an element's text is so common, I prefer writing custom commands like cy.getByTest
that can act like both cy.get
and cy.contains
commands.
1 | cy.getByTest('InventoryPageItem') |
Before we move to the cart page, let's confirm the inventory page updates the cart badge
1 | it('adds items to the cart', () => { |
Here are the two test ids we used
Let's test just the cart page. We can set the user local state with items in the cart and visit the page.
1 | it('shows the cart items', { viewportHeight: 1200 }, () => { |
By setting the data in the localStorage
we know exactly what to expect to see on the page. Thus we can loop through the list and confirm each CartItem
item.
If you use my plugin changed-test-ids you can see all test ids used in the source files and in the spec files. For example, let's list test ids used in the specs and also list all test ids not covered by specs.
1 | { |
Let's list test ids in the specs
1 | $ npm run specs |
Let's see all test ids without any tests
1 | $ npm run missing-tests |
Nice.
]]>sorry-cypress
.For more details see Cypress.io Blocking of Sorry Cypress and Currents.
If you want to split your multiple specs across multiple machines for free, I have created cypress-split plugin and described how to Run Cypress Specs In Parallel For Free. This solution does not "mimic" the Cypress Cloud API, thus it should not be banned.
☢️ If Cypress team DOES block cypress-split plugin, then I will have no choice but to block all my Cypress plugins (I have written 75+ Cypress plugins) if Cypress is run with
--record
flag.
All we need to do to use this plugin is to add it to the setupNodeEvents
callback.
1 | import { defineConfig } from 'cypress' |
The specs were split purely based on their names in the list, which can create unbalanced lists for machines. For example, let's take a project with five specs split across two machines. The first run splits the specs alphabetically:
Here is our GitHub Actions workflow file
1 | name: ci |
🎁 You can find the example application in the repo cypress-split-timings-example. In this examples I am using version v1 the
cypress-split
plugin.
Ughh, there is a spec spec-d.cy.js
that is much longer than others, and it makes the second machine take much longer. While the second machine is running spec-d.cy.js
and spec-e.cy.js
, the first machine is idle. It would be better to shift the spec spec-e.cy.js
to the first machine. Then we would save 10 seconds.
Let's tell cypress-split
to split specs based on previous run timings. Unfortunately, we do not have 3rd party service to keep track of spec timings, thus we need to do it ourselves. Let's set another environment variable SPLIT_FILE
on the test runners. Right now, it points at a non-existent file timings.json
in the root folder of the repo.
1 | ... |
Let's run the workflow again. The timings.json
file is not found yet, no big deal. The specs are split alphabetically.
1 | cypress-split: there are 5 found specs |
At the end of the run, the plugin prints JSON object with spec durations for the current machines.
The second machine prints its timings JSON
Note: the timings are logged to the terminal and saved in the local file on CI. Thus at the end of the test-run each machine has its own uncommitted SPLIT_FILE
file on disk.
You manually copy / paste both timings and merge them into a single timings.json
file and add to the repo.
1 | { |
With this file present in the repo, the CI runs and cypress-split
splits the specs based on durations. The first machine runs only the spec cypress/e2e/spec-d.cy.js
which takes by itself 60 seconds. The second machine runs the rest of the specs that together take up 40 seconds.
Nice. The spec split implementation can be found in the src/timings.js file in the cypress-split
plugin. Right now it is a simple greedy algorithm, but it seems to work just fine.
If specs change or new specs are added, the timings file becomes out-of-date. I have a couple of ideas how to update it periodically without relying on 3rd party service.
Note: if the spec is new and does not have duration in the timings file yet, the split algorithm assigns it an average duration of all existing specs.
The first strategy is to use a single long CI job that runs all specs and then commits the updated timings.json
file. It is very easy to commit changed files when using GitHub Actions CI.
1 | name: nightly |
The above workflow runs every day or when I start it manually from GitHub repo Actions tab. If there are any changes to the timings, the timings.json
file is committed and pushed to the repository using the wonderful reusable GitHub Action stefanzweifel/git-auto-commit-action.
Note: cypress-split
writes new timings file only if there are new specs or any durations are off by more than 10% from the existing ones.
Instead of running a single job with all specs to update the timings file, we can use my split
workflow from cypress-workflows repo. It works with cypress-split
alias cypress-split-merge
to merge downloaded timings files from parallel runs and output a single combined JSON as a GitHub Actions output. Here is an example workflow from cypress-split-timings-example repo.
1 | name: split |
The reusable workflow split
expands into multiple jobs. At the end, the merge-split-timings
job creates the merged-timings
output with all combined timings. We can commit the timings into the repo to be used for next CI run.
If you have any problems using cypress-split or cypress-workflows do not hesitate opening a GitHub issue. I will be happy to help.
Running all E2E tests should be fast and easy.
🎁 You can find the example application in the repo bahmutov/swag-store and the test in the repo bahmutov/swag-store-tests. I am using v1 of the plugin changed-test-ids.
When we open pull requests in the source repo, we want to build the application, maybe run some linting tools, and then trigger the end-to-end tests. Here is my example workflow, for simplicity I am not passing the build artifacts to the test workflow, just the current Git reference SHA.
1 | name: pr |
The step find-ids
looks at the changed source files src/**/*.js
and finds all testId=...
JSX attributes. The npx find-ids
command comes from changed-test-ids
NPM package alias.
1 | - name: Find test ids used in the changed source files |
These test ids are set as the outputs of the step and can be used in the next step to trigger workflow run using curl
command.
1 | -d '{"event_type":"specs-by-test-ids","client_payload": |
The changed-test-ids
even writes its summary, so you can see it in the GitHub Actions UI. For example, let's look at the pull request #1. The changed source file only touched a last name component.
Only this JSX file changed in this branch versus its parent branch main
, and the find-ids
tool detected the following testId=...
values: cancel,continue,firstName,lastName,postalCode
These test ids are passed in the trigger CI workflow event in the repo bahmutov/swag-store-tests
Finally, the user sees the detected ids in the action summary panel.
Let's run the right specs based on the test ids in the source files. In the repo bahmutov/swag-store-tests I use the same workflow to run the tests on push event, when manually starting the workflow (and we allow passing a list of test ids to us via UI), plus when triggering the workflow using an API call.
1 | name: ci |
I check out the application source code, install dependencies, and install the test repo's own dependencies. Now we can find the specs to run, depending on the test ids, if any.
1 | - name: Find specs for test ids |
In the workflow triggered from the swag-store
pull request we looked above, it finds the following specs and sets the list as the output:
If we find any specific specs, we run only those specs using the spec: ...
argument
1 | - name: Run found E2E tests |
The test finishes and we can see the output of the changed-test-ids
command npx find-ids --test-ids ... --specs ... --set-gha-outputs
I also pass the test ids as the Cypress Cloud run tag list, which makes finding the run in the list easy:
What if there are no detected test ids, or we simply want to run the tests? No problem, the "if - else" testing step runs all the specs if there are no steps.find-specs.outputs.specsToRunN
.
1 | - name: Run all E2E tests |
Picking specs to run by the test ids used in the changed source files might be useful when you have a lot of tests to pick from.
]]>You can find the introduction to the challenge in the free lesson Lesson n1: Cypress Pagination Challenge of my 🎓 Cypress Plugins course. In this blog post I will post my solutions to the challenge.
🎁 You can find the repo with the code challenge at bahmutov/cypress-pagination-challenge. Clone it, install dependencies, and start solving!
When evaluating a solution, we should think how robust the test is against the web application challenges:
We do not know how many times (if any) we need to click the "Next" button, it depends on the size of the table. We know that once the "Next" button has attribute disabled
, we should stop - we reach the end. Thus we use recursion based on the button's attribute.
1 | beforeEach(() => { |
Let's see the solution in action:
📺 You can watch this solution derived in the free lesson Lesson n2: Table pagination solution of my 🎓 Cypress Plugins course.
Of course, we could have used a different syntax to implement the same test. For example, we could get the attribute disabled
using Cypress cy.invoke
command and get the Next button explicitly again after checking if we are on the last page:
1 | function maybeClickNext() { |
The above solutions work correctly when the page starts on the last page.
Right now our Cypress Command Log looks pretty busy.
Let's clean it up. We can hide the intermediate commands in the recursive loop, limiting the Log to "Page N" and "Last page!" messages. We need to pass the current page number to the recursive function.
1 | function maybeClickNext(page = 1) { |
Tip: cy.invoke command puts the options in the first argument, since the method you are calling might take an unknown number of arguments.
The Log looks so much better now
The above solution is a recursive one. Thus we can write it even simpler using my plugin cypress-recurse.
1 | import { recurse } from 'cypress-recurse' |
The test runs as we expect.
You can find the derivation of the solution in the lesson Lesson n3: Pagination using cypress-recurse which I plan to make public in October.
We can better represent "if the button is enabled, click on it" using cypress-if plugin.
1 | // https://github.com/bahmutov/cypress-if |
You can find the derivation of the solution in the lesson Lesson n5: Pagination using cypress-if which I plan to make public in October.
We can write "normal" JavaScript code by adding async / await
support to Cypress using my cypress-await preprocessor. Just set it up in the cypress.config.js
file
1 | const { defineConfig } = require('cypress') |
And use the await
keyword before getting value from the page, like the disabled
attribute
1 | beforeEach(() => { |
You can find the derivation of the solution in the lesson Lesson n6: Paginate using the await keyword which I plan to make public in October.
Using cypress-await plugin is cool, but writing await cy....
everywhere is noisy. The plugin includes another spec file preprocessor that lets you not write await
in front of every cy
command.
1 | const { defineConfig } = require('cypress') |
Look at the simplicity in this spec
1 | beforeEach(() => { |
Simple and powerful. You can find the derivation of the solution in the lesson Lesson n7: Paginate using synchronous code which I plan to make public in October.
Let's use while
loop and JavaScript DOM methods to interact with the elements on the page. First, we get the button using cy.document command and document.querySelector
. Then we can send mouse event "click" to the button until the DOM element gets disabled.
1 | beforeEach(() => { |
📺 Watch this solution explained in the video Table Pagination Solution Using Plain DOM Methods
cy.should
solutionI will post a lesson with each solution derivation on my 🎓 Cypress Plugins course. At first the lesson will be private. After a week, I will make the lesson public and will upload the video to my YouTube channel.
🎁 You can find the example application shown in this blog post in the repo bahmutov/taste-the-sauce-test-ids. We will use plugin bahmutov/changed-test-ids to determine which specs to run.
Our application is a typical React web app. We have pages and components, and we use different test IDs to select elements on the page during testing.
1 | ... |
The above CheckOutStepOne
component has testId
attribute "lastName". When the application runs, this becomes the HTML attribute data-test="lastName"
.
We use Cypress specs to go through the application. Right now we have only a few high level specs
1 | cypress/ |
In the specs, I used two Cypress custom commands to find elements by data-test
attribute. These two commands are similar to cy.get and cy.contains commands. These commands select HTML elements on the page using the attribute or attribute plus element's text.
1 | // find element(s) by data-test attribute |
Here is a typical checkout test using these custom commands cy.getByTestId
and cy.containsTestId
1 | // part of the checkout spec |
Right now all tests are passing.
Let's say we open a pull request where we modify the CheckOutStepOne.jsx
source file just a bit.
1 | - placeholder="Last name" |
You can find this change in the pull request #3. Which specs should we run? Do we need to run the login-form.cy.js
and logout.cy.js
specs?
This is where the small utility changed-test-ids comes handy. I have installed it as a dev dependency
1 | $ npm i -D changed-test-ids |
We can use this utility to parse both JSX and Cypress specs to find data test attributes used in both files. For example, let's see all test IDs in the source files:
1 | $ npx find-ids --sources 'src/**/*.jsx' |
The changed-test-ids
installs NPM bin script find-ids
. The above command parses all src/**/*.jsx
source files to find all testId="..."
props and then outputs them.
What about our Cypress specs? We use custom commands cy.getByTestId
and cy.containsTestId
, so let's parse the specs looking for those commands and report all collected ids.
1 | $ npx find-ids --specs 'cypress/e2e/**/*.cy.js' \ |
So our specs matching the glob pattern cypress/e2e/**/*.cy.js
use 8 data test attributes. There are more test ids in the source files than in the specs, so some elements are not directly selected by our E2E specs. We can warn the user about this by using both --sources
and --specs
parameters and combining the above two commands:
1 | $ npx find-ids --sources 'src/**/*.jsx' \ |
Cool, let's use this test ID information to pick Cypress specs to run based on the test IDs in the changed source files in the pull request. I am using GitHub Actions, and I can extract the test IDs from the sources changed between the current branch and the main
branch.
1 | name: pr |
The step Find Cypress specs for changed source files
is crucial.
1 | - name: Find Cypress specs for changed source files |
It looks at the Git information, finds the changed source files, finds the test IDs used in those source files, and then picks only the specs that use these test IDs.
In the changed source file CheckOutStepOne.jsx
there are multiple testId
attributes: cancel
, continue
, firstName
, lastName
, etc. We have detected only one spec that covers at least some of them: checkout.cy.js
. This is the spec we should run. By using --set-gha-outputs
the output list of specs and the number of specs is saved in the GitHub Actions outputs values. We can then pass it to the cypress-io/github-action
step to run those specs only:
1 | - name: Run any detected Cypress specs |
Compared to running all specs, we saved some time, even on this tiny project :)
Beautiful.
I have described how the same plugin changed-test-ids
can be used to pass the detected test ids to pick tests to run, even if the tests reside in another repo. Read the blog post Pick Tests Using Test Ids From Another Source Repo.
To compute the changes between the current code and the default repository branch, we can only check out the current code and the default branch main
1 | - name: Checkout the current merge commit 🛎️ |
To compute the difference, remove the --parent
parameter
1 | - name: Find test ids used in the changed source files |
If you want to see the verbose debug logs, add DEBUG: changed-test-ids
environment variable tothe find-ids
step
1 | - name: Find test ids used in the changed source files |
1 | $ ls -la source/_posts/*.md | wc -l |
Instead of the testing pyramid, the posts form a category pyramid with a very wide base.
Here are a examples from each category:
The very first blog post was about JSHint. Here it is
Looking back those 2013 posts were about testing, quality, updating dependencies - things I still care and learn about today. I still write mostly for myself, because I forget how to do things otherwise. I am glad people find the blog useful and visit it more than a million times a month.
Here are the posts I myself consult a lot, either directly or via site: glebbahmutov.com/blog
search.
Tip: for anything I wrote or recorded regarding Cypress, I recommend searching via https://cypress.tips/search page. I scrape the content myself, so it finds pretty good results.
I found the following trick to make my posts really useful even after many years have passed. I keep updating them. You can see "Update 1: ...", "Update 2: ...", etc in some blog posts, like Picking JavaScript testing framework. Sometimes I use headings "Bonus 1: ...", "Bonus 2: ...", like in Trying Lighthouse post.
My second trick is to always link the source repo to the blog post. And of course, I automate dependency updates, so that the repo still works hopefully.
Well, I can't stop now, can I? I will keep writing on topics related mostly to software quality. And sometimes I will write about climate. Ohhh, and I got to add a dark color theme to this site, it is pretty hard to read at night.
]]>Now that I no longer have A/C units on the roof, my system is absolutely quiet, just air moving. No more noise or vibration on top of my bedroom. The system is pretty much silent, even the outside compressors are almost impossible to hear even standing next to them. The entire install took 2 days and was done by the Boston company SumZero Energy Systems. They have done a marvelous job removing my old system carefully and fitting the new one. I found them through the EnergySage website - they were one of the companies that contacted me, sent a person to evaluate the house and gave me a quote. I definitely recommend SumZero to everyone.
The entire system with installation cost me $28k. Massachusetts gives $10k rebate for the complete electric installation. The remaining $18k was financed by a long-term 5+ interest free Mass Save loan via a participating bank. With inflation, such loan is a very good deal. Monthly bills remain to be seen, but I have seen an estimate that today in Massachusetts modern heat pumps would be 13% cheaper to use than gas to heat a well-insulated house. Regardless, I do enjoy the quiet climate comfort plus knowing that I am not burning fossil gas to heat my house.
I would love to electrify my water heater and replace the fossil gas stove with an induction stove. Unfortunately, I have maxed out the electrical capacity running into my house. If I want to run an induction stove plus have an electrical water heater, I would need to upgrade the power line feeding into my house plus upgrade the fuse box. This is currently very expensive.
For more about electrifying United States, see this website https://www.rewiringamerica.org/
]]>🎁 You can find my example project in the repo bahmutov/cypress-image-to-cloud and its recorded test runs at Cypress Cloud project.
Imagine we take an image "hello.png" below and want to upload it to the Cloud.
Our first solution will read the image contents and then attaches it to the document using a temporary <img src="data..."
element. We can use the bundled jQuery under Cypress.$
to write the test.
1 | it('stores an image in Cypress cloud', () => { |
The <img>
appears and disappears, but its element screenshot is uploaded by the cy..screenshot('hello', { overwrite: true })
command
A better solution in my opinion can avoid creating an image element just to upload it. We can use Cypress after screenshot API to "replace" the screenshot taken with a path to our image. It does not matter what screenshot we take. I like using the entire runner because it is faster since it avoids taking multiple screenshots to be stitched together.
1 | const { defineConfig } = require('cypress') |
The corresponding spec needs to take a screenshot named "spec2-hello" to trigger our Hello image upload
1 | it('stores an image in Cypress cloud by returning path', () => { |
The details of the image are shown in the terminal output
Note: each cy.screenshot
command provides onAfterScreenshot
callback, but you cannot replace the image path in that callback, unfortunately.
You can find the results of recording these tests on the project's Cloud page. Go to the "Specs" view and click on the screenshots thumbnail buttons.
Click on the screenshot button to see your image.
Hope you find some use for this trick.
]]>await
keyword in front of every Cypress command. This complaint comes up again and again: why do I need to use cy.then(callback)
to work with the data from the application? Why can't I just do what other test runners do:1 | // I WANT TO DO THIS!!!! |
I have written that how Cypress declarative approach might be simpler and shorter to write. I have recorded videos on how to avoid pyramid of Doom of callbacks. Nothing really lowers people's frustration. They want await cy...
really badly.
Ok then. Let's do this. Cypress lets you use your own spec file preprocessor to bundle or transpile the test code. I wrote cypress-await that uses Babel to automatically transpile testing specs that use await cy...
into cy....then(callback)
1 | // your code |
The code still retries using the built-in assertions like cy.get
and the entire query, so you are not losing anything using this plugin. There is even a "sync" mode that allows you to skip writing await
in front of every Cypress command and simply get the data variable = cy...
Nice!
Just like any Cypress plugin following the README instructions.
1 | const { defineConfig } = require('cypress') |
You can even transpile some spec files using a minimatch or file path suffix. For example, the majority of our spec files might be "plain" Cypress standard, while only some using await
keyword.
1 | cypress/ |
We can set up the specPattern
to transpile only files that end with .await.cy.js
string
1 | setupNodeEvents(on, config) { |
You can debug the transpiled source to see what happens under the hood:
1 | setupNodeEvents(on, config) { |
The terminal will show the transpiled spec source code:
Hmm, one of the attractive parts of Cypress is its declarative syntax. Putting await
in front of every Cypress command just to remove it seems useless. The only time we really want the value is when we assign the result of executing a Cypress command to a variable. Thus my plugin cypress-await has another "sync" preprocessor. You use it the same way: point Cypress file preprocessor at the "sync" preprocessor.
1 | const { defineConfig } = require('cypress') |
Now you can write Cypress specs the way you probably wanted to write it all this time:
1 | it('checks the server using cy.request (sync)', () => { |
The plugin transforms every variable = cy....
assignment into cy....then(variable => ...)
and moves all statements after the assignment into the callback.
1 | // "sync" spec file |
I have a few examples of using the plugin in my bahmutov/cypress-todomvc-await-example repo. It is almost too boring:
Spying on the network request to know how many items to find on the page
1 | // network.sync.cy.js |
Comparing items on the page
1 | // spec.sync.cy.js |
📺 Watch video Await Cypress Command Results
]]>Note this slide is from my presentation The Shape Of Testing Pyramid shown at AssertJS in 2018. The founders of Cypress.io Brian Mann and Randall Kent were sitting front row during the presentation.
If we could somehow avoid cy.visit(url)
and instead use cy.mount(<MyComponent withProp={...} />)
to "start" the component for framework X, then we could apply all Cypress command and interact with the component in the full browser just like a real user would. No more "scratchpad" dev pages showing individual custom Date components, etc. Just a browser running your component and the test runner clicking, asserting, spying, intercepting network requests, etc.
Here is an example component test from my blog post My Vision for Component Tests in Cypress.
1 | import React from "react" |
Notice how we can simply pass stubs and spies and it just works? I also made a mount
function for each major framework: React, Vue, Svelte, etc. Angular was difficult, but other Angular developers pushed it over the finish line and it worked.
I made the very first commit to my bahmutov/cypress-react-unit-test
repo on December 30, 2017. The first working React component test was made New Year's eve on Dec 31st, 2017. Yup, I was pretty excited about it.
Even more specific - the first commit was made almost at midnight at 11:48 PM EST. I should have waited for the strike of midnight clock, Cinderella-style.
The first test that worked happened next day Dec 31st, 2018
1 | import { HelloWorld } from '../../src/hello-world.jsx' |
Of course, it was "Hello World!"
The original component testing was a separate Cypress plugin. Good thing I love writing Cypress plugins. Later Dmitriy Kovalenko who worked at Cypress.io twice helped figure out how to mount the components in the same iframe as the specs, as opposed to the E2E tests that Cypress mounts in a separate iframe. This made things like stubs and spies work, thanks Dmitriy 🙏
Note: I worked a lot on Cypress component testing outside the normal business hours. When I left the company I transferred whatever repos they asked (cypress-react-unit-test
, cypress-vue-unit-test
, cypress-grep
, etc) to the Cypress organization. My plugins had code coverage built-in, stubbing of ES module imports, and other things that I thought were important to have. Since I left Cypress.io I have not worked on these plugins or any Cypress code really, except for reporting found issues.
Cypress company incorporated the component testing into the main test runner. You can read the official Cypress component testing documentation here.
I initially named these libraries "cypress-X-unit-test" because in principle you mount the component and interact with it like you would test a small piece of code. Only instead of arguments, you send user commands like clicks. Instead of the result you read the result from the page or other behaviors.
1 | // "normal" unit test |
And here is a unit React component test
1 | it('logs user in', () => { |
But I am not too attached to the naming. Everyone pretty much objected to the "unit test" part, and the official name became "Component Testing".
I always thought that interacting with the component through the DOM like the real user is the best testing strategy. But sometimes you need to access the internal state of the component. This is where I wrote bahmutov/cypress-react-app-actions but this works just as well with end-to-end tests that access a React we app.
The timeline of events is how I remember and can reconstruct from the code and blog posts and presentations. If you think I am incorrect, please let me know and bring receipts.
Brian Mann responds that he and Chris talked about component testing and even tried it out in the test runner in 2016.
To better see, this is the included Slack message
Lovely. Except it is something they discussed and it never went anywhere. "The company greenlit the decision" is also weird to say. Of course, once I showed it again and again and again it was decided to make it part of the test runner. Again, everyone can see the 480 commits I made by myself with my best friend Renovate into the repo cypress-io/cypress-react-unit-test. Zero Brian, zero Chris until October 13, 2020.
What is really weird in Brian's response is the focus on "It was my idea, even discussed during funding". Never talked to me, I never saw those funding proposals. I would understand "Gleb, your code was bad, it was a prototype, we had to rewrite it all to make Component Testing the first-class feature". I would totally get it, no problem. But the current response... is just so weird for an open-source tool.
Here is something else I realized: you can check out the author
of any NPM package either by looking online at package.json
file or by using npm info <package name> author
command. Let's take a look, shall we?
Over these couple of years I tried to promote Cypress component testing a lot because I truly believe it is a good and powerful testing solution. Here are my blog posts, videos, presentations, and even courses that show why and how to do component testing.
Imagine a very simple application:
Simple. If everything goes well, the end-to-end test can be
1 | cy.contains('button', 'load') |
🎁 You can find the example application and the tests in my collection of Cypress examples.
Everything goes well. We do not know when the "loading..." message switches to "Loaded", but the test relies on Cypress retry-ability to keep checking the page until it find the text "Loaded"
1 | // even if it takes 3.9 seconds |
If the application might take 10 seconds to show the "Loaded" message, we can increase the retry-ability timeout:
1 | // keep trying to find the text "Loaded" on the page |
The most common situation for adding a timeout
parameter to a Cypress command like cy.contains
is to increase the time limit. But there are cases when we want to shrink the timeout period to zero or just a few milliseconds instead.
Imagine that our application instead of successfully loading the data, for some reason shows an unexpected warning message, and then completes successfully.
Hmm, notice the test still passed. We do not care about the unexpected "Something went wrong" message, we are looking for the "Loaded" text, and it eventually appear. How do we "catch" the expected elements? We cannot use negative assertions because A) we do not know all possible unexpected elements and B) we cannot control the timing when these elements appear to know when to check. Here is an updated test that still does not "notice" the warning:
1 | // 🚨 DOES NOT WORK |
We need something better. We need to really "shrink" the time between the "Loading..." element going away and the "Loaded" text appearing. Then any other element trying to sneak in will be caught.
Here is what we can do. We can confirm when the "Loading..." text goes away.
1 | cy.contains('button', 'load') |
Note: despite having "not.exist", cy.contains('button', 'Loading...').should('not.exist')
is an example of a positive assertion. We check that something happens in response to an action.
Ok, so we know the "Loading..." text is gone. Now we are looking to see the "Loaded" text there immediately.
1 | // the "Loading..." element goes away |
Beautiful - if there are no extra elements, the test passes. Now let's see what happens when a warning message does "insert" itself.
By setting the timeout: 0
we tell Cypress to really shrink the retries limit to make sure no unexpected element can sneak in.
You can watch this blog post explained in the video below
The user can enter the message and see it on the page.
The initial code is really simple; we take the textarea contents and add a new list item element.
1 | send$.addEventListener('click', () => { |
We can confirm the messages are added by writing a Cypress end-to-end test
1 | // https://github.com/bahmutov/cypress-slow-down |
Tip: I am using my plugin cypress-slow-down to add small pauses after each Cypress command, otherwise the video goes way too quickly.
The app looks great. Except people ask to be able to format messages. Let's allow bold and italics using HTML tags <b>
and <i>
. Ughh, time is short, so let's use HTML for this! We modify our app to use .innerHTML =
instead of .innerText =
, quickly and powerful.
1 | send$.addEventListener('click', () => { |
Let's test adding bold messages.
1 | it('adds a bold message', () => { |
Ok, we heard about hackers, so let's make sure these bad bad no-goodniks cannot simply add script code to our list of messages.
1 | it('injecting <script> tag does not work', () => { |
All good, the console.log
is NOT running at all, the browser does not treat the appended <script>
tag as code and does not execute it.
We are safe and sound.
Our nightmares are interrupted by a wake up call. Someone stole all user information by injecting JavaScript into their messages. How did they do it? Turns out there are a lot of places where the browser WILL execute JavaScript. Styles, attributes, CSS import directives... Here is what the hackers did:
1 | it('injects XSS via img onerror attribute', () => { |
Ughh, time to move to Canada and grow our own vegetables.
Our hackers entered text data and the browser simply executed it as code. In an ideal world, the browser would only execute code from our ./app.js
script, or at least from our website, as listed in the public/index.html
1 | <head> |
We want to disable all inline scripts and all inline JavaScript attributes, this would stop script injections. This is where the ](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) comes into play. We can tell the browser the list of allowed places to load JavaScript (and other things) from. For example, we could allow scripts to load from the domain itself (called "self") and from specific CDN servers and from nowhere else. Here is how to specify this using helmet Node.js plugin for Express.js
1 | const express = require('express') |
When we load our site in the browser, we see the CSP header
By default, Cypress strips this header to be able to inject its own inline script directive. But we can tell Cypress to preserve it and instead modify it to inject each specific script using a nonce.
1 | const { defineConfig } = require('cypress') |
We can verify the CSP header is present in the returned page using Cypress cy.request
command (an API test!!!!)
1 | it('serves Content-Security-Policy header', () => { |
If the CSP header is NOT present, then the XSS attack can succeed
1 | it('can strip CSP and allow injections', () => { |
What happens if the CSP policy header is present and the hacker tries to use the <img onerror="..." />
trick? Let's try in the regular browser.
Our CSP policy successfully prevented the browser from executing the script added by the hacker. Not only that, the XSS attack was reported as a CSP policy violation to the /security-attacks
endpoint on our server. We do not even have this endpoint yet, but that's ok - stopping the attack is the top priority. Let's verify the CSP policy works using a Cypress E2E test. We can even intercept the POST /security-attacks
network call and verify what the browser sends is correct.
1 | it('stops XSS and reports CSP violations', () => { |
Nice!
Note: if you are thinking "I will sanitize the user input to only allow <b>
and <i>
tags to be safe" - it is a very hard problem. CSP and whitelisting script sources is a much safer approach to security. At least use a hardened library like cure53/DOMPurify to sanitize the user input before inserting it in to the page.
cy.intercept
and cy.request
commands? Take my Cypress Network Testing Exercises course.cypress-slow-down
.Here is the JSX markup for the above page.
1 | import { PatternFormat, NumericFormat } from 'react-number-format' |
Simple, isn't it? The entered text is formatted according to the props. The formatted values are set as the current value attribute of the input elements.
Let's confirm the application is working as expected and indeed formats the phone number and the dollar amount.
First, I will write an end-to-end test. I install Cypress and cypress-real-events because nothing is real without it.
1 | $ npm i -D cypress cypress-real-events |
1 | // https://github.com/dmtrKovalenko/cypress-real-events |
The test enters raw text and verifies the react-number-format
logic formats the values correctly.
The react-number-format
components have a lot of features. We might need to investigate how a component behaves if the user tries to enter invalid information. For example, the price input field should ignore multiple .
, not allow entering negative numbers, and strip the leading zeroes. Let's write a test to confirm it.
1 | it('ignores invalid characters in price input', () => { |
Notice a curious behavior: the price input ignores the duplicate .
and all digits after the two decimals. But it does not strip the leading zero until the element loses its focus.
Let's describe this behavior in our test to avoid accidentally changing it in the future. We can use cy.blur command to remove focus from an element. Since we are working with the same DOM element <input data-cy=price ... />
we can simply chain our commands and assertions into a single chain.
1 | it('ignores invalid characters in price input', () => { |
Another unhappy test could verify the phone number input formatter removes all characters but 10 digits.
1 | it('ignores invalid characters in phone input', () => { |
Beautiful.
End-to-End tests are powerful. They check the entire user flow. But what about learning how a component works? In our app we are using the `` prop that should notify the parent component about the formatted user values. Does it work? Is it safe to upgrade the react-number-format
version?
1 | <NumericFormat |
Here is where the component tests come into play. Let's configure Cypress component testing; there is nothing to install, since it uses Vite.js and React already used by the application itself.
Step 1: change from E2E to Component testing type
Step 2: confirm the detected framework and bundler options
Step 3: create a new component spec
Here is my first test. It simply mounts the component to play with.
1 | import { NumericFormat } from 'react-number-format' |
In the component tests we can simply pass a Cypress Sinon stub function as the onValueChange
prop.
1 | import { NumericFormat } from 'react-number-format' |
We can see the stub function called 6 times. The first call is when we cleared the input element using cy.clear
command. Then we types 5 characters: '5', '0', '.', '9', and '9'. For each character, the component called our onValueChange
prop. Let's confirm each call.
1 | import { NumericFormat } from 'react-number-format' |
It is a little bit verbose, but we are checking quite a few separate calls.
We can make it shorter using cypress-map to extract all first arguments at once.
1 | import { NumericFormat } from 'react-number-format' |
It is up to you to decide which test variant is more readable.
We can simplify the above test. By using cy-spok assertion we can check properties from multiple objects in an array of calls at once. Here is the relevant assertion:
1 | // https://github.com/bahmutov/cy-spok |
I think it looks nicer than the deep.equal
or individual assertions.
📺 You can watch me code this example in the video Verify Cypress Component Prop Calls Using Stubs, cypress-map, and cy-spok.
]]>