Testing how an application renders a drawing with Cypress and Percy.io

Note: you can find the source code for this blog post at github.com/cypress-io/angular-pizza-creator and a live demo of the application at toddmotto.com/angular-pizza-creator/.

Note 2: the webinar video has been posted at www.youtube.com/watch?v=MXfZeE9RQDw and the slides at slides.com/bahmutov/visual-testing-with-percy.

Table of contents

• Ordering a pizza
• Custom commands
• Visual testing
• Visual testing workflow
• Conclusions

Ordering a pizza

There is a nice little web application at toddmotto.com/angular-pizza-creator/ for ordering a pizza. It does not actually order pizza, but it surely looks appetizing!

When we click on a topping, the order changes price, and the pizza rendering gets a new set of sliced toppings dropped. If we are building a web application like this, how do we test it?

First, we need to ensure that the we can build the pizza we want and that the order is going to cost us the right amount. We can write such test using a functional end-to-end test runner Cypress.io. Our first test will check the following user story:

• user needs to enter delivery information before they can place an order
• user needs to pick at least one topping before they can place an order
• the total order price should be correct
• when the order is placed, a window alert pops up with the message "Order placed"

At the end the web application looks like this

Here is our first cypress/integration/order-spec.js

/// <reference types="Cypress" />
context('Pizza Creator', () => {
  beforeEach(() => {
    // uses base url setting from cypress.json
    // which right now points at "localhost:3000"
    cy.visit('/')
  })

  it('orders custom pizza', function () {
    // enter delivery information
    cy.get('[formcontrolname="name"]').type('Joe')
    cy.get('[formcontrolname="email"]').type('[email protected]')
    cy.get('[formcontrolname="confirm"]').type('[email protected]')

    // without complete delivery information,
    // we should not be able to place the order
    cy.get('button[type="submit"]').should('be.disabled')

    cy.get('[formcontrolname="address"]').type('1 Pizza st')
    cy.get('[formcontrolname="postcode"]').type('12345')
    cy.get('[formcontrolname="phone"]').type('1234567890')

    // still cannot order pizza - need to pick toppings
    cy.get('button[type="submit"]').should('be.disabled')

    // add a few toppings
    cy.contains('label.pizza-topping', 'Pepperoni').click()
    cy.contains('label.pizza-topping', 'Onion').click()
    cy.contains('label.pizza-topping', 'Mozzarella').click()
    cy.contains('label.pizza-topping', 'Basil').click()

    // check the price and order pizza
    cy.contains('.pizza-summary__total-price', 'Total: $12.75')

    // let us confirm we can place our order,
    // but first, prepare for "window.alert" call
    cy.on('window:alert', cy.stub().as('alert'))

    // now the button should be enabled
    cy.get('button[type="submit"]')
      .should('be.enabled')
      .click()
    cy.get('@alert').should('have.been.calledWithExactly', 'Order placed')

    // scroll pizza view back into view
    cy.get('form')
      .scrollIntoView({})
      .should('be.visible')
  })
})

which passes locally

Custom commands

If we plan to write more tests, entering delivery and picking toppings actions will quickly lead to lots of duplicate test code. We can factor them out to custom commands, making our test code more readable and dry. I will write the following into cypress/support/index.js file:

cypress/support/index.js

/** Returns selector for a form control using name attribute */
const f = name => `[formcontrolname="${name}"]`

Cypress.Commands.add('enterForm', (name, text) => {
  // enter text into the form control without Command Log messages
  const quiet = { log: false }
  cy.get(f(name), quiet).type(text, quiet)
})

Cypress.Commands.add('enterDeliveryInformation', () => {
  cy.enterForm('name', 'Joe')
  cy.enterForm('email', '[email protected]')
  cy.enterForm('confirm', '[email protected]')
  cy.enterForm('address', '1 Pizza st')
  cy.enterForm('postcode', '12345')
  cy.enterForm('phone', '1234567890')
})

Cypress.Commands.add('pickToppings', (...toppings) => {
  toppings.forEach(name => {
    cy.contains('label.pizza-topping', name).click()
  })
})

The support file is bundled with each spec file, thus my cypress/integration/dry-spec.js can immediate use the new Cypress commands.

cypress/integration/dry-spec.js

it('orders custom pizza', function () {
  cy.enterDeliveryInformation()
  cy.pickToppings('Pepperoni', 'Onion', 'Mozzarella', 'Basil')

  // check the price and order pizza
  cy.contains('.pizza-summary__total-price', 'Total: $12.75')

  // let us confirm we can place our order,
  // but first, prepare for "window.alert" call
  cy.on('window:alert', cy.stub().as('alert'))

  // now the button should be enabled
  cy.get('button[type="submit"]')
    .should('be.enabled')
    .click()
  cy.get('@alert').should('have.been.calledWithExactly', 'Order placed')

  // scroll pizza view back into view
  cy.get('form')
    .scrollIntoView({})
    .should('be.visible')
})

Beautiful, I ❤️ readable functional tests.

Visual testing

Hmm, we can verify the order side of the application - but what about the beautiful animated pizza drawing? Are the toppings falling onto the pizza crust - or do they accidentally land outside the circle? And what if someone changes the pie from the mouth-watering #FFDC71 to much less appetizing #71FF71?

A functional test cannot catch all possible changes in style, color and position - there are just too many assertions to make. Instead we need to compare the result as an image - and we need to compare it to a "good" baseline image. As long as the images match and the functional tests pass, our pizza app is working.

When dealing with images, we need to think where the baseline images are going to be stored - they quickly become a nuisance as their number grows. Think how many binary images can a Git repository hold until it becomes a nightmare to clone.

Also a huge problem with image diffing is the process of reviewing them and approving the visual changes. I would prefer to have an online service that shows me and other team members the differences in a nice convenient manner. I don't want to manually download images from CI to view them!

We need a complete solution, so today I will look at Percy.io visual diffing service. I have signed up for free with my GitHub account and created a project percy.io/cypress-io/angular-pizza-creator that you can see for yourself. My setup follows the Percy Cypress tutorial.

In my project I have added @percy/cypress development dependency.

yarn add --dev @percy/cypress
...
success Saved 1 new dependency.
info Direct dependencies
└─ @percy/[email protected]

Then I have added a single line to my support file

cypress/support/index.js

import '@percy/cypress'
// the rest of my custom commands

The imported @percy/cypress adds its own custom command cy.percySnapshot(). I write a test that snapshots the pizza before adding any topics and after in visual-spec.js:

cypress/integration/visual-spec.js

it('draws pizza correctly', function () {
  cy.percySnapshot('Empty Pizza')

  cy.enterDeliveryInformation()
  const toppings = ['Pepperoni', 'Chili', 'Onion']
  cy.pickToppings(...toppings)

  // make sure the web app has updated
  cy.contains('.pizza-summary__total-price', 'Total: $12.06')
  cy.percySnapshot(toppings.join(' - '))

  // scroll pizza view back into view
  cy.get('form')
    .scrollIntoView({})
    .should('be.visible')
})

The test runs locally.

Just remember: the test should take a snapshot when the application has finished rendering; and not before. The web app might take a while to redraw - maybe it is sending data to the server, or processing a complex operation. Adding an assertion is usually enough to wait as long as necessary, but no longer, thanks to the retry-ability built into most Cypress commands. To prevent the snapshot from being taken too early, like before the toppings have been applied to the order, the above test uses cy.contains:

// make sure the web app has updated
cy.contains('.pizza-summary__total-price', 'Total: $12.06')
cy.percySnapshot(toppings.join(' - '))

I can ignore additional Percy messages in the Command Log - because I have not set up sending data for image diffing yet. In fact, I will not run image diffing locally - there is no need for it, due to asynchronous nature of image generation and comparison. Percy custom command just sends DOM snapshot and styles to Percy cloud, where the actual images are generated (across multiple browsers and resolutions) and then compared. In order to enable sending images, I need to change how I run Cypress. Usually one runs Cypress by simply executing npx cypress open or npx cypress run in headless mode. But when Percy runs it needs extra time - to send the DOM snapshots and styles to the Percy.io API. Thus I need to run Percy app, which will start Cypress and will make sure the image diffing starts, even if Cypress application finishes. The command should be:

npx percy exec -- cypress run

I don't need to change anything in my package.json file - because normally I just work with functional tests. Only my CI configuration file needs to change its test command to run Cypress through Percy. Since I almost always use Cypress CircleCI Orb to run my end-to-end tests, here is my circle.yml file.

circle.yml

version: 2.1
orbs:
  # import Cypress orb by specifying an exact version x.y.z
  # or the latest version 1.x.x using "@1" syntax
  cypress: cypress-io/cypress@1
workflows:
  build:
    jobs:
      # "cypress" is the name of the imported orb
      # "run" is the name of the job defined in Cypress orb
      - cypress/run:
          yarn: true
          # builds and starts the local application
          start: yarn setup && yarn start
          # waits for web application to load completely
          wait-on: 'http://localhost:3000'
          # runs the Cypress tests via "Percy exec"
          command: npx percy exec -- cypress run --record

Note: I am recording the Cypress test results and video using cypress run --record on the Cypress Dashboard, which is a separate service from Percy web application dashboard.

For results to be sent to the right Percy project, I grabbed the PERCY_TOKEN from the Percy web application and set it on CircleCI as an environment variable. Then I pushed my code. CircleCI runs build #13 which shows Percy start message:

Percy outputs messages to the terminal when snapshots are taken:

After the entire run, Percy application sends the request to generate images and compare them and exits:

I can open the displayed url https://percy.io/cypress-io/angular-pizza-creator/builds/1663756 and after a few seconds the "Pending ..." status changes to "Unreviewed". There are two desktop screenshots (and two more generated using Firefox) that I can see.

I approve the changes - now these snapshots become the official baseline images that will be used in the future for comparisons. All images are stored in the Percy cloud, and do not clutter the project's GitHub repository.

Let me change the pizza crust color to green and try pushing the commit. CircleCI build passes, and Percy web application shows that there are new changes - it has detected the change in color.

We can go into "baseline vs current image" view and toggle diff to see where the colors have changed.

Perfect, Percy web application catches the visual difference - but our tests have passed, haven't they?

Visual testing workflow

Visual tests with Percy do not fail Cypress tests. Instead they send the DOM snapshots and all page styles to the Percy cloud where

• the actual images will be generated on multiple browsers and resolutions
• new images are compared against baseline images

A project could have 100s of images, waiting for all of them in the Cypress test might mean a loooong test. Thus Percy suggests a different asynchronous workflow it its "How it works" guide.

1. Install Percy GitHub application github.com/marketplace/percy and link the project to the GitHub repository. This enables commit status reporting.

Percy recommends using pull requests to make any changes to the code. By default Percy project settings has the master branch as auto-approved. I had it turned off before to show image diffs, but now I will turn it back on.

2. For functional and visual changes I will open a pull request. Each pull request runs functional tests AND Percy sends back image diffing results as a GitHub status check. For example angular-pizza-creator/pull/2 automatically gets 2 commit checks:

3. Clicking on the failed Percy check "details" link brings me to the diff view:

Thus each pull request needs the functional tests to pass and for the team to review and approve the visual changes (if it makes sense) - and there could be 100s of visual changes triggered across all part of the project, even for a small style change!

4. If I click "Approve" button in Percy, it changes the GitHub commit status to pass and my pull request is good to go.

The pull request was merged into master and the new approved images become the new baseline images

The entire process is simple and convenient.

Conclusions

Running both functional and visual tests gives me a peace of mind. The chances of accidentally breaking the page layout or hiding an element, or making the app look hideous goes pretty much to zero. If you would like to know more about visual testing with Cypress.io and Percy.io check out these links:

• percy.io
• www.cypress.io
• Percy Cypress documentation

There are also a few open source alternatives for visual diffing that do not have the GitHub integration or the cloud component that Percy provides, check them out if you would like to do image diffing yourself: on.cypress.io/plugins#visual-testing