process

Visual testing for React components using open source tools

Comparing React components pixel by pixel to catch style problems

Let's take a look at a modern React application like this Sudoku game that you can play online at https://sudoku-raravi.now.sh/.

Talks

I have covered this topic in a JSNationLive 2020 presentation

Video (20 minutes)

Later I have shown an expanded version of this talk at Des Moines meetup

Video (1 hour video)

Sudoku game

The game is nicely done. There are different difficulty levels, game modes and it looks very polished.

It is a well-designed web application with responsive styles for 3 different browser widths

The application is made out of React components, which we can see in the src folder, or by using React DevTools. There is <App /> and <Game /> components, and lots of smaller components matching the sections of the user interface.

Implementation

Let's look at the code. Note: you can find my fork of the game at bahmutov/sudoku, which includes all code shown in this blog post.

src/index.js
1
2
3
4
5
import React from 'react';
import { render } from 'react-dom';
import { App } from './App';

render(<App />, document.getElementById('root'));

The App component creates the React context object, imports the application's styles and creates the Game component

src/App.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import React from 'react'
import { Game } from './Game'
import './App.css'
import { SudokuProvider } from './context/SudokuContext'

/**
 * App is the root React component.
 */
export const App = () => {
  return (
    <SudokuProvider>
      <Game />
    </SudokuProvider>
  )
}

The Game component is much larger source file, because it brings all the logic together

src/Game.js
1
2
3
4
5
6
7
8
9
10
11
12
import React, { useState, useEffect } from 'react'
import moment from 'moment'
import { Header } from './components/layout/Header'
import { GameSection } from './components/layout/GameSection'
import { StatusSection } from './components/layout/StatusSection'
import { Footer } from './components/layout/Footer'
import { getUniqueSudoku } from './solver/UniqueSudoku'
import { useSudokuContext } from './context/SudokuContext'

export const Game = () => {
  ...
}

The Game component renders all the components shown above, passing props that receive user events

src/Game.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
return (
  <>
    <div className={overlay?"container blur":"container"}>
      <Header onClick={onClickNewGame}/>
      <div className="innercontainer">
        <GameSection
          onClick={(indexOfArray) => onClickCell(indexOfArray)}
        />
        <StatusSection
          onClickNumber={(number) => onClickNumber(number)}
          onChange={(e) => onChangeDifficulty(e)}
          onClickUndo={onClickUndo}
          onClickErase={onClickErase}
          onClickHint={onClickHint}
          onClickMistakesMode={onClickMistakesMode}
          onClickFastMode={onClickFastMode}
        />
      </div>
      <Footer />
    </div>
  </>
)

Numbers component

Among individual smaller components there is Numbers component that you can use to enter numbers into the grid.

Numbers component

The Numbers component receives its inputs from the parent component in two ways: via a context and via props.

src/components/Numbers.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import React from 'react';
import { useSudokuContext } from '../context/SudokuContext';

/**
 * React component for the Number Selector in the Status Section.
 */
export const Numbers = (props) => {
  let { numberSelected } = useSudokuContext();
  return (
    <div className="status__numbers">
      {
        [1, 2, 3, 4, 5, 6, 7, 8, 9].map((number) => {
          if (numberSelected === number.toString()) {
            return (
              <div className="status__number status__number--selected"
                key={number}
                onClick={() => props.onClickNumber(number.toString())}>{number}</div>
            )
          } else {
            return (
              <div className="status__number" key={number}
                onClick={() => props.onClickNumber(number.toString())}>{number}</div>
            )
          }
        })
      }
    </div>
  )
}

For example, the currently selected number (if any) is grabbed from the context.

1
let { numberSelected } = useSudokuContext();

The click handler is grabbed from the props argument and used to send the clicked number back to the parent component.

1
2
3
4
5
export const Numbers = (props) => {
  ...
  <div className="status__number" key={number}
    onClick={() => props.onClickNumber(number.toString())}>{number}</div>
}

If we want to refactor the above component, how do we ensure that we don't break it? How do we ensure that it renders in exactly the same way, and that when the user clicks, the number is sent to the parent component? How do we write component tests?

In my view, the component receives its inputs, the props and the context, and generates some output. The output is both the DOM nodes the component renders, and the props.onClickNumber invocations on click. Let's make sure the component works this way.

cypress-react-unit-test

To test the React components I will install cypress-react-unit-test. This adaptor allows the Cypress test runner to mount React components like little mini web applications and then use the full Cypress API to interact with them. Our application uses react-scripts, thus the setup is trivial

cypress/support/index.js
1
require('cypress-react-unit-test/support')
cypress/plugins/index.js
1
2
3
4
module.exports = (on, config) => {
  require('cypress-react-unit-test/plugins/react-scripts')(on, config)
  return config
}
cypress.json
1
2
3
4
{
  "experimentalComponentTesting": true,
  "componentFolder": "src"
}

Let's write our first test.

src/components/Numbers.spec.js
1
2
3
4
5
6
7
8
9
10
11
import React from 'react'
import { mount } from 'cypress-react-unit-test'
import { Numbers } from './Numbers'
describe('Numbers', () => {
  it('shows all numbers', () => {
    mount(<Numbers />);
    [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
      cy.contains('.status__number', k)
    })
  })
})

Open Cypress with yarn cypress open or npx cypress open and run the test. There is no need to start the application, because we are working with an individual component, not with a page.

Hmm, the test passes, all the numbers are there. But the component does not look right!

First test to check all numbers are present

I think it is very important to see the component to understand what the users will experience. Our component needs styles - and the styles are in the src/app.css file. Because we point the cypress/plugins/index.js at the react-scripts settings, the spec files will use the same loaders and bundlers as the application code. Which means we can import app.css from the spec file too:

src/components/Numbers.spec.js
1
2
3
import { Numbers } from './Numbers'
import '../App.css'
...

Include App.css in the test file

This is better, but still not exactly the same. Since the CSS assumes certain document structure, we need to surround our <Numbers/> with elements with specific class names.

src/components/Numbers.spec.js
1
2
3
4
5
6
7
mount(
  <div className="innercontainer">
    <section className="status">
      <Numbers />
    </section>
  </div>
)

This looks much better - now we see what the user is going to see.

Numbers component styled during test

Super. We can also confirm that the component calls the prop onClickNumber when the user clicks a number in the DOM:

src/components/Numbers.spec.js
1
2
3
4
5
6
7
8
9
10
11
it('reacts to a click', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers onClickNumber={cy.stub().as('click')}/>
      </section>
    </div>
  )
  cy.contains('.status__number', '9').click()
  cy.get('@click').should('have.been.calledWith', '9')
})

In the test above we pass cy.stub and confirm it was called with expected number 9 when the user clicks on the DOM element.

Confirm the click happens

How does our component look when there is a selected number? In order to test this, we need to wrap the <Numbers /> component with a mock context provider.

src/components/Numbers.spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import {SudokuContext} from '../context/SudokuContext'
describe('Numbers', () => {
  it('shows selected number', () => {
    mount(
      <SudokuContext.Provider value={{ numberSelected: '4' }} >
        <div className="innercontainer">
          <section className="status">
            <Numbers />
          </section>
        </div>
      </SudokuContext.Provider>
    )
    cy.contains('.status__number', '4').should('have.class', 'status__number--selected')
  })
})

The number 4 does have the expected class status__number--selected and we do see the color difference.

Selected number 4

Ok, but do we need to look at the tests to tell if the component looks correctly? Do we need a human in the loop? What if we change the App.css file and accidentally break some other component? This game has its own polished style, it would be a shame to break it accidentally.

Visual testing

If we want to confirm the look of the component, we could assert every computed CSS property of every DOM node ... but that would be unbelievably brittle and hard to maintain. Instead, we could render the component into an image and look at it. While computers are not very good (yet) at understanding images, they are really good at comparing them. So let's generate screenshots of our component with different inputs and store those images with the code. Any time there is a pull request, we will repeat the above steps and then compare the new images pixel by pixel with saved good images.

There are many commercial services that do this, but in this blog post I will use open source image comparison plugin for Cypress called cypress-image-snapshot. We can install it as a dev dependency and include its files in the support and plugins files.

cypress/support/index.js
1
2
require('cypress-react-unit-test/support')
require('cypress-image-snapshot/command').addMatchImageSnapshotCommand()
cypress/plugins/index.js
1
2
3
4
5
module.exports = (on, config) => {
  require('cypress-react-unit-test/plugins/react-scripts')(on, config)
  require('cypress-image-snapshot/plugin').addMatchImageSnapshotPlugin(on, config)
  return config
}

Now we get a new Cypress command cy.matchImageSnapshot(<snapshot name>). Let's use it to create image snapshot of the Number component with selected digit "4".

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
it('shows selected number', () => {
  mount(
    <SudokuContext.Provider value={{ numberSelected: '4' }} >
      <div className="innercontainer">
        <section className="status">
          <Numbers />
        </section>
      </div>
    </SudokuContext.Provider>
  )
  cy.contains('.status__number', '4')
    .should('have.class', 'status__number--selected')
  cy.get('.status__numbers')
    .matchImageSnapshot('numbers-selected')
})

In the test we have confirmed the component has been rendered using an assertion

1
2
cy.contains('.status__number', '4')
  .should('have.class', 'status__number--selected')

Then we create an image with the entire .status__numbers DOM element rendered.

1
2
cy.get('.status__numbers')
  .matchImageSnapshot('numbers-selected')

The snapshots are saved in cypress/snapshots folder.

Numbers component with selected number 4

If we edit the CSS file src/App.css and change the text padding around the numbers like this:

1
2
3
4
.status__number {
-   padding: 12px 0;
+   padding: 10px 0;
}

The test runs - and the generated snapshot (under the hood cypress-image-snapshot uses cy.screenshot) is slightly different. It is hard to notice, so the pixel by pixel comparison done by the computer is perfect tool for automating it. The cypress-image-snapshot plugin produces the diff image in cypress/snapshots/**/__diff_output__ folder. The diff image has three parts: on the left is the original baseline image. On the right is the current image. In the middle there is a composite image highlighting the difference.

Padding difference is caught by image comparison

The visual snapshots replace a lot of individual assertions. For example, there is no need to confirm that individual numbers are present in the DOM. Instead we can confirm the number of DOM elements with the class (to make sure the component has rendered) and take a single snapshot.

src/components/Numbers.spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
it('shows all numbers', () => {
  mount(
    <div className="innercontainer">
      <section className="status">
        <Numbers />
      </section>
    </div>
  );
-  // trying to assert every number in the DOM
-  [1, 2, 3, 4, 5, 6, 7, 8, 9].forEach(k => {
-    cy.contains('.status__number', k)
-  })
+ cy.get('.status__number').should('have.length', 9)
+ cy.get('.status__numbers').matchImageSnapshot('all-numbers')
})

A single image snapshot can confirm so much - as long as the components render consistently from the same data. Let's look how to ensure it.

Controlling the clock

Inside the app, we have a timer that shows elapsed time since the start of the game

Game timer

src/App.spec.js
1
2
3
4
5
6
7
import { App } from './App'
it('shows the timer', () => {
  mount(<App />)
  // the timer starts at zero, so this is probably ok
  cy.contains('.status__time', '00:00')
    .matchImageSnapshot('timer-zero')
})

Hmm, how do we make sure we take consistent snapshot of a timer element that keeps changing? By controlling the clock using cy.clock and cy.tick.

src/App.spec.js
1
2
3
4
5
6
7
8
9
10
import { App } from './App'
it('shows the timer', () => {
  cy.clock()
  mount(<App />)
  cy.contains('.status__time', '00:00')
    .matchImageSnapshot('timer-zero')
  cy.tick(700 * 1000)
  cy.contains('.status__time', '11:40')
    .matchImageSnapshot('timer-passed')
})

Because we explicitly control the app's clock, the timer stays frozen at "00:00" when we take the first snapshot, and stays frozen at "11:40" when we take the second snapshot 700 seconds later. A nice bonus for this technique is that we can test an application with long actions in a blink of an eye.

Controlling the clock test

Deterministic board

If we can test the smaller components, why can't we mount the entire <App> and take an image snapshot? That would give us a lot of confidence.

src/App.spec.js
1
2
3
4
5
6
import { App } from './App'
it('shows the board', () => {
  mount(<App />)
  // DOES NOT WORK, JUST FOR DEMO
  cy.get('.container').matchImageSnapshot('the-game')
})

Unfortunately the above test would not work. We cannot just take a snapshot - the board is generated randomly every time the game starts. Every new board will produce a new image, breaking the test.

Random board

Let's look how the game generates the board data. The <App> component has the <Game> child component, which imports a function to generate the board.

src/Game.js
1
2
3
4
5
6
import { getUniqueSudoku } from './solver/UniqueSudoku'
...
function _createNewGame(e) {
  let [temporaryInitArray, temporarySolvedArray] = getUniqueSudoku(difficulty, e);
  ...
}

To generate the same board, we will stub the ES6 import getUniqueSudoku. I have intercepted the results from getUniqueSudoku() call using the DevTools and saved the two arrays as two JSON fixture files.

1
2
3
4
// cypress/fixtures/init-array.json
["0", "0", "9", "0", "2", "0", "0", ...]
// cypress/fixtures/solved-array.json
["6", "7", "9", "3", "2", "8", "4", ...]

Plugin cypress-react-unit-test comes with a Babel plugin that can mock ES6 module imports, and for any application that uses react-scripts it should work automatically. Let mock the getUniqueSudoku import and return the arrays loaded from the fixture files.

src/App.spec.js
1
2
3
4
5
6
7
8
9
10
11
12
import { App } from './App'
import * as UniqueSudoku from './solver/UniqueSudoku'
it('shows the board', () => {
  cy.fixture('init-array').then(initArray => {
    cy.fixture('solved-array').then(solvedArray => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.container').matchImageSnapshot('the-game')
})

The above test always generates the same board, freezes the clock at "00:00" and generates a consistent snapshot image. Building on top of this approach, we can write a test that plays a move, since we have a deterministic board to start with.

src/App.spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import { App } from './App'
import * as UniqueSudoku from './solver/UniqueSudoku'
it('plays one move', () => {
  cy.fixture('init-array').then(initArray => {
    cy.fixture('solved-array').then(solvedArray => {
      cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
    })
  })
  cy.clock()
  mount(<App />)
  cy.get('.game__cell').first().click()
  // we can even look at the solved array!
  cy.contains('.status__number', '6').click()
  cy.get('.game__cell').first()
    .should('have.class', 'game__cell--highlightselected')
  cy.get('.container').matchImageSnapshot('same-game-made-one-move')
})

Test that makes a move

While a screenshot is nice, the power of Cypress comes from its full browser experience and time-traveling debugging feature. The video below shows what every test command does as we hover it.

Tip: if a lot of tests load the same fixture, you can load it once and store in a closure variable.

src/App.spec.js
1
2
3
4
5
6
7
8
9
10
11
12
let initArray
let solvedArray
before(() => {
  cy.fixture('init-array').then(arr => initArray = arr)
  cy.fixture('solved-array').then(arr => solvedArray = arr)
})
it('plays one move', () => {
  cy.stub(UniqueSudoku, 'getUniqueSudoku').returns([initArray, solvedArray])
  cy.clock()
  mount(<App />)
  ...
})

Read blog post Import Cypress fixtures for details.

Local workflow

We can write visual tests for our components and generate consistent images by mocking data and the clock. Is this enough to use visual testing in our day-to-day life?

No.

If we take a visual snapshot in Cypress GUI opened with yarn cypress open on Mac we will get an image with a specific resolution, for example 1300x600 pixels. When we take the same snapshot in the headless mode using yarn cypress run we will get an image with only the half resolution in each dimension: 650x300 pixels. This is due to pixel density of the graphical application vs headless rendering mode. Thus I disable snapshots in the interactive mode from the support file.

cypress/support/index.js
1
2
3
4
5
6
7
8
9
import { addMatchImageSnapshotCommand } from 'cypress-image-snapshot/command';
if (Cypress.config('isInteractive')) {
  Cypress.Commands.add('matchImageSnapshot', () => {
    cy.log('Skipping snapshot 👀')
  })
} else {
  addMatchImageSnapshotCommand()
}
require('cypress-react-unit-test/support')

When the test is running, the Command Log shows the places where the snapshots would be taken.

Skipped snapshots

To really take a snapshot, I execute yarn cypress run, inspect any new snapshot files, add them to the source control and push the code to the remote repository.

If I need to update a saved image snapshot, I can delete the snapshot image and execute yarn cypress run to save new image. Or I can run Cypress with environment variable telling the cypress-image-snapshot to update every snapshot if there is difference.

1
$ CYPRESS_updateSnapshots=true yarn cypress run

Note: make sure to inspect every changed snapshot before committing, since all snapshots can be updated during the above command.

Continuous Integration

Even when saving image snapshots in the headless mode we can encounter problems. For example, if we save the image snapshots on Mac and compare them to the images generated on Linux CI, there will be tiny pixel differences due to font rendering, aliasing, and different browser versions.

Mac vs Linux font rendering

The image above shows the detected pixel differences between 00:00 rendered on Mac (left) and on Linux (right). Even these tiny differences still fail the test.

We can configure the cypress-image-snapshot command to accept a small percentage of different pixels per image.

1
2
3
4
5
6
7
// when adding matchImageSnapshot command
addMatchImageSnapshotCommand({
  failureThreshold: 0.03, // threshold for entire image
  failureThresholdType: 'percent', // percent of image or number of pixels
  customDiffConfig: { threshold: 0.1 }, // threshold for each pixel
  capture: 'viewport', // capture viewport in screenshot
});

I do not trust such thresholds, since they can miss actual visual problems. Instead I recommend generating image snapshots using exactly the same environment as the one on CI. This is simple to do using Docker containers from cypress-docker-images. Instead of running the command yarn cypress run to generate or update snapshots, we will use a Docker image locally.

package.json
1
2
3
4
5
6

{
  "scripts": {
     "docker:run": "docker run -it -v $PWD:/e2e -w /e2e cypress/included:4.5.0"
  }
}

Because the component tests do not require a server, we can run them with a single command using the pre-installed Cypress Docker image cypress/included:<version>. On CI we can use a Docker with exactly the same OS dependencies, fonts and browser version. Here is an example CI config

1
2
3
4
jobs:
  cypress-run:
    runs-on: ubuntu-latest
    container: cypress/browsers:node12.13.0-chrome80-ff74

The Docker image cypress/included:4.5.0 is built from cypress/browsers:node12.13.0-chrome80-ff74 image with Cypress globally installed. Thus the browser should render the same HTML pages in exactly the same way.

Pull request workflow

Let's discuss what happens with image snapshots during pull requests. When someone opens a PR with a new feature or a bug fix, it is beneficial to separate the functional tests from the visual tests. If a button has changed its appearance, the functional tests should pass, and the visual tests should fail. This will tell us quickly if there are visual differences due to layout, color, or style changes.

For this purpose, bahmutov/sudoku runs tests on CI with an environment variable that tells the cypress-image-snapshot plugin to generate image diffs, but not fail the command matchImageSnapshot.

1
2
3
4
5
- name: Cypress run 🧪
  uses: cypress-io/github-action@v1
  with:
    # let's go through the tests and generate all diffs
    env: failOnSnapshotDiff=false

At the end of the run we can find these diff images and report them separately using GitHub commit status check.

scripts/set-gh-check.js
1
2
3
4
5
6
async function getVisualDiffs() {
  return globby('cypress/snapshots/**/__diff_output__/**.png')
}
getVisualDiffs().then(list => {
  return setGitHubCommitStatus(list.length, envOptions)
}).catch(onError)

If there are any DIFF images, then the status check will be fail. The video below shows a pull request where at first there are visual changes. Then the tests are updated and the next commit has no visual differences.

Whenever there are visual differences, you can inspect the diff images if you store them on CI as test artifacts; the snapshots are stored in cypress/snapshots folder.

1
2
3
4
5
6
7
8
9
- name: Store snapshots 📸
  uses: actions/upload-artifact@v1
  with:
    name: snapshots
    path: cypress/snapshots

- name: Set commit status 🖼
  run: |
    node ./scripts/set-gh-check.js

Summary

In this blog post we have looked at writing React component tests and using visual diffing to compare the current image against the good baseline image. We must generate the same image, which we can do by mocking the data and the clock to be deterministic. We also need to ensure that the snapshots are generated using exactly the same environment to be pixel-perfect matches.

Finally, we briefly looked at the visual tests during pull request workflow. Looking back at the entire work, I conclude:

  • writing React component tests using cypress-react-unit-test is fast and easy
  • cypress-image-snapshot plugin allows one to do visual testing for free, but we have to do a lot of work to manage images, render them, and review them during pull requests

I would strongly recommend giving a commercial visual testing service a try: Applitools, Happo.io, and Percy.io have Cypress plugins and are free to try and cheap use.

More info

Take a look at the example Sudoku repository, and especially at the long list of short videos there: bahmutov/sudoku#videos. These videos show every part of the process I have described in this blog in more detail.

You should also read the Cypress visual testing guide.