Canvas Visual Testing with Retries

How to visually compare a rendered Canvas with retry-ability

Imagine you are drawing something into HTML5 Canvas element. You want to confirm that the application is working, so you compare the drawing with a previously saved good image. This blog post shows how to do this using Cypress with a bonus feature meant to solve visual flake: the visual diffing is retried until the rendering matches the saved image (or the command times out).

The smiley face

The example application in the bahmutov/monalego repo uses Legra.js to draw a smiley face. The face is stretching its smile

Animated canvas smile

The script to draw the face is below

public/smile.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
import Legra from 'https://unpkg.com/legra?module'

const ctx = document.querySelector('canvas').getContext('2d')
const legra = new Legra(ctx)

let direction = 1
let smileY = 12
const draw = () => {
ctx.clearRect(0, 0, 480, 480)
legra.circle(10, 10, 8, { filled: true, color: 'yellow' })
legra.rectangle(6, 6, 2, 2, { filled: true, color: 'green' })
legra.rectangle(13, 6, 2, 2, { filled: true, color: 'green' })
legra.polygon(
[
[10, 8],
[8, 11],
[12, 11],
],
{ filled: true, color: 'red' },
)

legra.quadraticCurve(5, 13, 10, smileY, 15, 13)
}

function drawNext() {
setTimeout(() => {
smileY += direction
draw()
if (smileY >= 20) {
// the last slide
return
}
drawNext()
}, 150)
}

drawNext()

We want to confirm the final rendering - the full smile, so we have saved the PNG at the end of the animation. How do we confirm the smile is rendered from now on?

Saving the canvas

First, let's save the canvas as a PNG image we can compare against the good image. We can convert the HTML5 canvas to data URL and then save the base64 string on disk using cy.writeFile command.

cypress/integration/utils.js
1
2
3
4
5
6
7
8
9
10
11
12
export const downloadPng = (filename) => {
expect(filename).to.be.a('string')

// the simplest way is to grab the data url and use
// https://on.cypress.io/writefile to save PNG file
return cy.get('canvas').then(($canvas) => {
const url = $canvas[0].toDataURL()
const data = url.replace(/^data:image\/png;base64,/, '')
cy.writeFile(filename, data, 'base64')
cy.wrap(filename)
})
}
cypress/integration/smile-spec.js
1
2
3
4
5
6
7
8
import { downloadPng } from './utils'

describe('Lego face', () => {
it('saves canvas as an image', () => {
cy.visit('/smile')
cy.wait(4000)
downloadPng('good-smile.png')
})

The file is saved - note we had to wait for 4 seconds to be safe the animation has fully finished and the canvas has finished rendering.

Cypress test saved the canvas after 4 seconds

I have moved the saved PNG file into images/darwin/smile.png file to be used as good "gold" image for the comparison.

Saved PNG image

Let's write the visual test.

Visual test using odiff

Our test should download the canvas as a PNG image and compare the new image against the saved good image. For fast comparison I will use odiff module from my co-worker Dmitriy Kovalenko @dmtrKovalenko. Since this is a Node operation I will call odiff from the plugins file.

cypress/plugins/index.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
const { compare } = require('odiff-bin')
const path = require('path')
const os = require('os')

const osName = os.platform()

/**
* @type {Cypress.PluginConfig}
*/
module.exports = (on, config) => {
// `on` is used to hook into various events Cypress emits
// `config` is the resolved Cypress config
on('task', {
async compare({ filename, options }) {
const baseFolder = 'images'
const baseImage = path.join(baseFolder, osName, path.basename(filename))
const newImage = filename
const diffImage = 'diff.png'
console.log(
'comparing base image %s to the new image %s',
baseImage,
newImage,
)
if (options) {
console.log('odiff options %o', options)
}
const started = +new Date()

const result = await compare(baseImage, newImage, diffImage, options)
const finished = +new Date()
const elapsed = finished - started
console.log('odiff took %dms', elapsed)

console.log(result)
return result
},
})
}

Here is our test that calls the image comparison using cy.task

cypress/integration/smile-spec.js
1
2
3
4
5
6
7
8
9
10
11
it('smiles broadly with wait', () => {
cy.visit('/smile')
cy.wait(4000)

downloadPng('smile.png').then((filename) => {
cy.log(`saved ${filename}`)
cy.task('compare', { filename }).should('deep.equal', {
match: true,
})
})
})

Visual test confirms the smile is the same

Meanwhile we can see the terminal output

1
2
3
comparing base image images/darwin/smile.png to the new image smile.png
odiff took 37ms
{ match: true }

The test must wait

Can we speed up the test? Can we remove the cy.wait(4000)? Well, if we remove it completely, the test fails - because the canvas has not rendered at all

Zero wait - the canvas is blank during the comparison

Here is the diff PNG image the odiff comparison generates showing in red the pixels different from the good image.

Zero wait - the pixel difference

Maybe we can only wait one second? No, the test still fails

The test fails if we wait only one second

The visual difference image shows a smiling image, but not enough to match our good image.

Visual difference after one second wait

We see the mismatched red pixels around the mouth - our smile is bad!

Visual flake

In general, "catching" the page at the right moment so that all its pixels are updated turns out to be tricky and flakey. There could be JavaScript and CSS animations still running, there could be asynchronous data loading, there could be little page alerts and badges suddenly showing up. I have written about debugging a flaky visual regression test and must say - it is no picnic.

Things like a text box that hasn't fully opened are common:

The text box hasn't expanded yet

Or the entire element is missing from the DOM

The DOM section hasn't rendered yet

Or the element is present, but its text hasn't been rendered yet

The element's text is missing

What do we do? How do we fight the visual flake?

Retry-ability

First, let me note that the flake in our case is due to comparing the image prematurely. The good image has been taken after the canvas has completely finished its animation. The good image was "approved" by the human user, thus we know it is good. Our test has a problem - it takes an image too soon.

A similar problem happens in E2E testing very often - the Cypress Test Runner cannot know when an element is ready to pass an assertion, thus it just retries again and again. Eventually the element changes its state and the assertion passes. If the assertion never passes, no matter how many times the command retries, then the command times out, and the test fails. This Cypress feature is called retry-ability and it is the keystone to fighting the test flake.

Here is a typical test that relies on the retrying the command until the assertion passes

1
2
cy.get('.todo-list li')     // command
.should('have.length', 2) // assertion

Here is the retry-ability in action. Notice how the application inserts the new todo items after a noticeable delay - maybe the items have to be saved at a remote server first, thus they appear in the DOM only after a few seconds.

The command is retried until the assertion passes

The command is retried until the page reaches the desired state. The page is similar to the smiling face, the page is changing - and the test keeps "grabbing" the elements and checking if there are two of them. Once there are two elements found, the test is done.

Visual retry-ability

The solution to visual flake is to retry getting the canvas and comparing it to the good image. Getting the canvas is similar to getting the DOM elements - it is an idempotent operation that is safe to run again and again. Our only challenge is to implement the retries using the same commands we used during the visual test above; the Cypress commands cy.writeFile and cy.task are NOT retried by default.

Luckily, I have written a small reusable module called cypress-recurse for recursively calling any series of Cypress commands until a predicate becomes true. You can read and watch how I coded it up in the Writing Cypress Recurse Function tutorial.

In our case the test is so simple, the code speaks for itself:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
it('smiles broadly', () => {
cy.visit('/smile')

recurse(
// a function with Cypress commands to perform
() => {
return downloadPng('smile.png').then((filename) => {
cy.log(`saved ${filename}`)
return cy.task('compare', { filename })
})
},
// a predicate function
({ match }) => match,
)
})

The test calls the first function that performs the Cypress commands. This function downloads the canvas PNG and calls the cy.task('compare'). The task yields an object with a property match. Our predicate function simply grabs that property. Once the match is true the visual test is done - the two images matched.

Visual retries using cypress-recurse and odiff

Of course, we can turn off such verbose logging. But here I kept it on to show how every iteration of saving the canvas and image comparison takes about 100-150ms. The odiff visual comparison only takes 25ms for a 480x480 pixel image - the rest of the time is taken by saving the image and the Cypress command overhead.

You can see how I am writing the above test in this video and find the full project at bahmutov/monalego

Of course we can refactor the code and make it into a reusable utility function:

1
2
3
4
5
import { looksTheSame } from './utils'
it('looks the same', () => {
cy.visit('/smile')
looksTheSame('smile.png')
})

Waiting for static canvas

One optimization we can do to avoid relatively expensive visual image retries, is to wait for the canvas to become static (unchanging) first, before doing the image comparison with the file on disk. An image diffing module like pixelmatch can compare canvas pixels right in the browser. Thus we can check the canvas against itself after N milliseconds. If there are pixel differences, then the canvas is still changing. We need to wait then try again. If the images match - the canvas is static and can be compared to an image on disk.

The utility function can still use the cypress-recurse module to implement such in-browser image diffing

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
import { recurse } from 'cypress-recurse'
import pixelmatch from 'pixelmatch'

function ensureCanvasStatic(selector = 'canvas') {
cy.log(`ensure the image in **${selector}** is static`)
const noLog = { log: false }

const delay = 300 // ms, when grabbing new image

// take the current image
return cy
.get(selector, noLog)
.then(($canvas) => {
const ctx1 = $canvas[0].getContext('2d')
const width = $canvas[0].width
const height = $canvas[0].height
let img1 = ctx1.getImageData(0, 0, width, height)
console.log('canvas is %d x %d pixels', width, height)

// initial delay to make sure we catch updates
cy.wait(delay, noLog)

return recurse(
// "work" function
() => {
return cy.get(selector, noLog).then(($canvas) => {
const ctx2 = $canvas[0].getContext('2d')
const img2 = ctx2.getImageData(0, 0, width, height)

const diff = ctx2.createImageData(width, height)
// number of different pixels
const number = pixelmatch(
img1.data,
img2.data,
diff.data,
width,
height,
{
threshold: 0.1,
},
)
console.log(number)

// for next comparison, use the new image
// as the base - this way we can get to the end
// of any animation
img1 = img2

return number
})
},
// predicate function
(numberOfDifferentPixels) => numberOfDifferentPixels < 10,
// recurse options
{
// by default uses the default command timeout
log: (numberOfDifferentPixels) =>
cy.log(`**${numberOfDifferentPixels}** diff pixels`),
delay,
},
)
})
.then(() => {
cy.log(`picture in **${selector}** is static`)
})
}

A typical test would call the ensureCanvasStatic function first, then the function looksTheSame:

1
2
3
4
5
6
7
8
9
10
11
12
13
import { looksTheSame } from './utils'

function ensureCanvasStatic(selector = 'canvas') {
...
}

it('can wait for canvas to become static', () => {
cy.visit('/smile')
ensureCanvasStatic('canvas')
// after this it is pretty much guaranteed to
// immediately pass the image diffing on the 1st try
looksTheSame('smile.png', false)
})

The GIF below shows the dwindling number of different pixels. As it reaches 0, the canvas finished animation and is ready to be compared against the baseline image on disk.

Waiting for canvas to finish animation

Happy Visual Testing!