Canvas Visual Testing with Retries

Imagine you are drawing something into HTML5 Canvas element. You want to confirm that the application is working, so you compare the drawing with a previously saved good image. This blog post shows how to do this using Cypress with a bonus feature meant to solve visual flake: the visual diffing is retried until the rendering matches the saved image (or the command times out).

• The smiley face
• Saving the canvas
• Visual test using odiff
• The test must wait
• Visual flake
• Retry-ability
• Visual retry-ability
• Waiting for static canvas

The smiley face

The example application in the bahmutov/monalego repo uses Legra.js to draw a smiley face. The face is stretching its smile

The script to draw the face is below

public/smile.js

import Legra from 'https://unpkg.com/legra?module'

const ctx = document.querySelector('canvas').getContext('2d')
const legra = new Legra(ctx)

let direction = 1
let smileY = 12
const draw = () => {
  ctx.clearRect(0, 0, 480, 480)
  legra.circle(10, 10, 8, { filled: true, color: 'yellow' })
  legra.rectangle(6, 6, 2, 2, { filled: true, color: 'green' })
  legra.rectangle(13, 6, 2, 2, { filled: true, color: 'green' })
  legra.polygon(
    [
      [10, 8],
      [8, 11],
      [12, 11],
    ],
    { filled: true, color: 'red' },
  )

  legra.quadraticCurve(5, 13, 10, smileY, 15, 13)
}

function drawNext() {
  setTimeout(() => {
    smileY += direction
    draw()
    if (smileY >= 20) {
      // the last slide
      return
    }
    drawNext()
  }, 150)
}

drawNext()

We want to confirm the final rendering - the full smile, so we have saved the PNG at the end of the animation. How do we confirm the smile is rendered from now on?

Saving the canvas

First, let's save the canvas as a PNG image we can compare against the good image. We can convert the HTML5 canvas to data URL and then save the base64 string on disk using cy.writeFile command.

cypress/integration/utils.js

export const downloadPng = (filename) => {
  expect(filename).to.be.a('string')

  // the simplest way is to grab the data url and use
  // https://on.cypress.io/writefile to save PNG file
  return cy.get('canvas').then(($canvas) => {
    const url = $canvas[0].toDataURL()
    const data = url.replace(/^data:image\/png;base64,/, '')
    cy.writeFile(filename, data, 'base64')
    cy.wrap(filename)
  })
}

cypress/integration/smile-spec.js

import { downloadPng } from './utils'

describe('Lego face', () => {
  it('saves canvas as an image', () => {
    cy.visit('/smile')
    cy.wait(4000)
    downloadPng('good-smile.png')
  })

The file is saved - note we had to wait for 4 seconds to be safe the animation has fully finished and the canvas has finished rendering.

I have moved the saved PNG file into images/darwin/smile.png file to be used as good "gold" image for the comparison.

Let's write the visual test.

Visual test using odiff

Our test should download the canvas as a PNG image and compare the new image against the saved good image. For fast comparison I will use odiff module from my co-worker Dmitriy Kovalenko @dmtrKovalenko. Since this is a Node operation I will call odiff from the plugins file.

cypress/plugins/index.js

const { compare } = require('odiff-bin')
const path = require('path')
const os = require('os')

const osName = os.platform()

/**
 * @type {Cypress.PluginConfig}
 */
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config
  on('task', {
    async compare({ filename, options }) {
      const baseFolder = 'images'
      const baseImage = path.join(baseFolder, osName, path.basename(filename))
      const newImage = filename
      const diffImage = 'diff.png'
      console.log(
        'comparing base image %s to the new image %s',
        baseImage,
        newImage,
      )
      if (options) {
        console.log('odiff options %o', options)
      }
      const started = +new Date()

      const result = await compare(baseImage, newImage, diffImage, options)
      const finished = +new Date()
      const elapsed = finished - started
      console.log('odiff took %dms', elapsed)

      console.log(result)
      return result
    },
  })
}

Here is our test that calls the image comparison using cy.task

cypress/integration/smile-spec.js

it('smiles broadly with wait', () => {
  cy.visit('/smile')
  cy.wait(4000)

  downloadPng('smile.png').then((filename) => {
    cy.log(`saved ${filename}`)
    cy.task('compare', { filename }).should('deep.equal', {
      match: true,
    })
  })
})

Meanwhile we can see the terminal output

comparing base image images/darwin/smile.png to the new image smile.png
odiff took 37ms
{ match: true }

The test must wait

Can we speed up the test? Can we remove the cy.wait(4000)? Well, if we remove it completely, the test fails - because the canvas has not rendered at all

Here is the diff PNG image the odiff comparison generates showing in red the pixels different from the good image.

Maybe we can only wait one second? No, the test still fails

The visual difference image shows a smiling image, but not enough to match our good image.

We see the mismatched red pixels around the mouth - our smile is bad!

Visual flake

In general, "catching" the page at the right moment so that all its pixels are updated turns out to be tricky and flakey. There could be JavaScript and CSS animations still running, there could be asynchronous data loading, there could be little page alerts and badges suddenly showing up. I have written about debugging a flaky visual regression test and must say - it is no picnic.

Things like a text box that hasn't fully opened are common:

Or the entire element is missing from the DOM

Or the element is present, but its text hasn't been rendered yet

What do we do? How do we fight the visual flake?

Retry-ability

First, let me note that the flake in our case is due to comparing the image prematurely. The good image has been taken after the canvas has completely finished its animation. The good image was "approved" by the human user, thus we know it is good. Our test has a problem - it takes an image too soon.

A similar problem happens in E2E testing very often - the Cypress Test Runner cannot know when an element is ready to pass an assertion, thus it just retries again and again. Eventually the element changes its state and the assertion passes. If the assertion never passes, no matter how many times the command retries, then the command times out, and the test fails. This Cypress feature is called retry-ability and it is the keystone to fighting the test flake.

Here is a typical test that relies on the retrying the command until the assertion passes

cy.get('.todo-list li')     // command
  .should('have.length', 2) // assertion

Here is the retry-ability in action. Notice how the application inserts the new todo items after a noticeable delay - maybe the items have to be saved at a remote server first, thus they appear in the DOM only after a few seconds.

The command is retried until the page reaches the desired state. The page is similar to the smiling face, the page is changing - and the test keeps "grabbing" the elements and checking if there are two of them. Once there are two elements found, the test is done.

Visual retry-ability

The solution to visual flake is to retry getting the canvas and comparing it to the good image. Getting the canvas is similar to getting the DOM elements - it is an idempotent operation that is safe to run again and again. Our only challenge is to implement the retries using the same commands we used during the visual test above; the Cypress commands cy.writeFile and cy.task are NOT retried by default.

Luckily, I have written a small reusable module called cypress-recurse for recursively calling any series of Cypress commands until a predicate becomes true. You can read and watch how I coded it up in the Writing Cypress Recurse Function tutorial.

In our case the test is so simple, the code speaks for itself:

it('smiles broadly', () => {
  cy.visit('/smile')

  recurse(
    // a function with Cypress commands to perform
    () => {
      return downloadPng('smile.png').then((filename) => {
        cy.log(`saved ${filename}`)
        return cy.task('compare', { filename })
      })
    },
    // a predicate function
    ({ match }) => match,
  )
})

The test calls the first function that performs the Cypress commands. This function downloads the canvas PNG and calls the cy.task('compare'). The task yields an object with a property match. Our predicate function simply grabs that property. Once the match is true the visual test is done - the two images matched.

Of course, we can turn off such verbose logging. But here I kept it on to show how every iteration of saving the canvas and image comparison takes about 100-150ms. The odiff visual comparison only takes 25ms for a 480x480 pixel image - the rest of the time is taken by saving the image and the Cypress command overhead.

You can see how I am writing the above test in this video and find the full project at bahmutov/monalego

Of course we can refactor the code and make it into a reusable utility function:

import { looksTheSame } from './utils'
it('looks the same', () => {
  cy.visit('/smile')
  looksTheSame('smile.png')
})

Waiting for static canvas

One optimization we can do to avoid relatively expensive visual image retries, is to wait for the canvas to become static (unchanging) first, before doing the image comparison with the file on disk. An image diffing module like pixelmatch can compare canvas pixels right in the browser. Thus we can check the canvas against itself after N milliseconds. If there are pixel differences, then the canvas is still changing. We need to wait then try again. If the images match - the canvas is static and can be compared to an image on disk.

The utility function can still use the cypress-recurse module to implement such in-browser image diffing

import { recurse } from 'cypress-recurse'
import pixelmatch from 'pixelmatch'

function ensureCanvasStatic(selector = 'canvas') {
  cy.log(`ensure the image in **${selector}** is static`)
  const noLog = { log: false }

  const delay = 300 // ms, when grabbing new image

  // take the current image
  return cy
    .get(selector, noLog)
    .then(($canvas) => {
      const ctx1 = $canvas[0].getContext('2d')
      const width = $canvas[0].width
      const height = $canvas[0].height
      let img1 = ctx1.getImageData(0, 0, width, height)
      console.log('canvas is %d x %d pixels', width, height)

      // initial delay to make sure we catch updates
      cy.wait(delay, noLog)

      return recurse(
        // "work" function
        () => {
          return cy.get(selector, noLog).then(($canvas) => {
            const ctx2 = $canvas[0].getContext('2d')
            const img2 = ctx2.getImageData(0, 0, width, height)

            const diff = ctx2.createImageData(width, height)
            // number of different pixels
            const number = pixelmatch(
              img1.data,
              img2.data,
              diff.data,
              width,
              height,
              {
                threshold: 0.1,
              },
            )
            console.log(number)

            // for next comparison, use the new image
            // as the base - this way we can get to the end
            // of any animation
            img1 = img2

            return number
          })
        },
        // predicate function
        (numberOfDifferentPixels) => numberOfDifferentPixels < 10,
        // recurse options
        {
          // by default uses the default command timeout
          log: (numberOfDifferentPixels) =>
            cy.log(`**${numberOfDifferentPixels}** diff pixels`),
          delay,
        },
      )
    })
    .then(() => {
      cy.log(`picture in **${selector}** is static`)
    })
}

A typical test would call the ensureCanvasStatic function first, then the function looksTheSame:

import { looksTheSame } from './utils'

function ensureCanvasStatic(selector = 'canvas') {
  ...
}

it('can wait for canvas to become static', () => {
  cy.visit('/smile')
  ensureCanvasStatic('canvas')
  // after this it is pretty much guaranteed to
  // immediately pass the image diffing on the 1st try
  looksTheSame('smile.png', false)
})

The GIF below shows the dwindling number of different pixels. As it reaches 0, the canvas finished animation and is ready to be compared against the baseline image on disk.

Happy Visual Testing!