process

Visual diffing flow for your pretty CLI applications

How to use Percy.io visual diffing service to prevent regressions in terminal programs.

Note: you can find the example source code in repo bahmutov/percy-for-cli-example.

I have been a big fan of visual testing for web applications. Since reliable rendering of web pages and comparing images is complicated, I am a big fan of just using an existing 3rd party service like Percy.io and Applitools. These services are fast, reliable, and just work.

I especially like the pull request review workflow. If a visual service detects a difference between the "gold" images it stores and the newly generated ones, it sets a failed commit check, and I know there are visual regressions. Anyone from the team can review the visual changes, comment, understand them. If the changes are really expected, the new images can be approved and become the new "gold" images to be compared against.

So that is all good and fun for web applications, but recently we have been refactoring CLI output from the Cypress Test Runner - and it is complicated. Our current output renders tables with multiple columns, uses terminal colors, text padding and alignment, hmm. We are using text snapshots, but they strip colors, and are relatively hard to judge or discuss as a team.

I want the same workflow for unit tests or CLI applications that output complex information to the terminal. That's why I have created this experiment:

  1. Capture CLI output from an app, for example from a test runner
  2. Convert ANSI control characters that set foreground and background colors to matching HTML styles. There are many small NPM utilities that do this.
  3. Send the generated HTML to Percy.io API
  • Percy thinks this came from a real DOM snapshot
  • it renders the terminal HTML in a real browser
  • generates an image and compares it to the "gold"

Easy peasy.

Here is the "standard" Mocha run in the terminal.

Mocha test run

Here is how we can spawn Mocha from a parent process, yet force colors

1
2
3
4
5
6
7
8
9
10
11
12
13
// force child process to output ANSI colors
// if possible using FORCE_COLOR
// commonly used via https://github.com/chalk/supports-color
// const child = spawn('node', ['./colors'], {
const spawn = require('child_process').spawn
const child = spawn(
  'node',
  ['./node_modules/.bin/mocha', './spec.js', '--reporter', 'spec'],
  {
    env: { ...process.env, FORCE_COLOR: '2' },
    cwd: process.cwd(),
  },
)

Here is how we convert ANSI characters to HTML using ansi-to-html

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// assuming the browser page is white
const options = {
  newline: true,
  bg: '#fff',
  fg: '#111',
}
const convert = new (require('ansi-to-html'))(options)

let html = ''

const htmlStream = function htmlStream(stream) {
  return stream.on('data', function(chunk) {
    html += convert.toHtml(chunk)
  })
}
child.stdout.setEncoding('utf8')
child.on('error', console.error)
child.stdout.on('end', () => {
  // send result to Percy
})
htmlStream(child.stdout)

The final HTML can be wrapped in <html> with utf8 meta flag and will look something like this

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
  </head>
  <body>
    <br /><br />
    example<br />
    <span style="color:#0A0"> ✓</span><span style="color:#555"> works A</span
    ><span style="color:#A00"> (1002ms)</span><br />
    <span style="color:#0A0"> ✓</span><span style="color:#555"> works B</span
    ><span style="color:#A00"> (1005ms)</span><br />
    <span style="color:#0A0"> ✓</span><span style="color:#555"> works C</span
    ><span style="color:#A00"> (1002ms)</span><br />
    <span style="color:#0AA"> - skips D</span><br /><br /><br /><span
      style="color:#5F5"
    >
    </span
    ><span style="color:#0A0"> 3 passing</span
    ><span style="color:#555"> (3s)</span><br /><span style="color:#0AA"> </span
    ><span style="color:#0AA"> 1 pending</span><br /><br />
  </body>
</html>

We can send this HTML to Percy API by running a local Percy agent, which runs by default on port 5338 and posting the generated HTML there. Here is how to run Percy and demo project.

package.json
1
2
3
4
5
6
7
8
9
10
11
12
13
{
  "scripts": {
    "demo": "node ./index",
    "demo-percy": "npx percy exec -- npm run demo"
  },
  "dependencies": {
    "ansi-to-html": "0.6.11",
    "axios": "0.19.0"
  },
  "devDependencies": {
    "@percy/script": "1.0.0",
  }
}

Sending is just a POST request

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// post HTML to the Percy agent
// follow "cy.request" code in
// https://github.com/percy/percy-cypress/blob/master/lib/index.ts
// and https://github.com/percy/percy-agent
const url = 'http://localhost:5338/percy/snapshot'
axios
  .post(url, {
    name: 'my example name',
    url: 'http://localhost/example',
    enableJavaScript: false,
    domSnapshot: html,
  })
  .catch(e => {
    console.error(e)
    throw e
  })

Percy API requires a private project token, I will inject it during run-time using as-a utility.

1
$ as-a percy-for-cli-example npm run demo-percy

The scripts runs and sends its terminal to visual diffing service as HTML string.

Demo run

You can find Percy dashboard for this project at percy.io/bahmutov/percy-for-cli-example. All images uploaded from master branch are auto-approved, while images uploaded from other branches are compared to the "gold" images. For example the last build has 1 image with detected visual difference.

Percy project dashboard

Someone from the team will have to go to the build and review the visual changes and either approve or reject them. In this case the difference is just in millisecond numbers - our snapshot needs to be sanitized before sending to Percy to avoid flagging trivial changes like this.

Percy shows visual difference in the terminal output