products

Testing a chart with Cypress and Applitools

End-to-end testing an SVG chart using Cypress.io test runner and image diffing Applitools plugin.

Note: you can find the source code for this blog post at github.com/bahmutov/chart-testing-example.

Charts

I have found a great JavaScript library for creating SVG charts github.com/frappe/charts with a demo website at https://frappe.io/charts. Creating a chart is as simple as including a single script tag and writing a little snippet like

1
2
3
4
5
<body>
  <div id="chart"></div>
  <script src="https://unpkg.com/[email protected]/dist/frappe-charts.min.iife.js"></script>
  <script src="app.js"></script>
</body>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// app.js
const data = {
  labels: ['winter', 'spring', 'summer', 'fall'],
  datasets: [
    {
      name: 'Sunny days',
      type: 'bar',
      values: [10, 20, 30, 25]
    }
  ]
}

const chart = new frappe.Chart('#chart', {
  title: 'Sunny days per year',
  data: data,
  type: 'bar',
  height: 250,
  colors: ['#7cd6fd']
})

which produces a bar chart like this:

Bar chart with tooltip on hover

Great, but how do I actually test this chart? How do I ensure that I accidentally do not break my code when upgrading from [email protected] to [email protected]?

Functional tests

First, let's start testing this SVG chart using Cypress.io test runner. I will install cypress, parcel-bundler (to server local site) and start-server-and-test (for starting the server and the tests).

package.json
1
2
3
4
5
6
7
8
9
10
11
12
{
  "scripts": {
    "start": "parcel server index.html",
    "cy:open": "cypress open",
    "dev": "start-test 1234 cy:open"
  },
  "devDependencies": {
    "cypress": "3.2.0",
    "parcel-bundler": "1.12.3",
    "start-server-and-test": "1.7.12"
  }
}

The first test should load the site and assure that the chart is visible

cypress/integration/first-spec.js
1
2
3
4
5
6
7
8
9
10
/// <reference types="Cypress" />
it('shows bar chart', () => {
  cy.visit('localhost:1234')
  cy.get('.frappe-chart')
    .should('be.visible')
    .and(chart => {
      // we can assert anything about the chart really
      expect(chart.height()).to.be.greaterThan(200)
    })
})

We can start Cypress GUI with npm run dev and see the first test pass

First passing test

Do you see delayed data load with animation? It happens after the test has already finished. I would like my test to only finish when all 4 bars are rendered, so I will change my test to find the four SVG rectangles.

cypress/integration/first-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
/// <reference types="Cypress" />
it('shows bar chart', () => {
  cy.visit('localhost:1234')
  cy.get('.frappe-chart')
    .should('be.visible')
    .and(chart => {
      // we can assert anything about the chart really
      expect(chart.height()).to.be.greaterThan(200)
    })
    // let the chart load by observing the rendered bars
    .find('g.dataset-0 rect')
    .should('have.length', 4)
})

The test now waits for the rectangles to appear - and then it finishes.

Waiting for 4 rectangles

Since we are always going to load the chart, let us move the cy.visit and rectangle check to beforeEach hook. Even better, we can move the URL to cypress.json settings file.

cypress.json
1
2
3
4
{
  "baseUrl": "http://localhost:1234",
  "viewportHeight": 400
}
cypress/integration/first-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
/// <reference types="Cypress" />
beforeEach(() => {
  cy.visit('/')
  // let the chart load by observing the rendered bars
  cy.get('.frappe-chart g.dataset-0 rect').should('have.length', 4)
})

it('shows bar chart', () => {
  cy.get('.frappe-chart')
    .should('be.visible')
    .and(chart => {
      // we can assert anything about the chart really
      expect(chart.height()).to.be.greaterThan(200)
    })
})

The beforeEach commands are shown in its own top section of the Command Log.

Loading chart before each test

Testing tooltip

When I hover over a bar in my chart, a tooltip pops up. Can I test that it does? Cypress does not support :hover yet, but looking at the event listeners for the SVG rect element I notice that the tooltip is connected to mousemove and mouseleave events.

Event listeners for `rect` element

Maybe I can get the tooltip to show up by using cy.trigger('mousemove') command?

1
2
3
cy.get('.frappe-chart g.dataset-0 rect')
  .eq(1) // pick the "spring" bar
  .trigger('mousemove')

Yes! The tooltip shows up. Let us confirm it - and because the tooltip is hidden with style="opacity: 0" attribute, we can assert that it is initially hidden, then shown, then hidden again by writing small utility functions tooltipHidden and tooltipVisible.

cypress/integration/tooltip-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
const tooltipHidden = () =>
  // initially the tooltip is not visible
  // because element is set to be hidden using attribute style="opacity:0"
  // we should check its visibility using "have.css" assertion
  cy.get('.graph-svg-tip').should('have.css', 'opacity', '0')

const tooltipVisible = () =>
  cy.get('.graph-svg-tip').should('have.css', 'opacity', '1')

it('shows and hides tooltip', () => {
  const rectangles = '.frappe-chart g.dataset-0 rect'

  tooltipHidden()
  cy.get(rectangles)
    .eq(1) // pick the "spring" bar
    .trigger('mousemove')
    .wait(1000)

  // tooltipVisible() returns the Cypress element chain
  // so we can add an assertion to check the text
  // shown in the tooltip
  tooltipVisible().should('contain', 'spring')

  cy.get(rectangles)
    .eq(1)
    .trigger('mouseleave')

  tooltipHidden()
})

I have added cy.wait(1000) to pause the test and make the tooltip visible.

Tooltip test: first the tooltip appears and after 1 second disappears

Tooltip for each bar

A different tooltip appears when the mouse moves over each bar. Let us test it. For now, I will hardcode the season labels in the test.

cypress/integration/seasons-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
it('shows tooltip for each season', () => {
  const rectangles = '.frappe-chart g.dataset-0 rect'

  const labels = ['winter', 'spring', 'summer', 'fall']

  ;[0, 1, 2, 3].forEach(k => {
    cy.get(rectangles)
      .eq(k)
      .trigger('mousemove')
      .wait(500)

    cy.get('.graph-svg-tip', { log: false }).should('contain', labels[k])
  })
})

The test runs through the four vertical bars and confirms the tooltip has the right label

Seasons labels

Reading labels at run-time

Instead of hardcoding the labels in the test, we can read the labels from the application at run-time. First, we need to expose the chart reference or its data object reference during Cypress tests. Here is my preferred way of doing this:

app.js
1
2
3
if (window.Cypress) {
  window.chart = chart
}

From now on, if you open DevTools during Cypress tests, and point the context at the application's iframe, you will be able to walk to the labels via window.chart object.

Access labels via `window.chart`

We can read these labels from our tests too using cy.window() and cy.its()

cypress/integration/seasons-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
it('shows tooltip for each defined label', () => {
  const rectangles = '.frappe-chart g.dataset-0 rect'

  // labels are accessed at run-time from the chart object
  cy.window()
    .its('chart.data.labels')
    // make sure we have a valid list with labels
    .should('have.length.gt', 0)
    .then(labels => {
      labels.forEach((label, k) => {
        cy.get(rectangles)
          .eq(k)
          .trigger('mousemove')
          .wait(500)

        cy.get('.graph-svg-tip', { log: false }).should('contain', label)
      })
    })
})

Super, the test is passing, just like before, but now the labels are not copied.

Reading labels at compile-time

Instead of exposing the application and reading labels during the test, we can factor the labels into its own JavaScript module and share them between the application and the test code. We are already using parcel-bundler to serve our code, so we can just extract the labels into its own file.

labels.js
1
export const labels = ['winter', 'spring', 'summer', 'fall']
app.js
1
2
3
4
5
6
7
8
9
10
11
import { labels } from './labels'
const data = {
  labels,
  datasets: [
    {
      name: 'Sunny days',
      type: 'bar',
      values: [10, 20, 30, 25]
    }
  ]
}

The application is bundled using parcel-bundler and served - just like before.

Cypress includes its own bundler, so we import labels from the spec file.

cypress/integration/labels-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
/// <reference types="Cypress" />
import { labels } from '../../labels'
beforeEach(() => {
  cy.visit('/')
  // let the chart load by observing the rendered bars
  cy.get('.frappe-chart g.dataset-0 rect').should('have.length', 4)
})

it('shows tooltip for each imported label', () => {
  const rectangles = '.frappe-chart g.dataset-0 rect'

  labels.forEach((label, k) => {
    cy.get(rectangles)
      .eq(k)
      .trigger('mousemove')
      .wait(500)

    cy.get('.graph-svg-tip', { log: false }).should('contain', label)
  })
})

Great, all working, and there is no code duplication.

Visual diffing with Applitools plugin

But what if the charts library publishes a new version, changing the appearance of the bar chart? Or what if someone comes and changes the beautiful blue color in app.js from #7cd6fd to the ugly #816c30? The functional tests like we have written above cannot check if the chart looks the same. Yes, we could check each positional property and every style property of every element on the page ... and that will super painful to write as a test.

Instead let's set up image diffing testing, in this case I will use Applitools Cypress plugin.

First, I will install the Applitools Cypress plugin

1
npm install @applitools/eyes-cypress --save-dev

Then I need to run

1
npx eyes-setup

Which should do 2 things: load Applitools from Cypress plugins file and load Applitools commands from the support file

cypress/plugins/index.js
1
2
3
4
5
6
module.exports = (on, config) => {
  // `on` is used to hook into various events Cypress emits
  // `config` is the resolved Cypress config
}
// added by Applitools
require('@applitools/eyes-cypress')(module)
cypress/support/index.js
1
2
// added by Applitools
import '@applitools/eyes-cypress/commands'

I then made an Applitools account using my GitHub account and grabbed my Applitools API key. The image below comes from the Applitools Cypress Quickstart tutorial.

personal Applitools API key

Then from the terminal I have exported the key once

1
export APPLITOOLS_API_KEY=<my api key>

And now I am good to go. I have updated my tooltip test to use cy.eye... commands created by the Applitools plugin.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
it('shows tooltip for each season', () => {
  // start new batch of images
  cy.eyesOpen({
    appName: 'chart-testing-example',
    batchName: 'tooltips'
  })

  const rectangles = '.frappe-chart g.dataset-0 rect'

  const labels = ['winter', 'spring', 'summer', 'fall']

  labels.forEach((label, k) => {
    cy.get(rectangles)
      .eq(k)
      .trigger('mousemove')
      .wait(500)

    cy.get('.graph-svg-tip', { log: false }).should('contain', label)

    // we limit the visual diff to the chart
    cy.eyesCheckWindow({
      sizeMode: 'selector',
      selector: '.frappe-chart'
    })
  })

  cy.eyesClose() // tell Applitools we are done
})

I run Cypress locally (with APPLITOOLS_API_KEY environment variable). A few seconds after the test completes, the Applitools web application dashboard shows the 4 images.

Applitools shows the new batch of images

Because this is a new batch, all images were automatically accepted and saved as baselines. Let us change the color of the bars to my least favorite color #816c30 and just push the code to CI. I am using CircleCI and I have already set the APPLITOOLS_API_KEY as an environment variable there.

The CI run has failed, see https://circleci.com/gh/bahmutov/chart-testing-example/4 - and it has failed after the Applitools cloud that actually does image rendering and comparison has reported the changes asynchronously back to the test runner. That's why the failures are not reported immediately during the test, but inside the after test callback.

Visual diff has failed

Let us go the Applitool dashboard to see why the test has failed visual comparison.

Visual diff showing changes

We can go into each image to see the difference with the saved baseline image. For example we can toggle (T) between the new image and the baseline.

Toggle mode

We can even highlight the changed areas (although it is pretty clear in this case where the changes are)

Highlight differences mode

There is one more interesting feature the Applitools analysis offers. I can click on the "<>" button which opens a root cause analysis view. In this view, if I show the "diff" of the images, it also shows WHY the images have changed. In our case it correctly shows the root of the problem: the change "style: fill" property!

Root cause analysis

I don't like the new color, so I will revert the commit and push again.

Conclusions

  • testing an SVG chart is pretty close to testing a regular DOM using Cypress
  • for visual regressions, consider using an image diffing tool. There are several choices, both commercial and open source

Note: you can find the source code for this blog post at github.com/bahmutov/chart-testing-example.