Drive-by Testing Array Explorer

TLDR; testing a popular web app using Cypress.

• Array Explorer
• The setup
• First test
• Second test
• Covering primary methods
• Use JavaScript
• Simplifying test using aliases
• Generating tests
• Fake time
• Need for speed
• How do you say test in ...
• Final thoughts

Array Explorer

Array Explorer is a great resource for learning JavaScript array operations. You can see it in action yourself at https://sdras.github.io/array-explorer/. Here is me adding items to an array and seeing the result and relevant documentation.

The source for this project is available at github.com/sdras/array-explorer and is written by wonderfully productive Sarah Drasner. I have noticed that the project does not have any tests. This is a great opportunity to quickly write a few end-to-end tests using Cypress test runner. Just a few high level tests can exercise the entire application as if a real user was interacting with the page.

Let us start testing!

Hint: Sarah has a sister project sdras/object-explorer that you could test the same way! That would be a great practice for anyone trying out Cypress.

The setup

First, I need to setup Cypress. Luckily it only takes a minute.

• I forked the sdras/array-explorer repo to get my own copy
• cloned the forked repository to local drive

$ git clone [email protected]:bahmutov/array-explorer.git
$ cd array-explorer
$ git checkout -b add-tests
$ npm install

If we execute npm start right now we will get a local app running at http://localhost:8080. Let us start testing it.

• I installed Cypress as a dev dependency, and set the test script command.

$ npm i -D cypress

package.json

{
  "scripts": {
    "test": "cypress run",
    "test:gui": "cypress open"
  }
}

During local development we will be running npm run test:gui which opens Cypress in GUI mode - it is really convenient to see what is going during our tests. On CI we will run Cypress tests in headless mode using cypress run command.

• when I opened Cypress using npm run test:gui for the very first time, it scaffolded the test folder cypress and a configuration file cypress.json.

First test

• We can rename new file example_spec.js to just spec.js and remove its contents to start from scratch. Notice that Cypress picks up file changes right away; click on the renamed file spec.js in Cypress file list. There no tests yet it says. Here is the first test I wrote.

cypress/integration/spec.js

describe('array-explorer', () => {
  beforeEach(() => {
    cy.visit('http://localhost:8080')
  })
  it('loads', () => {})
})

Cypress has noticed file changes and ran it right away. I see the local site in the Cypress iframe (the dev server npm start is still running). Everything is green!

Second test

Let us confirm that the page is greeting us with the name of the project "JavaScript Array Explorer". For finding unique text on the page, I usually use cy.contains(text) command. In this case this text might appear somewhere else on the page. To avoid accidents like this, let us make the selector more precise. Recently we have introduced CSS Selector Playground tool. Click on its target icon next to the url bar and hover over the text to see its "best" selector - in this case it suggests using h1 selector.

Let us update our first test to confirm that the text really greets us after load.

it('greets us', () => {
  cy.contains('h1', 'JavaScript Array Explorer')
})

Perfect, the test passes.

Covering primary methods

There are 7 primary methods Array Explorer teaches: from "add items or other arrays" to "something else". We really should create suites of tests for each primary method. In this example I will write just a few example tests. Let me start with "something else" primary method, that has "length of the array" secondary option.

As suggested by the "CSS Selector Playground" helper (or by looking up in the DevTools), the first drop down choice can be queried with #firstmethod selector. The method options can be set in the #methodoptions drop down. Here is our test.

context('something else', () => {
  it('shows length of an array', () => {
    cy.get('#firstmethod').select('something else')
    cy.get('#methodoptions').select('find the length of the array')
  })
})

While the test is running I see Cypress opening drop downs, finding the right choice, selecting it. If an element with a given selector was missing and did not appear for the command's timeout duration, an intelligent error message with a screenshot would appear. But so far everything is going great.

We need to add an assertion to the test. We have not confirmed that the app actually shows an example of the array length property. And we have not confirmed that the output text area is really showing the right answer. Let us do this. The app works in steps

1. Code appears (with slow type effect) in the first box .usage1
2. Then the answer appears in box .usage2 > .usage-code

Under the hood, the Array Explorer has input and output code as text already in the DOM, just hidden. The input code is supposed to "produce" the output text - it has console.log(...) statements, but of course they are not actually executing. So in our "length of array" example the input looks like this

let arr = [5, 1, 8];
console.log(arr.length);

The output text is 3 and is initially hidden. Only after delayed "type" effect it appears.

In our test, we can grab the input code as text, and the output text, and we should make sure the output value is visible and correct. To make sure the value is correct, I will eval the input text. Because the input uses console.(...) to "print" the result, before evaluating the code, I will set up a spy on console.log method using built-in [cy.spy][spy] method. Here is the entire test - it kind of looks scary, but it is really universal and can work with any array explorer example!

it('shows length of an array', () => {
  cy.get('#firstmethod').select('something else')
  cy.get('#methodoptions').select('find the length of the array')
  // compute the output from input
  let output
  cy
    .get('.exampleoutput2')
    .invoke('text')
    .then(t => {
      output = t
    })
  // set up spy on `console.log` before
  // we can call `eval(input code)`
  cy.spy(console, 'log')
  cy.get('.usage1').then(v => {
    const input = v.text()
    // evaluate the input code - we are already spying on console.log!
    eval(input)
    // the value comes from DOM - so it needs to be
    // converted before we can compare it to the
    // compute value
    expect(console.log).to.have.been.calledWith(JSON.parse(output))
    // make sure the output becomes visible
    cy
      .get('.exampleoutput2')
      .should('have.css', 'opacity', '1')
      // and the right value appears
      .and('contain', String(output))
  })
})

Here is the test in action. Notice an interesting detail that goes to the heart of what makes Cypress great - the intelligent waiting for assertion to pass. The last statement of the test grab the output element and sets up TWO assertions.

cy
  .get('.exampleoutput2')
  // answer becomes visible
  .should('have.css', 'opacity', '1')
  // and the right value appears
  .and('contain', String(output))

Tip: .and is equivalent to .should. It improves readability when making multiple assertions about the same element.

The video below shows the first asserts waits until the type effect finishes and the app makes the answer visible. Then the second assertion verifies that the right value is in the element.

Cypress fights the unpredictable nature of end-to-end tests by retrying commands for a period of time, and these options are very configurable. The default command timeout of 4 seconds works well for this example, but was too short for other ones. So I increased the command timeout by setting it in cypress.json file

cypress.json

{
  "viewportHeight": 800,
  "defaultCommandTimeout": 10000
}

Use JavaScript

As my test grows in complexity to handle multiple output values and different edge cases, the code becomes difficult to understand. Luckily, it is just JavaScript. I can refactor the test code as much as I like. I could move utility functions out to separate files and just require or import them. I can bring any NPM module as a dependency and include in my test code.

Back to our test. We can factor out the logic for running the input code sample and comparing it to the output text box. Here is this function with its main parts.

const confirmInputAndOutput = () => {
  // get expected output text
  // spy on console.log method
  // get the input code sample
  // eval(input code sample)
  // compare console.log from the spy
  //   with expected output text values
  // check if output box appears and contains output text
}
// example test
it('shows length of array', () => {
  cy.get('#firstmethod').select('something else')
  cy.get('#methodoptions').select('find the length of the array')
  confirmInputAndOutput()
})

Simplifying test using aliases

A very handy feature to make tests simpler is aliases. Aliases allow saving reference to a DOM element, data or XHR stubs to use later. Here is a fragment of the function that gets the output text and saves several aliases for future use.

const confirmInputAndOutput = () => {
  cy
    .get('.exampleoutput2').as('output')
    .invoke('text').then(removeComments).as('outputText')
    .then(parseText).as('outputValues')
  // the test goes on
}

In the above example we save reference to the DOM element as alias output, raw text without comments as alias outputText and parsed JavaScript values as outputValues. Later we can use the aliased values like this

// eval(input code sample)
// confirm console.log with expected values happened in order
cy.get('@outputValues').then(outputValues => {
  outputValues.forEach((value, k) => {
    expect(console.log.getCall(k)).to.have.been.calledWith(value)
  })
})
// make sure the output text actually appears
cy.get('@outputText').then(outputText => {
  cy.get('@output').should('have.css', 'opacity', '1')
  // the only difficulty is with multiline text where there might
  // be white space at the start of each line
  outputText
    .split('\n')
    .map(trim)
    .forEach(line => {
      cy.get('@output').should('contain', line)
    })
})

Aliases are great way to avoid deep nesting or temporary variables.

Generating tests

I wrote a couple of other tests: "fill array ...", "copy a sequence of elements ...". They all work the same way - set the desired method example and call confirmInputAndOutput. The spec file kind of looks boring.

context('something else', () => {
  beforeEach(() => {
    cy.get('#firstmethod').select('something else')
  })

  it('shows length of an array', () => {
    selectMethodOptions('find the length of the array')
    confirmInputAndOutput()
  })

  it('fills array with given value', () => {
    selectMethodOptions(
      'fill all the elements of the array with a static value'
    )
    confirmInputAndOutput()
  })

  it('copy a sequence of array elements within the array', () => {
    selectMethodOptions('copy a sequence of array elements within the array.')
    confirmInputAndOutput()
  })
})

All tests are going to look like this. Can we generate the tests from a list of options? Yes we can. Instead of writing individual it(..., () => {...}) calls, I placed all primary method names and corresponding option names into an object of arrays. A small helper function iterates over the arrays, creating tests.

const methods = {
  'order an array': [
    'reverse the order of the array',
    'sort the items of the array',
  ],
  'something else': [
    'find the length of the array',
    'fill all the elements of the array with a static value',
    'copy a sequence of array elements within the array.',
  ],
  // a lot more methods ...
}

Object.keys(methods).forEach(method => {
  context(method, () => {
    beforeEach(() => {
      cy.get('#firstmethod').select(method)
    })
    methods[method].forEach(secondary => {
      it(secondary, () => {
        selectMethodOptions(secondary)
        confirmInputAndOutput()
      })
    })
  })
})

One short spec file is running 20 tests checking all array method examples (except for "find" method that requires extra arguments)

Fake time

One test is failing. Can you spot the problem?

The input code sample uses current date!

let arr = [5, 1, 8];
let date = [new Date()];
const arrString = arr.toLocaleString();
const dateString = date.toLocaleString();
console.log(arrString, dateString);

The expected output is hardcoded to be "12/26/2017, 6:54:49 PM", which will never match the evaluated input code. What can we do? Ordinarily, Cypress tests use cy.clock to control the time inside the running application like this

beforeEach(() => {
  const now = new Date('12/26/2017, 6:54:49 PM').getTime()
  cy.clock(now)
  cy.visit('http://localhost:8080')
})

But our use case is different. It is NOT the application that is calling new Date(), but our unit test via eval.

eval(`
let arr = [5, 1, 8];
let date = [new Date()];
const arrString = arr.toLocaleString();
const dateString = date.toLocaleString();
console.log(arrString, dateString);
`)

Thus our test code must fake the date. Luckily, Cypress has Sinon bundled and available under Cypress.sinon property. Right before we evaluate our code we can fake the Date object.

var clock = Cypress.sinon.useFakeTimers(
  new Date('12/26/2017, 6:54:49 PM').getTime()
)
{
  // evaluate the input code - we are already spying on console.log!
  eval('const Date = clock.Date;' + sourceCode)
}
// don't forget to restore system clock
// otherwise good things will not happen
clock.restore()

Tricking eval to use local variable via closure is one of my favorite JS tricks. Aside from this, I changed the original example a little bit. I split the single console.log to be two statements to better match what every example is doing and what my tests expect.

Note the alises are highlighted when saved (pink and blue rounded labels output and outputValues). They are also highlighted with @... label when used.

Need for speed

The tests are passing, but there is one negative. They take too long to finish - 67 seconds! The main reason each tests takes a few seconds are the delays built into the app. There is half a second delay to let the user read the first line of the example code, then there is the "type" effect that uses gsap library to reveal the rest of the example code.

AppCode.vue

selectedUsage() {
  // initial delay
  setTimeout(() => {
    this.typeOut()
  }, 500)
},
typeOut() {
  let split = new SplitText(this.$refs.ex, { type: 'chars' }),
    split2 = new SplitText(this.$refs.ex2, { type: 'chars' }),
    tl = new TimelineMax()

  tl.add('start')
  // animation keyframes
}

The gsap is JavaScript animation library, so maybe if we can "speed" up application's timers we can zoom through the type animation and make our tests faster. While Sinon includes timers, I found lolex to be the most convenient library to work with fake timers. Luckily using this from our tests is very simple.

First, install lolex as a new dev dependency.

$ npm i -D lolex
+ [email protected]

Second, install fake timers into the application before the application loads. We can do this in the callback onBeforeLoad of the cy.visit method. It is important to pass the application's window as the target for stubbing timers.

let fakeClock
beforeEach(() => {
  cy.visit('http://localhost:8080', {
    onBeforeLoad: win => {
      fakeClock = lolex.install({
        target: win,
      })
    },
  })
})

Inside the test logic we can manually advance the clock after the source code appears and the app called its setTimeout (which is now stubbed).

const confirmInputAndOutput = () => {
  cy
    .get('.exampleoutput2').as('output')
    .invoke('text').then(removeComments).as('outputText')
    .then(parseText).as('outputValues')
    .then(() => {
      fakeClock.tick(10000)
    })
  // the test goes on
}

The generous 10 second time jump completes the initial delay of 500ms and instantly zooms through any typing animation the app makes. The result is awesome - total time to finish all tests goes down from 67 seconds to 27 seconds!

How do you say test in ...

The last thing I will do is write a test to make sure the language selector works. Because the language selection element does not have a good way to access it I have added data-attr-cy attribute

<select v-model="selectedLanguage" data-attr-cy="language">
  <option v-for="(val, key) in languages" :value="key">{{val.long}}</option>
</select>

The test can use this data attribute to get the right selection element. After selecting the desired language (I am picking my native Russian), we use the same test logic as before.

it('works in Russian', () => {
  cy.get('[data-attr-cy="language"').select('Russian')
  selectMethod('удалить элементы') // remove elements
  selectMethodOptions('первый элемент массива') // first element of the array
  confirmInputAndOutput()
})

Tip: we could have imported data for different languages directly from the store/<language> files. Then we could actually iterate and create full test suites for every language. After all, out tests are JavaScript and can share code with the web app itself!

Final thoughts

Here is the final result - the spec.js file and the screen recording of the running tests. The tests take longer than 27 seconds because my laptop is choking a little bit during while doing full screen recording.

I have opened a pull request sdras/array-explorer/pull/70 to merge these tests into the Array Explorer. Maybe these tests can inspire you to add tests to sdras/object-explorer?

You can find more information about Cypress from these links

• https://www.cypress.io/
• https://on.cypress.io/intro
• https://www.cypress.io/blog/
• https://glebbahmutov.com/blog/tags/cypress/
• https://on.cypress.io/talks-media