products

Testing Vue CLI reload

Testing dev server hot reload using Cypress.

Vue-cli test

Recently I came across vue-cli server reload test. The test uses Puppeteer to verify that vue-cli scaffolds a project correctly, that dev server is serving a Vue app, and that changing the app's source file reloads the page, and it shows updated text. Here is the main part of that test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
const runTest = async (url) => {
  // some setup code omitted
  const browser = activeBrowser = await puppeteer.launch(opts)
  const page = await browser.newPage()
  await page.goto(url)

  const assertText = async (selector, text) => {
    const value = await page.evaluate(() => {
      return document.querySelector('h1').textContent
    })
    expect(value).toMatch(text)
  }

  const msg = `Welcome to Your Vue.js App`
  await assertText('h1', msg)

  // test hot reload
  const file = await read(`src/App.vue`)
  await write(`src/App.vue`, file.replace(msg, `Updated`))

  await nextUpdate() // wait for child stdout update signal
  await sleep(1000) // give the client time to update, should happen in 1s
  await assertText('h1', `Updated`)

  await browser.close()
  activeBrowser = null
}

The above code certainly works. But even discounting all the code outside the function runTest, count how many lines of code are related to the test and how many lines of code are dealing with starting the browser, finding the element, waiting for the text and then cleaning up. Puppeteer is a great tool, but it is NOT a testing tool specifically; it is a general Chrome browser automation tool. Thus a tool specifically designed to be a testing tool around a real browser might be more convenient to use in this case.

Cypress

One such tool that specifically focuses on solving browser management, element selection, automated retries and other developer experience issues common during testing is Cypress.io (full disclosure: I have joined Cypress team in 2017 after being a super happy user for almost a year). Let me try writing a similar server test using Cypress for comparison.

I prefer working on a new project rather than forking vue-cli to avoid distractions of a large code base. Let us create a new Vue project to use as a test playground. After installing vue-cli globally, I need to scaffold a project with hot module reloading (you can find finished project at bahmutov/test-vue-cli).

1
2
3
4
5
6
7
8
9
10
$ vue list

  Available official templates:

  ★  browserify - A full-featured Browserify + vueify setup with hot-reload, linting & unit testing.
  ★  browserify-simple - A simple Browserify + vueify setup for quick prototyping.
  ★  pwa - PWA template for vue-cli based on the webpack template
  ★  simple - The simplest possible Vue setup in a single HTML file
  ★  webpack - A full-featured Webpack + vue-loader setup with hot reload, linting, testing & css extraction.
  ★  webpack-simple - A simple Webpack + vue-loader setup for quick prototyping.

Pick webpack-simple and run vue init with full template name (to get around 404 response I am getting today) vuejs-templates/webpack-simple.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ vue init vuejs-templates/webpack-simple

? Generate project in current directory? Yes
? Project name test-vue-cli
? Project description A Vue.js project
? Author Gleb Bahmutov
? License MIT
? Use sass? No

   vue-cli · Generated "test-vue-cli".

   To get started:

     npm install
     npm run dev

The project has been bootstrapped in the current folder. Install NPM dependencies and here are the files in the current folder

1
2
3
4
5
6
7
8
$ ls -l
total 32
-rw-r--r--    1  staff    327 Jan  7 14:27 README.md
-rw-r--r--    1  staff    206 Jan  7 14:27 index.html
drwxr-xr-x  640  staff  21760 Jan  7 14:29 node_modules
-rw-r--r--    1  staff    855 Jan  7 14:27 package.json
drwxr-xr-x    5  staff    170 Jan  7 14:27 src
-rw-r--r--    1  staff   1600 Jan  7 14:27 webpack.config.js

Install Cypress using npm i -D cypress and add two script commands.

package.json
1
2
3
4
5
6
7
8
{
  "scripts": {
    "dev": "cross-env NODE_ENV=development webpack-dev-server --open --hot",
    "build": "cross-env NODE_ENV=production webpack --progress --hide-modules",
    "cypress:open": "cypress open",
    "cypress:run": "cypress run"
  }
}

Open Cypress once with npm run cypress:open to scaffold its folder cypress.

Scaffolding Cypress on first run

I prefer deleting the created example_spec.js with many many examples of tests and starting from scratch. Let us verify that our default application shows the expected text "Welcome to Your Vue.js App" in element h1.

cypress/integration/spec.js
1
2
3
4
it('serves', () => {
  cy.visit('http://localhost:8080')
  cy.contains('h1', 'Welcome to Your Vue.js App')
})

Start the dev server with npm run dev and open Cypress with npm run cypress:open. We have a passing test!

First test

Now let us read the src/App.vue file, change the message passed to the Vue component and check that the browser shows the updated message. We are going to use cy.readFile and cy.writeFile methods.

1
2
3
4
5
6
7
8
9
10
11
const message = 'Welcome to Your Vue.js App'
const setNewText = source => source.replace(message, 'Updated')
it('serves', () => {
  cy.visit('http://localhost:8080')
  cy.contains('h1', message)
  cy
    .readFile('src/App.vue')
    .then(setNewText)
    .then(source => cy.writeFile('src/App.vue', source))
  cy.contains('h1', 'Updated')
})

The test passes, of course. We can see each step of the test in the command log on the left.

Updated text test

Here is the important part. Cypress is really optimized for testing. Do you want to see why the first assertion cy.contains('h1', message) passed? Hover over or click on step 2 to see the state of the DOM at that moment. We are finding the right text at the right location on the page - not accidentally matching some other text that might be on the page.

Contains text

Click on the next test command - READFILE. It shows a popup message "Printed output to your console". Open the DevTools (Cypress is after all just an Electron app, or it can control Chrome browser) to see the file read and its contents.

cy.readFile

Isn't this convenient during working with tests to see all data related to each step of the test?

Dealing with initial state

There is one more note that I must make. You should strive to make each test independent from any other test, even independent from its previous run. In our case we have changed the file src/App.vue, and if we reload the test, it fails to find the original text "Welcome to Your Vue.js App". Our previous run changed the source file, and our test did not start from the clean state.

Running test again fails

Again, the above screenshot shows the difference between a general browser automation tool and Cypress. The reason for the failure is communicated very clearly; Cypress team pays a lot of attention to good error messages. It should be enough to look at the screenshot to immediately see the problem. In this case the app shows "Updated" where "Welcome ..." text should have been.

By the way, Cypress takes video of the entire test run by default, and screenshot after failures automatically when running on CI with npm run cypress:run command; there is nothing to configure.

Ok, let us fix the test. We can modify the App.vue file after the test - but that is an anti-pattern to avoid. You never know if the test failed or crashed - the clean up after the test is not guaranteed. So let us make sure the test starts with expected file.

There are many things we could do in this case, but I pick using a fixture. Copy the original src/App.vue to cypress/fixtures/App.vue. Before our test, load the fixture and save it as src/App.vue.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
beforeEach(() => {
  cy.fixture('App.vue')
    .then(source => cy.writeFile('src/App.vue', source))
})
const message = 'Welcome to Your Vue.js App'
const setNewText = source => source.replace(message, 'Updated')
it('serves', () => {
  cy.visit('http://localhost:8080')
  cy.contains('h1', message)
  cy
    .readFile('src/App.vue')
    .then(setNewText)
    .then(source => cy.writeFile('src/App.vue', source))
  cy.contains('h1', 'Updated')
})

The test reliably passes any time we run it. Even better, we can add back our very first original "message" test.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
describe('Vue server', () => {
  const message = 'Welcome to Your Vue.js App'
  const setNewText = source => source.replace(message, 'Updated')
  const saveSource = source => cy.writeFile('src/App.vue', source)
  beforeEach(() => {
    cy.fixture('App.vue').then(saveSource)
    cy.visit('http://localhost:8080')
  })
  it('serves', () => {
    cy.contains('h1', message)
  })
  it('reloads on change', () => {
    cy
      .readFile('src/App.vue')
      .then(setNewText)
      .then(saveSource)
    cy.contains('h1', 'Updated')
  })
})

This way we will know exactly which dev server feature is broken if a test fails. First test tells us if the dev server is serving the bundled application. Second test tells us if the hot module replacement is working.

Two tests

Starting the server

The original Vue test also starts the dev server, grabs the output URL and runs the test. We believe starting and stopping server inside Cypress is an anti-pattern. Instead we recommend using start-server-and-test utility.

1
2
3
{
  "test": "start-server-and-test dev http://localhost:8080 cypress:run"
}

The npm test command starts the dev server, waits for the given url to respond with 200, then runs Cypress without GUI, and then shuts down the server. The url is hardcoded, but we could add extracting url from the first command's output to start-server-and-test tool if needed. For now I will just move the url from the spec file to cypress.json file.

spec.js
1
2
3
4
beforeEach(() => {
  cy.fixture('App.vue').then(saveSource)
  cy.visit('/')
})
1
2
3
4
{
  "defaultCommandTimeout": 10000,
  "baseUrl": "http://localhost:8080"
}

At runtime, we can even overwrite the base url using command line arguments or environment variable. For example, if the server is running at port 3000 we can run out tests against that port with this command.

1
CYPRESS_baseUrl=http://localhost:3000 npm run cypress:run

Final thoughts

The Vue test has 60 lines, including several blank ones. Almost equivalent Cypress test is 20 lines. But I think the smaller test size does not tell the whole story.

Puppeteer is a great general purpose Chrome automation tool, but lacks anything to make testing web applications easier. Cypress has smart selector API, automatic command retries, GUI with full interactive command history, server and object mocking, helpful errors, site blacklisting, command customization, plugins API, great CI support, screenshots and video recording, central dashboard and many other things currently on the roadmap.

Give Cypress 5 minutes, and I guarantee that you will not want to use any other tool for E2E testing. After all, here is what NASA says about Cypress

NASA tweets

Where to start