products

Code Coverage For Chat App

How to measure fullstack code coverage from Cypress tests.

In several blog posts I have shown how to test a Socket.io chat application using Cypress.io

Title Description
Test a Socket.io Chat App using Cypress | Simulates the second user by connecting to the chat server from the plugins file Run Two Cypress Test Runners At The Same Time | Launches two test runners, giving them separate specs to run Sync Two Cypress Runners via Checkpoints | Launches two test runners, which stay in sync by communicating via their own Socket.io server

In this blog post I will show how to collect code coverage in each case. From the code coverage reports, we will see that using separate test runners to simulate two users is not necessary. The application code is already exercised when using a separate socket connection to simulate the second user. Even a test with 1 user going through the user interface covers 100% of the code, because every message, even own message, goes through the server before being shown.

🎁 You can find the source code in the repo bahmutov/cypress-socketio-chat. You can also flip through the slides for the presentation covering this topic at slides.com/bahmutov/e2e-for-chat.

Code instrumentation

The application includes the source code using a script tag

index.html
1
<script src="scripts/app.js"></script>

From the server, we can instrument the scripts/app.js source code before sending

index.js
1
2
3
4
5
6
7
8
9
10
const { createInstrumenter } = require('istanbul-lib-instrument')
const instrumenter = createInstrumenter()

app.get('/scripts/app.js', function (req, res) {
  const filename = path.join(__dirname, 'scripts', 'app.js')
  const src = fs.readFileSync(filename, 'utf8')
  const instrumented = instrumenter.instrumentSync(src, filename)
  res.set('Content-Type', 'application/javascript')
  res.send(instrumented)
})

To check if the code has been instrumented, inspect the window.__coverage__ object from the DevTools console.

Code coverage object exists

We can also instrument the server code using nyc module following the Instrument backend code section of the Cypress docs.

1
2
$ npm i -D nyc
+ [email protected]

Change the start server command

package.json
1
2
- "start": "node .",
+ "start": "nyc --silent node .",

And expose the code coverage endpoint

index.js
1
2
3
4
5
// https://github.com/cypress-io/code-coverage#instrument-backend-code
/* istanbul ignore next */
if (global.__coverage__) {
  require('@cypress/code-coverage/middleware/express')(app)
}

To ensure the code coverage report always includes the client and the server code, add to the package.json "nyc" options

package.json
1
2
3
4
5
6
7
8
9
{
  "nyc": {
    "all": true,
    "include": [
      "index.js",
      "scripts/*.js"
    ]
  }
}

Tip: if you need to instrument your application code, find an example matching your situation among the examples in the Cypress code coverage plugin repo.

The first spec

Our first spec uses a single test runner to send the message to itself.

cypress/integration/first-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
/// <reference types="cypress" />

it('posts my messages', () => {
  // https://on.cypress.io/visit
  cy.visit('/', {
    onBeforeLoad(win) {
      // when the application asks for the name
      // return "Cy" using https://on.cypress.io/stub
      cy.stub(win, 'prompt').returns('Cy')
    },
  })

  // make sure the greeting message is shown
  cy.contains('#messages li i', 'Cy join the chat..').should('be.visible')

  // try posting a message
  cy.get('#txt').type('Hello there{enter}')
  cy.contains('#messages li', 'Hello there').contains('strong', 'Cy')
})

The generated HTML report shows full client-side code coverage.

First spec code coverage report

Drill down into the server file to see the two missed lines

First spec never disconnects from the server

Unfortunately, it is hard to test the user disconnect if our page is the only one present.

Mock socket spec

In another spec we replace the actual socket connection with the Mock Socket object.

cypress/integration/mock-socket-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
/// <reference types="cypress" />

import SocketMock from 'socket.io-mock'

describe('Mock socket', () => {
  // these tests "trick" the application by injecting
  // a mock socket from the test into the application
  // instead of letting the application connect to the real one
  const socket = new SocketMock()

  // store info about the client connected from the page
  let username
  let lastMessage
  socket.socketClient.on('username', (name) => {
    console.log('user %s connected', name)
    username = name
    // broadcast to everyone, mimicking the index.js server
    socket.socketClient.emit(
      'is_online',
      '🔵 <i>' + username + ' join the chat..</i>',
    )
  })

  socket.socketClient.on('chat_message', (message) => {
    console.log('user %s says "%s"', username, message)
    lastMessage = '<strong>' + username + '</strong>: ' + message
    socket.socketClient.emit('chat_message', lastMessage)
  })

  it('chats', () => {
    cy.intercept('/scripts/app.js', (req) => {
      // delete any cache headers to get a fresh response
      delete req.headers['if-none-match']
      delete req.headers['if-modified-since']

      req.continue((res) => {
        res.body = res.body.replace(
          "io.connect('http://localhost:8080')",
          'window.testSocket',
        )
      })
    }).as('appjs')

    // the browser is the 1st user
    const name = `Cy_${Cypress._.random(1000)}`

    cy.log(`User **${name}**`)
    cy.visit('/', {
      onBeforeLoad(win) {
        win.testSocket = socket
        cy.stub(win, 'prompt').returns(name)
      },
    })
    cy.wait('@appjs') // our code intercept has worked
      // verify we have received the username
      // use .should(callback) to retry
      // until the variable username has been set
      .should(() => {
        expect(username, 'username').to.equal(name)
      })

    // try sending a message via page UI
    cy.get('#txt').type('Hello there{enter}')
    cy.contains('#messages li', 'Hello there').contains('strong', name)

    // verify the mock socket has received the message
    cy.should(() => {
      expect(lastMessage, 'the right text').to.include('Hello there')
      expect(lastMessage, 'the sender').to.include(name)
    }).then(() => {
      // emit message from the test socket
      // to make sure the page shows it
      socket.socketClient.emit(
        'chat_message',
        '<strong>Cy</strong>: Mock socket works!',
      )

      cy.contains('#messages li', 'Mock socket works').contains('strong', 'Cy')
    })
  })
})

Because we do not run any socket commands on the server, our server-side coverage drops.

Code coverage summary when mocking the Socket

The server report shows no socket callbacks have executed.

When mocking the socket client-side, the server is not used

Second user via socket connection

Let's run the test that uses the UI page as the first user, while connecting to the server through another socket connection to simulate the 2nd user. For example, we can open that 2nd socket connection from the spec.

cypress/integration/socket-from-browser-spec.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
/// <reference types="cypress" />

const io = require('socket.io-client')

describe('Open 2nd socket connection', () => {
  it('communicates with 2nd user', () => {
    // the browser is the 1st user
    const name = `Cy_${Cypress._.random(1000)}`

    cy.log(`User **${name}**`)
    cy.visit('/', {
      onBeforeLoad(win) {
        cy.stub(win, 'prompt').returns(name)
      },
    })

    // make sure the greeting message is shown
    cy.contains('#messages li i', `${name} join the chat..`)
      .should('be.visible')
      .then(() => {
        // and now connect to the server using 2nd user
        // by opening a new Socket connection from the same browser window
        const secondName = 'Ghost'

        const socket = io.connect('http://localhost:8080')
        socket.emit('username', secondName)

        // keep track of the last message sent by the server
        let lastMessage
        socket.on('chat_message', (msg) => (lastMessage = msg))

        // the page shows that the second user has joined the chat
        cy.contains('#messages li i', `${secondName} join the chat..`).should(
          'be.visible',
        )

        // the second user can send a message and the page shows it
        const message = 'hello from 2nd user'
        socket.emit('chat_message', message)
        cy.contains('#messages li', message)

        // when the first user sends the message from the page
        // the second user receives it via socket
        const greeting = `Hello there ${Cypress._.random(10000)}`
        cy.get('#txt').type(greeting)
        cy.get('form').submit()

        // verify the web page shows the message
        // this ensures we can ask the 2nd user for its last message
        // and it should already be there
        cy.contains('#messages li', greeting).contains('strong', name)

        // place the assertions in a should callback
        // to retry them, maybe there is a delay in delivery
        cy.should(() => {
          // using "include" assertion since the server adds HTML markup
          expect(lastMessage, 'last message for 2nd user').to.include(greeting)
          expect(lastMessage, 'has the sender').to.include(name)
        })

        cy.log('**second user leaves**').then(() => {
          socket.disconnect()
        })
        cy.contains('#messages li i', `${secondName} left the chat..`).should(
          'be.visible',
        )
      })
  })
})

Note that this test disconnects the second user and confirms the page shows the right message.

The Cypress test UI

The fullstack code coverage achieves 100% for both the client and the server files.

The code coverage report shows 100% code coverage

The server really exercised all Socket commands.

The server coverage during the test

Run two test runners

Now let's switch to the more complicated way of verifying the chat between two users - by running two test runners. Does it give us any more confidence? Does it cover any more code lines? Well, it would be hard to cover more lines, since we already have reached 100% code coverage!

We will run two test runners and they will wait for each other using checkpoints. For example, here are the ends of the two spec files where the first user disconnects by going away from the page localhost:8080 and the second user confirms it sees the message "First left the chat"

cypress/pair/first-user.js
1
2
3
4
5
// disconnect from the chat by visiting the blank page
cy.window().then((win) => {
  win.location.href = 'about:blank'
})
cy.task('waitForCheckpoint', 'second user saw first user leave')
cypress/pair/second-user.js
1
2
3
4
5
// the first user will disconnect now
cy.contains('#messages li i', `${firstName} left the chat..`).should(
  'be.visible',
)
cy.task('checkpoint', 'second user saw first user leave')

The code coverage stays the same: more lines might be repeated, but no new lines can possible be added to the already full coverage.

Testing approach Fullstack code coverage percentage
Single spec 95%
Mock socket 75%
2nd user via socket 100%
Run two test runners 100%

Happy fullstack testing!