products

Immutable deploys with data and testing

How to roll over data when doing clean deploys using Zeit. Plus testing with Cypress!

Lately, I have been enjoying three great new tools: Zeit's now for clean fresh deploys at the speed of light, Cypress.io for quick and painless end to end browser testing and Feathers for my JavaScript real time server. In this blog post I will explain how to work around the existing limitations in Zeit deploys - mainly your data is temporary and not shared among deploys. Unlike Dokku each Zeit deploy only can write to a local temporary location, thus there is a problem where to store the data and how to roll it over to the new server.

Of course, you can use a data backend, like Firebase. But keeping the data local is usually faster and simpler for quick prototypes. I prefer to keep the database small and just copy it from the previous deploy to the new one.

Example application

Take Feathers chat app for example. It has two services messages and users. Both services use NeDB to store the data in a local file. NeDB is fast, simple to use and mimics MongoDB API for document storage and retrieval. For quick prototypes or simple projects it just works. The data for each service is stored in a plain text file; the local user login and password database looks like this

1
2
3
$ cat users.db
{"email":"[email protected]","password":"$2a$10$N6RIeQH6.VNUI...","_id":"vYF85ZL5ZEGtF1zJ"}
{"email":"[email protected]","password":"$2a$10$MQ2n7PVsM.h6H...","_id":"pnWbw8qNCOamNGQx"}

Notice that passwork is hashed and salted for security. Even if an attacker steals this database, it would take a long time to match a password to the value stored inside the DB. Consult Feathers security docs for details.

My copy of the chat application is at bahmutov/feathers-chat-app and has additional middleware feathers-nedb-dump I wrote to return a given service's database file. It can also replace the running service's database with uploaded value. Taken together this middleware allows to roll over data each time I deploy new "production" app to Zeit.

Middleware

The feathers-nedb-dump is simple to use. Just configure the secret token used to authorize the API call to return or receive the database, and add two routes.

config/default.json
1
2
3
{
  "dumb-db-secret": "ebd2d309-83d2-4857-8b02-b933c480c1a9"
}
1
2
3
4
5
6
7
8
9
10
// src/middleware/index.js
const dbSet = require('feathers-nedb-dump').set;
const dbDump = require('feathers-nedb-dump').get;
module.exports = function() {
  const app = this;
  // GET and POST to return and receive service database
  app.get('/db-dump/:service', dbDump(app));
  app.post('/db-set', dbSet(app));
  // other routes
};

You can use other API endpoints for obscurity, but more importantly, you have to use HTTPS to prevent token from leaking. Zeit always uses HTTPS, so this should be safe.

Deployment steps

Zeit environment restricts the applications to only write to /tmp folder. We can of course configure our Feathers chat application to do so

config/default.json
1
2
3
4
{
  "nedb": "/tmp/",
  "dumb-db-secret": "ebd2d309-83d2-4857-8b02-b933c480c1a9"
}

Imagine we have an existing application running at https://feathers-chat-app-one.now.sh placed there some time ago using the Zeit now command. The deployment has new users, and has chat messages. Now we have (hopefully) fixed a discovered bug and want to deploy new application version. Rather than overwriting the code at the existing url, we are going to deploy to a brand new environment. Let us say that now has given us a fresh server at a different url https://feathers-chat-app-two.now.sh. We want to copy data from -one.now.sh to -two.now.sh server. Just use the following script, which copies the two databases. I am using httpie instead of curl for a more user-friendly API; you can find this script in copy-data.sh file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
FROM_HOST=https://feathers-chat-app-one.now.sh
DEST_HOST=https://feathers-chat-app-two.now.sh

# from and to tokens could be different
# for example each deploy could generate random one
FROM_TOKEN=ebd2d309-83d2-4857-8b02-b933c480c1a9
DEST_TOKEN=ebd2d309-83d2-4857-8b02-b933c480c1a9

SERVICES=messages users

for NAME in messages users
do
  FILENAME=$NAME.db
  echo Copying $NAME to $FILENAME
  http $FROM_HOST/db-dump/$NAME \
    dumb-db-secret:$FROM_TOKEN > $FILENAME
  # we could transform the data here
  # for example adjusting its schema
  http -f POST $DEST_HOST/db-set \
    dumb-db-secret:$DEST_TOKEN \
    service=$NAME \
    db=@$FILENAME
done

After a few seconds the new server has a replica of the dataset from the previous server (which is still running!)

End to end testing new deployment

The web application is now running at two locations. Let us test the new deployment. The end to end testing has never been easier with Cypress. For example here is a test that logs into an existing account (possible because we rolled the data) and sends a test message.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
describe('Chat app', function(){
  it('logs in and sees messages', function(){
    cy.visit(Cypress.config('baseUrl'))
    cy.title().should('include', 'Feathers Chat')
    cy.contains('Login').click()
    cy.url().should('include', 'login.html')
    // email and password are hardcoded here
    // but should be read from env variables
    cy.get('input[name="email"]').clear().type('[email protected]')
    cy.get('input[name="password"]').clear().type('a')
    cy.contains('Login').click()
    cy.url().should('include', 'chat.html')
    // there should be messages already
    cy.get('.message').should('have.length.gt', 0)
    // send a message
    const random = Math.round(Math.random() * 1e+6)
    const msg = `This is a test ${random}`
    cy.get('input[name="text"]').type(msg + '{enter}')
    // make sure the test message has appeared
    cy.contains('.message', msg).should('exist')
  })
})

In Cypress we can observe the test execution, and each step provides a lot of information. For example the last assert verifies that the test message we just sent is actually present in the message list: cy.contains('.message', msg).should('exist'). When hovering over this assertion in the test runner column, we see the found element highligted in the iframe on the right.

chat test

By default, Cypress runs against the server base url specified in the cypress.json file. When testing new deployment we can specify a new url using either command line or environment variable.

1
$ CYPRESS_baseUrl=https://feathers-chat-app-two.now.sh cypress open

This makes it extremely easy to verify that the new deployment is functioning. After the test passes, we can copy the data again - this makes sure the new deployment has the very recent data snapshot, and removes any traces left by the end to end tests.

Switching on the new deploy

Once we have tested the new deploy, we can use zeit alias to point the public URL at the new deployment. The users should not notice any changes, since the data from the previous deployment has been copied, and the environment has been tested.

More information