products

Synching PouchdDB with remote DB

How to run a database locally that syncs with central one.

If we want to control our own data, we must know how to run our own database. To make this more attractive, let us pick a database that knows how to sync with our mobile or web application. Then we can get an advanced feature - offline data support almost without any extra work.

I will show how to run a Couch database locally and in a server (using Dokku), and then how to write a simple web application using VueJS that works with a local in-browser Pouch database. The Pouch database keeps the local browser in sync with the remote Couch database.

A basic knowledge of Docker is required to follow this blog post.

Running local CouchDB

Let us try out CouchDB locally inside a Docker container using the frodenas/couchdb image. To be able to easily replicate work, I saved the shell command into a file

1
2
3
4
5
6
7
8
9
# start the db inside a container
# with given username, password and DB name
docker run -d \
  --name couchdb \
  -p 5984:5984 \
  -e COUCHDB_USERNAME=iamuser \
  -e COUCHDB_PASSWORD=mypass \
  -e COUCHDB_DBNAME=c1 \
  frodenas/couchdb

This gave me a running DB named "c1" in about 15 seconds. Most of this time was spent by the docker downloading the base image binaries.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ ./setup-db.sh
Unable to find image 'frodenas/couchdb:latest' locally
latest: Pulling from frodenas/couchdb
012a7829fd3f: Pull complete
41158247dd50: Pull complete
916b974d99af: Pull complete
a3ed95caeb02: Pull complete
5dc5d08f5c98: Pull complete
1bee2d0b12c7: Pull complete
745c9cfc6b90: Pull complete
e7538cbaf2c8: Pull complete
0e8f353191c0: Pull complete
Digest: sha256:8b777ca1d60b7b5beab4178a8ebc412622ec43f2fec969dcf836bb11ac7ddad8
Status: Downloaded newer image for frodenas/couchdb:latest
7dd6baed4d70c50929490bf24d83110b186f49ed5261b17f726d67466d30e642
$ docker ps
7dd6baed4d70  frodenas/couchdb 0.0.0.0:5984->5984/tcp   couchdb

Right away you should be able to see empty database

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ curl http://localhost:5984/c1
{
  "db_name":"c1",
  "doc_count":0,
  "doc_del_count":0,
  "update_seq":0,
  "purge_seq":0,
  "compact_running":false,
  "disk_size":79,
  "data_size":0,
  "instance_start_time":"1472301143096818",
  "disk_format_version":6,
  "committed_update_seq":0
}

Let us connect to this database using the JavaScript PouchDB library.

1
2
3
const PouchDB = require('pouchdb')
const db = new PouchDB('http://localhost:5984/c1')
db.info().then(console.log).catch(console.error)

This gives us the same information as did the curl command.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ node index.js
{ db_name: 'c1',
  doc_count: 0,
  doc_del_count: 0,
  update_seq: 0,
  purge_seq: 0,
  compact_running: false,
  disk_size: 79,
  data_size: 0,
  instance_start_time: '1472301143096818',
  disk_format_version: 6,
  committed_update_seq: 0,
  host: 'http://localhost:5984/c1/',
  auto_compaction: false,
  adapter: 'http' }

Note: if the Docker is restarted, the new container with the CouchDB will be stopped. You can simple run docker restart 7dd6baed4d70 to get it running again.

Populating data from command line

Since the database is empty, let us insert a couple of fake todos.

1
2
3
4
5
6
7
8
9
10
11
12
13
function insertFakeTodos(n) {
  const generate = require('fake-todos')
  const items = generate(n)
  return db.bulkDocs(items)
}

function dbStats() {
  return db.info().then(console.log)
}

insertFakeTodos(3)
  .then(dbStats)
  .catch(console.error)

We should get 3 documents in the database after running this code.

1
2
3
4
5
$ node index.js
{ db_name: 'c1',
  doc_count: 3,
  ...
}

We can see them by running db.allDocs() method and printing the returned rows list

1
2
3
4
db.allDocs({include_docs: true})
  .then(r => r.rows)
  .then(console.log)
  .catch(console.error)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
$ node index.js
[ { id: '1161b310fc0d36dfe90091f25a000b9b',
    key: '1161b310fc0d36dfe90091f25a000b9b',
    value: { rev: '1-545cf4fb862701be2211ac1d53133eba' },
    doc:
     { _id: '1161b310fc0d36dfe90091f25a000b9b',
       _rev: '1-545cf4fb862701be2211ac1d53133eba',
       what: 'fix chess',
       due: 'tomorrow',
       done: false,
       id: '24214f90-e4e6-4636-834d-d1a8dbaeb3a4' } },
  { id: '1161b310fc0d36dfe90091f25a000fa8',
    key: '1161b310fc0d36dfe90091f25a000fa8',
    value: { rev: '1-a5944cd8118a184598cc8f27736a82c1' },
    doc:
     { _id: '1161b310fc0d36dfe90091f25a000fa8',
       _rev: '1-a5944cd8118a184598cc8f27736a82c1',
       what: 'clean fishing rod',
       due: 'tomorrow',
       done: false,
       id: '8ad8a85a-69db-4984-8cbd-bf290776caa1' } },
  { id: '1161b310fc0d36dfe90091f25a000ff8',
    key: '1161b310fc0d36dfe90091f25a000ff8',
    value: { rev: '1-b9007b3ca9a79a2a938fe11162a7103d' },
    doc:
     { _id: '1161b310fc0d36dfe90091f25a000ff8',
       _rev: '1-b9007b3ca9a79a2a938fe11162a7103d',
       what: 'avoid learning distant relatives',
       due: 'tomorrow',
       done: false,
       id: '6b46a90d-852a-420d-a311-4f81f7da981e' } } ]

You can also see this data by fetching it from the CouchDB directly using url curl http://localhost:5984/c1/_all_docs?include_docs=true or by opening the url http://localhost:5984/_utils/database.html?c1 in the browser.

PouchDB just uses the CouchDB as its data store.

Running PouchDB in the browser

CORS: controlling who can call the database

Note: this step is optional, as we are going to "hide" the database behind a dedicated server later on.

Before we can access our CouchDB from the browser, we need to enable cross domain requests. There is a tiny Node utility package to do this, which needs the user name and the password we have used when we created the CouchDB.

1
2
3
4
$ npm i -D add-cors-to-couchdb
$ $(npm bin)/add-cors-to-couchdb http://localhost:5984 \
  -u iamuser -p mypass
success

You can check the status of the CORS by going to the database web interface at http://localhost:5984/_utils/config.html, the selecting "login" and using the username and password from the database setup step. You should see the following settings enabled

1
2
3
4
5
cors
credentials: true
headers: accept, ...
methods: GET, PUT, ...
origins: *

You can limit the domains by editing the "origins" value and specifying a list of domains that can access the database directly.

Using the CouchDB from the page

Now let us create a page and fetch the documents just like we have done before.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<!DOCTYPE html>
<html>
<head>
  <title>PouchDB Todos</title>
  <script src="https://cdn.jsdelivr.net/pouchdb/5.4.5/pouchdb.min.js"></script>
</head>
<body>
  <div id="app"></div>
  <script>
    const db = new PouchDB('http://localhost:5984/c1')
    db.allDocs({include_docs: true})
      .then(r => r.rows)
      .then(console.log)
      .catch(console.error)
  </script>
</body>
</html>

If you open this index.html in the browser you should see the 3 items printed in the console. PouchDB works fine in the local Node environment, as well as in the browser.

Small Todo web application

Let us make a tiny web application to show the fetched Todos and be able to add new ones. I will use Vue.js for this, but I will not implement the full example. If you want a full example, see Vue examples page.

All we will do is fetch the items from the PouchDB and then put them into a table.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
<head>
  <title>PouchDB Todos</title>
  <script src="https://cdn.jsdelivr.net/pouchdb/5.4.5/pouchdb.min.js"></script>
  <script src="https://cdn.jsdelivr.net/vue/1.0.26/vue.min.js"></script>
</head>
<div id="app">
  <table>
    <thead><tr><th>Task</th><th>Done?</th></tr></thead>
    <tbody>
      <tr v-for="todo in todos">
        <td v-text="todo.doc.what"></td>
        <td v-text="todo.doc.done"></td>
      </tr>
    </tbody>
</div>
<script>
  var vue = new Vue({
    el: '#app',
    db: null,
    data: {
      todos: []
    },
    created() {
      this.db = new PouchDB('http://localhost:5984/c1')
      this.db.allDocs({include_docs: true})
        .then(r => r.rows)
        .then(list => {
          this.todos = list
        })
        .catch(console.error)
    }
  })
</script>

This gives us the list in the browser, but it is static - if there is an update to the data in the database, then new items do not appear in the web application; the user must reload the page to fetch the data again.

Data replication

Let us setup two way data replication - if the CouchDB gets new data, we want to the data to propagate to the PouchDB in the browser, and vice versa. The PouchDB has a good replication documentation page. In our case we want to keep the browser application in sync with the CouchDB, and even handle the case when the browser loses network connection (offline mode).

Instead of working directly with the remove CouchDB we will work against a local replica

1
2
3
4
5
6
7
8
9
10
this.db = new PouchDB('tododb')
var remoteDB = new PouchDB('http://localhost:5984/c1')
this.db.sync(remoteDB, {
  live: true,
  retry: true
}).on('change', function (change) {
  console.log('data change', change)
}).on('error', function (err) {
  console.log('sync error', err)
});

The web application works as before, but now we can stop the CouchDB Docker container but continue working like before in the browser.

1
2
$ docker stop 7dd6baed4d70
7dd6baed4d70

The data still loads in the page, but shows the failed network sync requests if we have XHR logging enabled in the DevTools

1
pouchdb.min.js:9 GET http://localhost:5984/c1/ net::ERR_CONNECTION_REFUSED

If we start the container, the errors stop.

1
$ docker restart 7dd6baed4d70

What happens if we create new data items in the CouchDB database? Let us use our Node program to add 3 more Todo items and observe the PouchDB event callbacks in the browser. The browser console shows 3 new events

1
data change Object {direction: "pull", change: Object}

The change property includes the actual new data object. While we could use the change directly to update the vue.data.todos list, we might as well just refetch the data, since this is now a local browser data fetch! Here is the relevant web application code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
var vue = new Vue({
  ...
  methods: {
    fetch() {
      this.db.allDocs({include_docs: true})
        .then(r => r.rows)
        .then(list => {
          this.todos = list
          console.log('fetched %d todos', list.length)
        })
        .catch(console.error)
    }
  },
  created() {
    this.db = new PouchDB('tododb')
    var remoteDB = new PouchDB('http://localhost:5984/c1')
    this.db.sync(remoteDB, {
      live: true,
      retry: true
    }).on('change', change => {
      console.log('data change', change)
      this.fetch()
    }).on('error', function (err) {
      console.log('sync error', err)
    });
    this.fetch() // fetch initial
  }
})

Here is the fetch in action; as the new items are inserted into the CouchDB, the browser PouchDB gets the updates and refreshes the list.

Run CouchDB in the cloud

I am a big fan of Dokku and will use it to run CouchDB on a personal DigitalOcean droplet. I have already described how to setup a Dokku instance, so I will skip the general introduction.

I will install the CouchDB plugin from the list of Dokku Community plugins. Since I do not have an application (CouchDB has built-in web server), I will just create the service and will expose it to the web.

1
2
3
sudo dokku plugin:install https://github.com/dokku/dokku-couchdb.git couchdb
dokku couchdb:create c1
dokku couchdb:expose c1 6690

The CouchDB is now running available at the direct IP address <ip4.com>:6690 which is not the best practice of course, but would work for now. Let us insert 3 todo records - just change the database url in the index.js. We can even use the domain name instead of IP4 address.

1
const db = new PouchDB('http://<domain.com>:6690/c1')

We can see the items using the CouchDB built-in web browser, just open in the browser url http://<domain.com>:6690/_utils/database.html?c1

Limit direct access via CORS

Note: again, you can safely skip this step as we are going to proxy calls to the database through a server.

We need to enable our browser client to call the CouchDB interface across the domain boundary. Login into the Dokku host and lookup the CouchDB password.

1
2
3
4
dokku couchdb:info c1
DSN: http://c1:<password>@dokku-couchdb-c1:5984/c1
Config dir: /var/lib/dokku/services/couchdb/c1/config
Data dir: /var/lib/dokku/services/couchdb/c1/data

The DSN field has the long random password generated for us when we created a CouchDB service instance. The username is the database name "c1".

Now go back to the CouchDB web configuration interface and change the "cors|credentials" value to "true". The at the bottom click "Add a new section" and add the following sections one by one (copy from local config created by add-cors-to-couchdb utility).

1
2
3
4
5
section | option    | value
-------------------------------
cors    | headers   | accept, authorization, content-type, origin, referer, x-csrf-token
cors    | methods   | GET, PUT, POST, HEAD, DELETE
cors    | origins   | *

If you are running the PouchDB only from a certain domain, you should enter a specific value instead of a wild card. Finally, change 'httpd' section value 'enable_cors' to "true".

The PouchDB running locally now should sync with the CouchDB running in the cloud!

Restricting database access

Exposing the database to the whole world is less than optimal. Plus we need to serve the web page somehow. We can solve both problems by writing a tiny static HTML page server + proxy to redirect the database requests. Here is a server code that is using Express framework. The server redirects any request to /db/ to a local CouchDB running at the standard port 5984.

1
2
3
4
5
6
7
8
9
10
11
12
const express = require('express')
const request = require('request')
// assuming CouchDB is running locally
const dbUrl = 'http://127.0.0.1:5984'
const app = express()
app.use(express.static('public'))
app.all('/db/*', function (req, res) {
  const fullUrl = dbUrl + req.url.replace(/^\/db/, '') // strip leading "/db"
  req.pipe(request(fullUrl)).pipe(res)
})
const port = process.env.PORT || 4600
app.listen(port)

The index.html page can be moved into folder /public and just go back to the server to get the data

1
2
3
console.log('app started')
this.db = new PouchDB('tododb')
var remoteDB = new PouchDB(window.location.origin + '/db/c1')

The application is working locally, let us deploy it.

Deploying server with hidden database

Log into your Dokku box and create new application

1
2
$ dokku apps create pouch-couch
Creating pouch-couch... done

Stop exposing Couch database directly to the outside world. Instead link it to the new pouch-couch app.

1
2
3
4
$ dokku couchdb:unexpose c1
-----> Service c1 unexposed
$ dokku couchdb:link c1 pouch-couch
COUCHDB_URL: http://c1:<long sha>@dokku-couchdb-c1:5984/c1

The command shows long unique URL string we should be using to access the Couch database from our proxy application. We should use this variable name in the server.js if available.

1
const dbUrl = process.env.COUCHDB_URL || 'http://127.0.0.1:5984/c1'

Add simple "Procfile" to the repo to let the hosting environment know we need a web process

1
2
3
cat "web: npm start" > Procfile
git add Procfile
git commit -m "add proc file"

Deploy the application to the Dokku server by pushing it as a Git remote

1
2
3
4
5
6
7
$ git remote add dokku dokku@<domain name>.com:pouch-couch
$ git push dokku master
...
=====> Application deployed:
       http://pouch-couch.<domain name>.com
To dokku@<domain name>.com:pouch-couch
   011273a..f1673ad  master -> master

If we browse to http://pouch-couch.<domain name>.com we will see an empty HTML page (there are no tasks yet), and if we browser to http://pouch-couch.<domain name>.com/db/ we can see the CouchDB database system information.

We should quickly protect the application from any interception by installing SSL. We can also protect the requests to the database in the server code if we wanted to.

Result

We got an isolated database only accessed via the Node server. The database and the server are running in two connected Docker containers on a single DigitalOcean "droplet" server. The web client (or multiple ones) works with local database in the browser, and the PouchDB syncs the data automatically.

pouch-couch architecture

Multiple clients can connect to the same url at the same time and have the data stay in sync, even when going offline and reconnecting. The PouchDB synchronization does the heavy lifting, while the web app only works with the local data source.

Additional information