products

Deploying private NPM modules to Zeit

How to bundle a server including private modules and static files.

Update Zeit has just released the official way to use private NPM modules, see the announcement.

I have been playing with Zeit.co - a super quick immutable deployment environment targeted at Node projects. If I have a server in a local folder, it could be deployed to new server almost instantly. It was a perfect solution for deploying a small web hook, or a chat server.

Yet there were two limitations:

  • lack of environment variables support; this was simple to work around using a separate private repo and a tool like as-a
  • the deployment tool does not install any private NPM dependencies due to lack of NPM authentication

This post shows how I worked around the second limitation by bundling the entire server code using browserify, preparing a single source file to be deployed. At the end there will be just a single JavaScript deployed, with zero NPM dependencies.

Example application

As an example I took a chat application server implemented on top of Feathers framework. The original example already had some modifications to make it work on Zeit

  1. No Socket.io on Zeit - I have left the REST api only
  2. Local data can be written into /tmp folder only.
  3. (Optional) the server can determine its own host url using environment variable NOW_URL

I have published the chat app on NPM under name feathers-chat-app-gleb, but to make the example more realistic, I have included a dummy private NPM dependency. The dependency @bahmutov/private-foo comes from public GitHub repo private-foo but the module has been published as a private scoped module on NPM registry. The chat application just prints the value exported by the private module

src/app.js
1
2
3
4
5
const privateFoo = require('@bahmutov/private-foo');
console.log('I include', privateFoo);
// npm start
// I include private foo
// Feathers application started on localhost:3030

How can we deploy this application to Zeit using the zeit now tool?

Authentication using NPM_TOKEN

First, let us try deploying the application as is.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
$ now -f
> Deploying "/Users/irinakous/git/feathers-chat-app"
> Using Node.js 6.2.1 (requested: `6`)
> Ready! https://feathers-chat-app-gleb-fhzggnzoij.now.sh (copied to clipboard) [2s]
...
NPM install errors (too fast too see)
...
> Building
> â–˛ npm install
> Installing package subarg@^1.0.0
> Installing package through2@^2.0.0
> Installing package xtend@^4.0.0
> Installing package bn.js@^4.0.0
> Installing package inherits@^2.0.1
> Installing package minimalistic-assert@^1.0.0
> Installing package duplexer2@~0.1.0
> Installing package readable-stream@^2.0.2
> Installing package acorn@^1.0.3
> Installing package defined@^1.0.0
> â–˛ npm start
> module.js:442
>     throw err;
>     ^
> Error: Cannot find module 'feathers'
>     at Function.Module._resolveFilename (module.js:440:15)
>     at Function.Module._load (module.js:388:25)
>     at Module.require (module.js:468:17)
>     at require (internal/module.js:20:19)
>     at Object.<anonymous> (/home/nowuser/src/src/app.js:4:21)

So the deployment fails, but the install log passes way too fast to see the problem. To read the exception message, let us record the terminal output using the awesome asciinema utility.

1
2
3
4
5
6
$ asciinema rec
$ now -f
$ exit
~ Asciicast recording finished.
~ Press <Enter> to upload, <Ctrl-C> to cancel.
https://asciinema.org/a/acp3tvh8mui1o7y6goox42t29

Opening the movie url, we can observe the detailed output, and even embed it below

Now we can see the install error clearly

1
2
3
4
5
message: 'Response code 404 (Not Found)',
host: 'registry.npmjs.org',
method: 'GET',
path: '/@bahmutov%2Fprivate-foo',
statusCode: 404

The install fails, as it should; the registry returns 404 when someone is requesting our private module. Can we copy the personal NPM auth token into the local .npmrc file to allow Zeit deploy to install the private module? If we keep the repo private, this could be acceptable solution. We can even git ignore the .npmrc file to avoid checking it into the repo (and even use a tool like ban-sensitive-files to prevent this from happening).

Yet, running the now tool shows that the local .npmrc file is ignored and the private scoped module is still NOT installed. Hmm.

Shrink packing

If we cannot install private dependencies from the NPM registry, we could try including the downloaded dependencies directly in the git repo. The best way to keep code dependencies together with source is shrinkpack.

First, use npm shrinkwrap command to "fix" the exact versions of all dependencies, including the private ones. We are excluding the dev dependencies though.

1
2
3
$ npm prune
$ npm shrinkwrap
wrote npm-shrinkwrap.json

The written file npm-shrinkwrap.json has every dependency resolution. Here is the start of the file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
  "name": "feathers-chat-app-gleb",
  "version": "1.0.0",
  "dependencies": {
    "@bahmutov/private-foo": {
      "version": "1.0.0",
      "from": "@bahmutov/private-foo@*"
    },
    "accepts": {
      "version": "1.3.3",
      "from": "accepts@>=1.3.3< 1.4.0",
      "resolved": "http://registry.npmjs.org/accepts/-/accepts-1.3.3.tgz"
    }
  }
}

Next, we will run the command to pack every resolved dependency back into the .tar.gz file (almost like it was downloaded from the NPM registry).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ npm install -g shrinkpack
$ shrinkpack
i 231 dependencies in npm-shrinkwrap.json
i 0 need removing from ./node_shrinkwrap
i 231 need adding to ./node_shrinkwrap
i 209 are in your npm cache
i 22 need downloading
i 1 have a missing "resolved" property
? @bahmutov/[email protected] has no "dist.tarball" in
  /git/feathers-chat-app/node_modules/@bahmutov/private-foo/package.json
? @bahmutov/[email protected] contacting registry...
âś“ set missing "resolved" property for @bahmutov/[email protected] to
  https://registry.npmjs.org/@bahmutov/private-foo/-/private-foo-1.0.0.tgz
...
shrinkpack +231 -0 ↓22 ✓1 00:07

All the .tar.gz files were placed into node_shrinkwrap folder

1
2
3
4
5
6
$ ls -l node_shrinkwrap/
total 9880
-rw-r--r--  1     576 Jul  1 08:52 @bahmutov-private-foo-1.0.0.tgz
-rw-r--r--  1    5102 Jul  1 08:52 accepts-1.3.3.tgz
-rw-r--r--  1  136743 Jul  1 08:52 acorn-1.2.2.tgz
...

The tool also updated the npm-shrinkwrap.json file, setting the paths to new local files

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "name": "feathers-chat-app-gleb",
  "version": "1.0.0",
  "dependencies": {
    "@bahmutov/private-foo": {
      "version": "1.0.0",
      "from": "@bahmutov/private-foo@*",
      "resolved": "./node_shrinkwrap/@bahmutov-private-foo-1.0.0.tgz"
    },
    "accepts": {
      "version": "1.3.3",
      "from": "accepts@>=1.3.3< 1.4.0",
      "resolved": "./node_shrinkwrap/accepts-1.3.3.tgz"
    }
  }
}

We can check in the node_shrinkwrap folder into our Git repository, and it will make npm install instant from now on, because the registry will NOT be used - all the module resolutions are present locally after cloning.

Trying now deployment command, and ... it does not work. The now command only uploads the JavaScript files, NOT the .tar.gz files. Even when I forced it to upload by moving the node_shrinkwrap folder into the src folder, the installation command did not resolve the local .tar.gz files. Maybe the now command does not respect the npm-shrinkwrap.json, or maybe there is some other reason. We need to find another way.

Bundling the server code

Our goal is to run the application. In the "normal" Node mode, each piece of JavaScript can be loaded from a separate file using require(<filename>) call. The <filename> is resolved relative to the path, or from the node_modules folder. If we cannot have some files loaded from the node_modules because the install failed, we can try "bundling" the source code into a single source file - just like we would for running an app inside a browser!

I installed browserify and used a simple build.js to pass options that build a bundle targeted for Node environment (instead of the browser).

build.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
var fs = require('fs')
var browserify = require('browserify')
var insertGlobals = require('insert-module-globals')
browserify('./src/index.js', {
    builtins: false,
    commondir: false,
    insertGlobalVars: {
        __filename: insertGlobals.vars.__filename,
        __dirname: insertGlobals.vars.__dirname,
        process: function() {
            return;
        },
    },
    browserField: false,
})
.bundle()
.pipe(fs.createWriteStream('./src/out.js'))

Let us build the bundle and see if we can run the project without node_modules folder.

1
2
3
4
5
6
7
8
9
10
11
12
$ node build.js
$ ls -l src/out.js
-rw-r--r--  1  1575733 Jul  1 09:09 src/out.js
$ node src/out.js
I include private foo
Feathers application started on localhost:3030
^C
$ mv node_modules tmp
$ node src/out.js
I include private foo
Feathers application started on localhost:3030
^C

I also had to move folders config and public into src for deployment to include them.

We have all the code needed inside a single file src/out.js. Let run use this bundle when running now. Oops, still there is a problem

1
2
3
4
5
6
7
8
> Error: Cannot find module '/git/feathers-chat-app/src/config/default'
>     at Function.Module._resolveFilename (module.js:440:15)
>     at Function.Module._load (module.js:388:25)
>     at Module.require (module.js:468:17)
>     at require (internal/module.js:20:19)
>     at s (/home/nowuser/src/src/out.js:1:176)
>     at /home/nowuser/src/src/out.js:1:367
>     at EventEmitter.<anonymous> (/home/nowuser/src/src/out.js:20848:26)

The feathers-configuration module loads config JSON files by looking through the files in the config folder. This fails, because the browserify does not include these - there is no way it can know which files to include! So we need to hack around the config and just load the files we need for this case ourselves. Change the src/app.js to load the config/default.json ourselves and set the properties.

app.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// app.configure(configuration(__dirname));
const defaults = require('./config/default')
Object.keys(defaults).forEach((key) => {
  // TODO adjust the paths, merge environment settings, etc
  var value = defaults[key]
  function isPath(s) {
    return /^\./.test(s)
  }
  if (isPath(value)) {
    console.log('resolving', value)
    value = path.resolve(__dirname, value)
    console.log('resolved', value)
  }
  app.set(key, value);
});

In the future I will refactor the feathers-configuration module to allow sending multiple config objects without loading them on the fly from the file system.

Next, I removed favicon middleware, since the .ico file is not bundled into JavaScript

app.js
1
2
3
app
// .use(favicon( path.join(app.get('public'), 'favicon.ico') ))
  .use('/', serveStatic( app.get('public') ))

At this point, the REST API is working on Zeit, which we can see if we try to grab the list of users, for example.

1
2
3
4
5
6
7
8
9
10
$ http https://feathers-chat-app-gleb-kmljntdwmf.now.sh/users
HTTP/1.1 401 Unauthorized
X-Powered-By: Express
{
    "className": "not-authenticated",
    "code": 401,
    "errors": {},
    "message": "Authentication token missing.",
    "name": "NotAuthenticated"
}

But if we try to open the deployed site in the browser we get 404! Why do we not get served the HTML file public/index.html?

Bundling the mock read-only file system

The deployed application does not include any non-javascript files from the public folder. We have the missing static pages there

1
2
3
4
5
6
7
8
$ ls -l src/public/
total 56
-rw-r--r--  1  3295 Jun 19 10:26 app.js
-rw-r--r--  1  3443 Jun 19 10:26 chat.html
-rw-r--r--  1  5533 Jun 18 23:24 favicon.ico
-rw-r--r--  1  1410 Jun 18 23:29 index.html
-rw-r--r--  1  1357 Jun 18 23:32 login.html
-rw-r--r--  1  1365 Jun 18 23:33 signup.html

Somehow we need to

  1. include the text contents of these files in the built bundle
  2. serve the bundled text for each file, as the server tries to read it from its "file system"

I love mocking environments using browserify, see blog posts Angular in WebWorker and Express in ServiceWorker. Our current problem looks like a fun little challenge.

The static folder in Feathers is served using the plain Express serve-static package, which uses send module to open a file stream to a file and pipe it back into the resource. If only these modules had a file system that pointed at contents from the bundle!

Luckily there is module mock-fs that does exactly this

1
2
3
4
var mock = require('mock-fs');
mock({
  'public/index.html': 'hi there'
});

During the build step we can collect all HTML files to be bundled into a single file, then require this mock object when we bundle the code. We are going to turn on the mocking, and the server will have no choice but return the bundled responses. I have described the steps on a tiny example in separate repo browserify-server. First, the build collects all public HTML and JS files and puts the text into a single JSON object.

build.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
const glob = require('glob')
const files = glob.sync('src/public/*.js').concat(glob.sync('src/public/*.html'))
const mockOptions = {}
files.forEach((fullName) => {
  mockOptions[fullName] = read(fullName, 'utf8')
})
const publicFiles = './public-bundle.json'
write(publicFiles,
  JSON.stringify(mockOptions, null, 2) + '\n', 'utf8')
console.log('saved public files contents to', publicFiles)
// browserify commands ...
browserify('./start.js', {
  ...
})

We also change the entry file for browserify to start.js, which right away requires the original index.js file, and mocks the file system using the file created.

start.js
1
2
3
4
require('./src/index')
const mockOptions = require('./public-bundle')
const mock = require('mock-' + 'fs')
mock(mockOptions)

To avoid getting into recursive bundling, I split the mock-fs module name in the require('mock-' + 'fs') expression; this is a common trick to stop browserify from messing with things it should leave alone.

We can find our HTML inside the bundle

1
2
$ grep "<title>" src/out.js
"src/public/chat.html": "<html>\n<head> ...

We can even run the bundled code with the folder src/public removed

1
2
$ mv src/public src/tmp
$ npm start

Deploying the bundle to zeit and ... in production the module mock-fs is not found! Seems Zeit does not let you mock the file system.

Ok, time to find one more work around.

Writing the temp file system

Zeit does allow you to write data into /tmp folder. This is where we write the messages database for example. We have the public folder bundled into our single application JavaScript file; we can just dump the files into the /tmp folder at the startup step. The server app then can serve the static files normally.

The build file still collects all .js and .html files in the public folder, and dumps their contents into a single .json file

build.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
function collectPublicFiles () {
  const glob = require('glob')
  const files = glob.sync('public/*.js').concat(glob.sync('public/*.html'))
  console.log('public files\n' + files.join('\n'))
  const read = require('fs').readFileSync
  const mockOptions = {}
  files.forEach((fullName) => {
    mockOptions[fullName] = read(fullName, 'utf8')
  })
  return mockOptions
}
function mockPublicFiles () {
  const mockOptions = collectPublicFiles()
  // console.log(mockOptions)
  const publicFiles = './public-bundle.json'
  write(publicFiles,
    JSON.stringify(mockOptions, null, 2) + '\n', 'utf8')
  console.log('saved public files contents to', publicFiles)
}
mockPublicFiles()
browserify('./start.js', {...})

Then start.js file dumps the collected contents into /tmp folder

start.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
function writePublicFiles () {
  const mockOptions = require('./public-bundle')
  const fs = require('fs')
  const outputFolder = '/tmp'
  const dirname = require('path').dirname
  const inTemp = require('path').join.bind(null, outputFolder)
  const mkdirp = require('mkdirp')
  const write = require('fs').writeFileSync
  Object.keys(mockOptions).forEach((name) => {
    const full = inTemp(name)
    console.log('writing file', full)
    const folder = dirname(full)
    mkdirp.sync(folder)
    write(full, mockOptions[name], 'utf8')
  })
}
writePublicFiles()
require('./src/index')

Let us deploy and see if we finally got a happy server

1
2
3
4
5
6
7
8
9
10
11
12
13
$ now
> Ready! https://feathers-chat-app-gleb-yhqzmzizjz.now.sh (copied to clipboard) [2s]
> Upload [====================] 100% 0.0s
> Sync complete (1.51MB) [8s]
> â–˛ npm run now-start
> writing file /tmp/public/app.js
> writing file /tmp/public/chat.html
> writing file /tmp/public/index.html
> writing file /tmp/public/login.html
> writing file /tmp/public/signup.html
> I include private foo
> Feathers application started on localhost:3030
> Deployment complete!

From the terminal let us fetch the index page

1
2
3
4
5
6
7
8
9
10
$ http https://feathers-chat-app-gleb-yhqzmzizjz.now.sh
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
<html>
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8">
    <title>Feathers Chat</title>
    ...
  </head>
</html>

These are not tears of sadness, these are tears of happiness

You can check the app yourself at the above url https://feathers-chat-app-gleb-yhqzmzizjz.now.sh.

As the final change, I added bundling and writing the config JSON files the same way to reuse the original feathers-configuration code.

Installing nothing

To speed up the deploys, we can skip the dependency deployment on Zeit. We have already bundled everything into the single src/out.js file; there is no need to run npm install when deploying. We can write a simple script to empty the dependencies and devDependencies in package.json

no-deps.js
1
2
3
4
5
6
'use strict'
const pkg = require('./package.json')
pkg.dependencies = pkg.devDependencies = []
const write = require('fs').writeFileSync
write('./package.json', JSON.stringify(pkg, null, 2) + '\n', 'utf8')
console.log('removed dependencies from package.json')

We will run the above script before the deployment, and will restore the package.json back to its full state after the deploy.

packate.json
1
2
3
4
5
6
7
8
9
{
  "scripts": {
    "now-start": "node src/out.js",
    "bundle": "node build.js",
    "predeploy": "npm run bundle && node no-deps.js",
    "deploy": "now",
    "postdeploy": "git checkout package.json"
  }
}

Let us try this. Without removing the superfluous node_modules the bundling and deployment takes 60 seconds.

1
2
3
4
5
6
7
8
9
$ time npm run plain-deploy
> Building
> â–˛ npm install
> Installing package subarg@^1.0.0
> Installing package through2@^2.0.0
> Installing package xtend@^4.0.0
...
> Deployment complete!
real  0m59.948s

With just the bundling and moving just the single unchanged file it takes 25 seconds.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
$ time npm run deploy
> [email protected] predeploy /git/feathers-chat-app
> npm run bundle && node no-deps.js
> [email protected] bundle /git/feathers-chat-app
> node build.js
public files
public/app.js
public/chat.html
public/index.html
public/login.html
public/signup.html
saved public files contents to ./public-bundle.json
removed dependencies from package.json
> [email protected] deploy /git/feathers-chat-app
> now
> Deploying "/git/feathers-chat-app"
> Using Node.js undefined (requested: `6`)
> Ready! https://feathers-chat-app-gleb-rjynpwiawt.now.sh (copied to clipboard) [2s]
> Upload [====================] 100% 0.0s
> Sync complete (837B) [989ms]
> Initializing…
> Building
> â–˛ npm install
> Error {
  Error: ENOENT: no such file or directory, scandir '/home/nowuser/src/node_modules'
> I include private foo
> Feathers application started on localhost:3030
> Deployment complete!
> [email protected] postdeploy /git/feathers-chat-app
> git checkout package.json
real  0m24.659s

Fast and simple.

Conclusion

I wish Zeit had support for private modules, yet lack of this feature was a great opportunity for hacking together a work around. Notice that I had pursued multiple solutions that lead to a dead end. Yet every solution that did not work taught me something new. One just has to be patient and keep coming up with new ideas, chipping away at the problem.

The final solution is a weird one; I am sure the original problem has to do with browserified bundle not being able to properly load the regular files due to some path mangling. I will not pursue this further, instead I hope to see the Zeit team implement installation of private NPM modules using proper authentication.

You can find the code at bahmutov/feathers-chat-app and see the running application at https://feathers-chat-app-gleb-rjynpwiawt.now.sh

If you like hacking the code like me, read How to become a better hacker blog post.