products

Web Packing the Internet

WebPack as a service for quick personal projects and examples.

Recently I had the pleasure of trying HyperApp.js and described the experience in Pure programming with Hyper App. One of the first things I tried was a simple example. In the HyperApp architecture the app is decoupled from the "view" generation. You can use any virtual DOM library to actually produce the page elements.

In my case I wanted to use hyperx so my HTML example page looked like this

1
2
3
4
5
<body>
  <script src="https://unpkg.com/hyperapp"></script>
  <script src="https://wzrd.in/standalone/hyperx"></script>
  <script src="app.js"></script>
</body>

The first script works reliably - unpkg.com is solid "NPM CDN". The second script https://wzrd.in worked at first, but then started returning 502 error. The https://wzrd.in is "Browserify as a service". It is a valuable service, but I wanted something that I could control in case it went down.

My second consideration was trying examples where several libraries had to be packed together to allow me to quickly try something. For example Cycle.js framework allows one to mix and match 3 different parts in one application: stream library, DOM layer and runner function. But there is no easy way to load these from a single script tag, thus one has to either pack these NPM modules locally and serve; or use one of the prepacked online playgrounds, like webpackbin.com. I do not like iframed playgrounds like this (jsfiddle, codepen, etc) because there is one level of iframed DOM indirection which makes exploring and working with the code harder for me.

Both choices require too much effort when all I want is to explore an example application. It is especially hard to justify when I want to include a new framework in an existing application to see / show how the two can inter-operate. Remember how simple it was to show Angular 1 in an existing static page or web application?

1
2
3
4
5
6
7
8
9
10
11
12
13
<script src="cdn/angular.js"></script>
<div ng-app="myApp">
  <ul ng-controller="Todo">
    <li ng-repeat="todo in todoList">
      {{todo.label}}
    </li>
  </ul>
</div>
<script>
  angular.module('myApp', [])
    .controller('Todo', ...)
  // BOOM App is working!
</script>

A single script needs to be included and everything is working, even in a static page. I want the same for modern frameworks that require bundling first.

Lib bundle

webpackbin.com does something interesting. It packs a specific list of modules for Cycle for example as a bundle "dll.js". For example https://www.webpackbin.com/bins/-KfROsIlNDdauJZGG8ep has all Cycle dependencies packed into https://cdn.jsdelivr.net/webpack/%40cycle%2Fdom%4016.0.0%2B%40cycle%2Frun%403.0.0%2Bxstream%4010.3.0/dll.js. Note that the bundle should be immutable, since the versions of the dependencies is specified in the url. Unfortunately, I could not use the bundle due to a WebPack limitation - I do not know where these three libs are in the packed list. I know they are there, but their specific indices are hard to figure out!

At this point, I stopped trying to load the webpackbin "dll.js" and instead decided to write my own webpacking micro-service.

Webpack as a service

I wrote web-packing - a tiny service one can easily host (Zeit.co Now works really well). The service allows playing with apps like Cycle.js without any problems.

Just give it a list of NPM modules to bundle and you will get an object window.packs. Every listed library will be there, camel cased. For example,

1
2
3
4
<script src="http://where-is-web-packing/hyperx"></script>
<script>
var hyperx = window.packs.hyperx
</script>

or

1
2
3
4
5
6
7
8
9
<script src="http://where-is-web-packing/@cycle/run&@cycle/dom"></script>
<script>
// instead of
// import {run} from '@cycle/run'
// import {div, label, input, hr, h1, makeDOMDriver} from '@cycle/dom'
// do this
const {run} = window.packs.run
const {div, label, input, hr, h1, makeDOMDriver} = window.packs.dom
</script>

Pretty simple.

Speed

Installing NPM modules and bundling takes time, usually several seconds. Doing this every time someone requests a bundle is very wasteful. We can take a few shortcuts to save time and avoid repetitive work. For example, we can cache created bundles to quickly return them.

We can go one step further. We can return the bundle with the following response header to store the created bundle in the browser's cache and every proxy in between. This ensures that a client gets the bundle instantly on reload.

1
cache-control:max-age=99999999, public, immutable

See Mozilla docs for info on each property.

And just for kicks, I return the new ServerTiming header that shows how long the service spent creating the bundle.

1
server-timing:install=2.024; "NPM install", bundle=0.136; "Webpack bundling"

In Chrome this looks like this

Server Timing

We could even push the bundles to a CDN if necessary, but that would require a little more work.

Time to hack!

Special note on Cycle.js

One can run Cycle.js application directly using universal bundles. See Install without npm This approach can be a little bit user unfriendly. For example the first example:

1
2
3
4
5
6
7
<script src="https://unpkg.com/@cycle/[email protected]/dist/cycle-run.js"></script>
<script src="https://unpkg.com/@cycle/[email protected]/dist/cycle-dom.js"></script>
<script>
const {run} = Cycle
const {div, label, input, hr, h1, makeDOMDriver} = CycleDOM
// rest of the code
</script>

The execution crashes on startup with the stack trace

1
2
Uncaught TypeError: Cannot read property 'default' of undefined
    at makeSinkProxies (cycle-run.js:33)

Only by looking at the code we can find the offending line:

1
sinkProxies[name_1] = xstream_1.default.createWithMemory();

We need the library xstream and turns out it needs to go first in the list of scripts

1
2
3
<script src="https://unpkg.com/[email protected]/dist/xstream.js"></script>
<script src="https://unpkg.com/@cycle/[email protected]/dist/cycle-run.js"></script>
<script src="https://unpkg.com/@cycle/[email protected]/dist/cycle-dom.js"></script>

Good, but not great.