products

Smaller published NPM modules

How to measure size and publish smaller modules to NPM.

By default, the entire module is published to the NPM registry, except for files listed inside your .gitignore file. This filters out the dependencies but still allows quite a lot of files to be pushed. The users then have to wait for the package to download on every install. See the complaint in this article. As an example, the author reports that the grunt module is a 6MB download.

Can you measure your own modules and make them smaller? I have decided to find out. I will use check-more-types as an example. It has just a few files, plus unit tests and documentation folder used to build the README markdown file.

First, we need to find out the size of the current package. NPM stores all files as tar files, thus we need to make one and see its size. You can make a tar archive from the current folder using npm pack . command. Let us make a tar file, show its contents and size using a script suggested by Mathias Bynens mathias

tarball="$(npm pack .)"; wc -c "${tarball}"; tar tvf "${tarball}"; rm "${tarball}";

I measured the tar size at the commit 3ada360

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ tarball="$(npm pack .)"; wc -c "${tarball}"; tar tvf "${tarball}"; rm "${tarball}";

   25184 check-more-types-2.1.2.tgz
-rw-r--r--  0 501    20       1977 Nov 19 13:55 package/package.json
-rw-r--r--  0 501    20         64 Nov 19 13:18 package/.npmignore
-rw-r--r--  0 501    20      19703 Nov 19 13:49 package/README.md
-rw-r--r--  0 501    20       1073 Nov 19 13:18 package/LICENSE
-rw-r--r--  0 501    20       2534 Nov 19 13:18 package/Gruntfile.js
-rw-r--r--  0 501    20      18204 Nov 19 13:18 package/check-more-types.js
-rw-r--r--  0 501    20       6723 Nov 19 13:49 package/check-more-types.min.js
-rw-r--r--  0 501    20        600 Nov 19 13:49 package/bower.json
-rw-r--r--  0 501    20        162 Nov 19 13:18 package/.travis.yml
-rw-r--r--  0 501    20       1756 Nov 19 13:18 package/.jshintrc
-rw-r--r--  0 501    20        655 Nov 19 13:18 package/docs/README.tmpl.md
-rw-r--r--  0 501    20       1936 Nov 19 13:18 package/docs/badges.md
-rw-r--r--  0 501    20        255 Nov 19 13:18 package/docs/footer.md
-rw-r--r--  0 501    20        240 Nov 19 13:18 package/docs/install.md
-rw-r--r--  0 501    20      13707 Nov 19 13:49 package/docs/use.md
-rw-r--r--  0 501    20        127 Nov 19 13:18 package/test/check-more-types-minified-spec.js
-rw-r--r--  0 501    20         78 Nov 19 13:18 package/test/check-more-types-spec.js
-rw-r--r--  0 501    20        467 Nov 19 13:18 package/test/load-under-node-test.js
-rw-r--r--  0 501    20        738 Nov 19 13:18 package/test/synthetic-browser-spec.js
-rw-r--r--  0 501    20      37754 Nov 19 13:18 package/test/unit-tests.js

Lots of files, and the packed size is 25184 bytes. Let us first automate this process a little. I found a good utility for reporting the same size, but avoiding the shell command - pkgfiles by Tim Oxley @secoif.

I installed this as dev dependency and added a prepublish script to the package.json

1
2
3
4
5
6
7
8
{
  "devDependencies": {
    "pkgfiles": "2.3.0"
  },
  "scripts": {
    "prepublish": "pkgfiles"
  }
}

The prepublish script will run every time on local install and when I run npm publish command. Let us see the output right now from npm run publish command

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
$ npm run prepublish

> [email protected] prepublish /Users/kensho/git/check-more-types
> pkgfiles


PATH                                    SIZE       %
.npmignore                              0 B        0%
test/check-more-types-spec.js           78 B       0%
test/check-more-types-minified-spec.js  127 B      0%
.travis.yml                             162 B      0%
docs/install.md                         240 B      0%
docs/footer.md                          255 B      0%
test/load-under-node-test.js            467 B      0%
bower.json                              600 B      1%
docs/README.tmpl.md                     655 B      1%
test/synthetic-browser-spec.js          738 B      1%
LICENSE                                 1.07 kB    1%
.jshintrc                               1.76 kB    2%
docs/badges.md                          1.94 kB    2%
package.json                            2.05 kB    2%
Gruntfile.js                            2.53 kB    2%
check-more-types.min.js                 6.72 kB    6%
docs/use.md                             13.71 kB   13%
check-more-types.js                     18.2 kB    17%
README.md                               19.7 kB    18%
test/unit-tests.js                      37.75 kB   35%

DIR                                     SIZE       %
docs/                                   16.79 kB   15%
test/                                   39.16 kB   36%
.                                       108.77 kB  100%

PKGFILES SUMMARY
Size on Disk with Dependencies  ~126.72 MB
Size with Dependencies          ~88.58 MB
Publishable Size                ~108.77 kB
Number of Directories           3
Number of Files                 20

Very detailed information, and the heaviest files are reported last, making it convenient to see in the terminal. The heaviest is the docs and tests folders - we don't actually want to publish them!

There are 3 ways to limit the files published to the NPM registry. We are using the default - the files listed in .gitignore are black listed. We could write a separate .npmignore file to black list more files. Or we could white list files inside package.json. I prefer to white list files. Note that some files (like package.json, README, etc) are always white listed.

1
2
3
4
5
6
7
{
  "files": [
    "bower.json",
    "check-more-types.js",
    "check-more-types.min.js"
  ]
}

Note that in the files list you can exclude files using ! notation. For example if we have a test subfolder inside src folder, we can exclude it

1
2
3
4
5
6
{
  "files": [
    "src",
    "!src/test"
  ]
}

If we had test files and production files in the same folder "src" we could ban individual files like this

1
2
3
4
5
6
{
  "files": [
    "src/*.js",
    "!src/*-spec.js"
  ]
}

The new commit is bc3e2a1, let us see the size break down

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ npm run prepublish

> [email protected] prepublish /Users/kensho/git/check-more-types
> pkgfiles


PATH                     SIZE      %
bower.json               600 B     1%
LICENSE                  1.07 kB   2%
package.json             2.15 kB   4%
check-more-types.min.js  6.72 kB   14%
check-more-types.js      18.2 kB   38%
README.md                19.7 kB   41%

DIR                      SIZE      %
.                        48.45 kB  100%

PKGFILES SUMMARY
Size on Disk with Dependencies  ~126.72 MB
Size with Dependencies          ~88.58 MB
Publishable Size                ~48.45 kB
Number of Directories           1
Number of Files                 6

The published size went down 55% - from 107 kB to 48 kB. This is just raw source size, let us see the change in the tar file. Unfortunately, using the npm pack . command will call our prepublish script and getting confused by its output. Thus I renamed the prepublish step for now before running the tar command again. I even added the tarball=... command to the NPM scripts (under the name size) for reuse.

1
2
3
4
5
6
7
8
9
10
11
12
13
kensho at Kenshos-MacBook-Air.local  ~/git/check-more-types on master*
$ npm run size

> [email protected] size /Users/kensho/git/check-more-types
> tarball="$(npm pack .)"; wc -c "${tarball}"; tar tvf "${tarball}"; rm "${tarball}";

   13179 check-more-types-2.1.2.tgz
-rw-r--r--  0 501    20       2256 Nov 19 14:09 package/package.json
-rw-r--r--  0 501    20      19703 Nov 19 13:58 package/README.md
-rw-r--r--  0 501    20       1073 Nov 19 13:18 package/LICENSE
-rw-r--r--  0 501    20      18204 Nov 19 13:58 package/check-more-types.js
-rw-r--r--  0 501    20       6723 Nov 19 13:58 package/check-more-types.min.js
-rw-r--r--  0 501    20        600 Nov 19 13:58 package/bower.json

Any dependent client will have to download only 13 kB instead of 25 kB, 50% size reduction down the wire!

Finally, I have decided to show the npm published size on every code push from the local repo to the remote master. I use my own [pre-git][pre-git] for managing Git hooks. I just added both the size and pkgfiles commands to the pre-push steps to run

npm install -D pre-git

1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
  "scripts": {
    "pkgfiles": "pkgfiles",
    "size": "tarball=\"$(npm pack .)\"; wc -c \"${tarball}\"; tar tvf \"${tarball}\"; rm \"${tarball}\";"
  },
  "config": {
    "pre-git": {
      "pre-push": [
        "npm run size",
        "npm run pkgfiles"
      ]
    }
  }
}

I bumped the package version from 2.1.2 to 2.2.0 to publish this smaller module.

To verify, I used a temp folder and NPM version 3.4.0

1
2
3
4
5
6
$ time npm i [email protected]
/private/tmp/test-small
└── [email protected]
real  0m2.706s
user  0m1.419s
sys 0m0.323s

Almost 3 seconds. Let us delete the node_modules folder and try the new package

$ rm -rf node_modules/

1
2
3
4
5
6
$ time npm i [email protected]
/private/tmp/test-small
└── [email protected]
real  0m1.716s
user  0m1.244s
sys 0m0.198s

We have shaved 1 second, almost 30% from the installation time for our single tiny module!

The final commands together in one place for simple copying (from kensho/check-more-types/package.json)

npm install -D pkgfiles pre-git

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
  "devDependencies": {
    "pkgfiles": "2.3.0",
    "pre-git": "1.3.0"
  },
  "scripts": {
    "pkgfiles": "pkgfiles",
    "size": "tarball=\"$(npm pack .)\"; wc -c \"${tarball}\"; tar tvf \"${tarball}\"; rm \"${tarball}\";"
  },
  "config": {
    "pre-git": {
      "pre-push": [
        "npm run size",
        "npm run pkgfiles"
      ]
    }
  }
}

or even a shorter "size" command with variable "t" instead of "tarball" to keep the filename of the temporary archive

1
2
3
4
"scripts": {
  "pkgfiles": "pkgfiles",
  "size": "t=\"$(npm pack .)\"; wc -c \"${t}\"; tar tvf \"${t}\"; rm \"${t}\";"
}