Smaller published NPM modules

How to measure size and publish smaller modules to NPM.

By default, the entire module is published to the NPM registry, except for files listed inside your .gitignore file. This filters out the dependencies but still allows quite a lot of files to be pushed. The users then have to wait for the package to download on every install. See the complaint in this article. As an example, the author reports that the grunt module is a 6MB download.

Can you measure your own modules and make them smaller? I have decided to find out. I will use check-more-types as an example. It has just a few files, plus unit tests and documentation folder used to build the README markdown file.

First, we need to find out the size of the current package. NPM stores all files as tar files, thus we need to make one and see its size. You can make a tar archive from the current folder using npm pack . command. Let us make a tar file, show its contents and size using a script suggested by Mathias Bynens mathias

tarball="$(npm pack .)"; wc -c "${tarball}"; tar tvf "${tarball}"; rm "${tarball}";

I measured the tar size at the commit 3ada360

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ tarball="$(npm pack .)"; wc -c "${tarball}"; tar tvf "${tarball}"; rm "${tarball}";

25184 check-more-types-2.1.2.tgz
-rw-r--r-- 0 501 20 1977 Nov 19 13:55 package/package.json
-rw-r--r-- 0 501 20 64 Nov 19 13:18 package/.npmignore
-rw-r--r-- 0 501 20 19703 Nov 19 13:49 package/README.md
-rw-r--r-- 0 501 20 1073 Nov 19 13:18 package/LICENSE
-rw-r--r-- 0 501 20 2534 Nov 19 13:18 package/Gruntfile.js
-rw-r--r-- 0 501 20 18204 Nov 19 13:18 package/check-more-types.js
-rw-r--r-- 0 501 20 6723 Nov 19 13:49 package/check-more-types.min.js
-rw-r--r-- 0 501 20 600 Nov 19 13:49 package/bower.json
-rw-r--r-- 0 501 20 162 Nov 19 13:18 package/.travis.yml
-rw-r--r-- 0 501 20 1756 Nov 19 13:18 package/.jshintrc
-rw-r--r-- 0 501 20 655 Nov 19 13:18 package/docs/README.tmpl.md
-rw-r--r-- 0 501 20 1936 Nov 19 13:18 package/docs/badges.md
-rw-r--r-- 0 501 20 255 Nov 19 13:18 package/docs/footer.md
-rw-r--r-- 0 501 20 240 Nov 19 13:18 package/docs/install.md
-rw-r--r-- 0 501 20 13707 Nov 19 13:49 package/docs/use.md
-rw-r--r-- 0 501 20 127 Nov 19 13:18 package/test/check-more-types-minified-spec.js
-rw-r--r-- 0 501 20 78 Nov 19 13:18 package/test/check-more-types-spec.js
-rw-r--r-- 0 501 20 467 Nov 19 13:18 package/test/load-under-node-test.js
-rw-r--r-- 0 501 20 738 Nov 19 13:18 package/test/synthetic-browser-spec.js
-rw-r--r-- 0 501 20 37754 Nov 19 13:18 package/test/unit-tests.js

Lots of files, and the packed size is 25184 bytes. Let us first automate this process a little. I found a good utility for reporting the same size, but avoiding the shell command - pkgfiles by Tim Oxley @secoif.

I installed this as dev dependency and added a prepublish script to the package.json

1
2
3
4
5
6
7
8
{
"devDependencies": {
"pkgfiles": "2.3.0"
},
"scripts": {
"prepublish": "pkgfiles"
}
}

The prepublish script will run every time on local install and when I run npm publish command. Let us see the output right now from npm run publish command

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
$ npm run prepublish

> [email protected] prepublish /Users/kensho/git/check-more-types
> pkgfiles


PATH SIZE %
.npmignore 0 B 0%
test/check-more-types-spec.js 78 B 0%
test/check-more-types-minified-spec.js 127 B 0%
.travis.yml 162 B 0%
docs/install.md 240 B 0%
docs/footer.md 255 B 0%
test/load-under-node-test.js 467 B 0%
bower.json 600 B 1%
docs/README.tmpl.md 655 B 1%
test/synthetic-browser-spec.js 738 B 1%
LICENSE 1.07 kB 1%
.jshintrc 1.76 kB 2%
docs/badges.md 1.94 kB 2%
package.json 2.05 kB 2%
Gruntfile.js 2.53 kB 2%
check-more-types.min.js 6.72 kB 6%
docs/use.md 13.71 kB 13%
check-more-types.js 18.2 kB 17%
README.md 19.7 kB 18%
test/unit-tests.js 37.75 kB 35%

DIR SIZE %
docs/ 16.79 kB 15%
test/ 39.16 kB 36%
. 108.77 kB 100%

PKGFILES SUMMARY
Size on Disk with Dependencies ~126.72 MB
Size with Dependencies ~88.58 MB
Publishable Size ~108.77 kB
Number of Directories 3
Number of Files 20

Very detailed information, and the heaviest files are reported last, making it convenient to see in the terminal. The heaviest is the docs and tests folders - we don't actually want to publish them!

There are 3 ways to limit the files published to the NPM registry. We are using the default - the files listed in .gitignore are black listed. We could write a separate .npmignore file to black list more files. Or we could white list files inside package.json. I prefer to white list files. Note that some files (like package.json, README, etc) are always white listed.

1
2
3
4
5
6
7
{
"files": [
"bower.json",
"check-more-types.js",
"check-more-types.min.js"
]
}

Note that in the files list you can exclude files using ! notation. For example if we have a test subfolder inside src folder, we can exclude it

1
2
3
4
5
6
{
"files": [
"src",
"!src/test"
]
}

If we had test files and production files in the same folder "src" we could ban individual files like this

1
2
3
4
5
6
{
"files": [
"src/*.js",
"!src/*-spec.js"
]
}

The new commit is bc3e2a1, let us see the size break down

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ npm run prepublish

> [email protected] prepublish /Users/kensho/git/check-more-types
> pkgfiles


PATH SIZE %
bower.json 600 B 1%
LICENSE 1.07 kB 2%
package.json 2.15 kB 4%
check-more-types.min.js 6.72 kB 14%
check-more-types.js 18.2 kB 38%
README.md 19.7 kB 41%

DIR SIZE %
. 48.45 kB 100%

PKGFILES SUMMARY
Size on Disk with Dependencies ~126.72 MB
Size with Dependencies ~88.58 MB
Publishable Size ~48.45 kB
Number of Directories 1
Number of Files 6

The published size went down 55% - from 107 kB to 48 kB. This is just raw source size, let us see the change in the tar file. Unfortunately, using the npm pack . command will call our prepublish script and getting confused by its output. Thus I renamed the prepublish step for now before running the tar command again. I even added the tarball=... command to the NPM scripts (under the name size) for reuse.

1
2
3
4
5
6
7
8
9
10
11
12
13
kensho at Kenshos-MacBook-Air.local  ~/git/check-more-types on master*
$ npm run size

> [email protected] size /Users/kensho/git/check-more-types
> tarball="$(npm pack .)"; wc -c "${tarball}"; tar tvf "${tarball}"; rm "${tarball}";

13179 check-more-types-2.1.2.tgz
-rw-r--r-- 0 501 20 2256 Nov 19 14:09 package/package.json
-rw-r--r-- 0 501 20 19703 Nov 19 13:58 package/README.md
-rw-r--r-- 0 501 20 1073 Nov 19 13:18 package/LICENSE
-rw-r--r-- 0 501 20 18204 Nov 19 13:58 package/check-more-types.js
-rw-r--r-- 0 501 20 6723 Nov 19 13:58 package/check-more-types.min.js
-rw-r--r-- 0 501 20 600 Nov 19 13:58 package/bower.json

Any dependent client will have to download only 13 kB instead of 25 kB, 50% size reduction down the wire!

Finally, I have decided to show the npm published size on every code push from the local repo to the remote master. I use my own [pre-git][pre-git] for managing Git hooks. I just added both the size and pkgfiles commands to the pre-push steps to run

npm install -D pre-git
1
2
3
4
5
6
7
8
9
10
11
12
13
14
{
"scripts": {
"pkgfiles": "pkgfiles",
"size": "tarball=\"$(npm pack .)\"; wc -c \"${tarball}\"; tar tvf \"${tarball}\"; rm \"${tarball}\";"
},
"config": {
"pre-git": {
"pre-push": [
"npm run size",
"npm run pkgfiles"
]
}
}
}

I bumped the package version from 2.1.2 to 2.2.0 to publish this smaller module.

To verify, I used a temp folder and NPM version 3.4.0

1
2
3
4
5
6
$ time npm i [email protected]
/private/tmp/test-small
└── [email protected]
real 0m2.706s
user 0m1.419s
sys 0m0.323s

Almost 3 seconds. Let us delete the node_modules folder and try the new package

$ rm -rf node_modules/
1
2
3
4
5
6
$ time npm i [email protected]
/private/tmp/test-small
└── [email protected]
real 0m1.716s
user 0m1.244s
sys 0m0.198s

We have shaved 1 second, almost 30% from the installation time for our single tiny module!

The final commands together in one place for simple copying (from kensho/check-more-types/package.json)

npm install -D pkgfiles pre-git
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
"devDependencies": {
"pkgfiles": "2.3.0",
"pre-git": "1.3.0"
},
"scripts": {
"pkgfiles": "pkgfiles",
"size": "tarball=\"$(npm pack .)\"; wc -c \"${tarball}\"; tar tvf \"${tarball}\"; rm \"${tarball}\";"
},
"config": {
"pre-git": {
"pre-push": [
"npm run size",
"npm run pkgfiles"
]
}
}
}

or even a shorter "size" command with variable "t" instead of "tarball" to keep the filename of the temporary archive

1
2
3
4
"scripts": {
"pkgfiles": "pkgfiles",
"size": "t=\"$(npm pack .)\"; wc -c \"${t}\"; tar tvf \"${t}\"; rm \"${t}\";"
}