Really painless modular development

Keeping project's dependencies up to date with next-update and changed

Previously I described how an excellent package management system included with nodejs solves giant project problem. A project can be quickly split into multiple separate modules, each linked to its dependencies through public or private repository, or even through direct git repo urls. The benefits are clear:

• each module can be a small logical unit, easy to understand and test
• updates to the module do not break its dependents, as long as they are released under new version tag.
• there are lots of excellent open sources packages already available (more than 40K as of October 2013)

Example

For example, a graphical calculator application might start as a main module, that depends on the arithmetic module and on the command line interface, both developed in parallel with the main application and hosted at a public repository.

// calculator's package.json
"name": "calculator",
"version": "0.0.1",
"dependencies": {
    "arithmetic": "0.0.1",
    "cli": "0.0.1"
}

Let's say the arithmetic module adds more testing, fixes internal bugs, maybe even refactors its public interface, then releases version 0.0.2. The main Calculator app does not have to switch to the new version right away, and might never do this at all!

Keeping dependencies up to date problem

Immediately, we see the problem with highly modular development: each module always keeps falling behind. Its dependencies keep releasing new versions, adding features, fixing bugs, maybe reacting to hardware changes, or even to the changes in their own dependencies.

In a single giant application, this problem does not appear, since anyone touching files inside the arithmetic folder would have to right away update files in the calculator folder. Instead the problem manifests itself in stagnating project, requiring tremendous effort to add each new feature. Give me a modular project any day over that ;)

I often use a lot of 3rd party dependencies in my projects, mainly because I hate writing code that someone already has designed, written and tested. For example, my project xplain that generates API docs from unit tests uses 20 3rd party libraries.

I use david-dm.org to see if my project's dependencies are out of date. It is a very simple system - just give it the github repo with package.json and it will compare the declared versions with the latest available ones. It even gives beautiful badges one can include in the project's README.md file

An important reason to keep your dependencies up to date is security. Tools from nodesecurity projects like grunt-nsp-shrinkwrap and grunt-retire warn you if a dependency has a known security risk and should be upgraded.

next-update and changed

David-dm.org shows if the dependencies are up to date. If they are not, I need to:

• test if upgrading to latest version breaks my application
• find what has changed to decide if upgrade is really necessary

I perform these tasks often enough to automate them.

next-update

next-update is a command line tool that checks if there are newer versions, installs them one by one and runs project's unit tests (you do have unit tests, right?) to see if upgrading a particular dependency is possible or not. At the end it reverts everything back to the declared versions and prints which modules can be updated. It even prints the command that would install the successful dependencies. In the Calculator app case, it might show something like:

$ next-update
current versions: arithmetic: 0.0.1, cli: 0.0.1
available: arithmetic: 0.0.2
testing [email protected]
    tests pass
reverting to [email protected]
You can update to latest versions using command:
npm install [email protected] --save

next-update by default runs npm test command, but you can specify any command to run as an option.

You can see the next-update in action in the screencast below

next-update-stats

I run free anonymous update stats service at next-update.herokuapp.com. Anytime next-update tests going from version X to Y of dependency N, it sends the result (true/false) to the stats service. Then you can lookup how likely a particular update is based on overall results.

next-updater

A CLI tool that uses next-update to update multiple repos / NPM modules.

$ next-update -u bahmutov --tag --publish --allow minor

The above command will update all packages by user bahmutov. Only minor and patch versions of dependencies will be checked. If upgrade is successful, new version will be tagged and published to NPM. The next-updater makes keeping multiple projects up to date a breeze.

changed

To quickly fetch version history for any NPM module, I wrote changed. It installs a module in a temp folder and looks for changes.md, history.md or similarly named file. As a last resort it will even accept README.md. Once found, the file is printed to the output console:

$ changed qunit-promises
what has changed in qunit-promises
  installing qunit-promises

0.0.11 / 2013-10-06
==================

  * ignoring build json files
  * syncing properties from package.json to bower.json automatically
  * added pre-push command

Keeping human readable log file with your project is a good practice, and I suggest using git changelog command that comes with git-extras to make it super simple.

dont-break

dont-break solves the opposite problem. It checks if the current working version of a package breaks any dependent module (either specific or top NPM downloads).

$ dont-break --top-downloads 10

Checks if the module in the current folder does not break tests in the top 10 most downloaded dependent projects. Useful to check before publishing new version.

Open questions

I use next-update a lot in my projects, and usually most projects with clearly defined stable public interfaces do not break my applications. I update the dependencies as soon as tests pass and next-update approves. Sometimes, if I am hesitant, I look up changes using changed to see if the effort to fix my project is worth the trouble.

Question 1

It would be very nice to automate the combination next-update + update all dependencies that do not break the tests. It could be a continuous integration job, running extensive tests, and committing the updated package.json after successful updates.

answer I wrote next-updater to update multiple repos / npm packages at once.

Question 2

The information is always pulled: calculator checks if a newer version of arithmetic is compatible or not. It is up to the dependent module to find the reason for the breaking build and fix it. The author of arithmetic has no idea that he broke a dependent module. In case of enterprise development, where teams develop modules in parallel, I would like to have a system that collects information from the unsuccessful next-update attempts and transmits the information upstream to the breaking dependency's author. This would make the integration and convergence process faster.

Question 3

In a single application, due to transitive nature of the dependencies, you might end up with several versions of the same module. NPM allows this, it will try to use the latest version (I think, the found info seems to contradict what I see during installs). It would be nice to run next-update recursively to get to consistent state along the entire dependency tree.