Many many moons ago in 2013 I have released next-update CLI tool and described it in the Really painless modular development blog post. The next-update automates dependency updates by installing each dependency one by one, running the project's tests, and if they still pass, keeping it. While this was exciting, there was one more trick up my sleeve - I had a companion service running online that collected anonymous upgrade stats. It kept just a tally of update pass/fail counts per NPM module. Something like this table for every dependency:
When next-update wanted to check if dependency X could be updated safely from version X to version Y it queried this online service and showed the stats to the user:
Great - across all projects, updating check-more-types from version 1.1.1 to version 1.7.3 was 74% successful without any user input. This number was computed from 31 projects that used next-update. On the other hand, updating check-types from 1.4.0 to version 3.3.0 was pretty much a guaranteed failure - none of the 26 projects that tried it were successful.
I loved this information - I strongly believe the wisdom of the crowds across the NPM users finds all possible edge cases.
Now I mostly just use Renovate App service to keep all my dependencies up to date. I have more than one hundred GitHub repos updated this way - and I am not afraid to let the bot auto merge updates if the tests pass.
Last week Renovate became even better. It now collects the anonymous update stats and in every pull request shows the information about other projects across the internet that updated the same dependency. Read the official announcement here. The feature is called "Merge Confidence" and every pull request shows the following:
As you can see, updating Prettier from v1 to v2 will probably require manual work - the automattic updates mostly failed for this breaking change.
On the other hand, upgrading Cypress from v5 to v6 is pretty smooth - almost 80% of projects did the update without breaking tests.
Most projects could upgrade automatically but have not merged the Renovate pull requests yet - thus the confidence is still low. The confidence will increase once more projects that successfully pass tests actually merge the dependency update. Here is an example of a patch dependency update with lots of projects passing their tests, and thus the wisdom of the crowds marking "high confidence".
By using these additional statistics you can update your dependencies with confidence. You also no longer have to rely on just your own tests, and you do not have to blindly trust the dependency authors to be diligent about semantic versioning - because you know how your own project and all other projects behave during upgrade.
See Merge Confidence documentation page for details.
👏 Rhys Atkins and WhiteSource Renovate Team.