products

Publish release notes to Slack

Using NPM hooks and Zeit to send the semantic release notes to Slack.

The semantic release notes

I love modular development using semantic-release and NPM. Using a pre-git hook, every commit has a well-formatted message like this

1
type(scope): short message, maybe even github issue close #n

When the commits are pushed to GitHub, the CI runs and after passing the tests it triggers the following npm run semantic-release script run.

1
2
3
4
5
{
  "scripts": {
    "semantic-release": "semantic-release pre && npm publish && semantic-release post"
  }
}

The semantic-release pre inspects the list of commits since the last published release, and if there are new features or fixes in the commit list, increments the package version, publishes the package to NPM and creates a new release tag on GitHub. The best part about these automatic release notes is their usefulness. All the non-essential commits are hidden; only the new features and fixes are listed. For example, [email protected] lists the following features

1.0.0 (2016-03-29)

Features

Again, all this happens automatically.

Waiting for the publish

There is a small problem when using semantic release. Since I no longer execute npm publish locally (the CI server does this after the build), there is a delay between the time I push the commits to the remote origin and the time the new version gets published. Usually I check the github interface to see when the tag appears. The browser page reload and waiting is very annoying.

I want a way to be notified when the new version has been published, and the message should include the beautiful release notes too!

NPM hooks

Recently NPM registry has introduced hooks for paying customers. You can register up to 100 hooks and be notified about different events: new packages published, starred or deprecated. You can listen for events related to an individual package, a single user, or an entire organization (using the scope name). Let us create a simple server that:

  • receive the "publish" message from the NPM registry
  • grabs the release notes from GitHub
  • publishes a message to a Slack channel.

There three API to connect, plus we need to run the server somewhere.

Where to deploy?

Long time ago, I deployed a simple server like this to Heroku. I used a free dyno, which was great for simple demos. Yet for this project a free dyno does not work very well - the free dyno can go to sleep, ignoring the incoming message. Plus, there is a little bit too much overhead in setting a simple application like this on Heroku.

Recently, I have tried using Dokku on a single DigitalOcean droplet. I love Dokku - I can host multiple Docker images on a single box, there are plugins for different databases, things just work. It also makes perfect financial sense - a single $10 droplet can run 10-20 apps, and as long as none is very CPU expensive, the apps keep chugging along.

Yet, even the Dokku requires manual setup. Is there something simpler?

Enter Zeit. Advertised as making cloud computing as simple as possible, it has certainly provoked a lot of interest among the web development crowd. This project was my opportunity to try it out.

Before we proceed, I must say that I will be using a paid Zeit account - it allows me to hide the source folder from my deploys, supports custom domains and costs very little. 1000 of deploys per month (every deploy gets a new uniq URL) for $14 is much much cheaper than buying droplets from DigitalOcean, or suffering through the AWS API (compared to the beautiful zeit now command)

Security

We are going to need a couple of tokens to get the data from NPM and GitHub and post it to Slack. First, let us grab the Slack access token and channel ID (NOT the channel's name!) The simplest way to grab these two pieces was

I had to go through these hoops because the NPM Slack example uses Web API to post messages (via @slack/client package). I would prefer using a Webhook instead via slack-node package. Maybe in the future I will make the switch.

To grab the GitHub release notes from public repos, we do not need much - just your GitHub user name. In order to get release information from private repos you need to make a personal access token with "repo" scope.

Finally, in order to subscribe to NPM events, you need to be a paying customer, install "wombat" CLI app using npm i -g wombat and login. To ensure secure delivery, you need to set a secret pass phrase when making each hook. For example no one knows what it's like... is a good shared secret for this experiment.

All together, here are the environment variables we need to set before running the server code

1
2
3
4
5
SLACK_API_TOKEN=xoxp-...
SLACK_CHANNEL=C1...
SHARED_SECRET=no one knows what it's like...
GITHUB_USERNAME=bahmutov
GITHUB_TOKEN=bb45...

We can export these variables before running the server. In the future, I will show a better way to handle the environment variables using as-a.

Server

The incoming messages from NPM need to reach our server. I just grabbed the single index.js from the npm-hook-slack repo - this will be the base for the future code modifications. Code just makes a Slack client, and a server using npm-hook-receiver

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
var makeReceiver = require('npm-hook-receiver')
var slack = require('@slack/client')
var web = new slack.WebClient(process.env.SLACK_API_TOKEN)
var server = makeReceiver({
  secret: process.env.SHARED_SECRET,
  mount: '/incoming'
})
// All hook events, with special handling for some.
server.on('hook', function onIncomingHook(hook)
{
  var pkg = hook.name.replace('/', '%2F');
  var type = hook.type;
  var change = hook.event.replace(type + ':', '');
  var message;
  var user = hook.change ? hook.change.user : '';
  switch (hook.event)
  {
  case 'package:star':
    message = `★ \<https://www.npmjs.com/~${user}|${user}\>
    starred :package: \<https://www.npmjs.com/package/${pkg}|${hook.name}\>`;
    break;
  }
  // more event handlers
  ...
  // finally send the formatted message
  web.chat.postMessage(process.env.SLACK_CHANNEL, message);
})

We can refactor the code, and even remove the unnecessary event handles, since we are only interested in package:publish event, etc. For now, let us connect the local server to the NPM registry hook. After receiving the package:publish event, the code fetches the latest GitHub release notes and sends them as a message to a Slack channel.

1
2
3
4
5
6
7
8
9
10
11
server.on('hook', function onIncomingHook(hook)
  switch (hook.event) {
    case 'package:publish':
      web.chat.postMessage(process.env.SLACK_CHANNEL, 'package published')
      getReleaseNotes()
        .then(function (notes) {
          web.chat.postMessage(process.env.SLACK_CHANNEL, notes)
        })
    break
  }
})

Let us start by running the server locally.

Local development

Before we deploy this code, let us start by testing the server code locally. Of course, in this case we get the server running at http://localhost:6666, which is not reachable from external clients. Let us tunnel to the local server using ngrok.

1
2
3
4
5
6
7
8
9
10
$ npm start
listening on 6666
$ ngrok http 6666
ngrok by @inconshreveable
Tunnel Status                 online
Version                       2.0.25/2.1.1
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://743e560f.ngrok.io -> localhost:6666
Forwarding                    https://743e560f.ngrok.io -> localhost:6666

Notice we are getting both http and https forwarding. Let us register the NPM hook. I will observe a repo of mine little-store.

1
2
$ wombat hook add little-store https://743e560f.ngrok.io/incoming "no one knows what it's like..."
+ little-store ➜ https://743e560f.ngrok.io/incoming

You can see all your registered hooks at any time

1
2
3
4
5
6
7
8
9
$ wombat hook ls
hooks you have 1 hook
┌──────────┬──────────────┬──────────────┬─────────────────────────────────────┐
│ id       │ type         │ target       │ endpoint                            │
├──────────┼──────────────┼──────────────┼─────────────────────────────────────┤
│ dwvydv16 │ package      │ little-store │ https://743e560f.ngrok.io/incoming  │
│          ├──────────────┴──────────────┴─────────────────────────────────────┤
│          │ never triggered                                                   │
└──────────┴───────────────────────────────────────────────────────────────────┘

To trigger the release, let us push a few dummy commits to the repo bahmutov/little-store. Even dummy commits would do, as long as we follow the semantic version commit format.

1
2
3
4
$ git commit --allow-empty -m "feat(dummy): this is new feature"
$ git commit --allow-empty -m "feat(dummy): this is another new feature"
$ git commit --allow-empty -m "fix(test): just fixed a bad bug"
$ git push origin master

The Travis CI build has finished and has determined the new published version 1.2.0, generating the release notes.

1.2.0 (2016-06-16)

Bug Fixes

Features

The NPM message has arrived (you can see this in ngrok terminal output or in its web interface running at 127.0.0.1:4040)

1
POST /incoming 200 OK

The request body has the event and the package information.

1
2
3
4
5
{
  "event":"package:publish",
  "name":"little-store",
  "...": "lots more"
}

From the payload we can grab the repo, and if it is hosted on GitHub, make a request to grab the latest release notes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
function githubReleaseUrl(payload) {
  const url = payload.repository.url
  const parsed = gh(url)
  const me = process.env.GITHUB_USERNAME
  const ghToken = process.env.GITHUB_TOKEN
  return `https://${me}:${ghToken}@api.github.com/repos/${parsed.repo}/releases/latest`
}
const releaseUrl = githubReleaseUrl(payload)
const options = {
  uri: releaseUrl,
  headers: {
    'User-Agent': process.env.GITHUB_USERNAME
  },
  json: true
}
const rp = require('request-promise')
rp(options)
  .then((response) => {
    // response.body is the release in Markdown format
  })

Since Slack does not follow Markdown, we need to convert the incoming message to the Slack format. I just do 2-step transformation Markdown -> HTML -> Slack format using marked and slackify-html.

1
2
3
4
const marked = require('marked')
const slackify = require('slackify-html')
const slackMessage = slackify(marked(response.body))
web.chat.postMessage(process.env.SLACK_CHANNEL, slackMessage)

The local server posts this message to the test channel

Slack message

Mission accomplished!

Now we can remove the NPM hook used for the local environment and deploy the permanent external server.

1
2
$ wombat hook rm dwvydv16
– little-store ✘ https://743e560f.ngrok.io/incoming

Deployment with now

The two best features about deploying using zeit now are simplicity and speed. You also get a unique url for each deployment - you can test the deploy, and only if it works switch a domain alias to point at the unique url.

First, start by installing the tool now and logging in.

1
2
$ npm install -g now
$ now

Hmm, the server throws an exception when trying to start. Seems it cannot find the environment variables we are using! Currently the now tool does not support the environment variables, thus we need to pass them inside the npm start script.

1
2
3
4
5
{
  "scripts": {
    "start": "SLACK_API_TOKEN=xoxp-... SLACK_CHANNEL=C1... ... node index.js"
  }
}

This is quickly becoming a very long and unwieldy command. As a work around, I use as-a to pass the variables. I just placed all variables into a local file .as-a.ini and run the node index.js by executing as-a. I like to keep the variables grouped in the .as-a.ini file

1
2
3
4
5
6
7
8
[slack]
SLACK_API_TOKEN=xoxp-...
SLACK_CHANNEL=C1...
[npm]
SHARED_SECRET=no one knows what it's like...
[gitub]
GITHUB_USERNAME=bahmutov
GITHUB_TOKEN=bb45...

The server "start" command is now much cleaner

1
2
3
4
5
{
  "scripts": {
    "start": "as-a slack,npm,github node index.js"
  }
}

We can deploy the server using now and it will give us a unique URL. I will use -f option to force a clean install.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ time now -f
> Deploying "/git/npm-hook-test"
> Using Node.js 6.2.1 (default)
> Ready! https://release-to-slack-fcwdzcxyev.now.sh (copied to clipboard) [1s]
> Initializing…
> Building
> ▲ npm install
> Installing package async@~1.0.0
> Installing package [email protected]
> Installing package [email protected]
> Installing package [email protected]
> Installing package [email protected]
> Installing package [email protected]
> Installing package [email protected]
> Installing package brace-expansion@^1.0.0
> Installing package balanced-match@^0.4.1
> Installing package [email protected]
> ▲ npm start
> {"listening on 6666"}
> Deployment complete!

real  0m17.839s
user  0m0.978s
sys   0m0.138s

18 seconds for a clean install, but if there are no dependency changes, or even code changes, the install takes a lot shorter.

1
2
3
4
5
6
7
8
9
10
$ time now
> Deploying "/git/npm-hook-test"
> Using Node.js undefined (default)
> Ready! https://release-to-slack-fcwdzcxyev.now.sh (copied to clipboard) [3s]
> Initializing…
> Deployment complete!

real  0m4.872s
user  0m0.694s
sys   0m0.102s

Nice! We can register an NPM hook with the given url, test it out, and then remove any unnecessary deployments.

1
2
3
4
5
6
7
8
$ wombat hook add little-store https://release-to-slack-fcwdzcxyev.now.sh/incoming "no one knows what it's like..."

$ now ls

release-to-slack

  75bUJhvsoS3NDCA7wPrTxihN      https://release-to-slack-fcwdzcxyev.now.sh      3m ago
  dFDDV9Da9lWeVpCitEj1znMY      https://release-to-slack-tylnttnrfy.now.sh      4m ago

Let us remove the older deploy dFDDV9Da9lWeVpCitEj1znMY.

1
2
3
4
5
$ now rm dFDDV9Da9lWeVpCitEj1znMY
> The following deployment will be removed permanently
  dFDDV9Da9lWeVpCitEj1znMY      https://release-to-slack-tylnttnrfy.now.sh      5m ago
> Are you sure? [yN] y
> Success! Deployment dFDDV9Da9lWeVpCitEj1znMY removed [771ms]

That is it. Zeit truly delivers on its premise of simple cloud computing, and there is even an API to automate the deployments.

Open source it!

Finally, I have open sourced the server at bahmutov/publish-release-notes. Because the deploy needs the environment variables to be present, I have created a separate private repo that keeps just .as-a.ini file and defines the start command.

private project package.json
1
2
3
4
5
6
7
8
9
10
11
12
{
  "name": "release-to-slack",
  "version": "1.0.0",
  "description": "Wraps release notes publish with private token variables",
  "scripts": {
    "start": "as-a slack,npm,gitub publish-release-notes"
  },
  "dependencies": {
    "as-a": "^1.3.1",
    "publish-release-notes": "^1.0.1"
  }
}

This way I can keep my tokens private for deploy, but the 99% of the code is open source. Once zeit supports external environment variables, this wrapper project will become unnecessary.

Conclusion

Creating personal "API to API" bridges has never been easier. Webhooks, notification messages, well documented APIs and one-click immutable deployments are breaking down the barriers to incremental software development. Instead of waiting for a large number of changes to be ready to deploy and test, I commit, test and deploy each tiny change - because the entire code deployment pipeline is simple to setup and use. Instead of manual sitting and waiting, each step notifies whoever is listening that it is done, allowing the quick propagation. In a sense, we now can build a reactive pipeline for code, removing the manual pull steps.