products

Homebrew CLI testing

You can easily perform CLI end to end testing by checking the exit codes.

Testing a CLI utility using Nodejs is extremely easy, even without using any 3rd party tool. Here is how I unit test a cli utility using a homebrew 10 liner.

The CLI test subject

A tiny command line script that outputs back passed command line argument, twice.

// index.js
if (!process.argv[2]) { process.exit(1); }
console.log('' + process.argv[2] + process.argv[2]);
// example
$ node index.js hello!
hello!hello!

Testing goals

The test cases need to confirm correct STDOUT output and exit codes.

node index.js //=> no output, exit code 1
node index.js foo // exit code 0, outputs foofoo

The testing engine should keep track of the number of running tests (because everything is being tested asynchronously) and exit with the number of failed assertions. If everything passes, the exit code should be 0.

Homebrew testing framework

While there are fine testing frameworks for Node, like mocha and my own gt, for this exercise I rolled a homebrew framework. Its biggest advantages are simplicity and easy adjustment to any specific project's needs. I start with writing a tiny utility function that would fail an assertion based on condition.

// exec-test.js
var failed = 0;
function check(name, condition, actual, expected, message) {
  if (!condition) {
    console.error('test "' + name + '" failed,', message);
    console.error('  Expected "' + expected + '", got "' + actual + '"');
    failed += 1;
  }
}

This is common pattern, for example QUnit.push is used the same way: our other assertions can report failures via single check call. We can even expose this function to the user code via assert object to allow creating custom assertions.

The main function exposed to the user is called execTest in the same file. Here is the book keeping part

// exec-test.js after check function
var running = 0;
function execTest(cmd, expectedExitCode, expr) {
  running += 1;

  // exec CLI command
  // when finishes decrement running counter
  if (!running) { process.exit(failed); } // all done
}
module.exports = execTest;

Here is the inside of the execTest function, I am using Node's child process to actually execute the shell command, in this case without any options.

function execTest(cmd, expectedExitCode, expr) {
  var exec = require('child_process').exec;
  running += 1;
  exec(cmd, function (err, stdout, stderr) {
    running -= 1;
    if (err) {
      check(cmd, err.code === expectedExitCode,
        err.code, expectedExitCode,
        'exit code\n---\n' + stderr + '---'); // 1
    } else {
      if (expr) {
        stdout = stdout.trim();
        check(cmd, expr.test(stdout),
        stdout, expr.toString(), 'output'); // 2
      }
    }
    if (!running) { process.exit(failed); }
  });
}

Notice how exit code comparison uses check function in // 1. The success condition is simple strict equality err.code === expectedExitCode. The order of actual vs expected arguments is not important, and is up to you in this case. I always forget which one is first.

Comparing program's output using regular expression in // 2 is more robust than string comparison. It only needs to happen if the exit code is unexpected. In this particular case, if the exit code is unexpected, we print the stderr output surrounding it by delimiters. This will come handly when we combine multiple tests into a hierarchy.

User tests

Lets write a few tests

// test.js
var execTest = require('./exec-test');
execTest('node index.js foo', 0, /^foofoo$/);
execTest('node index.js 2', 0, /^22$/);
execTest('node index.js', 1);

If all tests pass, there is no output. Boring, but useful. Lets change the expected outputs for two tests on purpose

// test.js with failing tests
var execTest = require('./exec-test');
execTest('node index.js foo', 0, /^foofoo$/);
execTest('node index.js 2', 0, /^55$/);
execTest('node index.js', 2);
// run
$ node test.js
test "node index.js" failed, exit code
---
---
  Expected "2", got "1"
test "node index.js 2" failed, output
  Expected "/^55$/", got "22"

Because these unit tests are so simple, we can easily place this command into package.json for both npm test and pre-commit.

"scripts": {
  "test": "node test.js"
},
"pre-commit": "node test.js"

Tests of tests

Notice that node test.js is a shell command. Thus we can test it using execTest itself! This might come handy when combining lots of unit tests into a hierarchy without using modules or suites of tests (because our tiny testing engine does not support them).

Lets write a top level module that runs other exec tests, with a single assertion for now

// all-tests.js
var execTest = require('./exec-test');
execTest('node test.js', 0);

This test does not check the output (because it is successful, there should not be any). It only checks that command exited cleanly. Since we left test.js with failing unit tests, here is what the top level test reports

$ node all-tests.js
test "node test.js" failed, exit code
---
test "node index.js" failed, exit code
---
---
  Expected "2", got "1"
test "node index.js 2" failed, output
  Expected "/^55$/", got "22"
---
  Expected "0", got "2"

The output might be a little more clear, but overall a powerful tool from just a few lines of JavaScript.

Missing

It would be easy to extend this engine with more assertions, but in my opinion the most important missing feature is the sequential test execution. Currently, due to Node's asynchronous nature, if you write 3 execTest commands, all 3 will run in parallel, and the order is not guaranteed. This might be fine in this case, but might break for larger programs. Luckily this feature would be easy to implement via a single 3rd party module, like async or cadence.