Testing async module setup

Unit testing asynchronous JavaScript code is tricky. One problem in particular was causing me pain: unit testing code that requires asynchronous initialization. Both Jasmine and QUnit provide great support for async tests, but not for initialization.

Problem

Here is a example to be tested that shows the problem.

var counter = 0;
function init() {
    setTimeout(function () {
        counter += 1;
    }, 1000);
}

The Jasmine unit test fails because it starts immediately, without waiting for beforeEach function to finish the initializaiton.

beforeEach(init);
it('was counter initialized?', function () {
    expect(counter).toEqual(1); // false
});

Same problem showing using QUnit framework via qunit-node

QUnit.module('async setup', {
    setup: init
});
QUnit.test('counter value', function () {
    // fails
    QUnit.equal(counter, 1, 'was the counter initialized?');
});

Solutions

Jasmine

Native Jasmine solution is limited, but at least it works:

beforeEach(function () {
    runs(init);
    waitsFor(function () {
        return counter > 0;
    });
});
it('counter was initialized', function () {
    expect(counter).toEqual(1); // true
});

I don't like this solution, because it requires shared state between the code tested (function init) and the testing code (through counter variable). I often write async code using promises (either Q or jQuery), and checking if the returned promise has been fulfilled still requires some boilerplate code

// init() returns promise
beforeEach(function () {
    var initialized;
    runs(function () {
        initialized = init();
    });
    waitsFor(function () {
        return initialized.isFulfilled();
    });
});
it('counter was initialized', function () {
    expect(counter).toEqual(1); // true
});

Other people have noticed this problem and proposed either an extension or a new runner.

I am not sure how Karma's Jasmine runner behaves in this case, but I don't see any code handling async suite setup case.

QUnit

Testing queue can be stopped and restarted inside the module's setup function:

// init() returns a promise
QUnit.module('async setup', {
    setup: function () {
        QUnit.stop();
        init().finally(QUnit.start);
    }
});

QUnit.test('counter value', function () {
    QUnit.equal(counter, 1, 'async setup works!');
});

I like the QUnit's minimal boilerplate solution, the boilerplate is minimal and the semantics is obvious. If only QUnit provided slightly better support for promises, then all boilerplate could be removed.

My solution

I have my own QUnit-compatible test runner gt. It accepts same syntax as QUnit, runs natively on nodejs, and has lots of extensions for performance testing and other nice stuff.

Adding async setup method support was trivial. If your setup function returns a promise, the engine will wait until the promise is fulfilled before proceeding:

// init() returns a promise
QUnit.module('async setup', {
    setup: init
});
QUnit.test('counter value', function () {
    QUnit.equal(counter, 1, 'was the counter initialized?');
    // success!
});

There is a similar support for async module's teardown method.

Under the hood

Peaking under the hood, the addition required only a superficial code change. Before adding support each unit test was executed using this code:

preTest();      // executes module's setup if any
this.executeTest(test, function () {
    verifyIntegrity();
    postTest(); // executes module's teardown if any
    callback(); // continues to next test
});

If preTest() returns either nothing or a promise, and we want to wait before it gets resolved, we can use Q's when function:

var self = this;
Q.when(preTest()).then(function () {
    self.executeTest(test, function () {
        verifyIntegrity();
        postTest();
        callback();
    });
}, onSetupError);

Conclusion

I think testing async code using Jasmine or QUnit will become more and more important. Both libraries are excellent, and will not be displaced by vows or mocha any time soon. If you are using QUnitJs and look to execute tests under nodejs it makes sense to look at gt. If you still need better async promise testing in the browser, take a look at qunit-promises