Randomize file order when testing

Keeping the same load order of files and tests can lead to hidden bugs.

Imagine a typical project with bunch of unit tests. Usually there is a configuration file that lists all source files to be loaded, and then the second list of test files. It could be a Gruntfile.js, karma.conf.js or something else.

We often keep adding new source files at the end of the list of files to be loaded before tests, or we keep the list sorted alphabetical by using a wildcard

// alphabetical order
files: [
  'src/**/*.js',
  'specs/**/*.js'
]

Some frameworks allow you to scramble the test execution order, but not the file load order. For example, QUnit allows to execute previously failed tests first Other frameworks, like Mocha allows you to sort test files before loading instead. None of the frameworks can detect the following situation (although global leak detection helps, but does not solve this).

// basic.js
window.foo = window.foo || {
  foo: 'foo'
};
// something.js
// we forget to initialize window.foo!
if (foo.foo !== 'foo') {
  throw new Error('Wrong foo');
}

In production we concatenate files alphabetically using a wildcard and everything works. We also list files to load for testing using wildcards

// Grunfile.js
concat: {
  prod: {
    files: '*.js'
  }
},
jasmine: {
  tests: {
    src: '*.js',
    options: {
      specs: 'specs/*.js'
    }
  }
}

Problem: we will never discover that we forgot to initialize foo object in file something.js. We could turn on global leak detection, but this will give us false positive - we need window.foo.

This problem can partially be attributes to the Engish language and a programmer's tendency to name the files to be included first starting with suffixes that lead alphabetically, like basic or _something.

Solution

Take each group of the files to be loaded separately (source, test files). Instead of using wildcards directly, grab all matching filenames and scramble them! If using grunt you can call grunt.file.expand to get all filenames matching a pattern. Then randomize its order using shuffle:

// Gruntfile.js
var _ = require('lodash');
jasmine: {
  tests: {
    src: _.shuffle(
      // all js files but skip any file ending with "-spec.js"
      grunt.file.expand('src/**/*.js, !src/**/*-spec.js')
    ),
    options: {
      specs: _.shuffle(
        grunt.file.expand('!src/**/*-spec.js')
      )
    }
  }
}

This will lead to flaky tests of course. But chances are if you dig into each failing test and print the files before running the tests, you will get to the true reason: some files are meant to load first. And then you will use require.js or browserify.