process

Code coverage by commit

Separate and update JavaScript code coverage information using tested-commits.

Code coverage is a great tool for making sure the code works against a set of unit tests. In this blog post I will show how to split the combined code coverage into coverage for specific commits. Having coverage by commit allows to make sure that a specific feature has been tested without drowning in the noise of the entire repo.

I implemented a tool that splits coverage for JavaScript code. It is available at bahmutov/tested-commits and can be installed from NPM using command npm install -g tested-commits. It can take coverage output from istanbul and split it into selected git repo commits. There are multiple tools that use istanbul to instrument JavaScript code: Karma, gt, was-tested.

tested-commits in action

The above screenshot shows tested-commits in action. It took coverage produced by Karma and the git repo information and generated individual coverage reports for each commit.

To show the tested-commits in action, I created a small repo bahmutov/foo-bar. You can follow the steps in this blog post to see the results yourself.

Step 1: clone the example repo

git clone https://github.com/bahmutov/foo-bar.git
cd foo-bar

The example repo has single page with two buttons. Each button runs a piece of javascript code. You can open the file index.html directly or run a local webserver (needed for future steps in this post). I recommend running the local server using http-server.

foo-bar screenshot

Step 2: look at repo commit history

You can quickly see the repo's 3 commits

$ git log --oneline
8b851be added karma unit tests
1b29608 added bar
0137e22 added foo

first commit

The first commit 0137e22 added foo created the index page with a single button and a function foo. You can see the code at github/foo-bar/0137e22.

index.html
1
2
3
4
5
6
7
<body>
  <h2>foo-bar application</h2>
  <p>Start the local server in this folder using <pre>http-server -p 3003</pre></p>
  <button id="foo">Click Foo</button>
  <p>See details at <a href="https://github.com/bahmutov/foo-bar">bahmutov/foo-bar</a>
  <script src="app.js"></script>
</body>

The application's javascript file app.js

app.js
1
2
3
4
5
6
7
8
9
10
11
function foo() {
  console.log('foo');
}
function onFooClick() {
  console.log('button foo clicked');
  foo();
  alert('Thank you for clicking the button foo');
}
document.addEventListener('DOMContentLoaded', function() {
  document.querySelector('#foo').addEventListener('click', onFooClick);
});

Think this commit as a typical feature implemented and pushed to the repo.

second commit

The second commit 1b29608 added bar adds a second button and separate function bar. You can see the updated code at github/foo-bar/1b29608.

index.html
1
2
<button id="foo">Click Foo</button>
<button id="bar">Click Bar</button> <-- added button foo -->

The app.js file with added code

app.js
1
2
3
4
5
6
7
8
9
10
11
12
// same code for 'foo' feature
function bar() {
  console.log('bar');
}
function onBarClick() {
  console.log('button bar clicked');
  bar();
  alert('Thank you for clicking the button bar');
}
document.addEventListener('DOMContentLoaded', function() {
  document.querySelector('#bar').addEventListener('click', onBarClick);
});

third commit

We add unit tests to the repo in the third commit 8b851be added karma unit tests. I created karma.conf.js and wrote a few jasmine unit tests that just execute foo and bar functions. You can see the code at github/foo-bar/8b851be.

app-spec.js
1
2
3
4
5
6
7
8
describe('app', function () {
  it('has foo', function () {
    foo();
  });
  it('has bar', function () {
    bar();
  });
});

You can run the unit tests and see the coverage results by installing dependencies npm install and running karma.

Karma run

We then open coverage/index.html in the browser we can see which parts of the app.js were covered by the unit tests

Karma coverage

Notice that our unit tests only executed source lines inside functions foo and bar, but not click event handlers.

Karma coverage plugin saved statement by statement coverage in a json file. It looks a little like

$ c coverage/Chrome\ 39.0.2171\ \(Mac\ OS\ X\ 10.9.4\)/coverage-final.json
{
    "./app.js": {
        "path":"./app.js",
        "s": {"1":1,"2":1,"3":1,"4":1,"5":0,"6":0, ...
    ...

s list in this structure shows statements that were executed during running the unit tests. Statements 1, 2, 3, 4 were covered, while statements 5 and 6 were not covered. There is a separate map that links statements to source code lines, but for our purpose it is an unimportant detail.

The code coverage displayed in the above covers the entire repo and does not separate information for the two features foo and bar. Thus it is easy to confuse average code coverage across the entire repo with both features tested at the same rate. But one commit could be tested completely, while the second one could be skipped completely. This can be dangerous when used together with the branching models commonly used in the industry.

Git branching and commit work flow

Every company I worked for used some variation of the following git branching model. There is a master and several branches used to deploy to specific environments. By default everyone is committing to the master. Periodically we deploy the master to the dev server and test the app. If everything is working, we merge commits from master to staging branch and deploy to the staging server. We test again and merge to the prod branch and deploy the code to production. The situation becomes a lot more complex when some things do not work and we cannot merge entire branch from master to staging. Instead we cherry pick specific commits to bring from master to staging. Similarly, we are sometimes forced to cherry pick fixes to be applied to the prod branch.

git branching model

The above image is taken from the excellent blog post A successful Git branching model.

Notice the following problem

We move the individual features from branch to branch using commits, but our coverage information is computed across the entire repo at the HEAD commit.

Split coverage by commit

When looking at the HEAD of the repo, we can see which commit is responsible for each line of code. A line by line break down can be shown using git blame command

$ git blame app.js -s
^0137e22  1) console.log('this is app.js');
^0137e22  2)
^0137e22  3) function foo() {
^0137e22  4)   console.log('foo');
^0137e22  5) }
^0137e22  6)
^0137e22  7) function onFooClick() {
^0137e22  8)   console.log('button foo clicked');
^0137e22  9)   foo();
^0137e22 10)   alert('Thank you for clicking the button foo');
^0137e22 11) }
^0137e22 12)
^0137e22 13) document.addEventListener('DOMContentLoaded', function() {
^0137e22 14)   document.querySelector('#foo').addEventListener('click', onFooClick);
^0137e22 15) });
^0137e22 16)
1b296080 17) function bar() {
1b296080 18)   console.log('bar');
1b296080 19) }
1b296080 20)
1b296080 21) function onBarClick() {
1b296080 22)   console.log('button bar clicked');
1b296080 23)   bar();
1b296080 24)   alert('Thank you for clicking the button bar');
1b296080 25) }
1b296080 26)
1b296080 27) document.addEventListener('DOMContentLoaded', function() {
1b296080 28)   document.querySelector('#bar').addEventListener('click', onBarClick);
1b296080 29) });

In app.js case, the first two commits are responsible for the entire file. You can get this information using bahmutov/ggit module.

Using tested-commits I can take extract the code coverage specific to a single commit, or to several commits. The example below will split coverage to the last 3 commits in the repo.

$ npm install -g tested-commits
$ tested-commits -r
id                                        message
----------------------------------------  ----------------------
8b851be467f4d904d245afe3abe8d29a8ba0eaac  added karma unit tests
1b296080cd7d15bb4faf752ac3727862f9618944  added bar
0137e22f816dd0e5738eb75ec46489bf1762332b  added foo
...
blames for [ 'app-spec.js', 'app.js', 'karma.conf.js' ]
=============================== Coverage summary ===============================
Statements   : 0% ( 0/7 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 0/0 )
Lines        : 0% ( 0/7 )
================================================================================
full dir /commits/foo-bar/8b851be467f4d904d245afe3abe8d29a8ba0eaac
=============================== Coverage summary ===============================
Statements   : 25% ( 2/8 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 0/0 )
Lines        : 25% ( 2/8 )
================================================================================
full dir /commits/foo-bar/1b296080cd7d15bb4faf752ac3727862f9618944
=============================== Coverage summary ===============================
Statements   : 22.22% ( 2/9 )
Branches     : 100% ( 0/0 )
Functions    : 100% ( 0/0 )
Lines        : 22.22% ( 2/9 )
================================================================================
full dir /commits/foo-bar/0137e22f816dd0e5738eb75ec46489bf1762332b

The program discovered 3 commits and 3 JavaScript files tracked in the git repo. The it split the files to parts modified in each commit. Each commit's report and coverage file is saved independently in separate folder. You can open every report using a single command

tested-commits --open
// opens HTML reports in the browser

Each report shows only the lines that were modified in that commit, other lines are ignored.

added foo coverage

The 0137e22 added foo commit's coverage report only shows the lines that survived into the HEAD commit. Everything else is ignored, similarly to white space or comments.

added bar coverage

Similarly, the coverage for commit 1b29608 added bar shows information only for the lines from that commit.

Update split information

Once we have split the coverage information into the distinct sets, we can update each set by bringing the new coverage information. For example, we can execute the Jasmine tests using Karma

$ karma start

It generates the combined line by line coverage saved as file coverage/<browser name>/coverage-final.json. Let us update each commit's coverage using this generated combined coverage file

$ tested-commits --update coverage/Chrome/coverage-final.json
updating split coverage from coverage/Chrome/coverage-final.json
...
coverage numbers changed

Again we can open the separate coverage reports using tested-commits --open command. Notice the changed information, for example the code block inside the function foo() has been covered by the unit tests. A couple of other statements have been covered too, increasing the commit's code coverage from 22% to 55%.

foo covered by unit tests

Similarly, the code block inside function bar() has been covered. The latest commit also increased code coverage because the app-spec.js source has been fully covered.

For commits that have been completely overwritten by other commits, the coverage automatically shows 100%, because there are zero lines.

Conclusions

Splitting coverage information by commit allowed us track the testing information associated with each feature. Before cherry picking a commit, we can actually check its test status to make sure we only move reliable code between the branches.

Related