products

Robustness testing using proxies

Mocking responses using turtle-run, service-turtle and ng-wedge.

What happens to your web application if one of the static libraries takes 10 seconds to load instead of 100ms? Does it stay blank? Does it crash? Does it show the page but then crashes when the user tries to interact with it?

What happens if the server responds with a 500 error code to an Ajax request? Do you silently swallow the error, and the user keeps waiting in vain? Does the web application crash? Or does it show a meaningful error message to the user? Is the error sent to the crash reporting service, like Sentry?

I consider the web sites that graciously handle run time errors like an occasional 500 server response to be well-built and trustworthy ones. It is a good sign that the team not only has finished working on the regular features but has also properly designed and tested the error handling code paths.

From my personal development experience testing all possible intermittent failures is very hard and time consuming. It is very difficult to simulate an occasional 500 error on demand - the feature testing against the staging or production environments assumes the servers are a black box. Adding the ability to simulate the server's failures will surely add unnecessary complexity to the server's code. It also does not allow simulating network delays and responses with an unexpected data formats (two things very common in the real world).

I have designed and coded 3 different types of proxies for simulating these specific types of failures. The proxies can be used without any or with minimal application code modifications and allow simulating server failures, network delays and returning data in an unexpected format. These proxies are:

  • turtle-run is a stand alone proxy that runs between the tester and server. It can slow down some requests, or return a specific status code or mock data based on url match.
  • service-turtle is a proxy that runs inside the latest Chrome browser using ServiceWorker technology. It can intercept any HTTP request and return a fake status, and even delay the mock response by a given time period.
  • ng-wedge is the last script that can modify the web application's code on the fly. It is the most flexible approach that does not require an intermediate proxy server or an experimental browser API. Unfortunately this approach assumes the tested application uses AngularJS framework and requires good development insight into the application's code. In my opinion it is most useful to the developer who is working on the application.

I will show how these proxies can be used to test the web application's robustness to the common errors using a small AngularJS application. The source code is available at bahmutov/github-user and the live demo is available at glebbahmutov.com/github-user.

github user screenshot

You can follow the application's testing by cloning the repo and starting with tag step-0

git clone [email protected]:bahmutov/github-user.git
cd github-user
git checkout step-0

I recommend that you use a static http server to run the web application, for example http-server

cd github-user
sudo npm install -g http-server
http-server -p 3003
// open localhost:3003 in your browser

The application should run in the local browser at localhost:3003. If you enter a github username and click "Load user info" there is separate Ajax request to https://api.github.com. The results should be displayed below the input box.

Testing against network delays using turtle-run

The first thing we want to test is how the application looks when a 3rd party library does not load at all or loads only after a delay. The github-user demo page requires only a single external script to load, and it is included with the repo itself.

1
2
3
4
5
6
7
<head>
  <script src="bower_components/angular/angular.js"></script>
</head>
<body ng-app="GithubUser" ng-controller="GithubUserController">
    <h1>github-user</h1>
    ...
</body>

What happens if the angular.js file does not load? Does the page stay blank? Usually we load the script angular.js from a CDN, so it is likely that it will ultimately load, but there might be a significant delay. Let us simulate the delay in loading angular.js using a proxy and see how the web page looks while the request is pending.

Install the turtle-run proxy and create a JSON file describing which request to delay. I created sample file and tagged the commit step-1

sudo npm install -g turtle-run
git checkout step-1

The turtles.json is very simple. It tells the proxy where to forward requests to, and for each matching url pattern what to return / delay amount. In this case angular.js script will be loaded after 5 seconds.

1
2
3
4
5
6
7
8
{
  "target": "http://localhost:3003",
  "port": 8008,
  "debug": true,
  "urls": {
    "angular.js": 5000
  }
}

While the local http-server is running at port 3003, start the turtle-run and pass it the turtle config json filename. We can start in the github-user folder using the default turtles.json filename

cd github-user
http-server -p 3003 &
turtle-run

Open the browser localhost:8008.

GET /
GET /bower_components/angular/angular.js
Connect-slow: delaying 5000 ms on url /bower_components/angular/angular.js

The turtle-run sits between the local server and the user's browser, intercepting every request and deciding how to proceed. Most of the requests are just forwarded to the target. The requests matching the /angular\.js/ regular expression are forwarded to the target server after 5 seconds. Note: the Ajax request to the Github API is NOT proxied by the turtle-run because it is a direct cross-domain request from the page to the Github API endpoint.

turtle-run overview

Notice that the page stays blank for 5 seconds before anything appears! This is because the proxy slows down bower_components/angular/angular.js script, returning the script after 5 seconds. The script is included in the head element and is blocking the rest of the page from loading.

We can solve this problem by moving the script to the bottom of the page (or by adding async attribute). After moving the angular.js to the bottom of the body tag, the page loads right away, despite the delay. Unfortunately, this shows a few new problems.

pending angular

We can hide the application until it is ready to interact with the user using ng-cloak directive. To make the directive work, we include the angular CSS file in the head element (this small file will be blocking load), and then apply the cloak directive to the angular part of the page.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
<head>
  <link rel="stylesheet" type="text/css" href="bower_components/angular/angular-csp.css">
</head>
<body>
    <h1>github-user</h1>
    <div ng-app="GithubUser" ng-controller="GithubUserController" ng-cloak>
      <input ng-model="username" /> <button id="load" ng-click="load()">Load user info</button>
      <div class="info" ng-show="info">{{ info | json }}</div>
    </div>
    <script src="bower_components/angular/angular.js"></script>
    <script>
        // our application's code
    </script>
</body>

Now even if the angular.js takes a long time to load, the user will see the initial page content. Then the Angular application will reveal itself. You can see the changed code at tag step-2.

Using turtle-run one can make the application more robust to network and server errors. Read Give browser a chance for other suggestions how to work around the delayed or missing page information.

Intercepting cross-domain Ajax calls with service-turtle

A stand alone proxy server like turtle-run has a downside - it cannot intercept Ajax calls from the page to other service endpoints. For example, the demo page github-user makes a request directly from the page to the Github API

1
$http.get('https://api.github.com/users/' + $scope.username)

Can we test how our page handles 500 error from the Github API? We can by modifying the application's code a little and introducing service-turtle. It is a proxy that sits behind the page and can intercept any request. The proxy is implemented using an experimental API called ServiceWorker, and requires running the Chrome browser (at least version 39) with an advanced flag enabled

Open chrome://flags/ in the Chrome browser
Enable "Enable experimental Web Platform features."
Restart Chrome

You can grab the code and run the application with the included service-turtle by checking out step-3 and running http-server locally using http-server -p 3003. When running the local server, the page loads turtle.js, which in turn initializes the service worker using script service-turtle.js. Every Ajax request from the page then generates fetch event inside the ServiceWorker.

service turtle

The worker can let the fetch event pass, letting the request through. The worker can also respond to the event with fake data using event.respondWith method.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// self - reference to the ServiceWorker itself
self.addEventListener('fetch', function (event) {
    // mockData - mock response for event.request.url
    var options = mockData.options || {};
    var responseOptions = {
      status: options.code || options.status || options.statusCode,
      headers: {
        'Access-Control-Allow-Origin': '*',
        'Content-Type': 'application/json; charset=utf-8'
      }
    };
    var body = JSON.stringify(options.body || options.data);
    var response = new Response(body, responseOptions);
    event.respondWith(response);
}

We can confirm that this is working by registering a mock response using turtle object the script registers globally

1
turtle.get('/users/', { code: 200, body: 'hi there' });

The object turtle registered from the bower_components/service-turtle/turtle.js script sends the arguments to the service worker by posting a message. The execution example (first the normal request, then the mocked one) is shown below.

service turtle mock

If we want to delay the mock response by N milliseconds, we can using Promise API. We just need to respond with a Promise object

1
2
3
4
5
6
7
8
self.addEventListener('fetch', function (event) {
    // response object created as above
    event.respondWith(new Promise(function (resolve) {
        setTimeout(function () {
            resolve(response);
        }, options.timeout);
    }));
}

Testing and handling server error responses

In the next example, we can mock /users request to return server error 500.

1
turtle.get('/users/', { code: 500 });

Notice that the error is only printed to the console log, no user message appears and the result of the previous fetch is still displayed. Thus the application will NOT handle Github server errors in a user-friendly manner.

service turtle 500

It is simple to fix: just put the error message into the result to be displayed

1
2
3
4
5
6
7
8
$http.get('https://api.github.com/users/' + $scope.username)
  .success(function (data) {
    ...
  })
  .error(function (err) {
    $scope.info = 'Could not fetch info for user ' + $scope.username;
    console.error(err);
  });

Now the user will know something bad has happened and will not wait or be confused by previous data.

I plan to extend service-turtle to cover POST, DELETE, PUT and PATCH methods to allow mocking any request. Even with just GET method support, the service-turtle seems a very powerful tool at the cost of including a very small script. The inclusion could be limited to the development and the staging environments for security. The service workers are already limited to be loaded from safe urls: localhost or https. This prevents me from demoing this feature from the Github pages, since it does not support SSL.

Another important thing to remember - I do not know what the actual Github server response is for 500 errors. Instead of a plain error string like I used in the mock, it could be an object:

1
2
3
4
5
{
    type: 'error',
    message: 'Something went wrong on our side',
    code: 500
}

It is important to know the response schema you might want to mock, otherwise you are in for a surprise.

Intercepting request on the fly using ng-wedge

What if we cannot run a proxy between the client and the server (like turtle-run works) and do not want to include a separate ServiceWorker (like service-turtle does)? What if we do not even want to reload our web application, but just want to mock a response? Seems like an impossible task - hooking into the live application and changing its behavior.

Luckily, in case of AngularJS application we can do such substitution on the fly. Other frameworks could potentially work too, but for now I use mostly AngularJS so I will concentrate on this framework. A typical application (like our github-user) has a $scope object. The $scope object is initialized inside a controller function. The controller also injects $http service object. This $http object is used to perform all Ajax requests.

1
2
3
4
5
6
7
8
9
10
11
12
13
controller('GithubUserController', function ctrl($scope, $http) {
    $scope.load = function () {
        $http.get('https://api.github.com/users/' + $scope.username)
            .success(function (data) {
                console.log(data);
                $scope.info = data;
            })
            .error(function (err) {
                $scope.info = 'Could not fetch info for user ' + $scope.username;
                console.error(err);
            });
        };
    });

At runtime the method $scope.load will be called when the user clicks on a button

1
<button id="load" ng-click="load()">Load user info</button>

We can get the reference to the actual $scope object from the browser's console at any moment.

1
2
3
4
5
6
7
8
var $scope = angular.element(document.getElementById('load')).scope();
// $scope is
{
    // internal properties ...
    $parent: Scope,
    load: function (),
    username: "bahmutov"
}

Mocking live $http call

One of the JavaScript's strengths is its dynamic nature. We can easily replace / wrap a method with another method at run time.

1
2
3
4
var $scope = angular.element(document.getElementById('load')).scope();
$scope.load = function () { console.log('hi'); };
// click on "load" button
// console prints "hi"

To mock responses, we do not want to replace the $scope.load method. Instead we want to somehow replace the $http object used inside $scope.load. We can replace it with our own "proxy" $http object that can pass most calls to the real $http service, but return mock responses for some requests. In a sense we want to mock a variable available to the $scope.load via lexical scope

1
2
3
4
5
6
7
8
9
// application code
controller('GithubUserController', function ctrl($scope, $http) {
    $scope.load = function () {
        $http.get();
    }
}
// browser console
var $scope = angular.element(document.getElementById('load')).scope();
// Hmm, how can we access the $http argument passed into ctrl function?

Fortunately this is possible via Faking lexical scope method. In principle we convert the actual method $scope.load to its source code using .toString(), then eval it back but prepend the source code with mock $http variable. Any variable eval does not find in the source are plucked from the lexical scope of the place where the eval is called.

1
2
3
4
5
6
7
var $scope = angular.element(document.getElementById('load')).scope();
var $http = {
    get: function () { console.log('I am fake $http'); }
};
$scope.load = eval($scope.load.toString());
// click "load" button
prints 'I am fake $http'

The code is modified in page, without any extras needed

ng-wedge-diagram

Driving an wedge into live production app

Let us go back to our demo at glebbahmutov.com/github-user. Open the browser console and copy / paste the entire file ng-wedge.js. ng-wedge operates on a specific $scope instance's method. First we make a mock object, then define a fake response for specific url. Let us return an error message after a delay of 2000ms

1
2
var mock = wedge('#load', 'load');
mock('get', 'https://api.github.com/users/bahmutov', 500, 'not working', 2000);

Here is how it looks in action

ng-wedge screenshot

Notice that there is no actual HTTP request in the network tab - the request is intercepted before the Ajax request happens. Any other Ajax request, even to the same Github API endpoint happening through other $scope objects would NOT be intercepted by ng-wedge. It only replaces a particular method on a specific $scope instance.

Summary

I created 3 different proxy tools for conveniently mocking server responses.

  • turtle-run - a stand alone proxy between the user and the server. Does not require any modifications to the server code to run. The user needs to access the proxy URL instead of the actual server url. The proxy cannot work with https servers or proxy cross-domain requests.
  • service-turtle - a good choice for local development. Requires using the latest version of Chrome browser and turning on an experimental feature support. It is powerful and simple to control on demand, the code is still under development.
  • ng-wedge - a good choice for a developer when investigating the production website's behavior. Does not require adding extra libraries or even reloading the page. Currently works with AngularJS web applications only, but could be adapted to other frameworks. Requires a "simple" controller code that can be reused in eval call.