Nagual

What is this?

This is a standalone HTTP server created to simulate various responses as if they are returned by a third party web service APIs. For example, the simulator can be configured to act as api.twitter.com or graph.facebook.com without your application knowing the difference. Its primary use is for functional (manual and automated) testing. Its secondary uses are monitoring and performance/stress testing.

Here is a presentation about Nagual, that I gave at Agile Testing Days 2016 conference.

How to install Nagual

1. Install node.js and npm
2. Clone this repository: git clone https://github.com/emanuil/nagual (or download the zip file)
3. Install the required npm packages: cd nagual && npm install
4. Run the simulator: sudo node proxy.js

You should see the following output

Starting the HTTP(S) protocol simulator...
Loaded stubs count per host:

c9555.r55.cf2.rackcdn.com: 1
api.instagram.com: 6
api.linkedin.com: 4
api.twitter.com: 12
blah.com: 0 Transparent mode only!
cdn2.clouddrive.com: 1
graph.facebook.com: 32
identity.api.rackspacecloud.com: 1
porto.zendesk.com: 4
storage101.ord1.clouddrive.com: 5
www.googleapis.com: 5

Listening on port 80
Listening on port 443

In order to start using the simualtor you need to peform the two steps in bold bellow.

Routing traffic through Nagual

After Nagual has started its time to route your application traffic through it. Depending on how your application is configured there are couple of ways to do this. The easiest is to add an entry to /etc/hosts file on your application server to redirect all traffic to the simulator.

For example if 192.168.8.21 is the IP address where Nagual is running and you want to simulate an external service host api.twitter.com, you should add the following line to /etc/hosts file on the server thats running your applications.

192.168.8.21 api.twitter.com

This will route all the traffic intended to the external service, first through the simulator. Depending on how you setup Nagual rules, the requests will either be simulated or forwarded to the real service.

If for some reason you can not edit /etc/hosts, your options would be (depending on the architecture of your application):

• to edit your application code
• to edit a config file
• to edit a configuration in your database
• to edit the DNS records

The main point is to ultimately redirect all the HTTP traffic that you need to simulate to Nagual.

Screen Capture

Since all the traffic passes through Nagual, you can monitor it in real time. This traffic is usually encrypted and visible only to the client and to the server. Since Nagual acts are MiTM proxy, it can see and modify everything.

Handling HTTPS traffic

Most of the interesting external services run over HTTPS. Your code (or the code in the 3rd party libraries) used to connect to the external services has SSL server verification enabled. On every connection, your application checks if it talks to the real api.twitter.com, instead of any other server with self signed SSL certificate. This way your app is making sure that it is not susceptible to man-in-the-middle attack. Nagual uses the same technique to do good.

When Nagual starts, it auto generates fake SSL certificate issued to api.twitter.com. However, it is signed by root certificate that is itself self-signed (not signed by any of the built in root CAs in your OS). The server SSL verification would still fail at this point.

In order to enable HTTPS traffic simulation you need to do one thing. Install Nagual’s root certificate as trusted on your app OS. Then your app will not complain that the certificate issued to api.twitter.com is signed by unknown authority.

Nagual includes a default root certificate root_certificate/rootCA.crt, that you can use it to test with. However if you’re serious about security you should generate and use your own, just put it in root_certificate directory (there need to be two files rootCA.crt and rootCA.key).

Here is how to install the root certificate (root_certificate/rootCA.crt) as trusted in your OS.

If your application is Java based you need to install the root certificate in the Java keystore.

A Simple Stub

The simplest possible stub looks like this:

module.exports = function() {
    this.method = 'GET';
    this.url = '/twitter/:id';

    this.token = {
        string: 'test-token',
        location: 'url'
    };

    this.getResponse = function() {

        return {
            'body': JSON.stringify({"name":"John Doe"})
        };
    };
};

Assume that there is an API located here example.com/twitter/:id. When called with GET request it returns the name associated with the twitter id from the URL (:id can be random integer). The curl request to call this API might be:

curl https://example.com/twitter/71231348&token=test-token

Nagual will catch these request and will return {"name":"John Doe"} in the body.

Stub Location

Stubs should be located in the stubs directory. The next sub directory should be either http or https. The next sub directory is name of the server (exactly the same as in the request URL). Next comes the stub file name. It can be free style, as long as it ends with .js. However using a convention is really helpful - e.g. http_verb_request_url.js. For example, the simple stub declared above should be located here:

stubs/https/example.com/get_twitter_id.js

Stub Properties

The stub follows the standard Node.js modules structure, and is executable JavaScript code that provides great flexibility. It has 4 mandatory properties that have to be defined:

• this.method The HTTP verb used in the request. It's usually GET, POST, DELETE, HEAD or OPTIONS. Case insensitive.

• this.url This is the URL which is to be matched. It can be defined as string or as JavaScript regular expression. Important: The URL matches only the part after the host and before the arguments (separated by ?). For example, if the request has this full URL:

  https://example.com/twitter/1234874?token=test-token-random-stuff

  then

  this.url matches this part only /twitter/1234874

  Three of the most used regular expressions to match RESTful stule APIs can be used:

  • :string matches a random sting, e.g. 'KKlkdjUjasd'
  • :number matches a random number, e.g. '98129830123'
  • :id matched any combination of all printable characters, e.g. 'token-123-abc-456'

  So if you want to match the above URL you can do it in two ways:

  • this.url = '/twitter/:number'
  • this.url = '/twitter/:id'

  If this does not suite you, you can always write a custom JavaScript Regex

• this.token This is the string that needs to be present in the specified location (url, headers or body). If you do not want to use it you can effective disable it, by matching all the URLS:

this.token = {
    string: '',
    location: 'url'
};

• this.getResponse() If all of the above stub properties match, this function is called to generate the fake response. It returns an object with one or more of the following properties:
  • body the response body, most of the time you'll only need to return only this property
  • statusCode if you don't set it, it's 200 by default
  • statusMessage if you don't set it, it's 'OK' by default
  • headers an object that defines one or more custom headers to be returned

When the method, url and the token properties defined in the stub all match to the incoming request, then instead of redirecting to the real service, Nagual generates the response by calling this.getResponse() function. If one or more of the method, url or token properties do not match, then the request is forwared to the real service. The real response is forwarded back unconditionally.

Monitoring

Monitoring is the second type of usage that Nagual was designed for. This can be totally separated activity than simulating responses. Nagual is completely transparent as far as your application is concerned.

Since all the HTTP traffic passes through Nagual it can easily be monitored. Nagual supports -s statsd_server command line switch in case you want to log the number of requests to your StatsD server and later to visualize them. Out of the box, Nagual provides four types of statistics for the passing HTTP traffic.

• Total HTTP requests passing through Nagual for one hour. Increased number of requests after code push might indicate a problem. It may even cause the external service to throttle you, or to increase your spending. The graph bellow show a peak around midnight, thats when a lot of off-hours jobs are scheduled for.

• The number of 4xx errors. This usually means that your application is not configured correctly when communicating to the external services. The graph bellow shows a moment when we calling a Facebook API with wrong parameters.

![The number of 4xx errors for a day](/screenshots/