HRRS (HTTP Request Record Suite) is a set of tools that you can leverage to record, transform, and replay HTTP requests in your Java EE and Spring web applications written in Java 8 or higher. In essence, HRRS bundles a servlet filter for recording (hrrs-servlet-filter) and standalone command-line Java applications for transforming (hrrs-distiller) and replaying (hrrs-replayer) the requests.

Rationale
Overview
Getting Started (setting up a Spring web application, running distiller and replayer)
Recorder Configuration
Recorder Performance
Replayer Reports (Dropwizard Metrics and JMeter reports)
Distiller & Replayer Debugging
F.A.Q.
Caveats
License

Rationale

Why would someone want to record HTTP requests as is? There are two major problems that HRRS is aiming to solve:

Realistic performance tests: Artificially generated test data falls short of covering many production states. Testing with unrealistic user behaviour can cause caches to misbehave. Or benchmarks might have used JSON/XML for simplicity, while the actual production systems communicate over a binary protocol such as Protocol Buffers or Thrift. These short comings undermine the reliability of performance figures and renders regression reports unusable. HRRS lets the production load to be stored and reflected back to the test environment for more credible test results.
Diagnosing production problems: It might not always be a viable option to remotely debug an instance for production surfacing problems. HRRS can be leveraged to record the problem on production and replay it on development environment for further inspection.
Warming up standby service caches: Standby systems are an inevitable part of modern software architectures: reliability, separation of read & write clusters, etc. While replacing primaries with secondary systems, a cold replacement is anticipated to initially yield a degraded performance, which might not be desirable for certain systems. HRRS can be used to warm up the secondaries prior to deployment and alleviate this problem.

Overview

HRRS Overview

HRRS ships the following artifacts:

hrrs-api: Basic API models and interfaces like HttpRequestHeader, HttpRequestRecord, HttpRequestRecordReader, HttpRequestRecordReaderSource, etc.
hrrs-servlet-filter: Basic servlet filter leveraging the functionality of the API interfaces.
hrrs-replayer: The command line replayer application.
hrrs-distiller: A command line tool to transform and/or filter stored HttpRequestRecords.

These artifacts provide interfaces for the potential concrete implementations. Fortunately, we provide one for you: File-based Base64 implementation. That is, HTTP request records are encoded in Base64 and stored in a plain text file. Following artifacts provide this functionality:

hrrs-serializer-base64: The reader/writer implementation using Base64.
hrrs-servlet-filter-base64: Servlet filter implementation using the Base64 serializer.
hrrs-replayer-base64: The command line replayer implementation using the Base64 serializer.
hrrs-distiller-base64: The command line distiller implementation using the Base64 serializer.

HRRS is designed with extensibility in mind. As of now, it only supports file sourced/targeted Base64 readers/writers. But all you need is a few lines of code to introduce your own serialization schemes powered by a storage backend (RDBMS, NoSQL, etc.) of your preference.

Source code also contains the following modules to exemplify the usage of HRRS with certain Java web frameworks:

hrrs-example-jaxrs
hrrs-example-spring

Getting Started

In order to start recording HTTP requests, all you need is to plug the HRRS servlet filter into your Java web application. Below, we will use Base64 serialization for recording HTTP requests in a Spring web application. (See examples directory for the actual sources and the JAX-RS example.)

Add the HRRS servlet filter Maven dependency to your pom.xml:

<dependency>
    <groupId>com.vlkan.hrrs</groupId>
    <artifactId>hrrs-servlet-filter-base64</artifactId>
    <version>${hrrs.version}</version>
</dependency>

In the second and last step, you expose the HRRS servlet filter as beans so that Spring can inject them as interceptors:

@Configuration
public class HrrsConfig {

    @Bean
    public HrrsFilter provideHrrsFilter() throws IOException {
        String tmpPathname = System.getProperty("java.io.tmpdir");
        String file = new File(tmpPathname, "hrrs-spring-records.csv").getAbsolutePath();
        String filePattern = new File(tmpPathname, "hrrs-spring-records-%d{yyyyMMdd-HHmmss-SSS}.csv").getAbsolutePath();
        RotationConfig rotationConfig = RotationConfig
                .builder()
                .file(file)
                .filePattern(filePattern)
                .policy(new ByteMatchingRotationPolicy((byte) '\n', 50_000))
                .build();
        return new Base64HrrsFilter(rotationConfig);
    }

    @Bean
    public ServletRegistrationBean provideHrrsServlet() {
        HrrsServlet hrrsServlet = new HrrsServlet();
        return new ServletRegistrationBean(hrrsServlet, "/hrrs");
    }

}

And that's it! The incoming HTTP requests will be recorded into writerTargetFile. (You can also run HelloApplication of examples/spring in your IDE to see it in action.) All you need to do is instructing the HRRS servlet to enable the recorder:

$ curl http://localhost:8080/hrrs
{"enabled": false}

$ curl -X PUT http://localhost:8080/hrrs?enabled=true

After a couple of GET /hello?name=<name> queries, let's take a quick look at the contents of the Base64-serialized HTTP request records:

$ zcat records.csv.gz | head -n 3
iz4mjlt9_8f89s  20170213-224106.477+0100  hello  POST  ABYvaGVsbG8/bmFtZT1UZXN0TmFtZS0xAAAABQAEaG9zdAAObG9jYWxob3N0OjgwODAACnVzZXItYWdlbnQAC2N1cmwvNy40Ny4wAAZhY2NlcHQAAyovKgAMY29udGVudC10eXBlAAp0ZXh0L3BsYWluAA5jb250ZW50LWxlbmd0aAACMTMAAAAAAAAAAAAAAA9yYW5kb20tZGF0YS0x//8=
iz4mjlui_1l3bw  20170213-224106.522+0100  hello  POST  ABYvaGVsbG8/bmFtZT1UZXN0TmFtZS0zAAAABQAEaG9zdAAObG9jYWxob3N0OjgwODAACnVzZXItYWdlbnQAC2N1cmwvNy40Ny4wAAZhY2NlcHQAAyovKgAMY29udGVudC10eXBlAAp0ZXh0L3BsYWluAA5jb250ZW50LWxlbmd0aAACMTMAAAAAAAAAAAAAAA9yYW5kb20tZGF0YS0z//8=
iz4mjlty_sicli  20170213-224106.502+0100  hello  POST  ABYvaGVsbG8/bmFtZT1UZXN0TmFtZS0yAAAABQAEaG9zdAAObG9jYWxob3N0OjgwODAACnVzZXItYWdlbnQAC2N1cmwvNy40Ny4wAAZhY2NlcHQAAyovKgAMY29udGVudC10eXBlAAp0ZXh0L3BsYWluAA5jb250ZW50LWxlbmd0aAACMTMAAAAAAAAAAAAAAA9yYW5kb20tZGF0YS0y//8=

(If you can't see any content yet, you can enforce flushing via curl -X POST http://localhost:8080/hrrs.)

Here each line corresponds to an HTTP request record and fields are separated by \t character. A line first starts with plain text id, timestamp, group name, and method fields. There it is followed by a Base64-encoded field containing the URL (including request parameters), headers, and payload. This simple representation makes it easy to employ well-known command line tools (grep, sed, awk, etc.) to extract a certain subset of records.

$ zcat records.csv.gz | head -n 1 | awk '{print $5}' | base64 --decode | hd
00000000  00 16 2f 68 65 6c 6c 6f  3f 6e 61 6d 65 3d 54 65  |../hello?name=Te|
00000010  73 74 4e 61 6d 65 2d 31  00 00 00 05 00 04 68 6f  |stName-1......ho|
00000020  73 74 00 0e 6c 6f 63 61  6c 68 6f 73 74 3a 38 30  |st..localhost:80|
00000030  38 30 00 0a 75 73 65 72  2d 61 67 65 6e 74 00 0b  |80..user-agent..|
00000040  63 75 72 6c 2f 37 2e 34  37 2e 30 00 06 61 63 63  |curl/7.47.0..acc|
00000050  65 70 74 00 03 2a 2f 2a  00 0c 63 6f 6e 74 65 6e  |ept..*/*..conten|
00000060  74 2d 74 79 70 65 00 0a  74 65 78 74 2f 70 6c 61  |t-type..text/pla|
00000070  69 6e 00 0e 63 6f 6e 74  65 6e 74 2d 6c 65 6e 67  |in..content-leng|
00000080  74 68 00 02 31 33 00 00  00 00 00 00 00 00 00 00  |th..13..........|
00000090  00 0f 72 61 6e 64 6f 6d  2d 64 61 74 61 2d 31 ff  |..random-data-1.|
000000a0  ff                                                |.|
000000a1

Once you start recording HTTP requests, you can setup logrotate to periodically rotate and compress the record output file. You can even take one step further and schedule a cron job to copy these records to a directory accessible by your test environment. There you can replay HTTP request records using the replayer provided by HRRS:

$ java \
    -jar /path/to/hrrs-replayer-base64-<version>.jar \
    --targetHost localhost \
    --targetPort 8080 \
    --threadCount 10 \
    --maxRequestCountPerSecond 1000

Hrrs

Install / Use

README

Table of Contents

Rationale

Overview

Getting Started