<img src="doc/images/logo.png" width="32" /> HttpRecorder for .Net

Note: This is a Work In Progress, check back soon for updates

Status: Build and runs, no NuGet yet, correctly saves and replays response.

Features

At its core the library is designed to record HttpResponseMessage results for any given HttpRequestMessage into a binary LZW compressed 'Cassette' file. This is primarily designed to support both the following use cases:

• Recording calls to an external API for replaying in test runs, allowing tests to be run offline and reliably/repeatably.
• Efficient caching and retrieval of responses to for a website (at either end).

Many of the libraries out there support one or the other, but fundamentally the problems are the same, and just require a better interface design.

Other core features I couldn't find elsewhere:

• Highly extensible, including allowing different backing stores.
• Fast, compact storage, that supports byte content not just strings. For this I'm using the MessagePack format (via MessagePack-CSharp) with recordings held in a compressed archive.
• Correctly instrument HttpClientFactory clients, and support custom HttpMessageHandlers
• Allowing request matching to be highly customisable (i.e. deciding which parts of a request to match on whilst ignoring other elements)
• Accurately rebuilding a HttpResponseMessage as it was initially returned (this is particularly releveant to the HttpResponseMessage.Content type which most libraries do not reproduce accurately)
• Supports recording of the HttpResponseMessage.RequestMessage (see RequestRecordMode and RequestPlaybackMode)
• Support all content types. [WIP]
• Parameterisation of Request/Response, as well as allowing a real request to be recorded and played back this could allow for dummy recordings to be made with parameters to take from the request and place in the response. [WIP]
• Secret hiding. This was the original driver for parameterisation and can actually be solved using it, the idea is that secrets would not be stored on the cassette (e.g. passwords, keys, etc.) and instead replaced with parameters. [WIP]
• Doesn't assume custom handlers don't mangle the response's request object [TODO]
• Support streaming of responses to disk, allowing for large disks and efficient memory usage - particularly useful for replaying large file downloads, etc. [TODO]

Examples

The starting point for any use is to create a WebApplications.HttpRecorder.Cassette. Cassettes can be kept for the lifetime of the application or a seasion, but ideally should be disposed once finished with to ensure the underlying store is dispoed (if owned by the cassette) and to dispose any internal locks.

Creating a Cassette can be as easy as:

using (Cassette cassette = new Cassette())
{
    ...
}

This will create a cassette capable of holding multiple recordings in a single file, which will be placed alongside the calling method's source file with an extension '_cassette.hrc'.

HttpClient instrumentation

The easiest way to instrument a System.Net.HttpClient is to have a Cassette create one for you:

using (Cassette cassette = new Cassette())
using (HttpClient client = cassette.GetClient())
{
    ...
}

At this point any messages you send and receive with the client will be recorded to the Cassette. If the Cassette already exists and contains a matching response, then it will replay the response without hiting the real endpoint.

Alternatively you can retrieve an instrumented System.Net.HttpMessageHandler from the Cassette:

using (Cassette cassette = new Cassette())
using (HttpMessageHandler handler = cassette.GetHttpMessageHandler())
using (HttpClient client = new HttpClient(handler))
{
    ...
}

The GetHttpMessageHandler method supports passing in an inner System.Net.HttpMessageHandler making it capable of instrumenting your own pipelines.

Record anywhere

Ultimately, all the Cassette's instrumentation overloads, ultimately call the core RecordAsync method which can be used to record and playback any response.

using (Cassette cassette = new Cassette())
{
    await cassette.RecordAsync(request, async (r, ct) => response, cancellationToken);
    
    // or if you have the request and response already
    await cassette.RecordAsync(request, response, cancellationToken);
    
    // or if you only have the response (uses response.RequestMessage)
    await cassette.RecordAsync(response, cancellationToken);    
}

Caching responses in middleware

TODO

Changing key matching

Under the hood the Recorder converts a HttpRequestMessage to a 22 character URI Safe (see Section 2.3 RFC 3986) hash. It does this by serializing the request to a byte[] and then running an MD5 hash over it (yes I know MD5 is 'insecure' but it is a fast hash and we're not using it for security but collision avoidance which it is more than adequate for!).

The key data is generated using a KeyGenerator, the default one being FullRequestKeyGenerator.Instance.

TODO: Complete explanation

Changing the backing store

The recorder reads and writes to a backing store, by default this is a single Zip Archive, however there are other options available.

TODO

Options

The Cassette accepts a CassetteOptions object on creation which supply default options for all recordings made to the cassette. It can also be overloaded when using GetClient or GetHttpMessageHandler to apply to recordings made by the client, and again on each individual call to RecordAsync. Options are applied over the top of the default options (CassetteOptions.Default).

You can create a CassetteOptions object using its constructor, for example:

using (Cassette cassette = new Cassette(defaultOptions: new CassetteOptions(
    // These are the default options anyway, so no need to do this
    mode: RecordMode.Auto,
    waitForSave: false,
    simulateDelay: TimeSpan.Zero,
    requestRecordMode: RequestRecordMode.Ignore,
    requestPlaybackMode: RequestPlaybackMode.Auto)))
{
    using (HttpClient client = cassette.GetClient(new CassetteOptions(
        // Force this client to overwrite recordings and wait for saves
        mode: RecordMode.Overwrite,
        waitForSave: true)))
    {
        ...
    }
    using (HttpClient client = cassette.GetClient(new CassetteOptions(
        // This client will only playback (or error if no matching requests are found), and will use the recorded delay.
        mode: RecordMode.Playback,
        simulateDelay: TimeSpan.MinValue)))
    {
        ...
    }
}

You can also use some of the helper options and combine them using the & operator, for example the following is functionally equivalent to the above example (though there are more object allocations, so this syntax is best used in places it will not be run frequently):

using (Cassette cassette = new Cassette(defaultOptions: CassetteOptions.Default))
{
    using (HttpClient client = cassette.GetClient(
        CassetteOptions.Overwrite & CassetteOptions.WaitUntilSaved))
    {
        ...
    }
    using (HttpClient client = cassette.GetClient(
        CassetteOptions.Playback & CassetteOptions.RecordedDelay))
    {
        ...
    }
}

RecordMode

The most useful is the RecordMode option which controls how system handles recordings:

| Option | Recording Found? | Outcome | Action | |---------------|:----------------:|:-------:|---------| | Default | :white_check_mark:/:negative_squared_cross_mark:| | Will use the default option for the cassette | | Auto | :negative_squared_cross_mark: | :record_button: | Will hit the endpoint and record the response (default) | | Auto | :white_check_mark: | :play_or_pause_button: | Will playback the response, without hitting the endpoint (default) | | Playback | :negative_squared_cross_mark: | :exclamation: | Throws a CassetteNotFoundException Exception | | Playback | :white_check_mark: | :play_or_pause_button: | Will playback the response, without hitting the endpoint | | Record | :negative_squared_cross_mark: | :record_button: | Will hit the endpoint and record the response | | Record | :white_check_mark: | :left_right_arrow: | Will hit the endpoint and not record the response | | Overwrite | :white_check_mark:/:negative_squared_cross_mark: | :record_button: | Will hit the endpoint and record the response | | None | :white_check_mark:/:negative_squared_cross_mark: | :left_right_arrow: | Will hit the endpoint and not record the response |

WaitForSave

The WaitForSave option when true will force the recorder to wait until the underlying store successfully saves