CsCheck

CsCheck is a C# random testing library inspired by QuickCheck.

It differs in that generation and shrinking are both based on PCG, a fast random number generator.

This gives the following advantages over tree based shrinking libraries:

• Automatic shrinking. Gen classes are composable with no need for Arb classes. So less boilerplate.
• Random testing and shrinking are parallelized. This and PCG make it very fast.
• Shrunk cases have a seed value. Simpler examples can easily be reproduced.
• Shrinking can be continued later to give simpler cases for high dimensional problems.
• Parallel testing and random shrinking work well together. Repeat is not needed.

See why you should use it, the comparison with other random testing libraries, or how CsCheck does in the shrinking challenge. In one shrinking challenge test CsCheck managed to shrink to a new smaller example than was thought possible and is not reached by any other testing library. CsCheck is the only random testing library that can always shrink to the simplest example (given enough time).

CsCheck also has functionality to make multiple types of testing simple and fast:

• Random testing
• Model-based testing
• Metamorphic testing
• Parallel testing
• Causal profiling
• Regression testing
• Performance testing
• Debug utilities
• Configuration
• Development

The following tests are in ~~xUnit~~ TUnit but could equally be used in any testing framework.

More to see in the Tests. There are also 1,000+ F# tests using CsCheck in MKL.NET.

No Reflection was used in the making of this product. CsCheck is close to being AOT compatible but 'generic recursion is AOT kryptonite'.

Generator Creation Example

Use Gen and its Linq methods to compose generators for any type. Here we create a Gen for json documents. More often it will simply be composing a few primitives and collections. Don't worry about shrinking as it's automatic and the best in the business.

static readonly Gen<string> genString = Gen.String[Gen.Char.AlphaNumeric, 2, 5];
static readonly Gen<JsonNode> genJsonValue = Gen.OneOf<JsonNode>(
    Gen.Bool.Select(x => JsonValue.Create(x)),
    Gen.Byte.Select(x => JsonValue.Create(x)),
    Gen.Char.AlphaNumeric.Select(x => JsonValue.Create(x)),
    Gen.DateTime.Select(x => JsonValue.Create(x)),
    Gen.DateTimeOffset.Select(x => JsonValue.Create(x)),
    Gen.Decimal.Select(x => JsonValue.Create(x)),
    Gen.Double.Select(x => JsonValue.Create(x)),
    Gen.Float.Select(x => JsonValue.Create(x)),
    Gen.Guid.Select(x => JsonValue.Create(x)),
    Gen.Int.Select(x => JsonValue.Create(x)),
    Gen.Long.Select(x => JsonValue.Create(x)),
    Gen.SByte.Select(x => JsonValue.Create(x)),
    Gen.Short.Select(x => JsonValue.Create(x)),
    genString.Select(x => JsonValue.Create(x)),
    Gen.UInt.Select(x => JsonValue.Create(x)),
    Gen.ULong.Select(x => JsonValue.Create(x)),
    Gen.UShort.Select(x => JsonValue.Create(x)));
static readonly Gen<JsonNode> genJsonNode = Gen.Recursive<JsonNode>((depth, genJsonNode) =>
{
    if (depth == 5) return genJsonValue;
    var genJsonObject = Gen.Dictionary(genString, genJsonNode.Null())[0, 5].Select(d => new JsonObject(d));
    var genJsonArray = genJsonNode.Null().Array[0, 5].Select(i => new JsonArray(i));
    return Gen.OneOf(genJsonObject, genJsonArray, genJsonValue);
});

Random testing

Sample is used to perform tests with a generator. Either return false or throw an exception for failure. Sample will aggressively shrink any failure down to the simplest example.
The default sample size is 100 iterations. Set iter: to change this or time: to run for a number of seconds.
Setting these from the command line can be a good way to run your tests in different ways and in Release mode.

Unit Single

[Test]
public void Single_Unit_Range()
{
    Gen.Single.Unit.Sample(f => f is >= 0f and <= 0.9999999f);
}

Long Range

[Test]
public void Long_Range()
{
    (from t in Gen.Select(Gen.Long, Gen.Long)
     let start = Math.Min(t.V0, t.V1)
     let finish = Math.Max(t.V0, t.V1)
     from value in Gen.Long[start, finish]
     select (value, start, finish))
    .Sample(i => i.start <= i.value && i.value <= i.finish);
}

Int Distribution

[Test]
public void Int_Distribution()
{
    int buckets = 70;
    int frequency = 10;
    int[] expected = Enumerable.Repeat(frequency, buckets).ToArray();
    Gen.Int[0, buckets - 1].Array[frequency * buckets]
    .Select(sample => Tally(buckets, sample))
    .Sample(actual => Check.ChiSquared(expected, actual));
}

Serialization Roundtrip

static void TestRoundtrip<T>(Gen<T> gen, Action<Stream, T> serialize, Func<Stream, T> deserialize)
{
    gen.Sample(t =>
    {
        using var ms = new MemoryStream();
        serialize(ms, t);
        ms.Position = 0;
        return deserialize(ms).Equals(t);
    });
}
[Test]
public void Varint()
{
    TestRoundtrip(Gen.UInt, StreamSerializer.WriteVarint, StreamSerializer.ReadVarint);
}
[Test]
public void Double()
{
    TestRoundtrip(Gen.Double, StreamSerializer.WriteDouble, StreamSerializer.ReadDouble);
}
[Test]
public void DateTime()
{
    TestRoundtrip(Gen.DateTime, StreamSerializer.WriteDateTime, StreamSerializer.ReadDateTime);
}

Shrinking Challenge

[Test]
public void No2_LargeUnionList()
{
    Gen.Int.Array.Array
    .Sample(aa =>
    {
        var hs = new HashSet<int>();
        foreach (var a in aa)
        {
            foreach (var i in a) hs.Add(i);
            if (hs.Count >= 5) return false;
        }
        return true;
    });
}

Recursive

record MyObj(int Id, MyObj[] Children);

[Test]
public void RecursiveDepth()
{
    int maxDepth = 4;
    Gen.Recursive<MyObj>((i, my) =>
        Gen.Select(Gen.Int, my.Array[0, i < maxDepth ? 6 : 0], (i, a) => new MyObj(i, a))
    )
    .Sample(i =>
    {
        static int Depth(MyObj o) => o.Children.Length == 0 ? 0 : 1 + o.Children.Max(Depth);
        return Depth(i) <= maxDepth;
    });
}

Classify

Change the return in Sample to a string to produce a summary classification table. All other optional parameters work the same but writeLine: is now mandatory.

[Test]
public void AllocatorMany_Classify()
{
    Gen.Select(Gen.Int[3, 30], Gen.Int[3, 15]).SelectMany((rows, cols) =>
        Gen.Select(
            Gen.Int[0, 5].Array[cols].Where(a => a.Sum() > 0).Array[rows],
            Gen.Int[900, 1000].Array[rows],
            Gen.Int.Uniform))
    .Sample((solution,
             rowPrice,
             seed) =>
    {
        var rowTotal = Array.ConvertAll(solution, row => row.Sum());
        var colTotal = Enumerable.Range(0, solution[0].Length).Select(col => solution.SumCol(col)).ToArray();
        var allocation = AllocatorMany.Allocate(rowPrice, rowTotal, colTotal, new(seed), time: 60);
        if (!TotalsCorrectly(rowTotal, colTotal, allocation.Solution))
            throw new Exception("Does not total correctly");
        return $"{(allocation.KnownGlobal ? "Global" : "Local")}/{allocation.SolutionType}";
    }, TUnitX.WriteLine, time: 900);
}

| | Count | % | Median | Lower Q | Upper Q | Minimum | Maximum | |--------------------|------:|--------:|------------:|------------:|------------:|------------:|------------:| | Global | 458 | 50.22% | | | | | | |   RoundingMinimum | 343 | 37.61% | 2.68ms | 0.50ms | 10.85ms | 0.03ms | 190.92ms | |   EveryCombination | 87 | 9.54% | 173.99ms | 16.80ms | 1,199.64ms | 0.20ms | 42,257.35ms | |   RandomChange | 28 | 3.07% | 59,592.98ms | 55,267.94ms | 59,901.58ms | 38,575.41ms | 60,107.64ms | | Local | 454 | 49.78% | | | | | | |   RoundingMinimum | 301 | 33.00% | 60,000.12ms | 60,000.04ms | 60,003.70ms | 60,000.02ms | 60,144.84ms | |   RandomChange | 90 | 9.87% | 60,000.06ms | 60,000.03ms | 60,004.41ms | 60,000.02ms | 60,136.59ms | |   EveryCombination | 63 | 6.91% | 60,000.10ms | 60,000.03ms | 60,001.29ms | 60,000.01ms | 60,019.36ms |

Model-based testing

Model-based is the most efficient form of random testing. Only a small amount of code is needed to fully test functionality. SampleModelBased generates an initial actual and model and then applies a random sequence of operations to both checking that the actual and model are still equal.

SetSlim Add

[Test]
public void
SetSlim_ModelBased()
{
    Gen.Int.Array.Select(a => (new SetSlim<int>(a), new HashSet<int>(a)))
    .SampleModelBased(
        Gen.Int.Operation<SetSlim<int>, HashSet<int>>(
            (ss, i) => ss.Add(i),
            (hs, i) => hs.Add(i)
        )
        // ... other operations
    );
}

Metamorphic testing

The second most efficient form of random testing is metamorphic which means doing something two different ways and checking they produce the same result. SampleMetamo