Fuzzy v2.1.0

A fuzzy ecosystem for evaluating the effect of numerical error on computational tools.

Fuzzy v2.1.0

Motivation

Computational analysis has become an essential component of science. The software tools used bear the weight of the countless models, discoveries, and interventions based upon them. However, computers were never intended to be perfect, and one doesn't need to look very far to find examples of where conceptually simple operations may fail due to floating point arithmetic (e.g. ...

$ python3 -c "print(sum([0.001] * 1000))"
1.0000000000000007

). While small issues like the one above are unlikely to cause catestrophic failures on their own, they can cascade over the course of an execution and it isn't uncommon to find a plane that requires a power cycle every two months or an unstable series that consistently converges to the wrong solution.

Fuzzy allows you to study the stability or trustworthiness of tools and pipelines. You start by instrumenting libraries in your pipeline which do the bulk of numerical heavy lifting, then run your tool multiple times, and finally analyze the variability in your results. This project aims to provide an environment in which the stability of programs can be evaluated, reducing the overhead on both tool developers and consumers.

Usage

Building & booting the environment

You can get started with Fuzzy quite simply, just launch a Docker container as follows:

docker run -ti verificarlo/fuzzy

If you're on a shared system and would prefer to use Singularity, that's no problem, just convert the container using the appropriate method for your system (e.g.).

If you would like to build the environment locally on your system, look at the Dockerfiles in docker/base/ to see how installation was performed. At the end of the build chain, you'll find instrumented versions of libmath, lapack, python3, numpy, and several other recompiled libraries.

An example for how to verify your installation for Python could be the following:

$ python3 -c "print([sum([.001]*1000) for _ in range(3)])"
[1.0, 0.9999999999999997, 1.0000000000000007, 1.0000000000000002]

Adding your software

If your code is written in, say, Python, then you can simply mount your code to this environment, run it using the contained interpreter, and experince the :fuzz:. If your code loads shared libraries, such as libmath or lapack, make sure that they are using the instrumented versions of these libraries made available in /usr/local/lib/. You can check whether this is the case using ldd </path/to/yourbinary>.

The next step is to make sure the environment is configured to introduce perturbations the way you expect. You can start with a configuration which performs perturbations akin to randomizing machine error with the following:

echo "libinterflop_mca.so -m mca --precision-binary32=24 --precision-binary64=53" > $VFC_BACKENDS_FROM_FILE

For more usage instructions on how to control how perturbations are introduced and where they occur within operations, please refer to the Verificarlo repository.

A simple sanity check to verify that your code is using the perturbed libraries is to dramatically reduce the precision in your configuration by changing the precision-binary32 and precision-binary64 values to something small (e.g. 1), boot up a debugging session in your environment (e.g. a Python shell, GDB, etc.), load a math library, and run a basic math function a few times (e.g. sin(1)) — you should get different results if your instrumentation is setup properly. Important: Don't forget to set the precision back to your desired level prior to performing your experiments! Fuzzy should print a log message to the terminal when you run your commands, including the configuration, so you can verify that the parameters were properly specified.

Fuzzy-libmath

Since version v0.8.0, fuzzy-libmath comes with three modes (standard, quad, and mpfr). Internally, the fuzzy-libmath computes the actual result of the math function using the standard library of the OS and then perturbs the output by adding MCA noise. Since the accuracy of the libm used can vary from one version to another (see this article), one can use higher precision to ensure accurate intermediate computations. A new mode called fast was introduced in version v0.9.1 that is equivalent to a MCA noise with a fixed virtual precision (t=24 for binary32, t=53 for binary64) that cannot be changed. These modes specify the library used:

fast: uses the standard libm provided by the OS with a virtual precision fixed to the last bit (fastest)
standard: uses the standard libm provided by the OS
quad: uses the libquadmath.so library
mpfr: uses the MPFR library with 113 bits of precision, equivalent to the binary128 precision (slowest)

Note that a higher precision implies a larger slowdown. The standard mode is the fastest and can be used if one does not require high accuracy. To switch from one version to another, please use the set-fuzzy-libmath tool already installed in the docker image as follow:

usage: set-fuzzy-libmath [-h] --version {no-fuzzy,standard,quad,mpfr}

optional arguments:
  -h, --help            show this help message and exit
  --version {no-fuzzy,standard,quad,mpfr}

Fuzzy libmath version to use:
        - no-fuzzy: Disable fuzzy libmath
        - fast:     Use the standard libmath available on the system.
                    Not possible to modify the virtual precision!
                    Fastest instrumentation but possibly not accurate
        - standard: Use the standard libmath available on the system.
                    Fast instrumentation but possibly not accurate
        - quad:     Use the libquadmath for intermediate results.
                    Slower than the standard version but more accurate.
        - mpfr:     Use the mpfr library for intermediate results.
                    Slowest version but gives the correct rounding.

[!TIP] The script build_fuzzy_libmath_dockerfile.sh helps you turn your Docker image into a fuzzy-libmath one.

usage: ./build_fuzzy_libmath_dockerfile.sh <DOCKER_IMAGE> <TAG> [FUZZY_IMAGE]
          <DOCKER_IMAGE>: Name of the base Docker image to build
          <TAG>:          Tag of the new image to build
          [FUZZY_IMAGE]:  Name of the fuzzy image to copy from (optional)
                          Requires a fuzzy version >= 0.9.1

To test your instrumentation:

Python

>>> import math
>>> from collections import Counter
>>> Counter( math.cos(42) for i in range(1000) )
Counter({-0.39998531498835127: 506, -0.3999853149883513: 249, -0.3999853149883512: 245})

Octave

>>> x = repmath(42, 1000);
>>> y = cos(x);
>>> printf("%.17f\n", unique(y))
-0.39998531498835133
-0.39998531498835127
-0.39998531498835121

Using Fuzzy in Multi-stage builds

Fuzzy provides a set of recompiled shared objects and tools that facilitate adding Monte Carlo Arithmetic to tools. If you've got a Docker container which relies on some of these libraries, you can easily add Fuzzy with a Multi-stage Docker build.

For example:

FROM verificarlo/fuzzy:latest as fuzzy

# Your target image
FROM user/image:version

# Copy libmath fuzzy environment from fuzzy image, for example
RUN mkdir -p /opt/mca-libmath/{fast,standard,quad,mpfr}
COPY --from=${2} /opt/mca-libmath/set-fuzzy-libmath.py /usr/local/bin/set-fuzzy-libmath
COPY --from=${2} /opt/mca-libmath/fast/libmath.so /opt/mca-libmath/fast/libmath.so
COPY --from=${2} /opt/mca-libmath/standard/libmath.so /opt/mca-libmath/standard/libmath.so
COPY --from=${2} /opt/mca-libmath/quad/libmath.so /opt/mca-libmath/quad/libmath.so
COPY --from=${2} /opt/mca-libmath/mpfr/libmath.so /opt/mca-libmath/mpfr/libmath.so
COPY --from=${2} /usr/local/lib/libinterflop* /usr/local/lib/

# If you will also want to recompile more libraries with verificarlo, add these lines
COPY --from=fuzzy /usr/l

Fuzzy

Install / Use

README