Clad

<div align="center"> <img width=200em alt="Clad" src="https://user-images.githubusercontent.com/6516307/193281076-3f5a4c7f-2f0d-4c05-b76b-b2d6719adf0a.svg"/>

Clad is a source-transformation automatic differentiation (AD) library for C++,<br/>implemented as a plugin for the Clang compiler.

Try Online | Binder | Usage | Installation | Further Reading | Documentation | Contributing

</div>

About Clad

Clad enables automatic differentiation (AD) for C++. It is based on LLVM compiler infrastructure and is a plugin for Clang compiler. Clad is based on source code transformation. Given C++ source code of a mathematical function, it can automatically generate C++ code for computing derivatives of the function. It supports both forward-mode and reverse-mode AD.Clad has extensive coverage of modern C++ features and a robust fallback and recovery system in place.

How to use Clad

Clad provides five API functions:

• clad::differentiate to use forward-mode AD.
• clad::gradient to use reverse-mode AD.
• clad::hessian to compute Hessian matrix using a combination of forward-mode and reverse-mode AD.
• clad::jacobian to compute Jacobian matrix using reverse-mode AD.
• clad::estimate-error to compute the floating-point error of the given program using reverse-mode AD.

API functions are used to label an existing function for differentiation. Both functions return a functor object containing the generated derivative which can be called via .execute method, which forwards provided arguments to the generated derivative function.

For a guide on compiling your clad-based programs, look here.

Forward mode - clad::differentiate

For a function f of several inputs and single (scalar) output, forward mode AD can be used to compute (or, in case of Clad, create a function) a directional derivative of f with respect to a single specified input variable. Derivative function created by the forward-mode AD is guaranteed to have at most a constant factor (around 2-3) more arithmetical operations compared to the original function.

clad::differentiate(f, ARGS) takes 2 arguments:

1. f is a pointer to a function or a method to be differentiated
2. ARGS is either:

• a single numerical literal indicating an index of independent variable (e.g. 0 for x, 1 for y)
• a string literal with the name of independent variable (as stated in the definition of f, e.g. "x" or "y"), and if the variable is an array the index needs to be specified, e.g. "arr[1]"

Generated derivative function has the same signature as the original function f, however its return value is the value of the derivative. Example:

#include "clad/Differentiator/Differentiator.h"
#include <iostream>

double f(double x, double y) { return x * y; }

int main() {
  // Call clad to generate the derivative of f wrt x.
  auto f_dx = clad::differentiate(f, "x");
  // Execute the generated derivative function.
  std::cout << f_dx.execute(/*x=*/3, /*y=*/4) << std::endl;
  // Dump the generated derivative code to standard output.
  f_dx.dump();
}

Reverse mode - clad::gradient

Reverse-mode AD allows computing the gradient of f using at most a constant factor (around 4) more arithmetical operations compared to the original function. While its constant factor and memory overhead is higher than that of the forward-mode, it is independent of the number of inputs. E.g. for a function having N inputs and consisting of T arithmetical operations, computing its gradient takes a single execution of the reverse-mode AD and around 4*T operations, while it would take N executions of the forward-mode, this requiring up to N*3*T operations.

clad::gradient(f, /*optional*/ ARGS) takes 1 or 2 arguments:

1. f is a pointer to a function or a method to be differentiated
2. ARGS is either:

• not provided, then f is differentiated w.r.t. its every argument
• a string literal with comma-separated names/indices of independent variables (e.g. "x", "y", "x, y", "y, x", "0, 1", "0, y", etc.)
• a SINGLE number representing the index of the independent variable

Since a vector of derivatives must be returned from a function generated by the reverse mode, its signature is slightly different. The generated function has void return type and same input arguments. The function has additional n arguments (where n refers to the number of arguments whose gradient was requested) of type T*, where T is the type of the corresponding original variable. Each of these variables stores the derivative of the elements as they appear in the orignal function signature. The caller is responsible for allocating and zeroing-out the gradient storage. Example:

auto f_grad = clad::gradient(f);
double dx = 0, dy = 0;
// After this call, dx and dy will store the derivatives of x and y respectively.
f_grad.execute(x, y, &dx, &dy);
std::cout << "dx: " << dx << ' ' << "dy: " << dy << std::endl;

// Same effect as before.
auto f_dx_dy = clad::gradient(f, "x, y"); 
auto f_dy_dx = clad::gradient(f, "y, x");

// The same effect can be achieved by using an array instead of individual variables.
double result2[2] = {};
f_dy_dx.execute(x, y, /*dx=*/&result2[0], /*dy=*/&result2[1]);
// note that the derivatives are mapped to the "result" indices in the same order as they were specified in the argument:
std::cout << "dy: " << result2[0] << ' ' << "dx: " << result2[1] << std::endl;

Hessian mode - clad::hessian

Clad can produce the hessian matrix of a function using its forward and reverse mode capabilities. Its interface is similar to reverse mode but differs when arrays are involved. It returns the matrix as a flattened vector in row major format.

clad::hessian(f, /*optional*/ ARGS) takes 1 or 2 arguments:

1. f is a pointer to a function or a method to be differentiated
2. ARGS is either:
   • not provided, then f is differentiated w.r.t. its every argument except in the case of arrays where it needs to be provided
   • a string literal with comma-separated names of independent variables (e.g. "x" or "y" or "x, y" or "y, x" or in case of arrays "x[0:2]")

The generated function has void return type and same input arguments. The function has an additional argument of type T*, where T is the return type of f. This variable stores the hessian matrix. The caller is responsible for allocating and zeroing-out the hessian storage. Example:

#include "clad/Differentiator/Differentiator.h"
#include <iostream>

double f(double x, double y) { return x * y; }
double g(double x, double y[2]) { return x * y[0] * y[1]; }

int main() {
    // Since we are differentiating variables that are not arrays the interface
    // is same as in reverse mode
    auto f_hess = clad::hessian(f);
    // The size of the resultant matrix should be the square of the 
    // number of independent variables
    double mat_f[4] = {0};
    
    // Execute the hessian function
    f_hess.execute(/*x=*/3, /*y=*/4, mat_f);
    std::cout << "[" << mat_f[0] << ", " << mat_f[1] << "\n  " 
                     << mat_f[2] << ", " << mat_f[3] << "]";
    
    // When arrays are involved the array indexes that are to be differentiated needs to be specified
    // even if the whole array is being differentiated
    auto g_hess = clad::hessian(g, "x, y[0:1]");
    // The rest of the steps are the same.
}

Jacobian mode - clad::jacobian

Clad can produce the jacobian of a function using its vectorized forward mode. It returns the jacobian matrix as a clad::matrix for every pointer/array parameter.

clad::jacobian(f, /*optional*/ ARGS) takes 1 or 2 arguments:

1. f is a pointer to a function or a method to be differentiated
2. ARGS is either:
   • not provided, then f is differentiated w.r.t. its every argument
   • a string literal with comma-separated names of independent variables (e.g. "x" or "y" or "x, y" or "y, x")

The generated function has void return type and same input arguments. For every pointer/array parameter arr, the function has an additional argument _d_vector_arr. Its type is clad::matrix<T>, where T is the pointee type of arr. These variables store their derivatives w.r.t. all inputs. Output parameters are supposed to have _clad_out_ prefix. The caller is responsible for allocating the matrices. Example:

#include "clad/Differentiator/Differentiator.h"
#include <iostream>

void h(double a,