Varz

Painless optimisation of constrained variables in AutoGrad, TensorFlow, PyTorch, and JAX

Requirements and Installation
Manual
Examples

Requirements and Installation

pip install varz

Manual

Basics

from varz import Vars

To begin with, create a variable container of the right data type. For use with AutoGrad, use a np.* data type; for use with PyTorch, use a torch.* data type; for use with TensorFlow, use a tf.* data type; and for use with JAX, use a jnp.* data type. In this example we'll use AutoGrad.

>>> vs = Vars(np.float64)

Now a variable can be created by requesting it, giving it an initial value and a name.

>>> vs.unbounded(np.random.randn(2, 2), name="x")
array([[ 1.04404354, -1.98478763],
       [ 1.14176728, -3.2915562 ]])

If the same variable is created again, because a variable with the name x already exists, the existing variable will be returned, even if you again pass it an initial value.

>>> vs.unbounded(np.random.randn(2, 2), name="x")
array([[ 1.04404354, -1.98478763],
       [ 1.14176728, -3.2915562 ]])

>>> vs.unbounded(name="x")
array([[ 1.04404354, -1.98478763],
       [ 1.14176728, -3.2915562 ]])

Alternatively, indexing syntax may be used to get the existing variable x. This asserts that a variable with the name x already exists and will throw a KeyError otherwise.

>>> vs["x"]
array([[ 1.04404354, -1.98478763],
       [ 1.14176728, -3.2915562 ]])
       
>>> vs["y"]
KeyError: 'y'

The value of x can be changed by assigning it a different value.

>>> vs.assign("x", np.random.randn(2, 2))
array([[ 1.43477728,  0.51006941],
       [-0.74686452, -1.05285767]])

By default, assignment is non-differentiable and overwrites data. The variable can be deleted by passing its name to vs.delete:

>>> vs.delete("x")

>>> vs["x"]
KeyError: 'x'

When a variable is first created, you can set the keyword argument visible to False if you want to make the variable invisible to the variable-aggregating operations vs.get_latent_vars and vs.get_latent_vector. These variable-aggregating operations are used in optimisers to get the intended collection of variable to optimise. Therefore, setting visible to False will prevent a variable from being optimised.

Finally, a variable container can be copied with vs.copy(). Copies are lightweight and share their variables with the originals. As a consequence, however, assignment in a copy will also mutate the original. Differentiable assignment, however, will not.

Naming

Variables may be organised by naming them hierarchically using .s. For example, you could name like group1.bar, group1.foo, and group2.bar. This is helpful for extracting collections of variables, where wildcards may be used to match names. For example, *.bar would match group1.bar and group2.bar, and group1.* would match group1.bar and group1.foo. See also here.

The names of all variables can be obtained with Vars.names, and variables can be printed with Vars.print.

Example:

>>> vs = Vars(np.float64)

>>> vs.unbounded(1, name="x1")
array(1.)

>>> vs.unbounded(2, name="x2")
array(2.)

>>> vs.unbounded(3, name="y")
array(3.)

>>> vs.names
['x1', 'x2', 'y']

>>> vs.print()
x1:         1.0
x2:         2.0
y:          3.0

Constrained Variables

Unbounded variables: A variable that is unbounded can be created using Vars.unbounded or Vars.ubnd.
```
>>> vs.ubnd(name="normal_variable")
0.016925610008314832
```
Positive variables: A variable that is constrained to be positive can be created using Vars.positive or Vars.pos.
```
>>> vs.pos(name="positive_variable")
0.016925610008314832
```
Bounded variables: A variable that is constrained to be bounded can be created using Vars.bounded or Vars.bnd.
```
>>> vs.bnd(name="bounded_variable", lower=1, upper=2)
1.646772663807718
```
Lower-triangular matrix: A matrix variable that is constrained to be lower triangular can be created using Vars.lower_triangular or Vars.tril. Either an initialisation or a shape of square matrix must be given.
```
>>> vs.tril(shape=(2, 2), name="lower_triangular")
array([[ 2.64204459,  0.        ],
       [-0.14055559, -1.91298679]])
```
Positive-definite matrix: A matrix variable that is contrained to be positive definite can be created using Vars.positive_definite or Vars.pd. Either an initialisation or a shape of square matrix must be given.
```
>>> vs.pd(shape=(2, 2), name="positive_definite")
array([[ 1.64097496, -0.52302151],
       [-0.52302151,  0.32628302]])
```
Orthogonal matrix: A matrix variable that is constrained to be orthogonal can be created using Vars.orthogonal or Vars.orth. Either an initialisation or a shape of square matrix must be given.
```
>>> vs.orth(shape=(2, 2), name="orthogonal")
array([[ 0.31290403, -0.94978475],
       [ 0.94978475,  0.31290403]])
```

These constrained variables are created by transforming some latent unconstrained representation to the desired constrained space. The latent variables can be obtained using Vars.get_latent_vars.

>>> vs.get_latent_vars("positive_variable", "bounded_variable")
[array(-4.07892742), array(-0.604883)]

To illustrate the use of wildcards, the following is equivalent:

>>> vs.get_latent_vars("*_variable")
[array(-4.07892742), array(-0.604883)]

Variables can be excluded by prepending a dash:

>>> vs.get_latent_vars("*_variable", "-bounded_*")
[array(-4.07892742)]

Automatic Naming of Variables

To parametrise functions, a common pattern is the following:

def objective(vs):
    x = vs.unbounded(5, name="x")
    y = vs.unbounded(10, name="y")
    
    return (x * y - 5) ** 2 + x ** 2

The names for x and y are necessary, because otherwise new variables will be created and initialised every time objective is run. Varz offers two ways to not having to specify a name for every variable: sequential and parametrised specification.

Sequential Specification

Sequential specification can be used if, upon execution of objective, variables are always obtained in the same order. This means that variables can be identified with their position in this order and hence be named accordingly. To use sequential specification, decorate the function with sequential.

Example:

from varz import sequential

@sequential
def objective(vs):
    x = vs.unbounded(5)  # Initialise to 5.
    y = vs.unbounded()   # Initialise randomly.
    
    return (x * y - 5) ** 2 + x ** 2

>>> vs = Vars(np.float64)

>>> objective(vs)
68.65047879833773

>>> objective(vs)  # Running the objective again reuses the same variables.
68.65047879833773

>>> vs.names
['var0', 'var1']

>>> vs.print()
var0:       5.0      # This is `x`.
var1:       -0.3214  # This is `y`.

Parametrised Specification

Sequential specification still suffers from boilerplate code like x = vs.unbounded(5) and y = vs.unbounded(). This is the problem that parametrised specification addresses, which allows you to specify variables as arguments to your function. Import from varz.spec import parametrised. To indicate that an argument of the function is a variable, as opposed to a regular argument, the argument's type hint must be set accordingly, as follows:

Unbounded variables:

@parametrised
def f(vs, x: Unbounded):
    ...

Positive variables:
```
@parametris
```

Varz

Install / Use

README

Varz

Requirements and Installation

Manual

Basics

Naming

Constrained Variables

Automatic Naming of Variables

Sequential Specification

Parametrised Specification