Lrn2

Stefan Lattner, Maarten Grachten, Carlos Eduardo Cancino Chacon

LRN2 - Framework (including the nn_bricks package)

This is a python package that provides tools for learning representations from data, implemented in the context of the Lrn2Cre8.eu project. Its goal is to simplify the creation and training of Neural Network architectures using theano, as well as reducing the effort in the associated data management. It includes the nn_bricks package, providing classes which can be combined in a building block like manner making it easy to create custom Neural Network (NN) architectures. Pre-defined layers can be created, stacked and trained using a configuration file.

Among others, lrn2 includes [Convolutional] Restricted Boltzmann Machines (RBMs), [Convolutional] Neural Networks and Auto-Encoders, [Gated] RNNs, LSTMs, corresponding training routines and cost functions, as well as common regularizers. Overview of available bricks.

The structure of lrn2 is designed to facilitate the creation of an end-to-end work flow. This work flow involves the following four phases:

1. Reading and preparing data from files
2. Defining and creating models using a configuration file
3. Training models to learning representations from the data
4. Using the trained models in specific use case scenarios

Files are organized into several subdirectories (under lrn2):

• nn_bricks: This package provides brick-like classes (i.e. predefined NN, RNN and RBM layers, several unit types, cost functions, regularizers, training algorithms, plotting functions, serialization methods, ...) which can be freely combined into mix-in classes to define different NN layers or stacks.
• data: Data holding and data import. Interfaces for different domains and formats.
• util: Several utility modules specific for this framework
• application: Code which can be used for special applications (like classification or visualization)

Links

• Documentation
• Tutorials
  • First steps
  • Implementing custom viewpoints
  • The configuration file
• Project website

Requirements

For convolution, NVIDIA cuDNN is strongly recommended.

Installation

To install the lrn2 framework (Linux/Mac), run the following command in your terminal

(optional) cd /usr/local/lib/python2.7/dist-packages/
sudo pip install -e git+https://github.com/OFAI/lrn2.git#egg=lrn2

or manually

git clone https://github.com/OFAI/lrn2.git
cd lrn2
git submodule init
git submodule update
sudo python [setup.py | setup_mac.py] install

or download the source and use it in your IDE (feel free to fork). Running the above commands may still help to install the required dependencies.

Test the lrn2 framework:

• In the (extracted) tar archive, browse to an example folder examples/[example_name]. Run one of the examples by entering python run_demo.py [any_run_keyword] config_model.ini [options], or with the --help flag for help on the specific example. For the mnist_ examples, no further parameters are needed. The other two examples need the path to a .csv file with (monophonic) melodies (e.g. the Essen Folksong Collection) as additional parameter. You can generate such a .csv file by using python create_csv_filelist.py in lrn2/util.
• If some import errors are shown, ensure that all dependencies are included in the environment variable $PYTHONPATH.

Uninstall lrn2

Run the following command in your terminal: sudo pip uninstall lrn2

Example

The example below shows the main workflow of the lrn2 framework (omitting data preparation for now). lrn2.nn_bricks.make.make_net takes a configuration file and creates NN layers accordingly. The layers are then packed into a stack with cross-entropy cost (custom stacks and layers can be defined using some bricks of the nn_bricks collection). Binding data to variables (input and target are some default variables) is done using an ordered dictionary. lrn2.nn_bricks.train.train_cached is one of the two training methods of the framework, which trains a whole stack or a single layer. The other, lrn2.nn_bricks.train.train_layer_wise could be used for e.g. greedy, layer-wise pretraining of an RBM stack or for layer-wise pre-training of an Auto-Encoder stack.

# Create NN layers (according to a configuration)
layers = make_net(config, input_shape = (28, 28))

# Pack layers in stack with cross entropy cost
stack = FFNNCrossEntropy(layers, name = 'mnist_classifier')

# Define input data (mnist_data and mnist_labels are numpy arrays)
data = OrderedDict((('input', mnist_data), ('target', mnist_labels)))

# Train the whole stack with backpropagation in discriminative task
train_cached(stack, data, config, run_keyword = 'mnist_classifier', re_train = True)

For an extended introduction, see the tutorial first steps.

Config File

A config file (using the ConfigObj library) determines the nets creation and training parameters. In the case below, we create a three-layered Convolutional Neural Network with Rectified Linear Units and Batch Normalization, a Max-Pooling layer between the first and the second layer, and a dense layer with Softmax units on top (with 10 neurons for 10 MNIST classes). The category [STACK] defines some training parameters for the full stack. Training parameters defined in the layer categories are only related to layer-wise training (except regularizers like the weight regularization defined via wl2, which remain part of the resulting cost function).

[OUTPUT]
output_path = nn_output     # Resulting files will go nn_output/[run_keyword]

[INPUT]
ngram_size = 1              # 1 for images (e.g. MNIST)

[TOPOLOGY]
[[Layer1]]
type_net = NNConvReLUBN     # class name of the layer in lrn.nn_bricks.make
                            # (Convolutional NN with ReLU units and Batch Normalization)
convolutional = True        # convolutional flag
filter_shape = 5, 5         # shape of conv filter
n_hidden = 32               # hidden maps
wl2 = 1e-3                  # wl2 weight regularization

[[LayerPooling1]]
type_net = MaxPooling       # class name of the layer in lrn.nn_bricks.make
                            # (Max-pooling layer)
convolutional = True        # convolutional flag
downsample = 2, 2           # downsample by 1/2 in each dimension
no_param_layer = True       # layer has no params

[[Layer2]]
type_net = NNConvReLUBN     # class name of the layer in lrn.nn_bricks.make
                            # (Convolutional NN with ReLU units and Batch Normalization)
convolutional = True        # convolutional flag
filter_shape = 5, 5         # shape of conv filter
n_hidden = 32               # hidden maps
wl2 = 1e-3                  # wl2 weight regularization

[[LayerToDense]]
type_net = ConvToDense      # class name of the layer in lrn.nn_bricks.make
                            # (Convolution to dense transition layer)
convolutional = True        # convolutional flag
no_param_layer = True       # layer has no params

[[LayerTop]]
type_net = NNSoftmaxBN      # class name of the layer in lrn.nn_bricks.make
                            # Dense NN layer with softmax units and batch normalization
n_hidden = 10               # 10 hidden units
wl2 = 0.001                 # wl2 weight regularization

[STACK]
img_interval = 25           # Plot params, histograms, cost curves every 25 epochs
dump_interval = 50          # Backup model parameters every 50 epochs

learning_rate = 5e-3        # Learning rate
momentum = 0.85             # Momentum during training
batch_size = 100            # Mini-batch size
epochs = 151                # Number of epochs to train
reduce_lr = False           # Don`t reduce learning rate (to zero over epochs)

The file can be imported using the method lrn2.util.config.get_config(config_file_path, specification_path), which returns a dictionary reflecting the config file content and structure. The specification_path (default is lrn2/util/config_spec.ini) points to a specification file, which defines the parameter defaults, variable types, and the config files structure (see default config_spec.ini). You can always add a new parameter (e.g. to be used with a custom layer) and use it immediately in a brick, as long as it receives **kwargs (kwargs['parameter_name']), however, if its variable type is not defined in util/config_spec.ini, it has to be cast to the desired type. In order to avoid casting, copy the config_spec.ini in your project folder, edit it accordingly and pass it in specification_path of get_config.

Specific layer parameters are forwarded to the respective layer constructors as **kwargs and should be passed on to most mix-in class constructors (those who take parameters at all). Parameters defined in the config file but not used by any mix-in class (e.g. wl2 is defined in config file, but the corresponding layer does not derive from WeightRegular) are ignored.

For further information on the configuration file, see Tutorial 3, Tutorial 1, or see the related [Documentation](http://lrn2cre8.ofai.at/lrn2/doc/lrn2.util.ht