DeepR: Build and Train Deep Learning Pipelines for Production

|pypi|_ |ci|_

.. |pypi| image:: https://img.shields.io/pypi/v/deepr.svg .. _pypi: https://pypi.python.org/pypi/deepr

.. |ci| image:: https://github.com/criteo/deepr/workflows/Continuous%20integration/badge.svg .. _ci: https://github.com/criteo/deepr/actions?query=workflow%3A%22Continuous+integration%22

DeepR is a library for Deep Learning on top of Tensorflow 1.x that focuses on production capabilities. It makes it easy to define pipelines (via the Job abstraction), preprocess data (via the Prepro abstraction), design models (via the Layer abstraction) and train them either locally or on a Yarn cluster. It also integrates nicely with MLFlow and Graphite, allowing for production ready logging capabilities.

It can be seen as a collection of generic tools and abstractions to be extended for more specific use cases. See the Use DeepR section for more information.

Submitting jobs and defining flexible pipelines is made possible thanks to a config system based off simple dictionaries and import strings. It is similar to Thinc config system <https://thinc.ai/docs>_ or gin config <https://github.com/google/gin-config>_ in a lot of ways.

To start with deepr read the blogpost <https://medium.com/criteo-labs/deepr-training-tensorflow-models-for-production-dda34a914c3b?source=friends_link&sk=91949017f33714dba3323956035f76e0>_ then go to quickstart on colab <https://colab.research.google.com/github/criteo/deepr/blob/master/docs/getting_started/quickstart.ipynb>_

Why a Deep Learning Library based on TF1.x

Tensorflow 1.x provides great production oriented capabilities, centered around the tf.Estimator API. It makes it possible to deploy models using a protobuf with no python code, and optimize computational graphs with XLA compilation.

Although DeepR comes with a Layer interface (most similar to google TRAX <https://github.com/google/trax>_ and very close to most modern frameworks) that makes it easy to define models using a functional programming approach, most of its capabilities are orthogonal to it. Most of the building blocks expect generic python types (for example, a Layer is merely a function fn(tensors, mode)).

Use DeepR

You can use DeepR as a simple python library, reusing only a subset of the concepts (the config system is generic for example) or build your own extension as a standalone python package that depends on deepr.

Have a look at the submodule examples <../deepr/examples>_ of deepr that illustrates what packages built on top of deepr would look like. It defines custom jobs, layers, preprocessors, macros as well as configs <../deepr/examples/multiply/configs>_. Once your custom components are packaged in a library, it is easy to run configs with

.. code-block::

deepr run config.json macros.json

MovieLens Example

You can try using DeepR on the MovieLens dataset, consisting of movie ratings aggregated by users. The submodule movielens <../deepr/examples/movielens>_ implements an AverageModel, a Transformer Model and a BPR loss as well as jobs to build and evaluate on this dataset.

You can jump to the notebook on Colab <https://colab.research.google.com/github/criteo/deepr/blob/master/docs/movielens/movielens.ipynb>_ or use the command line.

.. code-block::

pip install deepr[cpu] faiss_cpu
cd deepr/examples/movielens/configs
wget http://files.grouplens.org/datasets/movielens/ml-20m.zip
unzip ml-20m.zip
deepr run config.json macros.json

Installation

Prerequisites

Make sure you use ``python>=3.6`` and an up-to-date version of ``pip`` and ``setuptools``

.. code-block::

    python --version
    pip install -U pip setuptools

It is recommended to install ``deepr`` in a new virtual environment. For example

.. code-block::

    python -m venv deepr
    source deepr/bin/activate
    pip install -U pip setuptools
    pip install deepr[cpu]


Using Pip
~~~~~~~~~

If installing using pip and your own ``requirements.txt`` file, be aware that ``Tensorflow`` is listed in ``extras_require`` in the ``setup.py``, which means that ``pip install deepr`` WON'T INSTALL Tensorflow. This is because the Tensorflow requirement is different depending on the platform (GPU or CPU-only).

You can specify which extras to use using the ``[cpu]`` or ``[gpu]`` argument like in the following examples

.. code-block::

    pip install deepr[cpu]
    pip install deepr[gpu]
    pip install -e ".[cpu]"
    pip install -e ".[gpu]"

Or alternatively, pre-install Tensorflow separately like so

.. code-block::

    pip install tensorflow==1.15.2
    pip install deepr



From Source
~~~~~~~~~~~

First, clone the ``deepr`` repo on your local machine with

.. code-block::

    git clone https://github.com/criteo/deepr.git
    cd deepr

To install from source in editable mode, run

.. code-block::

    make install-cpu

Or to install on a GPU enabled machine

.. code-block::

    make install-gpu

To install development tools and test requirements, run


.. code-block::

    make install-dev

Test
----

To run unit tests in your current environment, run

.. code-block::

    make test

To run integration tests in your current environment, run

.. code-block::

    make integration

To run lint + unit and integration tests in a fresh virtual environment, run

.. code-block::

    make venv-lint-test-integration


Lint
----

To run ``mypy``, ``pylint`` and ``black --check``:

.. code-block::

    make lint

To auto-format the code using ``black``

.. code-block::

    make black


Command Line Tools
------------------

To get a list of available commands, run

.. code-block::

    deepr --help

Contributing
------------

See `CONTRIBUTING <CONTRIBUTING.rst>`_


Change log
----------

See `CHANGELOG <CHANGELOG.rst>`_

Main contributors
-----------------

Main contributors and maintainers for deepr are
`Guillaume Genthial <https://github.com/guillaumegenthial>`_,
`Romain Beaumont <https://github.com/rom1504>`_,
`Denis Kuzin <https://github.com/denkuzin>`_,
`Amine Benhalloum <https://github.com/bamine>`_