torch::deploy (MultiPy)

[!CAUTION] MultiPy has been unmaintained for some time and is going to be archived soon. We recommend users to look at the new Free Threaded CPython version available starting with CPthon 3.13 as a long term solution to enable efficient multi-threading inference in CPython. For users looking to serve LLMs on server, we recommend higher level solutions such as vLLM as a good alternative.

torch::deploy (MultiPy for non-PyTorch use cases) is a C++ library that enables you to run eager mode PyTorch models in production without any modifications to your model to support tracing. torch::deploy provides a way to run using multiple independent Python interpreters in a single process without a shared global interpreter lock (GIL). For more information on how torch::deploy works internally, please see the related arXiv paper.

To learn how to use torch::deploy see Installation and Examples.

Requirements:

• PyTorch 1.13+ or PyTorch nightly
• Linux (ELF based)
  • x86_64 (Beta)
  • arm64/aarch64 (Prototype)

ℹ️ torch::deploy is ready for use in production environments, but is in Beta and may have some rough edges that we're continuously working on improving. We're always interested in hearing feedback and usecases that you might have. Feel free to reach out!

Installation

Building via Docker

The easiest way to build deploy and install the interpreter dependencies is to do so via docker.

git clone --recurse-submodules https://github.com/pytorch/multipy.git
cd multipy
export DOCKER_BUILDKIT=1
docker build -t multipy .

The built artifacts are located in multipy/runtime/build.

To run the tests:

docker run --rm multipy multipy/runtime/build/test_deploy

Installing via pip install

We support installing both python modules and the runtime libs using pip install, with the caveat of having to manually install the C++ dependencies first. This serves as a single-command source build, essentially being a wrapper around python setup.py develop, once all the dependencies have been installed.

To start with, the multipy repo should be cloned first:

git clone --recurse-submodules https://github.com/pytorch/multipy.git
cd multipy

# (optional) if using existing checkout
git submodule sync && git submodule update --init --recursive

Installing System Dependencies

The runtime system dependencies are specified in build-requirements-{debian,centos8}.txt. To install them on Debian-based systems, one could run:

sudo apt update
xargs sudo apt install -y -qq --no-install-recommends <build-requirements-debian.txt

While on a Centos system:

xargs sudo dnf install -y <build-requirements-centos8.txt

Python Environment Setup

We support both conda and pyenv+virtualenv to create isolated environments to build and run in. Since multipy requires a position-independent version of python to launch interpreters with, for conda environments we use the prebuilt libpython-static=3.x libraries from conda-forge to link with at build time, and for virtualenv/pyenv we compile python with -fPIC to create the linkable library.

NOTE We support Python versions 3.7 through 3.10 for multipy; note that for conda environments the libpython-static libraries are available for 3.8 onwards. With virtualenv/pyenv any version from 3.7 through 3.10 can be used, as the PIC library is built explicitly.

<details> <summary>Click to expand</summary>

Example commands for installing conda:

curl -fsSL -v -o ~/miniconda.sh -O  https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
chmod +x ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda && \
rm ~/miniconda.sh

Virtualenv / pyenv can be installed as follows:

pip3 install virtualenv
git clone https://github.com/pyenv/pyenv.git ~/.pyenv

</details>

Installing python, pytorch and related dependencies

Multipy requires a version of pytorch > 1.13 to run models successfully, and we recommend fetching the latest stable release (1.13) / nightlies and also cuda, if required.

In a conda environment, we would do the following or similar depending on which version of pytorch we want:

conda create -n newenv
conda activate newenv
conda install python=3.8
conda install -c conda-forge libpython-static=3.8

# cuda
conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia

# cpu only
conda install pytorch torchvision torchaudio cpuonly -c pytorch

For a pyenv / virtualenv setup, one could do:

export CFLAGS="-fPIC -g"
~/.pyenv/bin/pyenv install --force 3.8.6
virtualenv -p ~/.pyenv/versions/3.8.6/bin/python3 ~/venvs/multipy
source ~/venvs/multipy/bin/activate
pip install -r dev-requirements.txt

# cuda
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116

# cpu only
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu

Running pip install

Once all the dependencies are successfully installed, most importantly including a PIC-library of python and the latest nightly of pytorch, we can run the following, in either conda or virtualenv, to install both the python modules and the runtime/interpreter libraries:

# from base multipy directory
pip install -e .

The C++ binaries should be available in /opt/dist.

Alternatively, one can install only the python modules without invoking cmake as follows:

INSTALL_PYTHON_ONLY=1 pip install  -e .

NOTE As of 10/11/2022 the linking of prebuilt static fPIC versions of python downloaded from conda-forge can be problematic on certain systems (for example Centos 8), with linker errors like libpython_multipy.a: error adding symbols: File format not recognized. This seems to be an issue with binutils, and the steps in https://wiki.gentoo.org/wiki/Project:Toolchain/Binutils_2.32_upgrade_notes/elfutils_0.175:unable_to_initialize_decompress_status_for_section.debug_info can help. Alternatively, the user can go with the virtualenv/pyenv flow above.

Development

Manually building multipy::runtime from source

Both docker and pip install options above are wrappers around the cmake build of multipy's runtime. For development purposes it's often helpful to invoke cmake separately.

See the install section for how to correctly setup the Python environment.

# checkout repo
git clone --recurse-submodules https://github.com/pytorch/multipy.git
cd multipy

# (optional) if using existing checkout
git submodule sync && git submodule update --init --recursive

# install python parts of `torch::deploy` in multipy/multipy/utils
INSTALL_PYTHON_ONLY=1 pip install -e .

cd multipy/runtime

# configure runtime to build/
cmake -S . -B build
# if you need to override the ABI setting you can pass
cmake -S . -B build -D_GLIBCXX_USE_CXX11_ABI=<0/1>

# compile the files in build/
cmake --build build --config Release -j

Running unit tests for multipy::runtime

We first need to generate the neccessary examples. First make sure your python environment has torch. Afterwards, once multipy::runtime is built, run the following (executed automatically for docker and pip above):

python multipy/runtime/example/generate_examples.py
./multipy/runtime/build/test_deploy

Examples

See the examples directory for complete examples.

Packaging a model for multipy::runtime

multipy::runtime can load and run Python models that are packaged with torch.package. You can learn more about torch.package in the torch.package documentation.

For now, let's create a simple model that we can load and run in multipy::runtime.

from torch.package import PackageExporter
import torchvision

# Instantiate some model
model = torchvision.models.resnet.resnet18()

# Package and export it.
with PackageExporter("my_package.pt") as e:
    e.intern("torchvision.**")
    e.extern("numpy.**")
    e.extern("sys")
    e.extern("PIL.*")
    e.extern("typing_extensions")
    e.save_pickle("model", "model.pkl", model)

Note that since "numpy", "sys", "PIL" were marked as "extern", torch.package will look for these dependencies on the system that loads this package. They will not be packaged with the model.

Now, there should be a file named my_package.pt in your working directory.

<br>

Load the model in C++

#include <multipy/runtime/deploy.h>
#include <multipy/runtime/path_environment.h>
#include <torch/script.h>
#include <torch/torch.h>

#include <iostream>
#include <memory>

int main(int argc, const char* argv[]) {
    if (argc != 2) {
        std::cerr << "usage: example-app <path-to-exported-script-module>\n";
        return -1;
    }

    // Start an interpreter manager governing 4 embedded interpreters.
    std::shared_ptr<multipy::runtime::Environment> env =
        std::make_shared<multipy::runtime::PathEnvironment>(
            std::getenv("PATH_TO_EXTERN_PYTHON_PACKAGES") // Ensure to set this environment variable (e.g. /home/user/anaconda3/envs/multipy-example/lib/python3.8/site-packages)
        );
    multipy::runtime::InterpreterManager manager(4, env);

    try {
        // Load the model from the multipy.package.
        multipy::runtime::Package package = manager.loadPackage(argv[1]);
        multipy::runtime: