<div align="center">

OpenModelZ

</div> <p align=center> <a href="https://discord.gg/KqswhpVgdU"><img alt="discord invitation link" src="https://dcbadge.vercel.app/api/server/KqswhpVgdU?style=flat"></a> <a href="https://twitter.com/TensorChord"><img src="https://img.shields.io/twitter/follow/tensorchord?style=social" alt="trackgit-views" /></a> <a href="https://docs.open.modelz.ai"><img src="https://img.shields.io/badge/docs.open.modelz.ai-455946.svg?style=socail&logo=googlechrome&logoColor=white" alt="docs" /></a> <a href="https://github.com/tensorchord/openmodelz#contributors-"><img alt="all-contributors" src="https://img.shields.io/github/all-contributors/tensorchord/openmodelz/main"></a> <a href="https://github.com/tensorchord/openmodelz/actions/workflows/CI.yaml"><img alt="CI" src="https://github.com/tensorchord/openmodelz/actions/workflows/CI.yaml/badge.svg"></a> <a href="https://badge.fury.io/py/openmodelz"><img src="https://badge.fury.io/py/openmodelz.svg" alt="PyPI version" height="20"></a> <a href='https://coveralls.io/github/tensorchord/openmodelz'><img src='https://coveralls.io/repos/github/tensorchord/openmodelz/badge.svg' alt='Coverage Status' /></a> </p>

What is OpenModelZ?

OpenModelZ ( mdz ) is tool to deploy your models to any cluster (GCP, AWS, Lambda labs, your home lab, or even a single machine).

Getting models into production is hard for data scientists and SREs. You need to configure the monitoring, logging, and scaling infrastructure, with the right security and permissions. And then setup the domain, SSL, and load balancer. This can take weeks or months of work even for a single model deployment.

You can now use mdz deploy to effortlessly deploy your models. OpenModelZ handles all the infrastructure setup for you. Each deployment gets a public subdomain, like http://jupyter-9pnxd.2.242.22.143.modelz.live, making it easily accessible.

<p align=center> <img src="https://user-images.githubusercontent.com/5100735/260630222-46e26e54-50c6-43ba-b3ea-2e64dd276f87.png" alt="OpenModelZ" width="1000"/> </p>

Benefits

OpenModelZ provides the following features out-of-the-box:

• 📈 Auto-scaling from 0: The number of inference servers could be scaled based on the workload. You could start from 0 and scale it up to 10+ replicas easily.
• 📦 Support any machine learning framework: You could deploy any machine learning framework (e.g. vLLM/triton-inference-server/mosec etc.) with a single command. Besides, you could also deploy your own custom inference server.
• 🔬 Gradio/Streamlit/Jupyter support: We provide a robust prototyping environment with support for Gradio, Streamlit, jupyter and so on. You could visualize your model's performance and debug it easily in the notebook, or deploy a web app for your model with a single command.
• 🏃 Start from a single machine to a cluster of machines: You could start from a single machine and scale it up to a cluster of machines without any hassle, with a single command mdz server start.
• 🚀 Public accessible subdomain for each deployment ( optional ) : We provision a separate subdomain for each deployment without any extra cost and effort, making each deployment easily accessible from the outside.

OpenModelZ is the foundational component of the ModelZ platform available at modelz.ai.

How it works

Get a server (could be a cloud VM, a home lab, or even a single machine) and run the mdz server start command. OpenModelZ will bootstrap the server for you.

$ mdz server start
🚧 Creating the server...
🚧 Initializing the load balancer...
🚧 Initializing the GPU resource...
🚧 Initializing the server...
🚧 Waiting for the server to be ready...
🐋 Checking if the server is running...
🐳 The server is running at http://146.235.213.84.modelz.live
🎉 You could set the environment variable to get started!

export MDZ_URL=http://146.235.213.84.modelz.live
$ export MDZ_URL=http://146.235.213.84.modelz.live

Then you could deploy your model with a single command mdz deploy and get the endpoint:

$ mdz deploy --image modelzai/gradio-stable-diffusion:23.03 --name sdw --port 7860 --gpu 1
Inference sd is created
$ mdz list
 NAME  ENDPOINT                                                 STATUS  INVOCATIONS  REPLICAS 
 sdw   http://sdw-qh2n0y28ybqc36oc.146.235.213.84.modelz.live   Ready           174  1/1      
       http://146.235.213.84.modelz.live/inference/sdw.default                                

Quick Start 🚀

Install mdz

You can install OpenModelZ using the following command:

pip install openmodelz

You could verify the installation by running the following command:

mdz

Once you've installed the mdz you can start deploying models and experimenting with them.

Bootstrap mdz

It's super easy to bootstrap the mdz server. You just need to find a server (could be a cloud VM, a home lab, or even a single machine) and run the mdz server start command.

Notice: We may require the root permission to bootstrap the mdz server on port 80.

$ mdz server start
🚧 Creating the server...
🚧 Initializing the load balancer...
🚧 Initializing the GPU resource...
🚧 Initializing the server...
🚧 Waiting for the server to be ready...
🐋 Checking if the server is running...
Agent:
 Version:       v0.0.13
 Build Date:    2023-07-19T09:12:55Z
 Git Commit:    84d0171640453e9272f78a63e621392e93ef6bbb
 Git State:     clean
 Go Version:    go1.19.10
 Compiler:      gc
 Platform:      linux/amd64
🐳 The server is running at http://192.168.71.93.modelz.live
🎉 You could set the environment variable to get started!

export MDZ_URL=http://192.168.71.93.modelz.live

The internal IP address will be used as the default endpoint of your deployments. You could provide the public IP address of your server to the mdz server start command to make it accessible from the outside world.

# Provide the public IP as an argument
$ mdz server start 1.2.3.4

You could also specify the registry mirror to speed up the image pulling process. Here is an example:

$ mdz server start --mirror-endpoints https://docker.mirrors.sjtug.sjtu.edu.cn

Create your first UI-based deployment

Once you've bootstrapped the mdz server, you can start deploying your first applications. We will use jupyter notebook as an example in this tutorial. You could use any docker image as your deployment.

$ mdz deploy --image jupyter/minimal-notebook:lab-4.0.3 --name jupyter --port 8888 --command "jupyter notebook --ip='*' --NotebookApp.token='' --NotebookApp.password=''"
Inference jupyter is created
$ mdz list
 NAME     ENDPOINT                                                   STATUS  INVOCATIONS  REPLICAS
 jupyter  http://jupyter-9pnxdkeb6jsfqkmq.192.168.71.93.modelz.live  Ready           488  1/1
          http://192.168.71.93/inference/jupyter.default                                                                         

You could access the deployment by visiting the endpoint URL. The endpoint will be automatically generated for each deployment with the following format: <name>-<random-string>.<ip>.modelz.live.

It is http://jupyter-9pnxdkeb6jsfqkmq.192.168.71.93.modelz.live in this case. The endpoint could be accessed from the outside world as well if you've provided the public IP address of your server to the mdz server start command.

Create your first OpenAI compatible API server

You could also create API-based deployments. We will use OpenAI compatible API server with Bloomz 560M as an example in this tutorial.

$ mdz deploy --image modelzai/llm-bloomz-560m:23.07.4 --name simple-server
Inference simple-server is created
$ mdz list
 NAME           ENDPOINT                                                         STATUS  INVOCATIONS  REPLICAS 
 jupyter        http://jupyter-9pnxdkeb6jsfqkmq.192.168.71.93.modelz.live        Ready           488  1/1      
                http://192.168.71.93/inference/jupyter.default                                                 
 simple-server  http://simple-server-lagn8m9m8648q6kx.192.168.71.93.modelz.live  Ready             0  1/1      
                http://192.168.71.93/inference/simple-server.default                                           

You could use OpenAI python package and the endpoint http://simple-server-lagn8m9m8648q6kx.192.168.71.93.modelz.live in this case, to interact with the deployment.

import openai
openai.api_base="http://simple-server-lagn8m9m8648q6kx.192.168.71.93.modelz.live"
openai.api_key="any"

# create a chat completion
chat_completion = openai.ChatCompletion.create(model="bloomz", messages=[
    {"role": "user", "content": "Who are you?"},
    {"role": "assistant", "content": "I am a student"},
    {"role": "user", "content": "What do you learn?"},
], max_tokens=100)

Scale your deployment

You could scale your deployment by using the mdz scale command.

$ mdz scale simple-server --replicas 3

The requests will be load balanced between the replicas of your deployment.

You could also tell the mdz to autoscale your deployment based on the inflight requests. Please check out the Autoscaling documentation for more details.

Debug your deployment

Sometimes you may want to debug your deployment. You could use the mdz logs command to get the logs of your deployment.

$ mdz logs simple-server
simple-server-6756dd67ff-4bf4g: 10.42.0.1 - - [27/Jul/2023 02:32:16] "GET / HTTP/1.1" 200 -
simple-