Infinity

Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali

Generate Convert Improve

Install / Use

/learn @michaelfeil/Infinity

About this skill

Quality Score

0/100

README

[![Contributors][contributors-shield]][contributors-url] [![Forks][forks-shield]][forks-url] [![Stargazers][stars-shield]][stars-url] [![Issues][issues-shield]][issues-url] [![MIT License][license-shield]][license-url]

Infinity ♾️

[![codecov][codecov-shield]][codecov-url] [![ci][ci-shield]][ci-url] [![Downloads][pepa-shield]][pepa-url] Docker pulls

Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models, clip, clap and colpali. Infinity is developed under MIT License.

Why Infinity

Deploy any model from HuggingFace: deploy any embedding, reranking, clip and sentence-transformer model from HuggingFace
Fast inference backends: The inference server is built on top of PyTorch, optimum (ONNX/TensorRT) and CTranslate2, using FlashAttention to get the most out of your NVIDIA CUDA, AMD ROCM, CPU, AWS INF2 or APPLE MPS accelerator. Infinity uses dynamic batching and tokenization dedicated in worker threads.
Multi-modal and multi-model: Mix-and-match multiple models. Infinity orchestrates them.
Tested implementation: Unit and end-to-end tested. Embeddings via infinity are correctly embedded. Lets API users create embeddings till infinity and beyond.
Easy to use: Built on FastAPI. Infinity CLI v2 allows launching of all arguments via Environment variable or argument. OpenAPI aligned to OpenAI's API specs. View the docs at https://michaelfeil.github.io/infinity on how to get started.

Latest News 🔥

[2025/07] Blackwell support
[2024/11] AMD, CPU, ONNX docker images
[2024/10] pip install infinity_client
[2024/07] Inference deployment example via Modal and a free GPU deployment
[2024/06] Support for multi-modal: clip, text-classification & launch all arguments from env variables
[2024/05] launch multiple models using the v2 cli, including --api-key
[2024/03] infinity supports experimental int8 (cpu/cuda) and fp8 (H100/MI300) support
[2024/03] Docs are online: https://michaelfeil.github.io/infinity/latest/
[2024/02] Community meetup at the Run:AI Infra Club
[2024/01] TensorRT / ONNX inference
[2023/10] Initial release

Getting started

Launch the cli via pip install

pip install infinity-emb[all]

After your pip install, with your venv active, you can run the CLI directly.

infinity_emb v2 --model-id BAAI/bge-small-en-v1.5

Check the v2 --help command to get a description for all parameters.

infinity_emb v2 --help

Launch the CLI using a pre-built docker container (recommended)

Instead of installing the CLI via pip, you may also use docker to run michaelf34/infinity. Make sure you mount your accelerator ( i.e. install nvidia-docker and activate with --gpus all).

port=7997
model1=michaelfeil/bge-small-en-v1.5
model2=mixedbread-ai/mxbai-rerank-xsmall-v1
volume=$PWD/data

docker run -it --gpus all \
 -v $volume:/app/.cache \
 -p $port:$port \
 michaelf34/infinity:latest \
 v2 \
 --model-id $model1 \
 --model-id $model2 \
 --port $port

The cache path inside the docker container is set by the environment variable HF_HOME.

Specialized docker images

<details> <summary>Docker container for CPU</summary> Use the `latest-cpu` image or `x.x.x-cpu` for slimer image. Run like any other cpu-only docker image. Optimum/Onnx is often the prefered engine.

docker run -it \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest-cpu \
v2 \
--engine optimum \
--model-id $model1 \
--model-id $model2 \
--port $port

</details> <details> <summary>Docker Container for ROCm (MI200 Series and MI300 Series)</summary> Use the `latest-rocm` image or `x.x.x-rocm` for rocm compatible inference. **This image is currently not build via CI/CD (to large), consider pinning to exact version.** Make sure you have ROCm is correctly installed and ready to use with Docker.

Visit Docs for more info.

</details> <details> <summary>Docker Container for Onnx-GPU, Cuda Extensions, TensorRT</summary> Use the `latest-trt-onnx` image or `x.x.x-trt-onnx` for nvidia compatible inference. **This image is currently not build via CI/CD (to large), consider pinning to exact version.**

This image has support for:

ONNX-Cuda "CudaExecutionProvider"
ONNX-TensorRT "TensorRTExecutionProvider" (may not always work due to version mismatch with ORT)
CudaExtensions and packages, e.g. Tri-Dao's pip install flash-attn package when using Pytorch.
nvcc compiler support

docker run -it \
-v $volume:/app/.cache \
-p $port:$port \
michaelf34/infinity:latest-trt-onnx \
v2 \
--engine optimum \
--device cuda \
--model-id $model1 \
--port $port

</details>

Using local models with Docker container

In order to deploy a local model with a docker container, you need to mount the model inside the container and specify the path in the container to the launch command.

Example:

git lfs install 
cd /tmp
mkdir models && cd models && git clone https://huggingface.co/BAAI/bge-small-en-v1.5
docker run -it   -v /tmp/models:/models  -p 8081:8081  michaelf34/infinity:latest v2  --model-id "/models/bge-small-en-v1.5" --port 8081

Advanced CLI usage

<details> <summary>Launching multiple models at once</summary>

Since infinity_emb>=0.0.34, you can use cli v2 method to launch multiple models at the same time. Checkout infinity_emb v2 --help for all args and validation.

Multiple Model CLI Playbook:

1. cli options can be repeated e.g. `v2 --model-id mode

Related Skills

node-connect

334.9k

Diagnose OpenClaw node connection and pairing failures for Android, iOS, and macOS companion apps

frontend-design

82.3k

Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, or applications. Generates creative, polished code that avoids generic AI aesthetics.

openai-whisper-api

334.9k

Transcribe audio via OpenAI Audio Transcriptions API (Whisper).

commit-push-pr

82.3k

Commit, push, and open a PR