<p align="center" width="50%"> <img src="docs/assets/logo.png" alt="LMFlow" style="width: 50%; min-width: 200px; display: block; margin: auto; background-color: transparent;"> </p>

LMFlow

<h4 align="center"> <p> <b>English</b> | <a href="https://github.com/OptimalScale/LMFlow/blob/main/docs/readme/README_zh-hans.md">简体中文</a> | <a href="https://github.com/OptimalScale/LMFlow/blob/main/docs/readme/README_es.md">Español</a> | <a href="https://github.com/OptimalScale/LMFlow/blob/main/docs/readme/README_jp.md">日本語</a> | <a href="https://github.com/OptimalScale/LMFlow/blob/main/docs/readme/README_ko.md">한국어</a> | <a href="https://github.com/OptimalScale/LMFlow/blob/main/docs/readme/README_hindi.md">हिंदी</a> <p> </h4>

An extensible, convenient, and efficient toolbox for finetuning large machine learning models, designed to be user-friendly, speedy and reliable, and accessible to the entire community.

<p align="center" width="100%"> <img src="docs/assets/features.png" alt="LMFlow-features" style="width: 100%; min-width: 300px; display: block; margin: auto;"> </p>

Latest News

[!IMPORTANT]

• :exclamation: [2025-07-09] We have a major update to LMFlow with full Accelerate support and extensive streamlining. If you're looking for the previous version, please use git checkout v0.0.10, or check out the v0.0.10 branch. View all releases here.

• [2024-12-02] Support Hymba, a new family of small language models featuring a hybrid-head parallel architecture. Check out Post-training Hymba for more details.
• [2024-07-01] 🏆 LMFlow receives the Best Demo Paper Award at NAACL 2024! 🎉
• [2024-06-30] Expanding Optimization Options! We now support custom optimizer training with a variety of optimizers. Dive into the details and try out the new features with our updated script at custom_optimizers.
• [2024-04-25] :rocket: Support conversation template! We've preset the latest Llama-3 and Phi-3 conversation templates as well as some frequently used templates such as chatml (see all templates here), and we are working on adding more preset templates. Adding corresponding --conversation_template in the shell script and you are all set! :rocket:

<details> <summary>More news...</summary>

• [2024-03-27] Support LISA, enabling 7B training in 24G memory without offloading!

• [2023-09-11] Support speculative decoding. Check out speculative_decoding for the usage and acceleration details.

• [2023-08-14] Support long context inference with position interpolation (Linear & NTK scaling ) for LLaMA models. Check out postion_interpolation for more details.

• [2023-08-07] Support Flash Attention-2. Check out flash_attention for more details.

• [2023-08-02] Support Llama2, ChatGLM2, and Baichuan models.

• [2023-07-23] LMFlow multimodal chatbot is now available! Support multimodal inputs of images and texts. Online Demo is also provided (We hold the service on a single GPU, hence one may experience "queuing" or "application busy" sometimes when multiple users are accessing at the same time, please wait and attempt again later when such event happens)

• [2023-06-22] LMFlow paper is out! Check out our implementation details at https://arxiv.org/abs/2306.12420

• [2023-06-16] Our finetuned Robin-33B-V2 scored an impressive 64.1 on the Huggingface LLM leaderboard in our offline evaluation, outperforming major open-source LLMs! All checkpoints (7B, 13B, 33B, and 65B) are released! Checkout the performance here.

• [2023-06-07] LMFlow is now officially available on PyPI! Install it with pip install lmflow-finetune!

• [2023-05-30] Release Robin-13B-v2 and Robin-33B-v2!

• [2023-05-15] Release LMFlow-data, the training dataset of Robin-7B-v2. A new test data is also released.

• [2023-05-09] Release Robin-7B-v2, achieving competitive performance on chitchat, commonsense reasoning and instruction-following tasks. Refer to our comprehensive study.

• [2023-05-08] Release LMFlow Benchmark, an automatic evaluation framework for open-source chat-style LLMs. Benchmark results on 31 popular models are reported. Participate in LMFlow Benchmark.

• [2023-04-21] Release Robin-7B (based on LLaMA-7B), and two models for commercial use: Parakeets-2.7B (based on GPT-NEO-2.7B) and Cokatoo-7B (based on StableLM-7B) Download here

• [2023-04-15] Inference: Support streaming output and ChatGLM.

• [2023-04-10] We propose a new alignment algorithm: Reward rAnked FineTuning (RAFT), which is more efficient than conventional (PPO-based) RLHF. [Paper]

• [2023-04-02] Web service is online!

• [2023-04-01] Release three instruction-tuned checkpoints and three medical checkpoints in model zoo: LLaMA-7B-tuned, LLaMA-13B-tuned, LLaMA-33B-tuned, LLaMA-7B-medical, LLaMA-13B-medical, and LLaMA-33B-medical.

• [2023-03-27] Support full tuning and lora tuning for all decoder models.

• [2023-03-27] Tasked tuned model beats ChatGPT on medical domain.

• [2023-03-27] Release code and checkpoints - version 0.0.1! Our tasked-tuned model beats ChatGPT on medical domain.

</details>

Table of Contents

• LMFlow
  • Latest News
  • Table of Contents
  • Quick Start
    • Setup
    • Prepare Dataset
    • Finetuning
      • Estimated Hardware Requirement
      • Full Finetuning
      • LISA
      • LoRA
    • Inference
    • Deployment
    • Evaluation
  • Supported Features
  • Support
  • License
  • Citation

Quick Start

Setup

Our package has been tested on Linux OS (Ubuntu 20.04). Other OS platforms (MacOS, Windows) are not fully tested, where you may encounter unexpected errors. If you are using LMFlow for the first time, we recommend you to try on a Linux machine or Google Colab.

git clone -b v1.0.0 https://github.com/OptimalScale/LMFlow.git
cd LMFlow
conda create -n lmflow python=3.9 -y
conda activate lmflow
conda install mpi4py
pip install -e .

<details><summary> Looking for a previous version? </summary>

git clone -b v0.0.10 https://github.com/OptimalScale/LMFlow.git
cd LMFlow
conda create -n lmflow python=3.9 -y
conda activate lmflow
conda install mpi4py
pip install -e .

</details> <details><summary> For CUDA versions 10.3-11.7 </s