<div align="center"> <img src="assets/guanaco.svg" width="300"/> <div>&nbsp;</div> </div>

<div align="center">

👋🤗🤗👋 Join our WeChat.

</div>

Easy and Efficient Fine-tuning LLMs --- 简单高效的大语言模型训练/部署

<div align="center">

中文 | English

</div>

Introduction

LLamaTuner is an efficient, flexible and full-featured toolkit for fine-tuning LLM (Llama3, Phi3, Qwen, Mistral, ...)

Efficient

• Support LLM, VLM pre-training / fine-tuning on almost all GPUs. LLamaTuner is capable of fine-tuning 7B LLM on a single 8GB GPU, as well as multi-node fine-tuning of models exceeding 70B.
• Automatically dispatch high-performance operators such as FlashAttention and Triton kernels to increase training throughput.
• Compatible with DeepSpeed 🚀, easily utilizing a variety of ZeRO optimization techniques.

Flexible

• Support various LLMs (Llama 3, Mixtral, Llama 2, ChatGLM, Qwen, Baichuan, ...).
• Support VLM (LLaVA).
• Well-designed data pipeline, accommodating datasets in any format, including but not limited to open-source and custom formats.
• Support various training algorithms (QLoRA, LoRA, full-parameter fune-tune), allowing users to choose the most suitable solution for their requirements.

Full-featured

• Support continuous pre-training, instruction fine-tuning, and agent fine-tuning.
• Support chatting with large models with pre-defined templates.

Table of Contents

• Easy and Efficient Fine-tuning LLMs --- 简单高效的大语言模型训练/部署
  • Introduction
  • Table of Contents
  • Supported Models
  • Supported Training Approaches
  • Supported Datasets
    • Data Preprocessing
  • Model Zoo
  • Requirement
    • Hardware Requirement
  • Getting Started
    • Clone the code
  • Getting Started
  • License
  • Acknowledgements
  • Citation

Supported Models

| Model | Model size | Default module | Template | | ---------------------------------------------------- | -------------------------------- | --------------- | --------- | | Baichuan | 7B/13B | W_pack | baichuan | | Baichuan2 | 7B/13B | W_pack | baichuan2 | | BLOOM | 560M/1.1B/1.7B/3B/7.1B/176B | query_key_value | - | | BLOOMZ | 560M/1.1B/1.7B/3B/7.1B/176B | query_key_value | - | | ChatGLM3 | 6B | query_key_value | chatglm3 | | Command-R | 35B/104B | q_proj,v_proj | cohere | | DeepSeek (MoE) | 7B/16B/67B/236B | q_proj,v_proj | deepseek | | Falcon | 7B/11B/40B/180B | query_key_value | falcon | | Gemma/CodeGemma | 2B/7B | q_proj,v_proj | gemma | | InternLM2 | 7B/20B | wqkv | intern2 | | LLaMA | 7B/13B/33B/65B | q_proj,v_proj | - | | LLaMA-2 | 7B/13B/70B | q_proj,v_proj | llama2 | | LLaMA-3 | 8B/70B | q_proj,v_proj | llama3 | | LLaVA-1.5 | 7B/13B | q_proj,v_proj | vicuna | | Mistral/Mixtral | 7B/8x7B/8x22B | q_proj,v_proj | mistral | | OLMo | 1B/7B | q_proj,v_proj | - | | PaliGemma | 3B | q_proj,v_proj | gemma | | Phi-1.5/2 | 1.3B/2.7B | q_proj,v_proj | - | | Phi-3 | 3.8B | qkv_proj | phi | | Qwen | 1.8B/7B/14B/72B | c_attn | qwen | | Qwen1.5 (Code/MoE) | 0.5B/1.8B/4B/7B/14B/32B/72B/110B | q_proj,v_proj | qwen | | StarCoder2 | 3B/7B/15B | q_proj,v_proj | - | | XVERSE | 7B/13B/65B | q_proj,v_proj | xverse | | Yi (1/1.5) | 6B/9B/34B | q_proj,v_proj | yi | | Yi-VL | 6B/34B | q_proj,v_proj | yi_vl | | Yuan | 2B/51B/102B | q_proj,v_proj | yuan |

Supported Training Approaches

| Approach | Full-tuning | Freeze-tuning | LoRA | QLoRA | | ---------------------- | ------------------ | ------------------ | ------------------ | ------------------ | | Pre-Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | Supervised Fine-Tuning | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | Reward Modeling | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | PPO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | DPO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | KTO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | ORPO Training | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |

Supported Datasets

As of now, we support the following datasets, most of which are all available in the Hugging Face datasets library.

<details><summary>Supervised fine-tuning dataset</summary>

• Stanford Alpaca
• Stanford Alpaca (Chinese)
• Hello-SimpleAI/HC3
• BELLE 2M (zh)
• BELLE 1M (zh)
• BELLE 0.5M (zh)
• BELLE Dialogue 0.4M (zh)
• BELLE School Math 0.25M (zh)
• BELLE Multiturn Chat 0.8M (zh)
• databricks-dolly-15k
• mosaicml/dolly_hhrlhf
• GPT-4 Generated Data
• Alpaca CoT
• UltraChat
• OpenAssistant/oasst1
• ShareGPT_Vicuna_unfiltered
• BIAI/OL-CC
• timdettmers/openassistant-guanaco
• Evol-Instruct
• OpenOrca
• Platypus
• OpenHermes

</details> <details><summary>Preference datasets</summary>

• DPO mixed (en&zh)
• Orca DPO Pairs (en)
• [HH-RLHF (en)](https://huggingface.co/datasets/A