Orion
Orion-14B is a family of models includes a 14B foundation LLM, and a series of models: a chat model, a long context model, a quantized model, a RAG fine-tuned model, and an Agent fine-tuned model. Orion-14B 系列模型包括一个具有140亿参数的多语言基座大模型以及一系列相关的衍生模型，包括对话模型，长文本模型，量化模型，RAG微调模型，Agent微调模型等。
Install / Use
/learn @OrionStarAI/OrionREADME
Table of Contents
- 📖 Model Introduction
- 🔗 Model Download
- 🔖 Model Benchmark
- 📊 Model Inference <img src="./assets/imgs/vllm.png" alt="vllm" style="margin: 0;display: initial;" height="20" /> <img src="./assets/imgs/llama_cpp.png" alt="llamacpp" style="margin: 0;display: initial;" height="20" />
- 📜 Declarations & License
- 🥇 Company Introduction
<a name="model-introduction"></a><br>
1. Model Introduction
-
Orion-14B series models are open-source multilingual large language models trained from scratch by OrionStarAI. The base model is trained on 2.5T multilingual corpus, including Chinese, English, Japanese, Korean, etc, and it exhibits superior performance in these languages. For details, please refer to tech report.
-
The Orion-14B series models exhibit the following features:
- Among models with 20B-parameter scale level, Orion-14B-Base model shows outstanding performance in comprehensive evaluations.
- Strong multilingual capabilities, significantly outperforming in Japanese and Korean testsets.
- The fine-tuned models demonstrate strong adaptability, excelling in human-annotated blind tests.
- The long-chat version supports extremely long texts, performing exceptionally well at a token length of 200k and can support up to a maximum of 320k.
- The quantized versions reduce model size by 70%, improve inference speed by 30%, with performance loss less than 1%.
- Orion-14B series models including:
- Orion-14B-Base: A multilingual large language foundational model with 14 billion parameters, pretrained on a diverse dataset of 2.5 trillion tokens.
- Orion-14B-Chat: A chat-model fine-tuned on a high-quality corpus aims to provide an excellence interactive experience for users in the large model community.
- Orion-14B-LongChat: The long-context version excels at handling extremely lengthy texts, performing exceptionally well at a token length of 200k and can support up to a maximum of 320k.
- Orion-14B-Chat-RAG: A chat-model fine-tuned on a custom retrieval augmented generation dataset, achieving superior performance in retrieval augmented generation tasks.
- Orion-14B-Chat-Plugin: A chat-model specifically tailored for plugin and function calling tasks, ideal for agent-related scenarios where the LLM acts as a plugin and function call system.
- Orion-14B-Base-Int4: A quantized base model utilizing 4-bit integer weights. It significantly reduces the model size by 70% and increases the inference speed by 30% while incurring a minimal performance loss of only 1%.
- Orion-14B-Chat-Int4: A quantized chat model utilizing 4-bit integer weights.
<a name="model-download"></a><br>
2. Model Download
Model release and download links are provided in the table below:
| Model Name | HuggingFace Download Links | ModelScope Download Links | OpenXLab Download Links | |-------------------------|-----------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------| | ⚾Orion-14B-Base | Orion-14B-Base | Orion-14B-Base | Orion-14B-Base | | 😛Orion-14B-Chat | Orion-14B-Chat | Orion-14B-Chat | Orion-14B-Chat | | 📃Orion-14B-LongChat | Orion-14B-LongChat | Orion-14B-LongChat | Orion-14B-LongChat | | 🔎Orion-14B-Chat-RAG | Orion-14B-Chat-RAG | Orion-14B-Chat-RAG | Orion-14B-Chat-RAG | | 🔌Orion-14B-Chat-Plugin | Orion-14B-Chat-Plugin | Orion-14B-Chat-Plugin | Orion-14B-Chat-Plugin | | 💼Orion-14B-Base-Int4 | Orion-14B-Base-Int4 | Orion-14B-Base-Int4 | Orion-14B-Base-Int4 | | 📦Orion-14B-Chat-Int4 | Orion-14B-Chat-Int4 | Orion-14B-Chat-Int4 | Orion-14B-Chat-Int4 |
<a name="model-benchmark"></a><br>
3. Model Benchmarks
3.1. Base Model Orion-14B-Base Benchmarks
3.1.1. LLM evaluation results on examination and professional knowledge
| Model | C-Eval | CMMLU | MMLU | AGIEval | Gaokao | BBH | |--------------------|----------|----------|----------|----------|----------|----------| | LLaMA2-13B | 41.4 | 38.4 | 55.0 | 30.9 | 18.2 | 45.6 | | Skywork-13B | 59.1 | 61.4 | 62.7 | 43.6 | 56.1 | 48.3 | | Baichuan2-13B | 59.0 | 61.3 | 59.5 | 37.4 | 45.6 | 49.0 | | QWEN-14B | 71.7 | 70.2 | 67.9 | 51.9 | 62.5 | 53.7 | | InternLM-20B | 58.8 | 59.0 | 62.1 | 44.6 | 45.5 | 52.5 | | Orion-14B-Base | 72.9 | 70.6 | 69.9 | 54.7 | 62.1 | 56.5 |
3.1.2. LLM evaluation results on language understanding and common knowledge
| Model |RACE-middle|RACE-high |HellaSwag | PIQA | Lambada | WSC | |--------------------|----------|----------|----------|----------|----------|----------| | LLaMA 2-13B | 63.0 | 58.9 | 77.5 | 79.8 | 76.5 | 66.3 | | Skywork-13B | 87.6 | 84.1 | 73.7 | 78.3 | 71.8 | 66.3 | | Baichuan 2-13B | 68.9 | 67.2 | 70.8 | 78.1 | 74.1 | 66.3 | | QWEN-14B | 93.0 | 90.3 | 80.2 | 79.8 | 71.4 | 66.3 | | InternLM-20B | 86.4 | 83.3 | 78.1 | 80.3 | 71.8 | 68.3 | | Orion-14B-Base | 93.2 | 91.3 | 78.5 | 79.5 | 78.8 | 70.2 |
3.1.3. LLM evaluation results of OpenCompass testsets
| Model | Average | Examination | Language | Knowledge | Understanding | Reasoning | |------------------|----------|----------|----------|----------|----------|----------| | LLaMA 2-13B | 47.3 | 45.2 | 47.0 | 58.3 | 50.9 | 43.6 | | Skywork-13B | 53.6 | 61.1 | 51.3 | 52.7 | 64.5 | 45.2 | | Baichuan 2-13B | 49.4 | 51.8 | 47.5 | 48.9 | 58.1 | 44.2 | | QWEN-14B | 62.4 | 71.3 | 52.67 | 56.1 | 68.8 | 60.1 | | InternLM-20B | 59.4 | 62.5 | 55.0 | 60.1 | 67.3 | 54.9 | |Orion-14B-Base| 64.3 | 71.4 | 55.0 | 60.0 | 71.9 | 61.6 |
3.1.4. Comparison of LLM performances on Japanese testsets
| Model |Average| JCQA | JNLI | MARC | JSQD | JQK | XLS | XWN | MGSM
