llama2-webui

Running Llama 2 with gradio web UI on GPU or CPU from anywhere (Linux/Windows/Mac).

• Supporting all Llama 2 models (7B, 13B, 70B, GPTQ, GGML, GGUF, CodeLlama) with 8-bit, 4-bit mode.
• Use llama2-wrapper as your local llama2 backend for Generative Agents/Apps; colab example.
• Run OpenAI Compatible API on Llama2 models.

Features

• Supporting models: Llama-2-7b/13b/70b, Llama-2-GPTQ, Llama-2-GGML, Llama-2-GGUF, CodeLlama ...
• Supporting model backends: transformers, bitsandbytes(8-bit inference), AutoGPTQ(4-bit inference), llama.cpp
• Demos: Run Llama2 on MacBook Air; Run Llama2 on free Colab T4 GPU
• Use llama2-wrapper as your local llama2 backend for Generative Agents/Apps; colab example.
• Run OpenAI Compatible API on Llama2 models.
• News, Benchmark, Issue Solutions

Contents

• Install
• Usage
  • Start Chat UI
  • Start Code Llama UI
  • Use llama2-wrapper for Your App
  • Start OpenAI Compatible API
• Benchmark
• Download Llama-2 Models
  • Model List
  • Download Script
• Tips
  • Env Examples
  • Run on Nvidia GPU
    • Run bitsandbytes 8 bit
    • Run GPTQ 4 bit
  • Run on CPU
    • Mac Metal Acceleration
    • AMD/Nvidia GPU Acceleration
• License
• Contributing

Install

Method 1: From PyPI

pip install llama2-wrapper

The newest llama2-wrapper>=0.1.14 supports llama.cpp's gguf models.

If you would like to use old ggml models, install llama2-wrapper<=0.1.13 or manually install llama-cpp-python==0.1.77.

Method 2: From Source:

git clone https://github.com/liltom-eth/llama2-webui.git
cd llama2-webui
pip install -r requirements.txt

Install Issues:

bitsandbytes >= 0.39 may not work on older NVIDIA GPUs. In that case, to use LOAD_IN_8BIT, you may have to downgrade like this:

• pip install bitsandbytes==0.38.1

bitsandbytes also need a special install for Windows:

pip uninstall bitsandbytes
pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.0-py3-none-win_amd64.whl

Usage

Start Chat UI

Run chatbot simply with web UI:

python app.py

app.py will load the default config .env which uses llama.cpp as the backend to run llama-2-7b-chat.ggmlv3.q4_0.bin model for inference. The model llama-2-7b-chat.ggmlv3.q4_0.bin will be automatically downloaded.

Running on backend llama.cpp.
Use default model path: ./models/llama-2-7b-chat.Q4_0.gguf
Start downloading model to: ./models/llama-2-7b-chat.Q4_0.gguf

You can also customize your MODEL_PATH, BACKEND_TYPE, and model configs in .env file to run different llama2 models on different backends (llama.cpp, transformers, gptq).

Start Code Llama UI

We provide a code completion / filling UI for Code Llama.

Base model Code Llama and extend model Code Llama — Python are not fine-tuned to follow instructions. They should be prompted so that the expected answer is the natural continuation of the prompt. That means these two models focus on code filling and code completion.

Here is an example run CodeLlama code completion on llama.cpp backend:

python code_completion.py --model_path ./models/codellama-7b.Q4_0.gguf

codellama-7b.Q4_0.gguf can be downloaded from TheBloke/CodeLlama-7B-GGUF.

Code Llama — Instruct trained with “natural language instruction” inputs paired with anticipated outputs. This strategic methodology enhances the model’s capacity to grasp human expectations in prompts. That means instruct models can be used in a chatbot-like app.

Example run CodeLlama chat on gptq backend:

python app.py --backend_type gptq --model_path ./models/CodeLlama-7B-Instruct-GPTQ/ --share True

CodeLlama-7B-Instruct-GPTQ can be downloaded from TheBloke/CodeLlama-7B-Instruct-GPTQ

Use llama2-wrapper for Your App

🔥 For developers, we released llama2-wrapper as a llama2 backend wrapper in PYPI.

Use llama2-wrapper as your local llama2 backend to answer questions and more, colab example:

# pip install llama2-wrapper
from llama2_wrapper import LLAMA2_WRAPPER, get_prompt 
llama2_wrapper = LLAMA2_WRAPPER()
# Default running on backend llama.cpp.
# Automatically downloading model to: ./models/llama-2-7b-chat.ggmlv3.q4_0.bin
prompt = "Do you know Pytorch"
answer = llama2_wrapper(get_prompt(prompt), temperature=0.9)

Run gptq llama2 model on Nvidia GPU, colab example:

from llama2_wrapper import LLAMA2_WRAPPER 
llama2_wrapper = LLAMA2_WRAPPER(backend_type="gptq")
# Automatically downloading model to: ./models/Llama-2-7b-Chat-GPTQ

Run llama2 7b with bitsandbytes 8 bit with a model_path:

from llama2_wrapper import LLAMA2_WRAPPER 
llama2_wrapper = LLAMA2_WRAPPER(
	model_path = "./models/Llama-2-7b-chat-hf",
  backend_type = "transformers",
  load_in_8bit = True
)

Check API Document for more usages.

Start OpenAI Compatible API

llama2-wrapper offers a web server that acts as a drop-in replacement for the OpenAI API. This allows you to use Llama2 models with any OpenAI compatible clients, libraries or services, etc.

Start Fast API:

python -m llama2_wrapper.server

it will use llama.cpp as the backend by default to run llama-2-7b-chat.ggmlv3.q4_0.bin model.

Start Fast API for gptq backend:

python -m llama2_wrapper.server --backend_type gptq

Navigate to http://localhost:8000/docs to see the OpenAPI documentation.

Basic settings

| Flag | Description | | ---------------- | ------------------------------------------------------------ | | -h, --help | Show this help message. | | --model_path | The path to the model to use for generating completions. | | --backend_type | Backend for llama2, options: llama.cpp, gptq, transformers | | --max_tokens | Maximum context size. | | --load_in_8bit | Whether to use bitsandbytes to run model in 8 bit mode (only for transformers models). | | --verbose | Whether to print verbose output to stderr. | | --host | API address | | --port | API port |

Benchmark

Run benchmark script to compute performance on your device, benchmark.py will load the same .env as app.py.:

python benchmark.py

You can also select the iter, backend_type and model_path the benchmark will be run (overwrite .env args) :

python benchmark.py --iter NB_OF_ITERATIONS --backend_type gptq

By default, the number of iterations is 5, but if you want a faster result or a more accurate one you can set it to whatever value you want, but please only report results with at least 5 iterations.

This colab example also show you how to benchmark gptq model on free Google Colab T4 GPU.

Some benchmark performance:

| Model | Precision | Device | RAM / GPU VRAM | Speed (tokens/sec) | load time (s) | | --------------------------- | --------- | ------------------ | -------------- | ------------------ | ------------- | | Llama-2-7b-chat-hf | 8 bit | NVIDIA RTX 2080 Ti | 7.7 GB VRAM | 3.76 | 641.36 | | Llama-2-7b-Chat-GPTQ | 4 bit | NVIDIA RTX 2080 Ti | 5.8 GB VRAM | 18.85 | 192.91 | | Llama-2-7b-Chat-GPTQ | 4 bit | Google Colab T4 | 5.8 GB VRAM | 18.19 | 37.44 | | llama-2-7b-chat.ggmlv3.q4_0 | 4 bit | Apple M1 Pro CPU | 5.4 GB RAM | 17.90 | 0.18 | | llama-2-7b-c