Cerebras Model Zoo

<picture> <source srcset="images/dark-image-nb.png" media="(prefers-color-scheme: dark)"> <source srcset="images/light-image.png" media="(prefers-color-scheme: light)"> <img src="images/light-image.png" alt="Cerebras banner"> </picture>

Introduction

The Cerebras Model Zoo is a collection of deep learning models and utilities optimized to run on Cerebras hardware. The repository provides reference implementations, configuration files, and utilities that demonstrate best practices for training and deploying models using Cerebras systems.

Key Features and Components

• CLI: The ModelZoo CLI is a comprehensive command-line interface that serves as a single entry point for all ModelZoo-related tasks. It streamlines workflows such as data preprocessing, model training, and validation.
• Models: Includes configuration files and reference implementations for a wide range of NLP, vision, and multimodal models, including Llama, Mixtral, DINOv2, and Llava. These are optimized for Cerebras hardware and follow best practices for performance and scalability.
• Data Preprocessing Tools: Scripts and utilities for preparing datasets for training, including tokenization, formatting, and batching for supported models.
• Checkpoint Converters and Porting Tools: Tools for converting between checkpoint formats (e.g., Cerebras ↔ HuggingFace) and porting PyTorch models to run on Cerebras systems.
• Advanced Features: Support for training optimizations such as custom training loops, custom model implementations, µParam (μP) scaling, rotary position embedding (RoPE) scaling for extended sequence lengths, and more.

Ready to Get Started?

• Reach out to us here to get access to Cerebras Hardware and ModelZoo!
• Install Cerebras ModelZoo by following the steps in our setup guide.
• Once you have ModelZoo installed, get started by pretraining or finetuning your first model!
• Visit our developer documentation for comprehensive guides on everything you can do with Cerebras ModelZoo.

Models in this repository

| Model | Code pointer | Model | Code pointer | |:--------------------------------|:-----------------------------------------------------------------------------:|:-----------------------------------------|:-----------------------------------------------------------------------------:| | BERT | Code | BLOOM | Code | | BTLM | Code | DiT | Code | | DINOv2 | Code | DPO | Code | | DPR | Code | ESM-2 | Code | | Falcon | Code | Gemma 2 | Code | | GPT-2 | Code | GPT-3 | Code | | GPT-J & GPT-Neox | Code | Jais | Code | | LLaMA | Code | LLaVA | Code | | Mistral | Code | Mixtral of Experts | Code | | Multimodal Simple | Code | SantaCoder | Code | | StarCoder | Code | Transformer | Code | | T5 | Code | | |

License

Apache License 2.0