Mmpretrain

<div align="center"> <img src="resources/mmpt-logo.png" width="600"/> <div>&nbsp;</div> <div align="center"> <b><font size="5">OpenMMLab website</font></b> ^{<a href="https://openmmlab.com"> <i><font size="4">HOT</font></i> </a>} &nbsp;&nbsp;&nbsp;&nbsp; <b><font size="5">OpenMMLab platform</font></b> ^{<a href="https://platform.openmmlab.com"> <i><font size="4">TRY IT OUT</font></i> </a>} </div> <div>&nbsp;</div>

📘 Documentation | 🛠️ Installation | 👀 Model Zoo | 🆕 Update News | 🤔 Reporting Issues

<img src="https://user-images.githubusercontent.com/36138628/230307505-4727ad0a-7d71-4069-939d-b499c7e272b7.png" width="400"/>

English | 简体中文

</div> </div> <div align="center"> <a href="https://openmmlab.medium.com/" style="text-decoration:none;"> <img src="https://user-images.githubusercontent.com/25839884/219255827-67c1a27f-f8c5-46a9-811d-5e57448c61d1.png" width="3%" alt="" /></a> <img src="https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png" width="3%" alt="" /> <a href="https://discord.gg/raweFPmdzG" style="text-decoration:none;"> <img src="https://user-images.githubusercontent.com/25839884/218347213-c080267f-cbb6-443e-8532-8e1ed9a58ea9.png" width="3%" alt="" /></a> <img src="https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png" width="3%" alt="" /> <a href="https://twitter.com/OpenMMLab" style="text-decoration:none;"> <img src="https://user-images.githubusercontent.com/25839884/218346637-d30c8a0f-3eba-4699-8131-512fb06d46db.png" width="3%" alt="" /></a> <img src="https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png" width="3%" alt="" /> <a href="https://www.youtube.com/openmmlab" style="text-decoration:none;"> <img src="https://user-images.githubusercontent.com/25839884/218346691-ceb2116a-465a-40af-8424-9f30d2348ca9.png" width="3%" alt="" /></a> <img src="https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png" width="3%" alt="" /> <a href="https://space.bilibili.com/1293512903" style="text-decoration:none;"> <img src="https://user-images.githubusercontent.com/25839884/219026751-d7d14cce-a7c9-4e82-9942-8375fca65b99.png" width="3%" alt="" /></a> <img src="https://user-images.githubusercontent.com/25839884/218346358-56cc8e2f-a2b8-487f-9088-32480cceabcf.png" width="3%" alt="" /> <a href="https://www.zhihu.com/people/openmmlab" style="text-decoration:none;"> <img src="https://user-images.githubusercontent.com/25839884/219026120-ba71e48b-6e94-4bd4-b4e9-b7d175b5e362.png" width="3%" alt="" /></a> </div>

Introduction

MMPreTrain is an open source pre-training toolbox based on PyTorch. It is a part of the OpenMMLab project.

The main branch works with PyTorch 1.8+.

Major features

• Various backbones and pretrained models
• Rich training strategies (supervised learning, self-supervised learning, multi-modality learning etc.)
• Bag of training tricks
• Large-scale training configs
• High efficiency and extensibility
• Powerful toolkits for model analysis and experiments
• Various out-of-box inference tasks.
  • Image Classification
  • Image Caption
  • Visual Question Answering
  • Visual Grounding
  • Retrieval (Image-To-Image, Text-To-Image, Image-To-Text)

https://github.com/open-mmlab/mmpretrain/assets/26739999/e4dcd3a2-f895-4d1b-a351-fbc74a04e904

What's new

🌟 v1.2.0 was released in 04/01/2023

• Support LLaVA 1.5.
• Implement of RAM with a gradio interface.

🌟 v1.1.0 was released in 12/10/2023

• Support Mini-GPT4 training and provide a Chinese model (based on Baichuan-7B)
• Support zero-shot classification based on CLIP.

🌟 v1.0.0 was released in 04/07/2023

• Support inference of more multi-modal algorithms, such as LLaVA, MiniGPT-4, Otter, etc.
• Support around 10 multi-modal datasets!
• Add iTPN, SparK self-supervised learning algorithms.
• Provide examples of New Config and DeepSpeed/FSDP with FlexibleRunner. Here are the documentation links of New Config and DeepSpeed/FSDP with FlexibleRunner.

🌟 Upgrade from MMClassification to MMPreTrain

• Integrated Self-supervised learning algorithms from MMSelfSup, such as MAE, BEiT, etc.
• Support RIFormer, a simple but effective vision backbone by removing token mixer.
• Refactor dataset pipeline visualization.
• Support LeViT, XCiT, ViG, ConvNeXt-V2, EVA, RevViT, EfficientnetV2, CLIP, TinyViT and MixMIM backbones.

This release introduced a brand new and flexible training & test engine, but it's still in progress. Welcome to try according to the documentation.

And there are some BC-breaking changes. Please check the migration tutorial.

Please refer to changelog for more details and other release history.

Installation

Below are quick steps for installation:

conda create -n open-mmlab python=3.8 pytorch==1.10.1 torchvision==0.11.2 cudatoolkit=11.3 -c pytorch -y
conda activate open-mmlab
pip install openmim
git clone https://github.com/open-mmlab/mmpretrain.git
cd mmpretrain
mim install -e .

Please refer to installation documentation for more detailed installation and dataset preparation.

For multi-modality models support, please install the extra dependencies by:

mim install -e ".[multimodal]"

User Guides

We provided a series of tutorials about the basic usage of MMPreTrain for new users:

• Learn about Configs
• Prepare Dataset
• Inference with existing models
• Train
• Test
• Downstream tasks

For more information, please refer to our documentation.

Model zoo

Results and models are available in the model zoo.

<div align="center"> <b>Overview</b> </div> <table align="center"> <tbody> <tr align="center" valign="bottom"> <td> <b>Supported Backbones</b> </td> <td> <b>Self-supervised Learning</b> </td> <td> <b>Multi-Modality Algorithms</b> </td> <td> <b>Others</b> </td> </tr> <tr valign="top"> <td> <ul> <li><a href="configs/vgg">VGG</a></li> <li><a href="configs/resnet">ResNet</a></li> <li><a href="configs/resnext">ResNeXt</a></li> <li><a href="configs/seresnet">SE-ResNet</a></li> <li><a href="configs/seresnet">SE-ResNeXt</a></li> <li><a href="configs/regnet">RegNet</a></li> <li><a href="configs/shufflenet_v1">ShuffleNet V1</a></li> <li><a href="configs/shufflenet_v2">ShuffleNet V2</a></li> <li><a href="configs/mobilenet_v2">MobileNet V2</a></li> <li><a href="configs/mobilenet_v3">MobileNet V3</a></li> <li><a href="configs/swin_transformer">Swin-Transformer</a></li> <li><a href="configs/swin_transformer_v2">Swin-Transformer V2</a></li> <li><a href="configs/repvgg">RepVGG</a></li> <li><a href="configs/vision_transformer">Vision-Transformer</a></li> <li><a href="configs/tnt">Transformer-in-Transformer</a></li> <li><a href="configs/res2net">Res2Net</a></li> <li><a href="configs/mlp_mixer">MLP-Mixer</a></li> <li><a href="configs/deit">DeiT</a></li> <li><a href="configs/deit3">DeiT-3</a></li> <li><a href="configs/conformer">Conformer</a></li> <li><a href="configs/t2t_vit">T2T-ViT</a></li> <li><a href="co