Smol Vision 🐣

Recipes for shrinking, optimizing, customizing cutting edge vision and multimodal AI models.

Latest examples 👇🏻

• Fine-tune Kosmos2.5 on OCR with bounding boxes
• Fine-tune Florence-2 on document question answering
• Fine-tune DINOv3 on image classification

Note: GitHub refuses to render notebooks for a long time now, so the notebooks of smol-vision with rich outputs now lives here. I still update this repository but it's inconvenient to read here.

| | Notebook | Description | |------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------| | Quantization/ONNX | Faster and Smaller Zero-shot Object Detection with Optimum | Quantize the state-of-the-art zero-shot object detection model OWLv2 using Optimum ONNXRuntime tools. | | VLM Fine-tuning | Fine-tune PaliGemma | Fine-tune state-of-the-art vision language backbone PaliGemma using transformers. | | Intro to Optimum/ORT | Optimizing DETR with 🤗 Optimum | A soft introduction to exporting vision models to ONNX and quantizing them. | | Model Shrinking | Knowledge Distillation for Computer Vision | Knowledge distillation for image classification. | | Quantization | Fit in vision models using Quanto | Fit in vision models to smaller hardware using quanto | | Speed-up | Faster foundation models with torch.compile | Improving latency for foundation models using torch.compile | | [NEW] VLM Fine-tuning | Fine-tune Florence-2 | Fine-tune Florence-2 on DocVQA dataset | | VLM Fine-tuning | QLoRA/Fine-tune IDEFICS3 or SmolVLM on VQAv2 | QLoRA/Full Fine-tune IDEFICS3 or SmolVLM on VQAv2 dataset | | VLM Fine-tuning (Script) | QLoRA Fine-tune IDEFICS3 on VQAv2 | QLoRA/Full Fine-tune IDEFICS3 or SmolVLM on VQAv2 dataset | | [NEW] VLM Fine-tuning | Grounded Fine-tuning | Grounded fine-tuning for vision-language models | | [NEW] Vision Model Fine-tuning | Fine-tune DINOv3 | Fine-tune DINOv3 for vision tasks | | Multimodal RAG | Multimodal RAG using ColPali and Qwen2-VL | Learn to retrieve documents and pipeline to RAG without hefty document processing using ColPali through Byaldi and do the generation with Qwen2-VL | | Multimodal Retriever Fine-tuning | Fine-tune ColPali for Multimodal RAG | Learn to apply contrastive fine-tuning on ColPali to customize it for your own multimodal document RAG use case | | Any-to-Any Fine-tuning | Fine-tune Gemma-3n for all modalities (audio-text-image) | Fine-tune Gemma-3n model to handle any modality: audio, text, and image. | | Any-to-Any RAG | Any-to-Any (Video) RAG with OmniEmbed and Qwen | Do retrieval and generation across modalities (including video) using OmniEmbed and Qwen. | | Speed-up/Memory Optimization | Vision language model serving using TGI (SOON) | Explore speed-ups and memory improvements for vision-language model serving with text-generation inference | | Quantization/Optimum/ORT | All levels of quantization and graph optimizations for Image Segmentation using Optimum (SOON) | End-to-end model optimization using Optimum |