---

name: huggingface-vision-trainer description: Trains and fine-tunes vision models for object detection (D-FINE, RT-DETR v2, DETR, YOLOS), image classification (timm models — MobileNetV3, MobileViT, ResNet, ViT/DINOv3 — plus any Transformers classifier), and SAM/SAM2 segmentation using Hugging Face Transformers on Hugging Face Jobs cloud GPUs. Covers COCO-format dataset preparation, Albumentations augmentation, mAP/mAR evaluation, accuracy metrics, SAM segmentation with bbox/point prompts, DiceCE loss, hardware selection, cost estimation, Trackio monitoring, and Hub persistence. Use when users mention training object detection, image classification, SAM, SAM2, segmentation, image matting, DETR, D-FINE, RT-DETR, ViT, timm, MobileNet, ResNet, bounding box models, or fine-tuning vision models on Hugging Face Jobs.

Vision Model Training on Hugging Face Jobs

Train object detection, image classification, and SAM/SAM2 segmentation models on managed cloud GPUs. No local GPU setup required—results are automatically saved to the Hugging Face Hub.

When to Use This Skill

Use this skill when users want to:

• Fine-tune object detection models (D-FINE, RT-DETR v2, DETR, YOLOS) on cloud GPUs or local
• Fine-tune image classification models (timm: MobileNetV3, MobileViT, ResNet, ViT/DINOv3, or any Transformers classifier) on cloud GPUs or local
• Fine-tune SAM or SAM2 models for segmentation / image matting using bbox or point prompts
• Train bounding-box detectors on custom datasets
• Train image classifiers on custom datasets
• Train segmentation models on custom mask datasets with prompts
• Run vision training jobs on Hugging Face Jobs infrastructure
• Ensure trained vision models are permanently saved to the Hub

Related Skills

• hugging-face-jobs — General HF Jobs infrastructure: token authentication, hardware flavors, timeout management, cost estimation, secrets, environment variables, scheduled jobs, and result persistence. Refer to the Jobs skill for any non-training-specific Jobs questions (e.g., "how do secrets work?", "what hardware is available?", "how do I pass tokens?").
• hugging-face-model-trainer — TRL-based language model training (SFT, DPO, GRPO). Use that skill for text/language model fine-tuning.

Local Script Execution

Helper scripts use PEP 723 inline dependencies. Run them with uv run:

uv run scripts/dataset_inspector.py --dataset username/dataset-name --split train
uv run scripts/estimate_cost.py --help

Prerequisites Checklist

Before starting any training job, verify:

Account & Authentication

• Hugging Face Account with Pro, Team, or Enterprise plan (Jobs require paid plan)
• Authenticated login: Check with hf_whoami() (tool) or hf auth whoami (terminal)
• Token has write permissions
• MUST pass token in job secrets — see directive #3 below for syntax (MCP tool vs Python API)

Dataset Requirements — Object Detection

• Dataset must exist on Hub
• Annotations must use the objects column with bbox, category (and optionally area) sub-fields
• Bboxes can be in xywh (COCO) or xyxy (Pascal VOC) format — auto-detected and converted
• Categories can be integers or strings — strings are auto-remapped to integer IDs
• image_id column is optional — generated automatically if missing
• ALWAYS validate unknown datasets before GPU training (see Dataset Validation section)

Dataset Requirements — Image Classification

• Dataset must exist on Hub
• Must have an image column (PIL images) and a label column (integer class IDs or strings)
• The label column can be ClassLabel type (with names) or plain integers/strings — strings are auto-remapped
• Common column names auto-detected: label, labels, class, fine_label
• ALWAYS validate unknown datasets before GPU training (see Dataset Validation section)

Dataset Requirements — SAM/SAM2 Segmentation

• Dataset must exist on Hub
• Must have an image column (PIL images) and a mask column (binary ground-truth segmentation mask)
• Must have a prompt — either:
  • A prompt column with JSON containing {"bbox": [x0,y0,x1,y1]} or {"point": [x,y]}
  • OR a dedicated bbox column with [x0,y0,x1,y1] values
  • OR a dedicated point column with [x,y] or [[x,y],...] values
• Bboxes should be in xyxy format (absolute pixel coordinates)
• Example dataset: merve/MicroMat-mini (image matting with bbox prompts)
• ALWAYS validate unknown datasets before GPU training (see Dataset Validation section)

Critical Settings

• Timeout must exceed expected training time — Default 30min is TOO SHORT. See directive #6 for recommended values.
• Hub push must be enabled — push_to_hub=True, hub_model_id="username/model-name", token in secrets

Dataset Validation

Validate dataset format BEFORE launching GPU training to prevent the #1 cause of training failures: format mismatches.

ALWAYS validate for unknown/custom datasets or any dataset you haven't trained with before. Skip for cppe-5 (the default in the training script).

Running the Inspector

Option 1: Via HF Jobs (recommended — avoids local SSL/dependency issues):

hf_jobs("uv", {
    "script": "path/to/dataset_inspector.py",
    "script_args": ["--dataset", "username/dataset-name", "--split", "train"]
})

Option 2: Locally:

uv run scripts/dataset_inspector.py --dataset username/dataset-name --split train

Option 3: Via HfApi().run_uv_job() (if hf_jobs MCP unavailable):

from huggingface_hub import HfApi
api = HfApi()
api.run_uv_job(
    script="scripts/dataset_inspector.py",
    script_args=["--dataset", "username/dataset-name", "--split", "train"],
    flavor="cpu-basic",
    timeout=300,
)

Reading Results

• ✓ READY — Dataset is compatible, use directly
• ✗ NEEDS FORMATTING — Needs preprocessing (mapping code provided in output)

Automatic Bbox Preprocessing

The object detection training script (scripts/object_detection_training.py) automatically handles bbox format detection (xyxy→xywh conversion), bbox sanitization, image_id generation, string category→integer remapping, and dataset truncation. No manual preprocessing needed — just ensure the dataset has objects.bbox and objects.category columns.

Training workflow

Copy this checklist and track progress:

Training Progress:
- [ ] Step 1: Verify prerequisites (account, token, dataset)
- [ ] Step 2: Validate dataset format (run dataset_inspector.py)
- [ ] Step 3: Ask user about dataset size and validation split
- [ ] Step 4: Prepare training script (OD: scripts/object_detection_training.py, IC: scripts/image_classification_training.py, SAM: scripts/sam_segmentation_training.py)
- [ ] Step 5: Save script locally, submit job, and report details

Step 1: Verify prerequisites

Follow the Prerequisites Checklist above.

Step 2: Validate dataset

Run the dataset inspector BEFORE spending GPU time. See "Dataset Validation" section above.

Step 3: Ask user preferences

ALWAYS use the AskUserQuestion tool with option-style format:

AskUserQuestion({
    "questions": [
        {
            "question": "Do you want to run a quick test with a subset of the data first?",
            "header": "Dataset Size",
            "options": [
                {"label": "Quick test run (10% of data)", "description": "Faster, cheaper (~30-60 min, ~$2-5) to validate setup"},
                {"label": "Full dataset (Recommended)", "description": "Complete training for best model quality"}
            ],
            "multiSelect": false
        },
        {
            "question": "Do you want to create a validation split from the training data?",
            "header": "Split data",
            "options": [
                {"label": "Yes (Recommended)", "description": "Automatically split 15% of training data for validation"},
                {"label": "No", "description": "Use existing validation split from dataset"}
            ],
            "multiSelect": false
        },
        {
            "question": "Which GPU hardware do you want to use?",
            "header": "Hardware Flavor",
            "options": [
                {"label": "t4-small ($0.40/hr)", "description": "1x T4, 16 GB VRAM — sufficient for all OD models under 100M params"},
                {"label": "l4x1 ($0.80/hr)", "description": "1x L4, 24 GB VRAM — more headroom for large images or batch sizes"},
                {"label": "a10g-large ($1.50/hr)", "description": "1x A10G, 24 GB VRAM — faster training, more CPU/RAM"},
                {"label": "a100-large ($2.50/hr)", "description": "1x A100, 80 GB VRAM — fastest, for very large datasets or image sizes"}
            ],
            "multiSelect": false
        }
    ]
})

Step 4: Prepare training script

For object detection, use scripts/object_detection_training.py as the production-ready template. For image classification, use scripts/image_classification_training.py. For SAM/SAM2 segmentation, use scripts/sam_segmentation_training.py. All scripts use HfArgumentParser — all configuration is passed via CLI arguments in script_args, NOT by editing Python variables. For timm model details, see references/timm_trainer.md. For SAM2 training details, see references/finetune_sam2_trainer.md.

Step 5: Save script, submit job, and report

1. Save the script locally to submitted_jobs/ in the workspace root (create if needed) with a descriptive name like training_<dataset>_<YYYYMMDD_HHMMSS>.py. Tell the user the path.
2. Submit using hf_jobs MCP tool (preferred) or HfApi().run_uv_job() — see directive #1 for both methods. Pass all config via script_args.
3. Report the job ID (from .id attribute), monitoring URL, Trackio dashboard (https://huggingface.co/spaces/{username}/trackio), expected time, and estimated cost.
4. Wait for user to request status checks — don't poll automatically. Training jobs run asynchronously and can take hours.

Critical directives

These rules prevent common failures. Follow them exactly.

1. Job submission: hf_jobs MCP tool vs Python API

hf_jobs() is an MCP tool, NOT a Python function. Do NOT try to import it from huggingface_hub. Call it as a tool:

hf_jobs("uv", {"script": training_script_content, "flavor": "a10g-large", "timeout": "4h", "secrets": {"HF_TOKEN": "$HF_TOKEN"}})

If hf_jobs MCP tool is unavailable, use the Python API directly:

from huggingface_hub import HfApi, get_token
api = HfApi()
job_info = api.run_uv_job(
    script="path/to/training_script.py",  # file PATH, NOT content
    script_args=["--dataset_name", "cppe-5", ...],
    flavor="a10g-large",
    timeout=14400,  # seconds (4 hours)
    env={"PYTHONUNBUFFERED": "1"},
    secrets={"HF_TOKEN": get_token()},  # MUST use get_token(), NOT "$HF_TOKEN"
)
print(f"Job ID: {job_info.id}")

Critical differences between the two methods:

| | hf_jobs MCP tool | HfApi().run_uv_job() | |---|---|---| | script param | Python code string or URL (NOT local paths) | File path to .py file (NOT content) | | Token in secrets | "$HF_TOKEN" (auto-replaced) | get_token() (actual token value) | | Timeout format | String ("4h") | Seconds (14400) |

Rules for both methods:

• The training script MUST include PEP 723 inline metadata with dependencies
• Do NOT use image or command parameters (those belong to run_job(), not run_uv_job())

2. Authentication via job secrets + explicit hub_token injection

Job config MUST include the token in secrets — syntax depen