SkillAgentSearch skills...

nanogpt-training

Train GPT-2 scale models (~124M parameters) efficiently on a single GPU. Covers GPT-124M architecture, tokenized dataset loading (e.g., HuggingFace Hub shards), modern optimizers (Muon, AdamW), mixed precision training, and training loop implementation.

Install / Use

npx skills add benchflow-ai/skillsbench --skill nanogpt-training

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

83/100

Supported Platforms

Zed

Tags

Our assessment of nanogpt-training

nanogpt-training scores 83/100 on our quality scale, 677th of 947 AI & Machine Learning skills we index.

Its SKILL.md is 3.4 KB long, well organised into 9 sections with 3 code examples: a solid amount of guidance for an agent.

With 1,813 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
26/30
Structure
18/20
Description
15/15
Adoption
14/20
Freshness
11/15

Maintenance, license and trust

  • The repository was last updated about 4 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
  • Our last check on 2026-11-06 found the source still online.
  • It is released under the Apache-2.0 license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 98/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-10-02. Automated pattern scan on 2026-10-02. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

nanogpt-training compared with similar skills

All 4 of these similar skills score higher than nanogpt-training; compare them before choosing.

SkillScoreStarsUpdatedFormat
nanogpt-training (this skill)by benchflow-ai831.8k4mo agoSKILL.md
claude-memby thedotmack100117.6ktodayCLAUDE.md
Understand-Anythingby Egonex-AI10092.8k1d agoCLAUDE.md
headroomby headroomlabs-ai10079.8ktodayCLAUDE.md
CowAgentby zhayujie10048.0ktodayCLAUDE.md

Frequently asked questions

How do I install nanogpt-training?
Run npx skills add benchflow-ai/skillsbench --skill nanogpt-training. The install tabs above show the steps for each supported agent.
Which AI agents does nanogpt-training work with?
It is written for Zed, as a SKILL.md file. Other agents that read the same format can often use it too.
Is nanogpt-training safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is Apache-2.0-licensed and scores 98/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is nanogpt-training still maintained?
The repository was last updated about 4 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.

name: nanogpt-training description: Train GPT-2 scale models (~124M parameters) efficiently on a single GPU. Covers GPT-124M architecture, tokenized dataset loading (e.g., HuggingFace Hub shards), modern optimizers (Muon, AdamW), mixed precision training, and training loop implementation.

NanoGPT Training

Overview

Training GPT-2 scale models (~124M parameters) efficiently on a single GPU. It provides:

  • GPT-124M Architecture: Standard transformer with RoPE and modern optimizations
  • Tokenized Datasets: Loading pre-tokenized shards from HuggingFace Hub or local files
  • Modern Optimizers: Muon optimizer with Newton-Schulz orthogonalization
  • Mixed Precision: bfloat16 training on A100 for 2x speedup

Training options:

  • Baseline GPT: Standard residual connections
  • Experimental residual variants: Optional alternative residual schemes for stability/efficiency

Quick Reference

| Topic | Reference | |-------|-----------| | Model Architecture | GPT Architecture | | Data Loading | Tokenized Data | | Optimizers | Optimizers | | Training Loop | Training Loop | | Hyperparameters | Hyperparameters |

Installation

pip install torch einops numpy huggingface_hub

Minimal Example

import modal

app = modal.App("gpt-training")

image = modal.Image.debian_slim(python_version="3.11").pip_install(
    "torch", "einops", "numpy", "huggingface_hub"
)

@app.function(gpu="A100", image=image, timeout=3600)
def train():
    import torch
    from dataclasses import dataclass

    @dataclass
    class GPTConfig:
        block_size: int = 1024
        vocab_size: int = 50257
        n_layer: int = 12
        n_head: int = 12
        n_embd: int = 768
        dropout: float = 0.0
        bias: bool = False

    # Download data, build model, train
    # ... (see references for full implementation)

    return {"final_loss": final_loss}

@app.local_entrypoint()
def main():
    results = train.remote()
    print(results)

Common Imports

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.cuda.amp import autocast, GradScaler
from dataclasses import dataclass
from einops import rearrange, repeat, reduce
import numpy as np
import math

When to Use What

| Scenario | Approach | |----------|----------| | Standard GPT training | Use baseline model with standard residuals | | Stability experiments | Try alternative residual variants or extra streams | | Small experiments | Use T4/A10G GPU | | Full training | Use A100 with bfloat16 | | Custom data | Modify the dataset loader class | | Different model size | Adjust GPTConfig parameters |

Metrics to Monitor

| Metric | Typical Signal | Notes | |--------|----------------|-------| | Validation loss | Steady decrease | Absolute value depends on dataset/tokenizer | | Grad norm | Moderate, stable range | Large spikes indicate instability | | Training stability | Smooth curves | Frequent spikes suggest LR/batch issues | | Throughput | Consistent tokens/sec | Use for comparing configs |

External Resources

  • nanoGPT: https://github.com/karpathy/nanoGPT
  • build-nanogpt: https://github.com/karpathy/build-nanogpt
  • modded-nanogpt: https://github.com/KellerJordan/modded-nanogpt
  • FineWeb-Edu token shards: https://huggingface.co/datasets/karpathy/fineweb-edu-100B-gpt2-token-shards

Related Skills

View on GitHub
GitHub Stars1.8k
CategoryAI
Updated3mo ago
Forks368

Languages

PDDL

Trust signals

98/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

1 info