SkillAgentSearch skills...

trl

Reference for the TRL (Transformer Reinforcement Learning) library codebase. Use proactively before reading or editing any file under `trl/` so you have the intended contracts and invariants in mind, not just what the current code says.

Install / Use

npx skills add benchflow-ai/skillsbench --skill trl

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

83/100

Supported Platforms

Zed

Tags

Our assessment of trl

trl scores 83/100 on our quality scale, 678th of 947 AI & Machine Learning skills we index.

Its SKILL.md is 3.9 KB long, well organised into 9 sections with 2 code examples: a solid amount of guidance for an agent.

With 1,813 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
26/30
Structure
18/20
Description
15/15
Adoption
14/20
Freshness
11/15

Maintenance, license and trust

  • The repository was last updated about 4 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
  • Our last check on 2026-11-06 found the source still online.
  • It is released under the Apache-2.0 license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 98/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-10-02. Automated pattern scan on 2026-10-02. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

trl compared with similar skills

All 4 of these similar skills score higher than trl; compare them before choosing.

SkillScoreStarsUpdatedFormat
trl (this skill)by benchflow-ai831.8k4mo agoSKILL.md
claude-memby thedotmack100117.6ktodayCLAUDE.md
Understand-Anythingby Egonex-AI10092.8k1d agoCLAUDE.md
headroomby headroomlabs-ai10079.8ktodayCLAUDE.md
CowAgentby zhayujie10048.0ktodayCLAUDE.md

Frequently asked questions

How do I install trl?
Run npx skills add benchflow-ai/skillsbench --skill trl. The install tabs above show the steps for each supported agent.
Which AI agents does trl work with?
It is written for Zed, as a SKILL.md file. Other agents that read the same format can often use it too.
Is trl safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is Apache-2.0-licensed and scores 98/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is trl still maintained?
The repository was last updated about 4 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.

name: trl description: Reference for the TRL (Transformer Reinforcement Learning) library codebase. Use proactively before reading or editing any file under trl/ so you have the intended contracts and invariants in mind, not just what the current code says. Covers trainer hierarchy (SFT, DPO, GRPO, KTO), shared utility functions (selective_log_softmax, decode_and_strip_padding, padding helpers), configuration system, model wrappers, and how data flows through any TRL trainer.

TRL Library Reference

Package Structure

TRL is organized around a trainer hierarchy that extends Hugging Face transformers.Trainer.

trl/
├── trainer/
│   ├── grpo_trainer.py       # GRPOTrainer
│   ├── grpo_config.py        # GRPOConfig
│   ├── sft_trainer.py        # SFTTrainer (supervised fine-tuning)
│   ├── dpo_trainer.py        # DPOTrainer (direct preference optimization)
│   ├── kto_trainer.py        # KTOTrainer (Kahneman-Tversky optimization)
│   ├── online_dpo_trainer.py # OnlineDPOTrainer
│   ├── utils.py              # Shared utilities (log probs, decoding, padding)
│   └── ...
├── models/
│   └── modeling_value_head.py  # Value head for PPO-style trainers
├── data_utils.py
├── commands/                   # CLI entry points
└── ...

Trainer Hierarchy

All TRL trainers extend transformers.Trainer:

transformers.Trainer
├── SFTTrainer          # Supervised fine-tuning
├── DPOTrainer          # Direct preference optimization
├── GRPOTrainer         # Group relative policy optimization
├── KTOTrainer          # Kahneman-Tversky optimization
└── OnlineDPOTrainer    # Online DPO

Each trainer overrides compute_loss with its specific objective, and RL-based trainers (GRPO, OnlineDPO) additionally override training_step to add a generation phase before the optimization step.

Shared Utility Functions (trainer/utils.py)

These utilities are used across multiple trainers. Read the source before modifying; the contracts below are what callers rely on.

selective_log_softmax(logits, index)

Memory-efficient per-token log-probability. Equivalent in value to F.log_softmax(logits, dim=-1).gather(...) at the selected token positions, but avoids materializing the full vocab-sized tensor.

Contract:

  • Input: logits [B, T, V], index [B, T]
  • Output: log_probs [B, T], each entry a valid log-probability (i.e. non-positive)
  • Must agree with F.log_softmax to within numerical tolerance on the same inputs

decode_and_strip_padding(input_ids, tokenizer)

Converts a batch of token ID tensors into the cleaned text strings that the reward function will score.

Contract:

  • Input: input_ids [B, T], tokenizer
  • Output: list[str] of length B
  • Strips padding and decoder artefacts
  • Handles any reasoning-block conventions the library supports; the exact policy for complete, incomplete, and absent reasoning markers is defined in the implementation

Other Utilities

  • pad / pad_to_length — Pad tensors to equal or specific lengths
  • Various tokenizer helpers for batch processing

Configuration System

All TRL configs extend transformers.TrainingArguments. Each trainer adds its own fields:

| Config | Trainer | Key fields | |--------|---------|------------| | SFTConfig | SFTTrainer | max_seq_length, packing, dataset_text_field | | DPOConfig | DPOTrainer | beta, loss_type, reference_free | | GRPOConfig | GRPOTrainer | num_generations, beta, epsilon, reward_functions | | KTOConfig | KTOTrainer | beta, desirable_weight, undesirable_weight |

Available References

| File | Contents | When to load | |------|----------|-------------| | references/trl-codebase.md | Module-by-module guide to TRL source: detailed breakdown of each trainer, model wrappers, data utilities, and CLI commands | When navigating unfamiliar parts of TRL beyond the trainer layer, or when you need details about a specific non-GRPO trainer |

Related Skills

View on GitHub
GitHub Stars1.8k
CategoryAI
Updated3mo ago
Forks368

Languages

PDDL

Trust signals

98/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

1 info