robot-reinforcement-learning
Reinforcement learning for robots: when RL beats imitation or classical control, MDP design (observations, action spaces, control rate, terminated vs truncated), reward design and reward hacking, algorithm choice, massively parallel sim training, safe real-robot RL, and debugging training curves
Install / Use
npx skills add arpitg1304/robotics-agent-skills --skill robot-reinforcement-learningInstalls into whichever agent you are using.
SKILL.md
Installable skill definition
Quality Score
Category
Education & ResearchSupported Platforms
Our assessment of robot-reinforcement-learning
robot-reinforcement-learning scores 89/100 on our quality scale, 166th of 416 Education & Research skills we index (top 40%).
Its SKILL.md is 21 KB long, well organised into 26 sections with 2 code examples: a thorough specification that gives an agent plenty to work with.
It has 405 GitHub stars, a meaningful sign that others use it.
Maintenance, license and trust
- The repository was last updated 3 days ago, so robot-reinforcement-learning is actively maintained.
- It is released under the Apache-2.0 license, a permissive license that allows use, modification and commercial use with attribution.
- Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.
Safety scan
No issues foundOur scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands.
Automated pattern scan on 2026-11-22. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.
robot-reinforcement-learning compared with similar skills
All 4 of these similar skills score higher than robot-reinforcement-learning; compare them before choosing.
| Skill | Score | Stars | Updated | Format |
|---|---|---|---|---|
| robot-reinforcement-learning (this skill)by arpitg1304 | 89 | 405 | 3d ago | SKILL.md |
| taste-skillby Leonxlnx | 100 | 120.0k | today | CLAUDE.md |
| Agent-Reachby Panniantong | 100 | 114.0k | today | CLAUDE.md |
| headroomby headroomlabs-ai | 100 | 80.5k | today | CLAUDE.md |
| last30days-skillby mvanhorn | 100 | 72.6k | today | CLAUDE.md |
Frequently asked questions
- How do I install robot-reinforcement-learning?
- Run
npx skills add arpitg1304/robotics-agent-skills --skill robot-reinforcement-learning. The install tabs above show the steps for each supported agent. - Which AI agents does robot-reinforcement-learning work with?
- It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
- Is robot-reinforcement-learning safe to use?
- Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. It is Apache-2.0-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
- Is robot-reinforcement-learning still maintained?
- The repository was last updated 3 days ago, so robot-reinforcement-learning is actively maintained.
Skill content
View source on GitHubname: robot-reinforcement-learning description: > Reinforcement learning for robots: when RL beats imitation or classical control, MDP design (observations, action spaces, control rate, terminated vs truncated), reward design and reward hacking, algorithm choice, massively parallel sim training, safe real-robot RL, and debugging training curves. Use when the user mentions reinforcement learning, RL, PPO, SAC, TD3, GAE, reward function, reward shaping, curriculum, domain randomization, sim-to-real, actuator network, asymmetric actor-critic, teacher-student distillation, rsl_rl, rl_games, skrl, Stable-Baselines3, CleanRL, Brax training, FastTD3, HIL-SERL, SERL, RLPD, DrQ, residual RL, DSRL, RL fine-tuning of a BC, diffusion or VLA policy, time-out bootstrapping, entropy collapse, value loss explosion, NaN during training, or evaluating RL with seeds and IQM. Also for Isaac Lab, mjlab or Playground locomotion and manipulation policies.
Robot Reinforcement Learning
Library landscape checked 2026-11-18. Versions, defaults and paper claims below were read from the official repositories and arXiv on that date; anything marked heuristic is practitioner experience, not a guarantee. Re-check versions before pinning.
RL on robots fails far more often from a wrong MDP, a hackable reward or a sim that does not match the robot than from the choice of algorithm. Spend effort in that order.
When to Use This Skill
- Deciding whether a robot task should use RL, imitation learning, classical control, or a hybrid
- Writing or reviewing an RL environment: observations, actions, rewards, terminations, resets
- Choosing PPO vs SAC/TD3, and rsl_rl vs rl_games vs skrl vs SB3 vs CleanRL vs Brax
- Training locomotion or manipulation in massively parallel GPU simulation
- Planning sim-to-real: domain randomisation, system identification, latency, actuator models
- Running RL on a physical robot (HIL-SERL style) with safety layers and learned rewards
- Fine-tuning a behaviour-cloned, diffusion or VLA policy with RL
- Diagnosing training that plateaus, collapses, produces NaNs, or exploits the reward
- Reporting results with multiple seeds and confidence intervals
Reference Files
Load these only when the task needs the detail:
| File | Load when | |---|---| | references/mdp-design.md | Full observation-group and action-space tables | | references/code-examples.md | Env skeleton, GAE, obs normaliser, IQM / CI code | | references/reward-design.md | Isaac Lab reward template; suspected reward hacking | | references/hybrid-methods.md | RL with demos, a base controller, or a BC / diffusion / VLA policy | | references/algorithm-mechanics.md | PPO internals, off-policy methods, asymmetric critics, distillation | | references/hyperparameters.md | Setting or tuning PPO / SAC / distillation hyperparameters; comparing to shipped Isaac Lab and SB3 values | | references/library-comparison.md | Choosing or switching RL libraries; interop bugs (obs groups, time-out flags, autoreset, normaliser export) | | references/sim-to-real-randomization.md | Writing domain randomisation, latency buffers, system identification, or a pre-hardware reality-gap audit | | references/real-world-rl.md | RL on hardware: safety layers, resets, success classifiers, interventions | | references/debugging.md | A training symptom is not in the short table in section 7 |
1. Mental Model
An RL project has four things you design, in decreasing order of how often they are the bug:
┌───────────────┐ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐
│ Data │ │ MDP │ │ Reward │ │ Algorithm │
│ distribution │ > │ obs, actions, │ > │ terms, weights│ > │ PPO/SAC, nets,│
│ sim + DR + │ │ rate, resets, │ │ curriculum, │ │ hyperparams │
│ resets │ │ terminations │ │ termination │ │ │
└───────────────┘ └───────────────┘ └───────────────┘ └───────────────┘
The optimiser will find the cheapest route to reward inside the simulator you give it, including routes you did not intend. Treat every learned behaviour as "correct for the MDP I wrote" until proven otherwise.
Is RL the Right Tool?
| Situation | Prefer | Why |
|---|---|---|
| Accurate model, cost is easy to write, need guarantees or interpretability | Classical control (MPC, WBC, IK + trajectory optimisation) | No training, predictable, certifiable |
| Easy to teleoperate, reward hard to write, perception-heavy manipulation | Imitation learning (see imitation-learning, lerobot) | Demos encode intent directly |
| Dynamics hard to model but cheap to simulate (legged locomotion, in-hand manipulation, agile flight) | RL in parallel sim + sim-to-real | Policy learns robustness the model cannot express |
| Demonstrations impossible or below the required performance (speed, recovery, super-human agility) | RL | RL can exceed the demonstrator |
| BC policy works 60-80% and you need 95%+ on one setup | Hybrid: RL fine-tuning on real or sim data | Cheaper than many more demos when success is detectable |
| Reward needs a human to judge every episode | Imitation, or learned reward with heavy validation | Real-world RL needs automatic reward |
Hybrid approaches (RLPD, residual RL, DSRL, RL for VLAs): references/hybrid-methods.md.
2. MDP Design
Observations
Groups (proprioception, task / command, exteroception, history, privileged for critic / teacher only), contents and rules: references/mdp-design.md.
- Scale every input to roughly unit range (hand-chosen scales or a running normaliser). Raw joint velocities of 20 rad/s next to quaternions of magnitude 1 slow learning.
- Base linear velocity is easy in sim and hard on hardware. Isaac Lab's velocity task observes it (with +-0.1 noise); a deployed policy needs a state estimator that matches, or drop it.
- If the time limit is part of the task (must finish by T), add remaining time to the observation (Pardo et al., arXiv 1712.00378). If it is only a training convenience, leave it out and bootstrap at truncation.
Action Spaces
Options (joint position target, the default for legged and humanoid; joint delta; torque; EE delta via IK / impedance / OSC; discrete gripper), pros and cons: references/mdp-design.md.
- Action scale is a hyperparameter. Too large: the initial std (often 1.0) slams joints to limits on step one. Too small: the policy cannot reach the needed poses and saturates.
- Clip in the environment, then apply. Log the fraction of clipped actions.
- PPO's Gaussian is unbounded; SAC's is tanh-squashed to [-1, 1]. Rescale to physical units in one place, shared by training and deployment code.
Control Rate
- Legged default (heuristic, matches Isaac Lab and legged_gym): physics at 200 Hz
(
dt=0.005), policy at 50 Hz (decimation=4), PD inside the physics loop. - Real-robot manipulation RL: commonly 10-30 Hz with EE deltas (LeRobot's HIL-SERL env config
defaults to
fps=30). - Higher policy rate means more steps per second of behaviour: raise gamma so the effective
horizon
1/(1-gamma)steps still covers the behaviour (0.99 at 50 Hz is ~2 s).
Terminations, Truncations and the Time-Out Bug
| Event | Gymnasium flag | Bootstrap? | Example |
|---|---|---|---|
| MDP genuinely ends (absorbing state) | terminated=True | No: future value is 0 | Robot fell, object dropped off table, joint limit violated |
| Episode cut for convenience | truncated=True | Yes: use V(final observation) | Time limit, curriculum reset, env reset for logging |
Treating a time-out as terminal teaches the critic that states near the time limit are worth
almost nothing, so the policy learns time-dependent, myopic behaviour and the value loss never
settles. Every major stack has a channel for this: Isaac Lab marks the time-out termination
with time_out=True, rsl_rl reads extras["time_outs"] and adds gamma * V to the reward on
those steps, SB3 replay buffers read info["TimeLimit.truncated"]
(handle_timeout_termination=True by default). Custom wrappers break it most often.
Gymnasium 1.x vector envs default to AutoresetMode.NEXT_STEP: the step after an episode ends
returns the reset observation, and that transition must be masked out. In SAME_STEP mode,
bootstrap from info["final_obs"], never from the returned (already reset) observation.
Runnable stdlib env skeleton, GAE with correct terminated / truncated handling and a running observation normaliser: references/code-examples.md. Always run a scripted or random policy through a new env before training; it catches sign errors, unreachable goals and broken resets in seconds instead of hours.
Resets
- Sim: randomise initial states broadly (Isaac Lab scales default joint positions by 0.5-1.5 at reset). Start some episodes from states along demonstrations or reference motions (reference state initialisation, DeepMimic) when the task has hard-to-reach phases.
- Early termination on falls or unrecoverable states speeds learning by not wasting samples, but interacts with reward sign (section 3).
- Real robots: resets are the bottleneck; see section 6.
3. Reward Design
Structure
r_t = sum_i w_i * term_i(s_t, a_t, s_{t+1})
task terms: bounded tracking kernels, e.g. exp(-||err||^2 / sigma), success bonus
regularisers: small penalties on action rate, torque, joint acceleration, power, limits
- Dense vs sparse: dense rewards learn fast but encode your solution (and your mistakes). Sparse success rewards are honest but need demos, curricula or very cheap samples. Real-world RL (SERL/HIL-SERL) uses sparse binary success plus demonstrations.
- Potential-based shaping
F = gamma * Phi(s') - Phi(s)(Ng, Harada and Russell, ICML 1999) provably preserves the optimal policy. Use it for "distance to goal" style shaping; set Phi to 0 at terminal states. Arbitrary shaping terms do not have this guarantee. - Normalise term magnitudes: log each term's per-episode sum at initialisation. If one penalty dominates the task term before learning starts, the first thing learned is "do nothing".
- Curricula: ramp terrain difficulty, command ranges, penalty weights (start small, increase once the task is solved), and randomisation ranges (automatic domain randomisation, arXiv 1910.07113). Advance on measured success, not on iteration count.
Reward and Termination Interact
| Reward sign per step | Termination | What the agent learns | Fix |
|---|---|---|---|
| Mostly negative | Failure ends episode | Fall early to stop paying penalties | Make per-step reward positive (legged_gym's only_positive_rewards=True clips the total at zero for this reason), or add a termination penalty larger than the remaining penalties |
| Mostly positive | Success ends episode | Avoid success; hover near the goal | Success bonus larger than the reward left in the episode, or make success non-terminal |
| Any | Time-out treated as terminal | Time-dependent, myopic behaviour | Bootstrap truncation (section 2) |
Reward Hacking: Detection
Hack examples and the Isaac Lab reward template: references/reward-design.md.
Detection habits:
- Log every reward term separately per episode (Isaac Lab logs
Episode_Reward/<term>). - Record a rollout video every N iterations and actually watch it.
- Track physical sanity metrics not in the reward: success from an independent
Truncated for display — read the full file on GitHub.
Related Skills
taste-skill
120.0kAgent skills that give coding agents taste: art direction, reference images and a browser check instead of generic AI pages
Agent-Reach
114.0kGive your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
headroom
80.5kCompress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
last30days-skill
72.6kAI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary
Languages
Trust signals
From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.
