SkillAgentSearch skills...

robot-reinforcement-learning

Reinforcement learning for robots: when RL beats imitation or classical control, MDP design (observations, action spaces, control rate, terminated vs truncated), reward design and reward hacking, algorithm choice, massively parallel sim training, safe real-robot RL, and debugging training curves

Install / Use

npx skills add arpitg1304/robotics-agent-skills --skill robot-reinforcement-learning

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

89/100

Supported Platforms

Universal

Our assessment of robot-reinforcement-learning

robot-reinforcement-learning scores 89/100 on our quality scale, 166th of 416 Education & Research skills we index (top 40%).

Its SKILL.md is 21 KB long, well organised into 26 sections with 2 code examples: a thorough specification that gives an agent plenty to work with.

It has 405 GitHub stars, a meaningful sign that others use it.

Substance
30/30
Structure
18/20
Description
15/15
Adoption
11/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated 3 days ago, so robot-reinforcement-learning is actively maintained.
  • It is released under the Apache-2.0 license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands.

Automated pattern scan on 2026-11-22. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

robot-reinforcement-learning compared with similar skills

All 4 of these similar skills score higher than robot-reinforcement-learning; compare them before choosing.

SkillScoreStarsUpdatedFormat
robot-reinforcement-learning (this skill)by arpitg1304894053d agoSKILL.md
taste-skillby Leonxlnx100120.0ktodayCLAUDE.md
Agent-Reachby Panniantong100114.0ktodayCLAUDE.md
headroomby headroomlabs-ai10080.5ktodayCLAUDE.md
last30days-skillby mvanhorn10072.6ktodayCLAUDE.md

Frequently asked questions

How do I install robot-reinforcement-learning?
Run npx skills add arpitg1304/robotics-agent-skills --skill robot-reinforcement-learning. The install tabs above show the steps for each supported agent.
Which AI agents does robot-reinforcement-learning work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is robot-reinforcement-learning safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. It is Apache-2.0-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is robot-reinforcement-learning still maintained?
The repository was last updated 3 days ago, so robot-reinforcement-learning is actively maintained.

name: robot-reinforcement-learning description: > Reinforcement learning for robots: when RL beats imitation or classical control, MDP design (observations, action spaces, control rate, terminated vs truncated), reward design and reward hacking, algorithm choice, massively parallel sim training, safe real-robot RL, and debugging training curves. Use when the user mentions reinforcement learning, RL, PPO, SAC, TD3, GAE, reward function, reward shaping, curriculum, domain randomization, sim-to-real, actuator network, asymmetric actor-critic, teacher-student distillation, rsl_rl, rl_games, skrl, Stable-Baselines3, CleanRL, Brax training, FastTD3, HIL-SERL, SERL, RLPD, DrQ, residual RL, DSRL, RL fine-tuning of a BC, diffusion or VLA policy, time-out bootstrapping, entropy collapse, value loss explosion, NaN during training, or evaluating RL with seeds and IQM. Also for Isaac Lab, mjlab or Playground locomotion and manipulation policies.

Robot Reinforcement Learning

Library landscape checked 2026-11-18. Versions, defaults and paper claims below were read from the official repositories and arXiv on that date; anything marked heuristic is practitioner experience, not a guarantee. Re-check versions before pinning.

RL on robots fails far more often from a wrong MDP, a hackable reward or a sim that does not match the robot than from the choice of algorithm. Spend effort in that order.

When to Use This Skill

  • Deciding whether a robot task should use RL, imitation learning, classical control, or a hybrid
  • Writing or reviewing an RL environment: observations, actions, rewards, terminations, resets
  • Choosing PPO vs SAC/TD3, and rsl_rl vs rl_games vs skrl vs SB3 vs CleanRL vs Brax
  • Training locomotion or manipulation in massively parallel GPU simulation
  • Planning sim-to-real: domain randomisation, system identification, latency, actuator models
  • Running RL on a physical robot (HIL-SERL style) with safety layers and learned rewards
  • Fine-tuning a behaviour-cloned, diffusion or VLA policy with RL
  • Diagnosing training that plateaus, collapses, produces NaNs, or exploits the reward
  • Reporting results with multiple seeds and confidence intervals

Reference Files

Load these only when the task needs the detail:

| File | Load when | |---|---| | references/mdp-design.md | Full observation-group and action-space tables | | references/code-examples.md | Env skeleton, GAE, obs normaliser, IQM / CI code | | references/reward-design.md | Isaac Lab reward template; suspected reward hacking | | references/hybrid-methods.md | RL with demos, a base controller, or a BC / diffusion / VLA policy | | references/algorithm-mechanics.md | PPO internals, off-policy methods, asymmetric critics, distillation | | references/hyperparameters.md | Setting or tuning PPO / SAC / distillation hyperparameters; comparing to shipped Isaac Lab and SB3 values | | references/library-comparison.md | Choosing or switching RL libraries; interop bugs (obs groups, time-out flags, autoreset, normaliser export) | | references/sim-to-real-randomization.md | Writing domain randomisation, latency buffers, system identification, or a pre-hardware reality-gap audit | | references/real-world-rl.md | RL on hardware: safety layers, resets, success classifiers, interventions | | references/debugging.md | A training symptom is not in the short table in section 7 |

1. Mental Model

An RL project has four things you design, in decreasing order of how often they are the bug:

  ┌───────────────┐   ┌───────────────┐   ┌───────────────┐   ┌───────────────┐
  │ Data          │   │ MDP           │   │ Reward        │   │ Algorithm     │
  │ distribution  │ > │ obs, actions, │ > │ terms, weights│ > │ PPO/SAC, nets,│
  │ sim + DR +    │   │ rate, resets, │   │ curriculum,   │   │ hyperparams   │
  │ resets        │   │ terminations  │   │ termination   │   │               │
  └───────────────┘   └───────────────┘   └───────────────┘   └───────────────┘

The optimiser will find the cheapest route to reward inside the simulator you give it, including routes you did not intend. Treat every learned behaviour as "correct for the MDP I wrote" until proven otherwise.

Is RL the Right Tool?

| Situation | Prefer | Why | |---|---|---| | Accurate model, cost is easy to write, need guarantees or interpretability | Classical control (MPC, WBC, IK + trajectory optimisation) | No training, predictable, certifiable | | Easy to teleoperate, reward hard to write, perception-heavy manipulation | Imitation learning (see imitation-learning, lerobot) | Demos encode intent directly | | Dynamics hard to model but cheap to simulate (legged locomotion, in-hand manipulation, agile flight) | RL in parallel sim + sim-to-real | Policy learns robustness the model cannot express | | Demonstrations impossible or below the required performance (speed, recovery, super-human agility) | RL | RL can exceed the demonstrator | | BC policy works 60-80% and you need 95%+ on one setup | Hybrid: RL fine-tuning on real or sim data | Cheaper than many more demos when success is detectable | | Reward needs a human to judge every episode | Imitation, or learned reward with heavy validation | Real-world RL needs automatic reward |

Hybrid approaches (RLPD, residual RL, DSRL, RL for VLAs): references/hybrid-methods.md.

2. MDP Design

Observations

Groups (proprioception, task / command, exteroception, history, privileged for critic / teacher only), contents and rules: references/mdp-design.md.

  • Scale every input to roughly unit range (hand-chosen scales or a running normaliser). Raw joint velocities of 20 rad/s next to quaternions of magnitude 1 slow learning.
  • Base linear velocity is easy in sim and hard on hardware. Isaac Lab's velocity task observes it (with +-0.1 noise); a deployed policy needs a state estimator that matches, or drop it.
  • If the time limit is part of the task (must finish by T), add remaining time to the observation (Pardo et al., arXiv 1712.00378). If it is only a training convenience, leave it out and bootstrap at truncation.

Action Spaces

Options (joint position target, the default for legged and humanoid; joint delta; torque; EE delta via IK / impedance / OSC; discrete gripper), pros and cons: references/mdp-design.md.

  • Action scale is a hyperparameter. Too large: the initial std (often 1.0) slams joints to limits on step one. Too small: the policy cannot reach the needed poses and saturates.
  • Clip in the environment, then apply. Log the fraction of clipped actions.
  • PPO's Gaussian is unbounded; SAC's is tanh-squashed to [-1, 1]. Rescale to physical units in one place, shared by training and deployment code.

Control Rate

  • Legged default (heuristic, matches Isaac Lab and legged_gym): physics at 200 Hz (dt=0.005), policy at 50 Hz (decimation=4), PD inside the physics loop.
  • Real-robot manipulation RL: commonly 10-30 Hz with EE deltas (LeRobot's HIL-SERL env config defaults to fps=30).
  • Higher policy rate means more steps per second of behaviour: raise gamma so the effective horizon 1/(1-gamma) steps still covers the behaviour (0.99 at 50 Hz is ~2 s).

Terminations, Truncations and the Time-Out Bug

| Event | Gymnasium flag | Bootstrap? | Example | |---|---|---|---| | MDP genuinely ends (absorbing state) | terminated=True | No: future value is 0 | Robot fell, object dropped off table, joint limit violated | | Episode cut for convenience | truncated=True | Yes: use V(final observation) | Time limit, curriculum reset, env reset for logging |

Treating a time-out as terminal teaches the critic that states near the time limit are worth almost nothing, so the policy learns time-dependent, myopic behaviour and the value loss never settles. Every major stack has a channel for this: Isaac Lab marks the time-out termination with time_out=True, rsl_rl reads extras["time_outs"] and adds gamma * V to the reward on those steps, SB3 replay buffers read info["TimeLimit.truncated"] (handle_timeout_termination=True by default). Custom wrappers break it most often.

Gymnasium 1.x vector envs default to AutoresetMode.NEXT_STEP: the step after an episode ends returns the reset observation, and that transition must be masked out. In SAME_STEP mode, bootstrap from info["final_obs"], never from the returned (already reset) observation.

Runnable stdlib env skeleton, GAE with correct terminated / truncated handling and a running observation normaliser: references/code-examples.md. Always run a scripted or random policy through a new env before training; it catches sign errors, unreachable goals and broken resets in seconds instead of hours.

Resets

  • Sim: randomise initial states broadly (Isaac Lab scales default joint positions by 0.5-1.5 at reset). Start some episodes from states along demonstrations or reference motions (reference state initialisation, DeepMimic) when the task has hard-to-reach phases.
  • Early termination on falls or unrecoverable states speeds learning by not wasting samples, but interacts with reward sign (section 3).
  • Real robots: resets are the bottleneck; see section 6.

3. Reward Design

Structure

r_t = sum_i w_i * term_i(s_t, a_t, s_{t+1})
      task terms:  bounded tracking kernels, e.g. exp(-||err||^2 / sigma), success bonus
      regularisers: small penalties on action rate, torque, joint acceleration, power, limits
  • Dense vs sparse: dense rewards learn fast but encode your solution (and your mistakes). Sparse success rewards are honest but need demos, curricula or very cheap samples. Real-world RL (SERL/HIL-SERL) uses sparse binary success plus demonstrations.
  • Potential-based shaping F = gamma * Phi(s') - Phi(s) (Ng, Harada and Russell, ICML 1999) provably preserves the optimal policy. Use it for "distance to goal" style shaping; set Phi to 0 at terminal states. Arbitrary shaping terms do not have this guarantee.
  • Normalise term magnitudes: log each term's per-episode sum at initialisation. If one penalty dominates the task term before learning starts, the first thing learned is "do nothing".
  • Curricula: ramp terrain difficulty, command ranges, penalty weights (start small, increase once the task is solved), and randomisation ranges (automatic domain randomisation, arXiv 1910.07113). Advance on measured success, not on iteration count.

Reward and Termination Interact

| Reward sign per step | Termination | What the agent learns | Fix | |---|---|---|---| | Mostly negative | Failure ends episode | Fall early to stop paying penalties | Make per-step reward positive (legged_gym's only_positive_rewards=True clips the total at zero for this reason), or add a termination penalty larger than the remaining penalties | | Mostly positive | Success ends episode | Avoid success; hover near the goal | Success bonus larger than the reward left in the episode, or make success non-terminal | | Any | Time-out treated as terminal | Time-dependent, myopic behaviour | Bootstrap truncation (section 2) |

Reward Hacking: Detection

Hack examples and the Isaac Lab reward template: references/reward-design.md.

Detection habits:

  • Log every reward term separately per episode (Isaac Lab logs Episode_Reward/<term>).
  • Record a rollout video every N iterations and actually watch it.
  • Track physical sanity metrics not in the reward: success from an independent

Truncated for display — read the full file on GitHub.

Related Skills

View on GitHub
GitHub Stars405
CategoryEducation
Updated3d ago
Forks51

Languages

Python

Trust signals

100/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

No cautions