SkillAgentSearch skills...

evaluating-cosmos-policy

Evaluates NVIDIA Cosmos Policy on LIBERO and RoboCasa simulation environments

Install / Use

npx skills add Orchestra-Research/AI-Research-SKILLs --skill cosmos-policy

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

95/100

Supported Platforms

Universal

Our assessment of evaluating-cosmos-policy

evaluating-cosmos-policy scores 95/100 on our quality scale, 33rd of 421 Education & Research skills we index (top 8%).

Its SKILL.md is 14 KB long, well organised into 19 sections with 17 code examples: a thorough specification that gives an agent plenty to work with.

With 14,543 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
30/30
Structure
20/20
Description
12/15
Adoption
18/20
Freshness
15/15

Maintenance, license and trust

  • The repository was last updated 36 days ago, so evaluating-cosmos-policy is actively maintained.
  • Our last check on 2026-11-15 found the source still online.
  • It is released under the MIT license, a permissive license that allows use, modification and commercial use with attribution.
  • Its trust signals score 100/100, with no cautions. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-09-27. Automated pattern scan on 2026-09-27. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

evaluating-cosmos-policy compared with similar skills

All 4 of these similar skills score higher than evaluating-cosmos-policy; compare them before choosing.

SkillScoreStarsUpdatedFormat
evaluating-cosmos-policy (this skill)by Orchestra-Research9514.5k36d agoSKILL.md
Agent-Reachby Panniantong100112.3k8d agoCLAUDE.md
last30days-skillby mvanhorn10068.0ktodayCLAUDE.md
x402by browser-use100121.0k3d agoSKILL.md
algorithmic-artby anthropics100177.9k55d agoSKILL.md

Frequently asked questions

How do I install evaluating-cosmos-policy?
Run npx skills add Orchestra-Research/AI-Research-SKILLs --skill evaluating-cosmos-policy. The install tabs above show the steps for each supported agent.
Which AI agents does evaluating-cosmos-policy work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is evaluating-cosmos-policy safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It is MIT-licensed and scores 100/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is evaluating-cosmos-policy still maintained?
The repository was last updated 36 days ago, so evaluating-cosmos-policy is actively maintained.

name: evaluating-cosmos-policy description: Evaluates NVIDIA Cosmos Policy on LIBERO and RoboCasa simulation environments. Use when setting up cosmos-policy for robot manipulation evaluation, running headless GPU evaluations with EGL rendering, or profiling inference latency on cluster or local GPU machines. version: 1.0.0 author: Orchestra Research license: MIT tags: [Cosmos Policy, VLA, Robotics, LIBERO, RoboCasa, Simulation, Evaluation, Profiling, EGL Rendering] dependencies: [torch>=2.1.0, mujoco>=3.0.0, robosuite>=1.4.0, "robocasa @ git+https://github.com/moojink/robocasa-cosmos-policy.git", transformers>=4.40.0, "cosmos-policy @ git+https://github.com/NVlabs/cosmos-policy.git"]

Cosmos Policy Evaluation

Evaluation workflows for NVIDIA Cosmos Policy on LIBERO and RoboCasa simulation environments from the public cosmos-policy repository. Covers blank-machine setup, headless GPU evaluation, and inference profiling.

Quick start

Run a minimal LIBERO evaluation using the official public eval module:

uv run --extra cu128 --group libero --python 3.10 \
  python -m cosmos_policy.experiments.robot.libero.run_libero_eval \
    --config cosmos_predict2_2b_480p_libero__inference_only \
    --ckpt_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B \
    --config_file cosmos_policy/config/config.py \
    --use_wrist_image True \
    --use_proprio True \
    --normalize_proprio True \
    --unnormalize_actions True \
    --dataset_stats_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B/libero_dataset_statistics.json \
    --t5_text_embeddings_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B/libero_t5_embeddings.pkl \
    --trained_with_image_aug True \
    --chunk_size 16 \
    --num_open_loop_steps 16 \
    --task_suite_name libero_10 \
    --num_trials_per_task 1 \
    --local_log_dir cosmos_policy/experiments/robot/libero/logs/ \
    --seed 195 \
    --randomize_seed False \
    --deterministic True \
    --run_id_note smoke \
    --ar_future_prediction False \
    --ar_value_prediction False \
    --use_jpeg_compression True \
    --flip_images True \
    --num_denoising_steps_action 5 \
    --num_denoising_steps_future_state 1 \
    --num_denoising_steps_value 1 \
    --data_collection False

Core concepts

What Cosmos Policy is: NVIDIA Cosmos Policy is a vision-language-action (VLA) model that uses Cosmos Tokenizer to encode visual observations into discrete tokens, then predicts robot actions conditioned on language instructions and visual context.

Key architecture choices:

| Component | Design | |-----------|--------| | Visual encoder | Cosmos Tokenizer (discrete tokens) | | Language conditioning | Cross-attention to language embeddings | | Action prediction | Autoregressive action token generation |

Public command surface: The supported evaluation entrypoints are cosmos_policy.experiments.robot.libero.run_libero_eval and cosmos_policy.experiments.robot.robocasa.run_robocasa_eval. Keep reproduction notes anchored to these public modules and their documented flags.

Compute requirements

| Task | GPU | VRAM | Typical wall time | |------|-----|------|-------------------| | LIBERO smoke eval (1 trial) | 1x A40/A100 | ~16 GB | 5-10 min | | LIBERO full eval (50 trials) | 1x A40/A100 | ~16 GB | 2-4 hours | | RoboCasa single-task (2 trials) | 1x A40/A100 | ~18 GB | 10-15 min | | RoboCasa all-tasks | 1x A40/A100 | ~18 GB | 4-8 hours |

When to use vs alternatives

Use this skill when:

  • Evaluating NVIDIA Cosmos Policy on LIBERO or RoboCasa benchmarks
  • Profiling inference latency and throughput for Cosmos Policy
  • Setting up headless EGL rendering for robot simulation on GPU clusters

Use alternatives when:

  • Training or fine-tuning Cosmos Policy from scratch (use official Cosmos training docs)
  • Working with OpenVLA-based policies (use fine-tuning-openvla-oft)
  • Working with Physical Intelligence pi0 models (use fine-tuning-serving-openpi)
  • Running real-robot evaluation rather than simulation

Workflow 1: LIBERO evaluation

Copy this checklist and track progress:

LIBERO Eval Progress:
- [ ] Step 1: Install environment and dependencies
- [ ] Step 2: Configure headless EGL rendering
- [ ] Step 3: Run smoke evaluation
- [ ] Step 4: Validate outputs and parse results
- [ ] Step 5: Run full benchmark if smoke passes

Step 1: Install environment

git clone https://github.com/NVlabs/cosmos-policy.git
cd cosmos-policy
# Follow SETUP.md to build and enter the supported Docker container.
# Then, inside the container:
uv sync --extra cu128 --group libero --python 3.10

Step 2: Configure headless rendering

export CUDA_VISIBLE_DEVICES=0
export MUJOCO_EGL_DEVICE_ID=0
export MUJOCO_GL=egl
export PYOPENGL_PLATFORM=egl

Step 3: Run smoke evaluation

uv run --extra cu128 --group libero --python 3.10 \
  python -m cosmos_policy.experiments.robot.libero.run_libero_eval \
    --config cosmos_predict2_2b_480p_libero__inference_only \
    --ckpt_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B \
    --config_file cosmos_policy/config/config.py \
    --use_wrist_image True \
    --use_proprio True \
    --normalize_proprio True \
    --unnormalize_actions True \
    --dataset_stats_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B/libero_dataset_statistics.json \
    --t5_text_embeddings_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B/libero_t5_embeddings.pkl \
    --trained_with_image_aug True \
    --chunk_size 16 \
    --num_open_loop_steps 16 \
    --task_suite_name libero_10 \
    --num_trials_per_task 1 \
    --local_log_dir cosmos_policy/experiments/robot/libero/logs/ \
    --seed 195 \
    --randomize_seed False \
    --deterministic True \
    --run_id_note smoke \
    --ar_future_prediction False \
    --ar_value_prediction False \
    --use_jpeg_compression True \
    --flip_images True \
    --num_denoising_steps_action 5 \
    --num_denoising_steps_future_state 1 \
    --num_denoising_steps_value 1 \
    --data_collection False

Step 4: Validate and parse results

import json
import glob

# Find latest evaluation result from the official log directory
log_files = sorted(glob.glob("cosmos_policy/experiments/robot/libero/logs/**/*.json", recursive=True))
with open(log_files[-1]) as f:
    results = json.load(f)

print(results)

Step 5: Scale up

Run across all four LIBERO task suites with 50 trials:

for suite in libero_spatial libero_object libero_goal libero_10; do
  uv run --extra cu128 --group libero --python 3.10 \
    python -m cosmos_policy.experiments.robot.libero.run_libero_eval \
      --config cosmos_predict2_2b_480p_libero__inference_only \
      --ckpt_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B \
      --config_file cosmos_policy/config/config.py \
      --use_wrist_image True \
      --use_proprio True \
      --normalize_proprio True \
      --unnormalize_actions True \
      --dataset_stats_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B/libero_dataset_statistics.json \
      --t5_text_embeddings_path nvidia/Cosmos-Policy-LIBERO-Predict2-2B/libero_t5_embeddings.pkl \
      --trained_with_image_aug True \
      --chunk_size 16 \
      --num_open_loop_steps 16 \
      --task_suite_name "$suite" \
      --num_trials_per_task 50 \
      --local_log_dir cosmos_policy/experiments/robot/libero/logs/ \
      --seed 195 \
      --randomize_seed False \
      --deterministic True \
      --run_id_note "suite_${suite}" \
      --ar_future_prediction False \
      --ar_value_prediction False \
      --use_jpeg_compression True \
      --flip_images True \
      --num_denoising_steps_action 5 \
      --num_denoising_steps_future_state 1 \
      --num_denoising_steps_value 1 \
      --data_collection False
done

Workflow 2: RoboCasa evaluation

Copy this checklist and track progress:

RoboCasa Eval Progress:
- [ ] Step 1: Install RoboCasa assets and verify macros
- [ ] Step 2: Run single-task smoke evaluation
- [ ] Step 3: Validate outputs
- [ ] Step 4: Expand to multi-task runs

Step 1: Install RoboCasa

git clone https://github.com/moojink/robocasa-cosmos-policy.git
uv pip install -e robocasa-cosmos-policy
python -m robocasa.scripts.setup_macros
python -m robocasa.scripts.download_kitchen_assets

This fork installs the robocasa Python package expected by Cosmos Policy while preserving the patched environment changes used in the public RoboCasa eval path. Verify macros_private.py exists and paths are correct.

Step 2: Single-task smoke evaluation

uv run --extra cu128 --group robocasa --python 3.10 \
  python -m cosmos_policy.experiments.robot.robocasa.run_robocasa_eval \
    --config cosmos_predict2_2b_480p_robocasa_50_demos_per_task__inference \
    --ckpt_path nvidia/Cosmos-Policy-RoboCasa-Predict2-2B \
    --config_file cosmos_policy/config/config.py \
    --use_wrist_image True \
    --num_wrist_images 1 \
    --use_proprio True \
    --normalize_proprio True \
    --unnormalize_actions True \
    --dataset_stats_path nvidia/Cosmos-Policy-RoboCasa-Predict2-2B/robocasa_dataset_statistics.json \
    --t5_text_embeddings_path nvidia/Cosmos-Policy-RoboCasa-Predict2-2B/robocasa_t5_embeddings.pkl \
    --trained_with_image_aug True \
    --chunk_size 32 \
    --num_open_loop_steps 16 \
    --task_name TurnOffMicrowave \
    --obj_instance_split A \
    --num_trials_per_task 2 \
    --local_log_dir cosmos_policy/experiments/robot/robocasa/logs/ \
    --seed 195 \
    --randomize_seed False \
    --deterministic True \
    --run_id_note smoke \
    --use_variance_scale False \
    --use_jpeg_compression True \
    --flip_images True \
    --num_denoising_steps_action 5 \
    --num_denoising_steps_future_state 1 \
    --num_denoising_steps_value 1 \
    --data_collection False

Step 3: Validate outputs

  • Confirm the eval log prints the expected task name, object split, and checkpoint/config values.
  • Inspect the final Success rate: line in the log.

Step 4: Expand scope

Increase --num_trials_per_task or add more tasks. Keep --obj_instance_split fixed across repeated runs for comparability.


Workflow 3: Blank-machine cluster launch

Cluster Launch Progress:
- [ ] Step 1: Clone the public repo and enter the supported runtime
- [ ] Step 2: Sync the benchmark-specific dependency group
- [ ] Step 3: Export rendering and cache environment variables before eval

Step 1: Clone and enter the supported runtime

git clone https://github.com/NVlabs/cosmos-policy.git
cd cosmos-policy
# Follow SETUP.md, start the Docker container, and enter it before continuing.

Step 2: Sync dependencies

uv sync --extra cu128 --group libero --python 3.10
# or, for RoboCasa:
uv sync --extra cu128 --group robocasa --python 3.10
# then install the Cosmos-compatible RoboCasa fork:
git clone https://github.com/moojink/robocasa-cosmos-policy.git
uv pip install -e robocasa-cosmos-policy

Step 3: Export runtime environment

export CUDA_VISIBLE_DEVICES=0
export MUJOCO_EGL_DEVICE_ID=0
export MUJOCO_GL=egl
export PYOPENGL_PLATFORM=egl
export HF_HOME=${HF_HOME:-$HOME/.cache/huggingface}
export TRANSFORMERS_CACHE=${TRANSFORMERS_CACHE:-$HF_HOME}

Expected performance benchmarks

Reference values from official evaluation (tied to specific setup and seeds):

| Task Suite | Success Rate | Notes | |-----------|-------------|-------| | LIBERO-Spatial | 98.1% | Official LIBERO spatial result | | LIBERO-Object | 100.0% | Official LIBERO object result | | LIBERO-Goal | 98.2% | Official LIBERO goal result | | LIBERO-Long | 97.6% | Official LIBERO long-horizon result | | LIBERO-Average | 98.5% | Official average across LIBERO suites | | RoboCasa | 67.1% | Official RoboCasa average result |

Reproduction note: Published success rates still depend on checkpoint choice, task suite, seeds, and simulator setup. Record the exact command and environment alo

Truncated for display — read the full file on GitHub.

Related Skills

View on GitHub
GitHub Stars14.5k
CategoryEducation
Updated1mo ago
Forks1.0k

Languages

TeX

Trust signals

100/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

No cautions