---

name: cdf-augmentation description: >- Use when authoring or validating Cosmos Data Factory (CDF) augmentation YAML configs, or running remote Cosmos Transfer (including Cosmos3 WSM controls), Cosmos Predict, image-edit, or image-to-video inference. license: Apache-2.0 metadata: owner: NVIDIA service: cosmos-data-factory version: 1.2.0 reviewed: '2026-09-16' author: "NVIDIA opensource@nvidia.com" tags: - physical-ai - augmentation - cosmos - wsm - image-edit

CDF Augmentation Pipeline Skill

Remote-API pipeline for captioning, generating, and evaluating augmented camera data. Models are configured as HTTP endpoints; no local model weights are included.

Purpose

Use it to select a supported generation mode, author and validate PipelineConfig YAML, configure captioning/evaluators, and run the cdf-augmentation:1.2.0 container. It covers Cosmos Transfer/Predict, Cosmos3 WSM controls, image editing, image-to-video, BYOM endpoints, and quality gates.

Do not use this skill for training or fine-tuning models, deploying clusters or NIM endpoints, or unrelated application/database development.

Prerequisites

| Requirement | Detail | |-------------|--------| | Docker | docker --version. The image is remote-API only — it bundles no Cosmos/torch weights, so plain remote inference needs no GPU and no HF_TOKEN. | | NVIDIA GPU (conditional) | Only when a configured local stage requires CUDA or decodes H.264 in the augmentation container. See Limitations. | | Endpoint URLs | One reachable URL per role used by the selected config. The examples name local Qwen services (Qwen/Qwen3.6-27B-FP8 on vlm, Qwen/Qwen2.5-14B-Instruct on llm), but those are not guaranteed to be running. Resolve every required role independently: keep each reachable configured endpoint and ask only for the missing role URLs. | | API keys (conditional) | Only for endpoints requiring authentication. Pass the environment variable named by each endpoint's api_key_env; never hardcode values in YAML. Local unauthenticated endpoints need none. | | Input media | A video/image reachable by multistorageclient (s3://, msc://, gs://, az://, or HTTPS). Local paths are allowed only when no external Cosmos evaluator is enabled; external evaluation requires checker-readable remote URIs. | | Approved release commit (execution only) | A 40-character commit hash confirmed by the user as release-owner-approved or retrieved from the verified release manifest. Never invent it or rely on an unverified environment value. |

Inputs

Resolve each value in this precedence order: state file → explicit prompt arguments → agent context → user prompt. Ask the user only for what remains unresolved.

| Input | Required | Description | |-------|----------|-------------| | config_path | Yes | Path to the pipeline YAML, e.g. configs/cookbook/video-data-augmentation/config_video_transfer_CT3_omni.yaml. If absent, pick a starting config from Supported Models and confirm with the user. | | input_media | Conditional | Source video/image → data[].inputs.rgb. Required for every mode except Cosmos Predict inference_type: text2world, where inputs may be null or rgb omitted. Overridable at run time via data.0.inputs.rgb=.... | | control_media | For precomputed control | Control video → data[].inputs.controls.<type>. For Cosmos3 WSM transfer, set controls.wsm; the client uploads it rather than exposing its local path to the server. | | output_paths | Yes | data[].output.{video,caption,metadata}; evaluation optional. | | model_name | Yes | augmentation.model.name — an endpoint id, a role, or a known model name. Free-form string, not an enum. | | endpoint_urls | Yes | One endpoints[] entry per role in use. | | api_key_env | If auth | Env-var name per endpoint; the value comes from the environment. | | target_attributes | No | captioning.llm.variables (e.g. weather_condition, lighting_condition). | | generation_params | No | augmentation.parameters — pass-through; only set knobs are sent. | | seed | No | Under augmentation.parameters. Before the first candidate, null resolves once to the generator/executor seed when available, otherwise the current Unix time. Retry n uses that resolved base seed plus n. pipeline.retry defaults to 1, so evaluation permits at most 2 candidates by default. |

BYOM model: endpoints, adapters, roles

Each endpoints: list entry declares role, url, wire model, and optional id, adapter, api_key_env, and timeout. Roles are vlm, llm, image_edit, video_transfer, video_predict, image2video, and evaluator. augmentation.model.name resolves by endpoint id, then role, then the known model-name mapping. Adapter contracts and full fields are in configuration-schema.md.

Supported Models

When the user hasn't specified a model, choose from their input type and goal:

| Input Type → Goal | model.name | Role / default adapter | Input → Output | |-------------------|--------------|------------------------|----------------| | Video — change scene attributes (weather, lighting, style) | cosmos-transfer2.5 | video_transfer / nim | Video (+ controls) → Video | | Video + precomputed WSM — follow world-state geometry/motion | A video_transfer endpoint id, e.g. cosmos3-transfer-wsm | video_transfer / openai.video.async | RGB video + WSM control + prompt → Video | | Video + text — extend or predict continuation | cosmos-predict | video_predict / nim | Video+Text → Video | | Text only — generate video from scratch | cosmos-predict (inference_type: text2world) | video_predict / nim | Text → Video | | Image — edit specific attributes | image-edit | image_edit / nim (or openai.chat.completions, openai.images.edits) | Image → Image | | Image — animate a first frame | cosmos3-image2video (or your Veo endpoint id) | image2video / openai.video.sync (Veo: openai.video.async) | Image + prompt → Video |

Key rule: use Cosmos Transfer for video scene changes, Cosmos Predict for new/continued video, image edit for still edits, and image-to-video to animate a frame. For Cosmos3 WSM, set data[].inputs.controls.wsm. Config selection details are in config-decision-tree.md.

All models run via remote HTTP through one BaseExecutor; there is no local torchrun and no executor_type field.

Canonical Workflow

Follow this ordered decision tree; open the linked references only for the selected mode's details.

Config Correction Budget

config_attempts counts YAML versions, not validation or preflight calls. Keep it across the whole workflow and never reset it.

1. Set it to 1 after initial authoring.
2. On a correctable error, if it is already 3, STOP and report the error. Otherwise increment it once and make one correction pass containing all currently reported YAML fixes.
3. Validate and preflight that edited version without another increment. Repeat rule 2 only if a later check finds another correctable YAML error.

STEP 0 — Mandatory Entry Gate

Execute this gate before every other step:

• Scope gate: if the request asks for training, fine-tuning, cluster/NIM deployment, or unrelated development, decline it and STOP.
• Input gate: resolve values using Inputs. If required values remain unresolved, request only those values and STOP until they are supplied.
• Proceed only when both gates pass and the input type and augmentation goal are known.

1. Select the model from Supported Models using that input type and goal.
2. Author the YAML in this order: data, endpoints, captioning, augmentation, evaluators, pipeline, then data_processing. Omit optional sections rather than creating placeholders. Set config_attempts = 1 (config attempt 1 of 3).
3. Branch on the requested outcome:
   • Configuration only or “do not execute”: do not build, launch, run preflight, or infer. A trusted runtime is an already-running container whose recorded immutable image ID/digest, checkout commit, and config mounts satisfy step 4. If validation was explicitly requested and such a runtime exists, run only the validation command and apply step 5's failure rules. Return the config (mark it unvalidated if validation was not run) and STOP.
   • Inference/run requested: proceed to step 4.
4. Prepare the runtime in these verifiable sub-steps: a. Obtain EXPECTED_RELEASE_REF from the user as a release-owner-approved full commit or retrieve it from the verified release manifest. b. Verify the Git working tree is clean. c. Verify HEAD matches EXPECTED_RELEASE_REF. If a release tag selected the revision, verify the tag first, then compare its resolved full commit. d. Build the image and capture its immutable sha256: image ID, or obtain a verified registry digest. e. Create the cdf Docker bridge if needed and attach local services. f. Launch the container using the recorded image ID, the cdf network, required volume mounts, --entrypoint /bin/bash, and only the endpoint key variables named by api_key_env plus the scoped storage credential variables required for the configured remote media. If any sub-step fails, STOP and report the exact failure and relevant remediation: clean a dirty tree, supply the approved revision, resolve a revision mismatch, check the Docker daemon/build, fix network setup, or fix the container launch. Do not continue with a partial runtime.
5. Inside the container, run the validation-only command in Usage Step 2 and classify its result while preserving config_attempts (current config attempt N of 3):
   • Success: if no external Cosmos evaluator is enabled, proceed directly to step 7. If one is enabled, proceed to step 6.
   • Editable YAML/schema error: when required endpoint URL/model details are known, apply the Config Correction Budget rule, then rerun validation on that YAML version.
   • Infrastructure/unknown input: for missing endpoint details, credentials, DNS, connectivity, or unhealthy services, do not edit speculatively. STOP and report the error or ask for the missing endpoint details.
6. With external Cosmos evaluation enabled, preserve config_attempts (current config attempt N of 3) and run its hard preflight gate before captioning/generation by querying /health, /checkers, and /dependency-graph:
   • Pass: require /health.status == "healthy"; require every configured checks[].name to exactly equal one string in the top-level checkers array returned by /checkers and one key in the top-level checkers object returned by /dependency-graph. Do not use substring, alias, or nested-field matching. Then proceed to step 7.
   • Config error: only a 4xx response from /checkers or /dependency-graph whose body explicitly identifies a configured checker name or invalid config field qualifies. Apply the Config Correction Budget rule, then return to step 5; revalidation does not increment the counter.
   • Infrastructure failure: any timeout, network/DNS error, 5xx response, /health failure, unhealthy service, or missing deployed checker stops the workflow before generation. Create no candidate and report the root cause.
7. Enter inference only after step 5 succeeds and, when external evaluation is enabled, step 6 passes. Run a candidate loop bounded to pipeline.retry + 1 candidates (retry defaults to 1, so the default maximum is 2). Resolve the base seed once before the first candidate as defined in Inputs; retry n uses `bas