---

name: cdf-image-attribute-augmentation description: Run the CDF Orchestration Image Attribute Augmentation DAG on Kubernetes - person-crop clothing augmentation, attribute search, and augmented dataset generation. Select for requests about image attribute augmentation, person attribute search, person re-identification data, clothing augmentation, attribute captions, augmentation payloads, run status, or result retrieval. Runs environment setup first when controller readiness is unknown. Not for video or defect-image generation. version: "1.0.1" license: CC-BY-4.0 AND Apache-2.0 metadata: owner: NVIDIA service: cosmos-data-factory version: "1.0.1" reviewed: "2026-10-09" author: NVIDIA tags: - physical-ai - cdf-orchestration - image-attribute-augmentation - cosmos

CDF Orchestration — Image Attribute Augmentation

Run the Image Attribute Augmentation DAG end to end: person-crop input preparation, cosmos image-edit augmentation, cosmos post-processing, event and person attribute search, augmented dataset generation, and result retrieval.

Component details

This workflow skill and its references are the primary guide for configuring and running the workflow. Consult a component skill only when the requested configuration or diagnosis requires component-specific details not covered here. Routine payload authoring with documented fields, preflight, triggering, monitoring, and result retrieval do not require component skill access.

When additional component guidance is needed, consult that component's documentation, and do not invent keys or values. The links below are pinned to the latest main commit of each component repository as of 2026-10-20; a component skill installed separately may describe a different version.

| Component | Covers | Documentation at that commit | |---|---|---| | cdf-augmentation | Image-edit configuration, clothing variables and distributions, captioning, verification, and augmentation errors | cdf-augmentation skill | | cdf-auto-labeling | Event and person attribute-search configuration and errors | cdf-auto-labeling skill |

Use only documentation at these commits; do not substitute a moving branch or another version.

If the repository is inaccessible, continue tasks supported by this workflow's guidance. Identify the unavailable reference and the unresolved component-specific detail only when that missing guidance prevents completing the requested work; do not invent component settings or treat component access as a prerequisite for ordinary workflow use.

Apply component guidance through this workflow's supported payload fields and DAG constraints. Component configuration and the Airflow trigger payload are different contracts; component capabilities do not automatically become supported DAG options. Keep execution through this workflow's existing procedure. If the pinned guidance differs from the deployed component version or DAG integration, report the mismatch before proposing the affected settings.

DAG selection

The workflow builds one DAG per compute platform from airflow/dags/workflows/image_attribute_augmentation_dag/:

| Platform | DAG ID | Manifest | |---|---|---| | Kubernetes | image_attribute_augmentation_dag_k8s | image_attribute_augmentation_k8s_manifest.yaml |

Kubernetes is the only platform whose manifest is checked in, so image_attribute_augmentation_dag_k8s is the only DAG this repository registers. A DAG is registered only if its manifest exists; a missing manifest means the DAG is absent from Airflow rather than broken. List the DAGs Airflow actually loaded before triggering, and never name a DAG ID that is not in that list.

There is a single end-to-end pipeline — there are no augmentation-only or labeling-only DAG variants. If a user asks for augmentation without attribute search, tell them the checked-in DAG does not offer that flow rather than inventing a DAG ID.

Manual payload entry in the Airflow UI

If the user wants to enter their own payload directly in the Airflow UI rather than have you construct and trigger one, your job is limited to getting them to the UI: confirm controller readiness, ensure make port-forward is running (see airflow-direct-api.md), and report the reachable URL. Do not render a payload, run preflight, or trigger a run yourself in this case — the user is doing that from the UI. Resume monitoring (step 6 below) once they tell you a run has been triggered; you can find it via the Airflow API without needing the payload they used.

Scope

Before building any payload, collect all of the following from the user. Never invent an endpoint URL or bucket path — use only values the user supplied.

| Required | Field | What to ask | |---|---|---| | Always | input_path | S3 (or HTTP/HTTPS) URL whose immediate subdirectories are person-ID folders | | Always | output_directory | Writable S3 URL where results should be written | | Always | service mode | external (user provides endpoint URLs) or internal (DAG deploys services in-cluster) | | External mode | cosmos.vlm_service_url | Full HTTPS URL for the VLM inference endpoint | | External mode | cosmos.llm_service_url | Full HTTPS URL for the LLM inference endpoint | | External mode | cosmos.image_edit_service_url | Full HTTPS URL for the image-edit inference endpoint | | Optional | max_imgs | Number of person-ID folders to process (default: 1; 0 or negative = all) | | Optional | cosmos.num_augmentation | Clothing variants per person (default: 1) | | Optional | cosmos.variable_distribution | Clothing attribute distribution file path (see payload-contract.md) |

If the user does not provide a required value, ask for it explicitly before proceeding. Never invent one. Values the user supplied earlier in the session, including for a previous run, may be reused; list each reused value in the payload shown for confirmation in step 3, so the user confirms it again before anything is submitted.

Always run the following readiness checks before triggering a run. The checks are short-circuiting — stop at the first failure and route to the environment-setup skill immediately.

Before any check, establish the cluster connection. The cluster is reached only through credentials the user supplies — they are never part of the repository. Check whether the cluster credential file path is already exported in the shell environment; if not, ask the user for the absolute path before running any cluster command. Never assume a path or fall back to any on-disk default — see setup-and-preflight.md for the full procedure.

The controller (Airflow) and DAG compute tasks run on the same cluster unless a different remote cluster connection was configured. GPU capacity is checked on this cluster.

1. Controller pods — check that the Airflow controller pods (not DAG task pods) are Running. DAG task pods in Pending or Failed state are normal and must not be mistaken for controller failures:

   kubectl get pods -n sdg-workflow -l "release=sdg-workflow-controller"
   

   All pods matching the release=sdg-workflow-controller label must be Running. If the namespace is absent, this is a first-install condition — route to the environment-setup skill, do not diagnose further.

2. Airflow API — reachable only if check 1 passes. First establish AIRFLOW_URL from the Kubernetes ClusterIP. That works from a cluster node; from a workstation, fall back to a port-forward as described in airflow-direct-api.md:

   AIRFLOW_URL="http://$(kubectl get svc -n sdg-workflow \
     sdg-workflow-controller-api-server \
     -o jsonpath='{.spec.clusterIP}'):8080"
   

   Then confirm the target DAG is loaded and is_paused: False. See airflow-direct-api.md for the full auth + check sequence. If the API is unreachable, route to the environment-setup skill.

3. Pools — only if check 2 passes. Derive the required pool names from the manifest; do not assume them. Pool names are deployment-specific: they come from the manifest's deployment.profiles.*.pool and are created by deploy/values.yaml's airflowPools.pools, so a customer on different hardware or with renamed pools will not match whatever this repository ships today. See airflow-direct-api.md for the derivation snippet, which also adds the mode-dependent augmentation pool the DAG selects at runtime (external_image_edit_service_pool for external, default_pool for internal — set in the DAG's CosmosTaskGroup(...) wiring, not in the manifest). On the currently checked-in manifest this resolves to default_pool, k8s_rtx6000_1, and external_image_edit_service_pool. Every derived pool must exist with open slots.

4. Compute-cluster GPUs — check the cluster (using the cluster connection established above):

   # Per node: allocatable GPUs minus GPUs requested by pods in every namespace.
   kubectl get nodes -o json > /tmp/iaa-nodes.json
   kubectl get pods -A --field-selector=status.phase!=Succeeded,status.phase!=Failed \
     -o json > /tmp/iaa-pods.json
   python3 - <<'PY'
   import json
   alloc = {n["metadata"]["name"]: int(n["status"].get("allocatable", {}).get("nvidia.com/gpu", 0))
            for n in json.load(open("/tmp/iaa-nodes.json"))["items"]}
   used = dict.fromkeys(alloc, 0)
   for p in json.load(open("/tmp/iaa-pods.json"))["items"]:
       node = p["spec"].get("nodeName")
       if node in used:
           for c in p["spec"].get("containers", []):
               used[node] += int(c.get("resources", {}).get("requests", {}).get("nvidia.com/gpu", 0))
   for node in sorted(alloc):
       print(f"{node}: {alloc[node] - used[node]} free / {alloc[node]} allocatable")
   print("cluster free GPUs:", sum(alloc.values()) - sum(used.values()))
   PY
   

   The compute cluster is shared — other users' runs and other namespaces may hold GPUs. Report GPUs as free-versus-total from this calculation, not just allocatable. Pending pods are not counted because they hold no node yet; mention any that are waiting for GPUs, since they will compete for the next free one. External mode needs no GPUs for inference — every task pod (augmentation, cosmos_post_processing, event_and_person_attribute_search) runs on a CPU profile, unlike EVG's k8s_gpu_task-profiled auto-labeling stages.

   Internal mode deploys two GPU endpoints, not three. service_lifecycle has exactly two services — vlm_service and image_edit_service — because one deployed Qwen serves both the multimodal and the text-only traffic: in internal mode the payload model assigns llm_service_url = vlm_service_url and llm_model = vlm_model. There is no separate llm_service endpoint to deploy. So one replica of each needs two allocatable GPUs, more only if a replicas value is raised. Derive this from the manifest rather than trusting this number — count components whose deployment_profile resolves to a profile with configuration.gpu: true, summing gpu_count where set (default 1).

5. Stale failed pods — before triggering, check for accumulated failed pods in the compute namespace and report them. They are retained by design and do not affect run correctness, but they consume namespace quota and clutter log searches: