---

name: wiztelemetry-tracing description: Use when working with WizTelemetry Tracing extension for KubeSphere, including installation, configuration, and tracing query API

WizTelemetry Tracing

Overview

WizTelemetry Tracing is an extension component in the KubeSphere Observability Platform that provides distributed tracing functionality based on the OpenTelemetry standard.

When to Use

• Installing or configuring the WizTelemetry Tracing extension
• Understanding tracing architecture (Generator + Operator + Collector + Agent)
• Using the tracing query API to query traces, spans, service graphs
• Configuring OpenTelemetry auto-instrumentation for applications

Components

| Component | Description | Default Enabled | |-----------|-------------|-----------------| | generator | WizTelemetry Tracing Generator: generates service graphs from tracing data (StatefulSet) | true | | operator | OpenTelemetry Operator: manages OpenTelemetry Collector and auto-instrumentation | true | | collector | OpenTelemetry Collector for WizTelemetry: receives traces, exports to Vector and OpenSearch | true | | agent | WizTelemetry Tracing Agent: collects tracing data from local log files (DaemonSet) | false | | demo | OpenTelemetry Demo: generates sample tracing data for demonstration | false |

Dependencies

• WizTelemetry Platform Service (whizard-telemetry): Required
• OpenSearch (opensearch): Required

Installation

Prerequisites

REQUIRED: Complete all steps in order before generating InstallPlan.

Step 1: Get Available Clusters and Confirm Target

CRITICAL: DO NOT proceed until target clusters are determined.

Step 1.1: Get available clusters

kubectl get clusters -o jsonpath='{.items[*].metadata.name}'

Step 1.2: Determine target clusters

• If user explicitly specified target clusters in the request -> Use those clusters directly, proceed to Step 2
• If user did NOT specify target clusters -> Ask user to confirm which clusters to deploy to, then proceed to Step 2

Ask user (if not specified):

Available clusters: host, dev
Which clusters do you want to deploy WizTelemetry Tracing to?

Step 2: Get Latest Version (if not provided by user)

MUST do this to get the latest version:

kubectl get extensionversions -n kubesphere-system -l kubesphere.io/extension-ref=wiztelemetry-tracing -o jsonpath='{range .items[*]}{.spec.version}{"\n"}{end}' | sort -V | tail -1

This outputs the latest version (e.g., 1.0.6). Note this down - you'll use it in the InstallPlan.

Step 3: Get Generator Endpoint

Get a node IP from the target cluster:

kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="ExternalIP")].address}' 2>/dev/null || \
kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}'

This outputs a node IP (e.g., 192.168.1.100). Use this as the generator endpoint.

If multiple shards (e.g., generator.shardCount: 3), generate all endpoints:

NODE_IP="<NODE_IP>"
for i in 0 1 2; do
  PORT=$((32318 + i))
  echo "  - http://${NODE_IP}:${PORT}"
done

Replace <NODE_IP> with the actual node IP from the previous command.

Step 4: Confirm Configuration with User

Before creating the InstallPlan, confirm the following with the user:

I'll install WizTelemetry Tracing with the following configuration:
- Version: <VERSION>
- Target clusters: <TARGET_CLUSTERS>
- Generator endpoint: http://<NODE_IP>:32318
- OpenSearch endpoint: <OPENSEARCH_ENDPOINT>

Do you want to proceed? (yes/no)

If user confirms, proceed to create the InstallPlan. If not, adjust the configuration based on user feedback.

Step 5: Create InstallPlan

CRITICAL: InstallPlan metadata.name MUST be wiztelemetry-tracing. DO NOT use any other name.

CRITICAL: config field is YAML format. You MUST:

• Use the config structure exactly as shown in the template
• DO NOT add configuration fields that are not shown in the template
• DO NOT modify the structure or hierarchy

CRITICAL: All placeholders MUST be replaced with actual values. DO NOT leave them as placeholders.

Based on your selections:

• Target clusters: User-confirmed cluster names
• OpenSearch endpoint: User-provided (default: https://opensearch-cluster-data.kubesphere-logging-system.svc:9200)
• OpenSearch credentials: User-provided (default user: admin)

Template

apiVersion: kubesphere.io/v1alpha1
kind: InstallPlan
metadata:
  name: wiztelemetry-tracing
  namespace: kubesphere-system
spec:
  extension:
    name: wiztelemetry-tracing
    version: <VERSION>  # From Step 2
  enabled: true
  upgradeStrategy: Manual
  config: |
    global:
      generator:
        endpoints:
          - http://<HOST_NODE_IP>:32318
      storage:
        opensearch:
          auth:
            strategy: basic
            user: <OPENSEARCH_USER>
            password: <OPENSEARCH_PASSWORD>
          endpoints:
            - <OPENSEARCH_ENDPOINT>
    generator:
      shardCount: 1
      service:
        nodePort: 32318
  clusterScheduling:
    placement:
      clusters:
        - <TARGET_CLUSTERS>

Replace placeholders:

• <VERSION>: From Step 2 (e.g., 1.0.6)
• <HOST_NODE_IP>: Node IP from Step 3 (e.g., 192.168.1.100)
• <OPENSEARCH_USER>: OpenSearch username
• <OPENSEARCH_PASSWORD>: OpenSearch password
• <OPENSEARCH_ENDPOINT>: OpenSearch endpoint (e.g., https://opensearch-cluster-data.kubesphere-logging-system.svc:9200)
• <TARGET_CLUSTERS>: User-confirmed cluster names

Enable Agent (Log File Collection)

To enable tracing data collection from local log files, set agent.enabled: true in the config. This deploys a DaemonSet that scans log files and forwards traces to the generator.

Enable Demo

To enable the OpenTelemetry Demo (generates sample trace data), set demo.enabled: true in the config.

Send Traces to Tempo

To forward traces to Tempo instead of OpenSearch, modify the collector.collector.config:

collector:
  enabled: true
  collector:
    config: |
      exporters:
        otlp:
          endpoint: <TEMPO_DISTRIBUTOR_GRPC_ENDPOINT>
          headers:
            x-scope-orgid: wiztelemetry-tracing-ks
          tls:
            insecure: true
      service:
        pipelines:
          traces:
            exporters:
              - otlp
              - otlphttp

Multiple Generator Instances

To scale the generator for higher throughput, increase generator.shardCount and update global.generator.endpoints:

global:
  generator:
    endpoints:
      - http://<HOST_NODE_IP>:32318
      - http://<HOST_NODE_IP>:32319
      - http://<HOST_NODE_IP>:32320

generator:
  shardCount: 3
  service:
    nodePort: 32318

shardCount must match the number of endpoints in global.generator.endpoints. The nodePort of each shard increases in sequence starting from the configured service.nodePort.

Configuration Parameters

Global Parameters

DNS Service

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | global.dnsService | string | coredns | DNS service name |

Generator Global Parameters

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | global.generator.endpoints | list | ["http://<ip>:32318"] | Endpoints of all generator shards. Tracing data is routed to different shards by traceId. |

OpenSearch Storage Parameters

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | global.storage.opensearch.auth.strategy | string | basic | Auth strategy | | global.storage.opensearch.auth.user | string | admin | OpenSearch username | | global.storage.opensearch.auth.password | string | admin | OpenSearch password | | global.storage.opensearch.endpoints | list | | OpenSearch endpoint URLs | | global.storage.opensearch.index.prefix | string | wiz-tracing-span | Index prefix | | global.storage.opensearch.index.timestring | string | %Y.%m.%d | Index time format (strftime pattern) |

Generator Parameters

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | generator.shardCount | int | 1 | Number of generator shards. Must equal the number of entries in global.generator.endpoints. | | generator.image.tag | string | v1.0.2 | Generator image tag | | generator.image.imagePullPolicy | string | IfNotPresent | Image pull policy | | generator.service.nodePort | int | 32318 | NodePort of the first shard. Subsequent shards use consecutive ports. | | generator.resources.limits.cpu | string | 2 | CPU limit | | generator.resources.limits.memory | string | 2000Mi | Memory limit | | generator.resources.requests.cpu | string | 100m | CPU request | | generator.resources.requests.memory | string | 100Mi | Memory request |

Operator Parameters

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | operator.enabled | bool | true | Enable OpenTelemetry Operator | | operator.fullnameOverride | string | wiztelemetry-tracing-operator | Override name for the operator deployment |

Collector Parameters

ISM Policy Parameters

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | collector.ism_policy.enable | bool | true | Enable OpenSearch Index State Management policy | | collector.ism_policy.span_index_pattern | string | *wiz-tracing-span* | Index pattern for span indices | | collector.ism_policy.service_index_pattern | string | *wiz-tracing-service* | Index pattern for service indices | | collector.ism_policy.min_index_age | string | 7d | Minimum index retention period | | collector.ism_policy.span_index_priority | int | 9950 | Index priority for span indices |

Collector Deployment Parameters

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | collector.enabled | bool | true | Enable OpenTelemetry Collector | | collector.collector.mode | string | deployment | Deployment mode (deployment or daemonset) | | collector.collector.replicas | int | 2 | Number of collector replicas | | collector.collector.upgradeStrategy | string | automatic | Upgrade strategy |

Agent Parameters

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | agent.enabled | bool | false | Enable Tracing Agent (DaemonSet) | | agent.dockerRootDir | string | /var/lib/docker | Docker root directory | | agent.config.scanInterval | string | 1m | Log file scan interval | | agent.config.deletionDelay | string | 5m | Delay before deleting processed files | | agent.config.logPath | string | /app/logs/*/*trace*.log | Path pattern for trace log files | | agent.config.podLabelSelector | map | {} | Filter pods by label selector | | agent.config.namespaceLabelSelector | map | {} | Filter namespaces by label selector | | agent.config.includeNamespaces | list | [] | List of namespaces to include | | agent.config.excludeNamespaces | list | [] | List of namespaces to exclude | | agent.resources.limits.cpu | string | 500m | CPU limit | | agent.resources.limits.memory | string | 500Mi | Memory limit | | agent.resources.requests.cpu | string | 10m | CPU request | | agent.resources.requests.memory | string | 10Mi | Memory request | | agent.vector.resources.limits.cpu | string | 2 | Vector sidecar CPU limit | | agent.vector.resources.limits.memory | string | 2000Mi | Vector sidecar memory limit | | agent.vector.resources.requests.cpu | string | 100m | Vector sidecar CPU request | | agent.vector.resources.requests.memory | string | 100Mi | Vector sidecar memory request |