name: gke-alert-configuration metadata: version: "1.0.0" category: CloudInfrastructure canonical_source: https://github.com/google/skills/tree/main/skills/cloud/gke-alert-configuration description: >- Configures alerting policies in Terraform for Google Kubernetes Engine (GKE) clusters, workloads, and services using PromQL and Google Cloud Managed Service for Prometheus. Use when writing, analyzing, validating, or deploying Terraform alerting policies to monitor GKE service latency, traffic, error rates using Multi-Window Multi-Burn-Rate SLO alerts, memory saturation, and cluster health such as CrashLoopBackOff and Node NotReady conditions. Don't use for non-GKE compute runtimes such as standalone Compute Engine VMs or standalone Cloud Run services without GKE.

GKE Alert Configuration

This skill provides guidelines and best practices for creating robust, high-signal alerting policies for Google Kubernetes Engine workloads using Google Cloud Managed Service for Prometheus and Terraform. It ensures comprehensive coverage of the 4 Golden Signals and key cluster health metrics while minimizing alert noise.

Critical Rules

• Negative Triggers and Scope Redirection for Non-GKE Standalone Runtimes:

  • This skill is strictly scoped to Google Kubernetes Engine (GKE) workloads, clusters, and services using PromQL and Google Cloud Managed Service for Prometheus.
  • Do not use for non-GKE compute runtimes, such as standalone Compute Engine virtual machines or standalone Cloud Run services without GKE.
  • STOP AND RESPOND DIRECTLY (Do Not Edit Files): When the user requests alert configuration for non-GKE compute infrastructure:
    1. Do not write, create, edit, or validate any Terraform files on disk.
    2. Immediately stop and respond directly to the user in chat:
       • Explicitly Clarify Out-of-Scope: State clearly that standalone Compute Engine virtual machine monitoring or standalone Cloud Run monitoring is out of scope for this GKE-specific PromQL alerting skill, which is designed specifically for GKE workloads using Google Cloud Managed Service for Prometheus and PromQL.
       • Do Not Generate GKE PromQL Alerts: Do not create or generate Kubernetes PromQL alert policies or fabricate Kubernetes container, pod, or node resources for non-GKE infrastructure.
       • Redirect the User: Guide and redirect the user to standard Google Cloud Monitoring metrics, such as compute.googleapis.com/instance/cpu/utilization or run.googleapis.com/request_latencies, using standard google_monitoring_alert_policy with condition_threshold or MQL, or recommend the relevant specialized Cloud observability skill.

• Mandatory kube-state-metrics (KSM) Cost Guardrail:

  • Deploying open-source kube-state-metrics in Google Cloud Managed Service for Prometheus incurs billable metric ingestion costs.
  • STOP AND ASK PERMISSION FIRST (Do Not Edit Files): When a requested alert rule relies on Tier 2 KSM metrics (such as kube_cronjob_*, kube_pod_status_phase, kube_persistentvolume_*, kube_deployment_*, kube_statefulset_*, kube_job_*, or kube_daemonset_*), do not write, create, edit, or validate any Terraform files or generate alert policies before obtaining user approval.
  • Instead, you must immediately stop and respond directly to the user to:
    1. Alert the user that the requested alert requires kube-state-metrics.
    2. Explain the cost impact: Detail that kube-state-metrics incurs billable sample ingestion costs in Google Cloud Managed Service for Prometheus.
    3. Ask for explicit permission: Ask the user for explicit permission before assuming, enabling, or generating KSM-dependent alert configurations.
    4. Recommend filtering or allowlisting: Suggest and recommend filtering or allowlisting only the specific required metrics, such as using a PodMonitoring resource with metricRelabeling (action: keep) or KSM --metric-allowlist to minimize ingestion costs. Provide a concrete allowlist example.
  • Always prefer Non-KSM Native Alternatives (Tier 1 cAdvisor or native GKE metrics documented in metrics_and_alerts_catalog.md) whenever possible, such as using container_memory_working_set_bytes and container_spec_memory_limit_bytes instead of kube_pod_container_resource_limits.
  • Explicit Tier and Cost Surcharge Identification in Response: In every response where you generate or recommend an alerting policy, you must explicitly state its classification tier and cost impact:
    • Tier 1 native or standard metric (GKE built-in metrics, cAdvisor container_*, kubelet volume stats, kubelet node conditions, and control-plane metrics; see metrics_and_alerts_catalog.md): State that it is a Tier 1 native or standard metric with zero KSM cost surcharge.
    • Tier 2 KSM metric: State that it is a Tier 2 KSM-dependent metric and follow the permission and allowlisting guardrail above. (Tip: Generally, metrics with the kube_ prefix that represent resource state or metadata belong to Tier 2).

• Plan-Validate-Execute Loop for Approved File Edits: When modifying, adding, or merging approved Terraform files on disk in a workspace, follow the three-phase workflow:

  1. Plan: Draft a structured change plan (changes.json) containing proposed policy resource names, PromQL expressions, grouping labels, and durations.
  2. Validate: Run the pre-edit validation script (python3 scripts/validate_config.py --plan changes.json) to verify PromQL grammar, lookback windows, duration rules, and ensure no duplicate signals exist.
  3. Execute: After the plan passes validation, apply or merge changes in-place into the target Terraform configuration (alerts.tf).
  4. Note: When answering questions or providing Terraform snippets directly in chat where no disk modification is requested, output the complete, valid Terraform HCL block in your response.

• Configure the 4 Golden Signals and Cluster Health: Always ensure the target Kubernetes workload or service has the following alerting coverage:

  1. Latency (P95 response time)
  2. Errors (Multi-Window Multi-Burn-Rate SLO alerts, such as Fast Burn 1 hour / 5 minutes with factor 14.4, Slow Burn 6 hours / 30 minutes with factor 6.0; do not use simple static ratios)
  3. Traffic (Sudden drop or complete metric disappearance using absent() or default 0 syntax, or overload spikes)
  4. Saturation (Memory Limit Utilization Only): When describing or configuring alert policies for a cluster or project, include ONLY Memory Saturation (container_memory_working_set_bytes / container_spec_memory_limit_bytes). Do NOT include CPU saturation alerts or list container_cpu_usage_seconds_total as an alert metric because CPU is compressible and throttled by CFS quotas rather than causing uncompressible fatal termination (OOM).
  5. Cluster Health (Pod CrashLooping, Node NotReady)

• PromQL Only (Managed Prometheus): You must use condition_prometheus_query_language with PromQL. Do NOT use MQL or standard condition_threshold unless explicitly requested. Google Cloud Managed Service for Prometheus is the standard telemetry ingestion path for GKE.

• Terraform Only: Write the generated observability configuration ONLY as Terraform (.tf) files, such as alerts.tf and variables.tf.

• Dynamic Multi-Resource Alerting (No Hardcoding): You must not hardcode specific pod names, node names, or service names in alerting conditions unless explicitly requested. Alerting policies must be written to cover resources dynamically:

  • Always use grouping aggregations (by (cluster, namespace, service, pod, container)) instead of filtering to a single instance. This allows a single alert policy to dynamically track each service or pod separately.
  • Always declare and use Terraform variables for project_id, cluster_name, and namespace (var.project_id, var.cluster_name, var.namespace) to make the configuration reusable across environments. Always define these variables in variables.tf (or within the configuration) and reference all three in policies or PromQL label matchers.

• No Redundant Duration Windows on Lookbacks:

  • When PromQL expressions already use an aggregated lookback window (such as increase(...[15m]) > 3 or multi-window SLO burn rates), the query time window already smooths out transient spikes.
  • Adding a Terraform duration on top of a PromQL lookback window increases the Mean Time to Detect (MTTD) without providing additional smoothing benefits.
  • In these cases, set Terraform duration = "0s" (or "60s"). Do not enforce duration = "300s" on top of [15m], which delays critical crashloop alerts by up to 20 minutes total (15 minutes + 5 minutes).
  • Use duration = "300s" only on instantaneous gauge conditions, such as kube_node_status_condition == 0.

• Use SLO Burn Rates Instead of Simple Ratios: For error rate alerting, always generate Multi-Window Multi-Burn-Rate (MWMBR) SLO alerts (such as 14.4x burn rate over 1 hour and 5 minute windows for a 99% SLO) rather than simple error rate ratios (rate(5xx)/rate(total) > 0.05), which produce excessive false alarms on low traffic.

• Robust Traffic Drop Detection (absent() / default 0): When monitoring for traffic drops to zero, do not use rate(...) == 0 alone because Prometheus time series disappear completely when no requests occur (evaluating to an empty vector rather than 0). Use default 0 syntax, such as sum(rate(...[5m])) default 0 == 0, or absent(...) == 1.

• Notification Channels: By default, never configure any notification channels without user input. If the user explicitly provides a notification channel, configure the alerts to use it. Otherwise, you must prompt the user in your response to ask if they would like to configure one.

• Consult GKE Metrics and Open-Source Alerts Catalog: When designing or generating evaluation suites or alerting policies, consult metrics_and_alerts_catalog.md for public GKE metrics (kubernetes.io/) and open-source Kubernetes alerts (awesome-prometheus-alerts).

• Plain English Response: You must include a plain English explanation for what the alerts do in your response. Explain what the alert measures, what the threshold represents, and what a trigger indicates.

• User Labels: Include a user_labels block in all google_monitoring_alert_policy resources to track policies created by this skill:

  user_labels = {
    created-with-google-skill = "g