GKE Reliability Skill | Agent Skills

GKE Reliability

Routing Note: To generate Kubernetes YAML manifests (Deployment, StatefulSet, Service, ConfigMap, HTTPRoute, PodDisruptionBudget), open gke-manifest-generation/SKILL.md.

This reference covers high availability and reliability configuration for GKE clusters and workloads.

MCP Tools: get_cluster, get_k8s_resource, describe_k8s_resource, apply_k8s_manifest, list_k8s_events

Golden Path Reliability Defaults

| Setting | Golden Path Value | Notes | | ---------------- | --------------------- | -------------------------------- | | Cluster type | Regional (4 zones: | Control plane replicated across | : : us-central1-a/b/c/f) : zones : | Upgrade strategy | SURGE (maxSurge: 1) | Rolling upgrades with extra | : : : capacity : | Auto-repair | true | Unhealthy nodes replaced | : : : automatically : | Auto-upgrade | true | Nodes follow control plane | : : : version : | Release channel | REGULAR | Balanced freshness and stability | | Stateful HA | Enabled | Leader election for stateful | : : : workloads :

Workflows

1. Verify Cluster High Availability

# MCP (preferred)
get_cluster(name="projects/<PROJECT>/locations/<REGION>/clusters/<CLUSTER>",
  readMask="location,locations,nodePools.locations")

# gcloud fallback
gcloud container clusters describe <CLUSTER> --region <REGION> \
  --format="json(location, locations)" \
  --quiet

If location is a region (e.g., us-central1), the control plane is regional
If locations has multiple entries, nodes span multiple zones

2. Pod Disruption Budgets (PDBs)

PDBs ensure minimum pod availability during voluntary disruptions (node upgrades, autoscaler scale-down).

Check existing PDBs:

# MCP (preferred)
get_k8s_resource(parent="...", resourceType="poddisruptionbudget")

# kubectl fallback
kubectl get pdb --all-namespaces

Create PDB:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: default
spec:
  minAvailable: 2       # Or use maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app

Every production Deployment with 2+ replicas should have a PDB.

3. Health Probes

Every production container should have liveness and readiness probes. Startup probes are recommended for slow-starting apps.

Check existing probes:

# MCP (preferred)
describe_k8s_resource(parent="...", resourceType="deployment", name="<APP>", namespace="<NS>")

# kubectl fallback
kubectl get deployment <APP> -n <NS> -o yaml | grep -E "livenessProbe|readinessProbe|startupProbe"

Recommended probe configuration:

spec:
  containers:
  - name: app
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /readyz
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
    startupProbe:             # For slow-starting apps
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 30    # 30 * 5s = 150s max startup time

Readiness: Determines when a pod can accept traffic
Liveness: Determines when to restart a container
Startup: Disables liveness/readiness until the app is ready (prevents premature restarts)

4. Graceful Shutdown

Ensure applications handle SIGTERM and drain in-flight requests:

spec:
  terminationGracePeriodSeconds: 30    # Default; increase for long-running requests
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 5"]  # Allow LB to deregister

5. Topology Spread Constraints

Distribute pods across zones and nodes to survive failures:

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: my-app

Zone spread (DoNotSchedule): Hard requirement -- pods must be balanced across zones
Node spread (ScheduleAnyway): Best-effort -- prefer distribution but don't block scheduling

6. Replicas

| Workload Type | Minimum Replicas | Reason | | -------------------- | -------------------- | ------------------------------ | | Stateless web/API | 2 | Survive single pod/node | : : : failure : | Critical services | 3 | Survive zone failure with zone | : : : spread : | Stateful (databases) | 3 (with replication) | Application-level quorum | | Batch/jobs | 1 | Ephemeral by nature |

Best Practices & Production Guidelines

Regional clusters for production: Always use regional clusters to survive zone failures.
PDBs for everything: Every production workload with 2+ replicas needs a PodDisruptionBudget (PDB) to protect against voluntary disruptions.
Probes with Explicit Timeouts: Every production container must have both liveness and readiness probes defined. Always explicitly define initialDelaySeconds, periodSeconds, and timeoutSeconds for all probes. Never rely on the Kubernetes default timeout of 1 second if your application requires more, but always set a strict limit to prevent hanging connections.
Zone spreading: Use topology spread constraints to distribute pods across failure domains (zones and nodes).
Graceful shutdown: Handle SIGTERM and set appropriate terminationGracePeriodSeconds with a preStop sleep hook to allow load balancer deregistration.
Maintenance windows: Schedule upgrades during low-traffic periods (see the gke-upgrades skill).

Agent Skills: GKE Reliability

Install this agent skill to your local

Skill Files