Ensure Node Auto-Repair Is Enabled For GKE Nodes

More Info:

Enable node auto-repair so GKE automatically repairs nodes that fail health checks, keeping nodes healthy and reducing manual maintenance. Unhealthy nodes can degrade workload availability.

Risk Level

Low

Address

Security

Compliance Standards

• CIS GKE

Triage and Remediation

Remediation

Remediation

Manual Steps

1. On any machine with gcloud installed and access to the project, list all node pools per cluster and location to scope what to review:

   gcloud container clusters list --project PROJECT_ID \
     --format="table(name,location)"

   For each cluster:

   gcloud container node-pools list \
     --cluster CLUSTER_NAME \
     --location LOCATION \
     --project PROJECT_ID \
     --format="table(name,config.machineType,initialNodeCount)"

2. For each node pool, retrieve and record its management settings, including auto-repair status:

   gcloud container node-pools describe POOL_NAME \
     --cluster CLUSTER_NAME \
     --location LOCATION \
     --project PROJECT_ID \
     --format json | jq '.name,.config.machineType,.initialNodeCount,.management'

   Focus on .management.autoRepair.

3. Decide which node pools must have auto-repair enabled, considering:

   • Workload criticality (production vs. dev/test)
   • Tolerance for automatic node replacement and potential pod rescheduling
   • Any special nodes where manual intervention is required (e.g., tightly coupled with external systems, stateful workloads without proper PodDisruptionBudgets).

4. For each node pool where .management.autoRepair is false and automatic health-based repair is acceptable, enable node auto-repair:

   gcloud container node-pools update POOL_NAME \
     --cluster CLUSTER_NAME \
     --location LOCATION \
     --project PROJECT_ID \
     --enable-autorepair

5. For any node pool where you intentionally choose not to enable auto-repair, document:

   • The node pool name and cluster
   • The business/technical justification (e.g., legacy workloads, special ops runbooks)
   • The alternative process for detecting and repairing unhealthy nodes (monitoring alerts, manual runbooks).

6. Verify the final state for all node pools after changes:

   gcloud container node-pools describe POOL_NAME \
     --cluster CLUSTER_NAME \
     --location LOCATION \
     --project PROJECT_ID \
     --format json | jq '.management'

   Confirm .autoRepair is true for node pools where it should be enabled and that exceptions are explicitly documented.

Using kubectl

kubectl cannot enable or configure GKE node auto-repair because it is a managed control-plane / node pool setting controlled via Google Cloud (console, gcloud, or IaC), not a Kubernetes API object. Use the cloud provider configuration as described in the Manual Steps section to remediate this finding.

Automation

#!/usr/bin/env bash
# Check GKE node auto-repair status for all clusters in a project (optionally filtered by location).
# Requires: gcloud, jq

set -euo pipefail

PROJECT_ID="<YOUR_PROJECT_ID>"   # e.g. my-prod-project
LOCATION_FILTER=""               # e.g. "us-central1" or leave empty for all locations

gcloud config set project "${PROJECT_ID}" >/dev/null 2>&1

echo "Listing clusters in project: ${PROJECT_ID}" >&2

# Get all clusters (optionally filtered by location)
if [[ -n "${LOCATION_FILTER}" ]]; then
  CLUSTERS_JSON=$(gcloud container clusters list \
    --project "${PROJECT_ID}" \
    --format=json \
    --filter="location:${LOCATION_FILTER}")
else
  CLUSTERS_JSON=$(gcloud container clusters list \
    --project "${PROJECT_ID}" \
    --format=json)
fi

if [[ "${CLUSTERS_JSON}" == "[]" || -z "${CLUSTERS_JSON}" ]]; then
  echo "No clusters found for project ${PROJECT_ID} (location filter: '${LOCATION_FILTER}')" >&2
  exit 0
fi

echo "cluster,location,node_pool,auto_repair_enabled" 

echo "${CLUSTERS_JSON}" | jq -r '.[] | @base64' | while read -r cluster_b64; do
  _jq() { echo "${cluster_b64}" | base64 --decode | jq -r "${1}"; }

  CLUSTER_NAME=$(_jq '.name')
  LOCATION=$(_jq '.location')

  # List node pools for this cluster
  NODEPOOLS_JSON=$(gcloud container node-pools list \
    --cluster "${CLUSTER_NAME}" \
    --location "${LOCATION}" \
    --project "${PROJECT_ID}" \
    --format=json)

  if [[ "${NODEPOOLS_JSON}" == "[]" || -z "${NODEPOOLS_JSON}" ]]; then
    echo "${CLUSTER_NAME},${LOCATION},<no-node-pools>,N/A"
    continue
  fi

  echo "${NODEPOOLS_JSON}" | jq -r '.[] | @base64' | while read -r np_b64; do
    _np_jq() { echo "${np_b64}" | base64 --decode | jq -r "${1}"; }

    NP_NAME=$(_np_jq '.name')
    AUTO_REPAIR=$(_np_jq '.management.autoRepair')

    # AUTO_REPAIR will typically be "true" or "false", but handle null explicitly
    if [[ "${AUTO_REPAIR}" == "true" ]]; then
      AUTO_REPAIR_STATUS="true"
    elif [[ "${AUTO_REPAIR}" == "false" ]]; then
      AUTO_REPAIR_STATUS="false"
    else
      AUTO_REPAIR_STATUS="null"
    fi

    echo "${CLUSTER_NAME},${LOCATION},${NP_NAME},${AUTO_REPAIR_STATUS}"
  done
done

How to run (from any machine with gcloud access to the project):

1. Save as check-gke-autorepair.sh and make executable:

   chmod +x check-gke-autorepair.sh

2. Edit the PROJECT_ID (and optionally LOCATION_FILTER) variables in the script.
3. Run:

   ./check-gke-autorepair.sh

Example output:

cluster,location,node_pool,auto_repair_enabled
prod-cluster,us-central1,default-pool,true
prod-cluster,us-central1,spot-pool,false
dev-cluster,us-central1-b,default-pool,null

Interpretation (what indicates a problem):

• Any row where auto_repair_enabled is false means that node auto-repair is explicitly disabled for that node pool and should be reviewed.
• Any row where auto_repair_enabled is null (missing) means the management settings could not be determined as expected and should be investigated manually.
• For CISGKE 5.5.2, all production node pools should typically show auto_repair_enabled as true unless there is a documented exception.