Skip to main content

Ensure Node Auto-Repair Is Enabled For Gke Nodes

More Info:​

Nodes In A Degraded State Are An Unknown Quantity And So May Pose A Security Risk

Risk Level​

Medium

Address​

Security

Compliance Standards​

  • CIS GKE

Triage and Remediation​

Remediation​

Manual Steps

  1. List all node pools and identify those without auto-repair

    • Run on: any machine with gcloud configured and access to the project.
    gcloud container node-pools list \
      --cluster CLUSTER_NAME \
      --location LOCATION \
      --project PROJECT_ID

    For each NODE_POOL_NAME in the output, gather its management settings:

    gcloud container node-pools describe NODE_POOL_NAME \
      --cluster CLUSTER_NAME \
      --location LOCATION \
      --project PROJECT_ID \
      --format json | jq '.name, .management'

    Record which pools have "autoRepair": false or missing.

  2. Review workload criticality and disruption tolerance per node pool

    • For each node pool with auto-repair disabled, determine what runs on it (e.g., critical system workloads, stateful apps, batch jobs) and the tolerance for node reboots or replacement.
    • If the pool hosts critical, always-on, or security-sensitive workloads, plan to enable auto-repair unless there is a strong operational reason not to.

  3. Check for alternative health and repair mechanisms if you decide not to enable auto-repair

    • Confirm whether there are other processes that detect and remediate unhealthy nodes (e.g., external automation, strict SLOs, manual rotation).
    • If such mechanisms are weak, undocumented, or slow, favor enabling auto-repair to reduce the risk of long-lived degraded nodes.

  4. Plan maintenance window and communication for enabling auto-repair

    • Enabling auto-repair can trigger node recreation when nodes are deemed unhealthy, which may cause pod rescheduling and brief interruptions.
    • Ensure pod disruption budgets, readiness/liveness probes, and multi-zone/replicated deployments are in place so workloads tolerate node replacement.

  5. Enable node auto-repair on selected node pools

    • Run on: any machine with gcloud configured and access to the project.
    • For each node pool you decide should have auto-repair:
    gcloud container node-pools update NODE_POOL_NAME \
      --cluster CLUSTER_NAME \
      --location LOCATION \
      --project PROJECT_ID \
      --enable-autorepair

  6. Verify auto-repair is enabled for the intended node pools

    • Re-run for each updated node pool:
    gcloud container node-pools describe NODE_POOL_NAME \
      --cluster CLUSTER_NAME \
      --location LOCATION \
      --project PROJECT_ID \
      --format json | jq '.management'
    • Confirm "autoRepair": true is present for all node pools where you decided to enforce auto-repair, and document any remaining exceptions with justification.
Using kubectl

kubectl cannot enable or configure GKE node auto-repair because it is a managed control-plane / cloud provider setting applied at the node pool level, not a Kubernetes API object. To remediate this finding, make the change via the GCP console, gcloud CLI, or your IaC as described in the Manual Steps section.

Automation
#!/usr/bin/env bash
#
# Report GKE node auto-repair status for all node pools in all clusters
# in the current gcloud project.
#
# Requirements:
#   - gcloud CLI authenticated and configured with a project
#   - jq installed
#
# Run on: any machine with gcloud and jq installed and access to the project.

set -euo pipefail

PROJECT_ID="$(gcloud config get-value project 2>/dev/null || true)"
if [[ -z "${PROJECT_ID}" ]]; then
  echo "ERROR: No active gcloud project configured. Run:"
  echo "  gcloud config set project YOUR_PROJECT_ID"
  exit 1
fi

echo "Using project: ${PROJECT_ID}"
echo

# List all GKE clusters (both zonal and regional)
CLUSTERS_JSON="$(gcloud container clusters list --project "${PROJECT_ID}" --format=json)"

if [[ "$(echo "${CLUSTERS_JSON}" | jq 'length')" -eq 0 ]]; then
  echo "No GKE clusters found in project ${PROJECT_ID}."
  exit 0
fi

# Header
printf "%-40s %-20s %-40s %-15s %-15s\n" \
  "CLUSTER" "LOCATION" "NODE_POOL" "AUTO-REPAIR" "AUTO-UPGRADE"
printf "%0.s-" {1..140}
echo

# Iterate through clusters and their node pools
echo "${CLUSTERS_JSON}" | jq -r '.[] | @base64' | while read -r cluster_b64; do
  _jq() {
    echo "${cluster_b64}" | base64 --decode | jq -r "${1}"
  }

  CLUSTER_NAME="$(_jq '.name')"
  LOCATION="$(_jq '.location')"

  # Get all node pools for this cluster
  NPS_JSON="$(gcloud container node-pools list \
    --cluster "${CLUSTER_NAME}" \
    --location "${LOCATION}" \
    --project "${PROJECT_ID}" \
    --format=json)"

  if [[ "$(echo "${NPS_JSON}" | jq 'length')" -eq 0 ]]; then
    printf "%-40s %-20s %-40s %-15s %-15s\n" \
      "${CLUSTER_NAME}" "${LOCATION}" "<no node pools>" "N/A" "N/A"
    continue
  fi

  echo "${NPS_JSON}" | jq -r '.[] | @base64' | while read -r np_b64; do
    _np_jq() {
      echo "${np_b64}" | base64 --decode | jq -r "${1}"
    }

    NP_NAME="$(_np_jq '.name')"
    MGMT_JSON="$(_np_jq '.management')"
    AUTO_REPAIR="$(echo "${MGMT_JSON}" | jq -r '.autoRepair // "false"')"
    AUTO_UPGRADE="$(echo "${MGMT_JSON}" | jq -r '.autoUpgrade // "false"')"

    printf "%-40s %-20s %-40s %-15s %-15s\n" \
      "${CLUSTER_NAME}" "${LOCATION}" "${NP_NAME}" "${AUTO_REPAIR}" "${AUTO_UPGRADE}"
  done
done

cat <<'EOF'

Interpretation:

- For CIS GKE 5.5.2, node pools where AUTO-REPAIR == "false" (or missing)
  are problematic and should be reviewed.
- Expected/healthy state for compliance: AUTO-REPAIR == "true" for all
  security-sensitive or production node pools.

Use this script's output to identify node pools that need manual review and
a policy decision on enabling auto-repair. Do not assume auto-repair can be
blindly enabled everywhere without considering workload and maintenance
requirements.
EOF

Additional Reading:​