> ## Documentation Index > Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Ensure Node Auto-Repair Is Enabled For GKE Nodes ### More Info: Enable node auto-repair so GKE automatically repairs nodes that fail health checks, keeping nodes healthy and reducing manual maintenance. Unhealthy nodes can degrade workload availability. ### Risk Level Low ### Address Security ### Compliance Standards * CIS GKE ### Triage and Remediation ### Remediation 1. On any machine with `gcloud` installed and access to the project, list all node pools per cluster and location to scope what to review: ```bash theme={null} gcloud container clusters list --project PROJECT_ID \ --format="table(name,location)" ``` For each cluster: ```bash theme={null} gcloud container node-pools list \ --cluster CLUSTER_NAME \ --location LOCATION \ --project PROJECT_ID \ --format="table(name,config.machineType,initialNodeCount)" ``` 2. For each node pool, retrieve and record its management settings, including auto-repair status: ```bash theme={null} gcloud container node-pools describe POOL_NAME \ --cluster CLUSTER_NAME \ --location LOCATION \ --project PROJECT_ID \ --format json | jq '.name,.config.machineType,.initialNodeCount,.management' ``` Focus on `.management.autoRepair`. 3. Decide which node pools must have auto-repair enabled, considering: * Workload criticality (production vs. dev/test) * Tolerance for automatic node replacement and potential pod rescheduling * Any special nodes where manual intervention is required (e.g., tightly coupled with external systems, stateful workloads without proper PodDisruptionBudgets). 4. For each node pool where `.management.autoRepair` is `false` and automatic health-based repair is acceptable, enable node auto-repair: ```bash theme={null} gcloud container node-pools update POOL_NAME \ --cluster CLUSTER_NAME \ --location LOCATION \ --project PROJECT_ID \ --enable-autorepair ``` 5. For any node pool where you intentionally choose not to enable auto-repair, document: * The node pool name and cluster * The business/technical justification (e.g., legacy workloads, special ops runbooks) * The alternative process for detecting and repairing unhealthy nodes (monitoring alerts, manual runbooks). 6. Verify the final state for all node pools after changes: ```bash theme={null} gcloud container node-pools describe POOL_NAME \ --cluster CLUSTER_NAME \ --location LOCATION \ --project PROJECT_ID \ --format json | jq '.management' ``` Confirm `.autoRepair` is `true` for node pools where it should be enabled and that exceptions are explicitly documented. `kubectl` cannot enable or configure GKE node auto-repair because it is a managed control-plane / node pool setting controlled via Google Cloud (console, `gcloud`, or IaC), not a Kubernetes API object. Use the cloud provider configuration as described in the Manual Steps section to remediate this finding. ```bash theme={null} #!/usr/bin/env bash # Check GKE node auto-repair status for all clusters in a project (optionally filtered by location). # Requires: gcloud, jq set -euo pipefail PROJECT_ID="" # e.g. my-prod-project LOCATION_FILTER="" # e.g. "us-central1" or leave empty for all locations gcloud config set project "${PROJECT_ID}" >/dev/null 2>&1 echo "Listing clusters in project: ${PROJECT_ID}" >&2 # Get all clusters (optionally filtered by location) if [[ -n "${LOCATION_FILTER}" ]]; then CLUSTERS_JSON=$(gcloud container clusters list \ --project "${PROJECT_ID}" \ --format=json \ --filter="location:${LOCATION_FILTER}") else CLUSTERS_JSON=$(gcloud container clusters list \ --project "${PROJECT_ID}" \ --format=json) fi if [[ "${CLUSTERS_JSON}" == "[]" || -z "${CLUSTERS_JSON}" ]]; then echo "No clusters found for project ${PROJECT_ID} (location filter: '${LOCATION_FILTER}')" >&2 exit 0 fi echo "cluster,location,node_pool,auto_repair_enabled" echo "${CLUSTERS_JSON}" | jq -r '.[] | @base64' | while read -r cluster_b64; do _jq() { echo "${cluster_b64}" | base64 --decode | jq -r "${1}"; } CLUSTER_NAME=$(_jq '.name') LOCATION=$(_jq '.location') # List node pools for this cluster NODEPOOLS_JSON=$(gcloud container node-pools list \ --cluster "${CLUSTER_NAME}" \ --location "${LOCATION}" \ --project "${PROJECT_ID}" \ --format=json) if [[ "${NODEPOOLS_JSON}" == "[]" || -z "${NODEPOOLS_JSON}" ]]; then echo "${CLUSTER_NAME},${LOCATION},,N/A" continue fi echo "${NODEPOOLS_JSON}" | jq -r '.[] | @base64' | while read -r np_b64; do _np_jq() { echo "${np_b64}" | base64 --decode | jq -r "${1}"; } NP_NAME=$(_np_jq '.name') AUTO_REPAIR=$(_np_jq '.management.autoRepair') # AUTO_REPAIR will typically be "true" or "false", but handle null explicitly if [[ "${AUTO_REPAIR}" == "true" ]]; then AUTO_REPAIR_STATUS="true" elif [[ "${AUTO_REPAIR}" == "false" ]]; then AUTO_REPAIR_STATUS="false" else AUTO_REPAIR_STATUS="null" fi echo "${CLUSTER_NAME},${LOCATION},${NP_NAME},${AUTO_REPAIR_STATUS}" done done ``` How to run (from any machine with `gcloud` access to the project): 1. Save as `check-gke-autorepair.sh` and make executable: ```bash theme={null} chmod +x check-gke-autorepair.sh ``` 2. Edit the `PROJECT_ID` (and optionally `LOCATION_FILTER`) variables in the script. 3. Run: ```bash theme={null} ./check-gke-autorepair.sh ``` Example output: ```text theme={null} cluster,location,node_pool,auto_repair_enabled prod-cluster,us-central1,default-pool,true prod-cluster,us-central1,spot-pool,false dev-cluster,us-central1-b,default-pool,null ``` Interpretation (what indicates a problem): * Any row where `auto_repair_enabled` is `false` means that node auto-repair is explicitly disabled for that node pool and should be reviewed. * Any row where `auto_repair_enabled` is `null` (missing) means the management settings could not be determined as expected and should be investigated manually. * For CISGKE 5.5.2, all production node pools should typically show `auto_repair_enabled` as `true` unless there is a documented exception.