Skip to main content

Gke Clusters Are Not Running Using The Compute Engine

More Info:​

Create And Use Minimally Privileged Service Accounts To Run Gke Cluster Nodes Instead Of Using The Compute Engine Default Service Account. Unnecessary Permissions Could Be Abused In The Case Of A Node Compromise.

Risk Level​

High

Address​

Security

Compliance Standards​

  • CIS GKE

Triage and Remediation​

Remediation​

Manual Steps

  1. Create a minimally privileged node service account (run on any machine with gcloud configured)

    PROJECT_ID="$(gcloud config get-value project)"
    gcloud iam service-accounts create gke-node-sa \
      --display-name="GKE Node Service Account"

    NODE_SA_EMAIL="$(gcloud iam service-accounts list \
      --format='value(email)' \
      --filter='displayName=GKE Node Service Account')"

  2. Grant the required roles to the new service account (run on any machine with gcloud configured)

    gcloud projects add-iam-policy-binding "$PROJECT_ID" \
      --member "serviceAccount:${NODE_SA_EMAIL}" \
      --role roles/monitoring.metricWriter

    gcloud projects add-iam-policy-binding "$PROJECT_ID" \
      --member "serviceAccount:${NODE_SA_EMAIL}" \
      --role roles/monitoring.viewer

    gcloud projects add-iam-policy-binding "$PROJECT_ID" \
      --member "serviceAccount:${NODE_SA_EMAIL}" \
      --role roles/logging.logWriter

  3. Create a new node pool using the new service account (run on any machine with gcloud configured)
    Replace the placeholders with your values.

    CLUSTER_NAME="YOUR_CLUSTER_NAME"
    LOCATION="YOUR_CLUSTER_LOCATION"   # e.g. us-central1, us-central1-a
    NEW_NODE_POOL_NAME="gke-nodepool-min-sa"

    gcloud container node-pools create "$NEW_NODE_POOL_NAME" \
      --cluster="$CLUSTER_NAME" \
      --location="$LOCATION" \
      --service-account="${NODE_SA_EMAIL}"

  4. Migrate workloads to the new node pool (run on any machine with kubectl access)

    • Option A (cordon and drain all old default-SA nodes): 
      # Identify old node pool name(s)
      gcloud container node-pools list \
        --cluster="$CLUSTER_NAME" \
        --location="$LOCATION"

      # For each OLD_NODE_POOL, cordon and drain its nodes
      OLD_NODE_POOL="OLD_NODE_POOL_NAME"

      kubectl get nodes -l cloud.google.com/gke-nodepool="${OLD_NODE_POOL}" -o name | \
        xargs -I {} kubectl cordon {}

      kubectl get nodes -l cloud.google.com/gke-nodepool="${OLD_NODE_POOL}" -o name | \
        xargs -I {} kubectl drain {} --ignore-daemonsets --delete-emptydir-data
    • Confirm pods are rescheduled onto nodes in gke-nodepool-min-sa: 
      kubectl get pods -A -o wide | grep gke-nodepool-min-sa

  5. Delete the old node pool that uses the default service account (run on any machine with gcloud configured)

    OLD_NODE_POOL="OLD_NODE_POOL_NAME"

    gcloud container node-pools delete "$OLD_NODE_POOL" \
      --cluster="$CLUSTER_NAME" \
      --location="$LOCATION" \
      --quiet

  6. Verify remediation (run on any machine with gcloud configured)

    PROJECT_ID="$PROJECT_ID"
    CLUSTER_NAME="$CLUSTER_NAME"
    LOCATION="$LOCATION"
    NODE_POOL="$(gcloud container clusters describe "$CLUSTER_NAME" \
      --location "$LOCATION" \
      --project "$PROJECT_ID" \
      --format='value(nodePools[0].name)')"

    SA="$(gcloud container node-pools describe "$NODE_POOL" \
      --cluster "$CLUSTER_NAME" \
      --location "$LOCATION" \
      --project "$PROJECT_ID" \
      --format='value(config.serviceAccount)')"

    if [[ "$SA" == "default" ]]; then
      echo 'DEFAULT_SA_IN_USE'
    else
      echo 'DEFAULT_SA_NOT_IN_USE'
      echo "Node pool: $NODE_POOL using SA: $SA"
    fi
Using kubectl

kubectl cannot change which Google Cloud service account GKE nodes use; that setting is managed at the cluster/node pool level via the Google Cloud console, gcloud CLI, or IaC. Use those cloud‑provider tools to create a minimally privileged node service account, migrate workloads to a new node pool that uses it, and delete the old default‑SA node pool as described in the Manual Steps section.

Automation
#!/usr/bin/env bash
# Remediation automation for:
# "Ensure GKE clusters are not running using the Compute Engine default service account"
#
# Runs on: any machine with gcloud installed and access to the GCP project.
# Requirements:
#   - gcloud configured (gcloud auth login / gcloud auth application-default login)
#   - roles/container.admin, roles/iam.serviceAccountAdmin, roles/resourcemanager.projectIamAdmin
#
# WHAT THIS SCRIPT DOES (idempotent):
#   1. Ensures a minimally privileged node service account exists in each target project.
#   2. Ensures that SA has the required IAM roles.
#   3. For each specified GKE cluster:
#        - Ensures a node pool exists that uses that SA (creates if missing).
#        - Optionally cordons/drains workloads from default-SA node pools and deletes them.
#   4. Re-runs the CIS audit logic to verify default SA is not in use.
#
# SAFE TO RE-RUN:
#   - SA creation is no-op if it already exists.
#   - IAM bindings are applied additively.
#   - Node pool creation is skipped if pool already exists.
#
# NOTE:
#   - Node pool replacement is per-cluster and may take several minutes.
#   - Deleting old node pools is disruptive (nodes are recreated, pods rescheduled).
#   - Set DELETE_OLD_DEFAULT_POOLS=false to only create the new node pool and leave
#     migration/deletion to a manual change window.

set -euo pipefail

############################################
# USER CONFIGURATION
############################################

# Comma-separated list of cluster specs: "project:location:cluster_name"
# location is either a zone (e.g. us-central1-a) or a region (e.g. us-central1)
# Example:
#   TARGET_CLUSTERS="myproj-1:us-central1:prod-cluster,myproj-1:us-central1-a:dev-cluster"
TARGET_CLUSTERS="${TARGET_CLUSTERS:-}"

# Name to use for the GKE node service account (per project)
NODE_SA_NAME="${NODE_SA_NAME:-gke-node-sa}"

# Display name for the SA (for human readability)
NODE_SA_DISPLAY_NAME="${NODE_SA_DISPLAY_NAME:-GKE Node Service Account}"

# Name to use for the new node pool created with the secure SA
NEW_NODE_POOL_NAME="${NEW_NODE_POOL_NAME:-secure-node-pool}"

# Whether to delete existing node pools that use the default service account (true/false)
DELETE_OLD_DEFAULT_POOLS="${DELETE_OLD_DEFAULT_POOLS:-false}"

# Whether to cordon and drain nodes in old pools before deletion (true/false)
# Only relevant if DELETE_OLD_DEFAULT_POOLS=true and kubectl access is configured.
CORDON_AND_DRAIN_OLD_POOLS="${CORDON_AND_DRAIN_OLD_POOLS:-true}"

############################################
# HELPER FUNCTIONS
############################################

log()  { printf '[%s] %s\n' "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" "$*" >&2; }
die()  { log "ERROR: $*"; exit 1; }

require_cmd() {
  for c in "$@"; do command -v "$c" >/dev/null 2>&1 || die "Command not found: $c"; done
}

get_project_number() {
  local project_id="$1"
  gcloud projects describe "$project_id" --format='value(projectNumber)'
}

ensure_node_sa() {
  local project_id="$1"

  log "Ensuring node service account '$NODE_SA_NAME' exists in project '$project_id'"

  local existing_email
  existing_email="$(gcloud iam service-accounts list \
    --project "$project_id" \
    --filter="displayName:$NODE_SA_DISPLAY_NAME" \
    --format='value(email)' || true)"

  local sa_email
  if [[ -n "$existing_email" ]]; then
    sa_email="$existing_email"
    log "  Found existing SA by display name: $sa_email"
  else
    sa_email="${NODE_SA_NAME}@${project_id}.iam.gserviceaccount.com"
    if gcloud iam service-accounts describe "$sa_email" --project "$project_id" >/dev/null 2>&1; then
      log "  SA already exists by name: $sa_email"
    else
      log "  Creating SA: $sa_email"
      gcloud iam service-accounts create "$NODE_SA_NAME" \
        --project "$project_id" \
        --display-name "$NODE_SA_DISPLAY_NAME"
    fi
  fi

  echo "$sa_email"
}

ensure_sa_roles() {
  local project_id="$1"
  local sa_email="$2"

  log "Ensuring IAM roles for SA '$sa_email' in project '$project_id'"

  local roles=(
    "roles/monitoring.metricWriter"
    "roles/monitoring.viewer"
    "roles/logging.logWriter"
  )

  for role in "${roles[@]}"; do
    log "  Binding $role"
    gcloud projects add-iam-policy-binding "$project_id" \
      --member "serviceAccount:$sa_email" \
      --role "$role" \
      --quiet >/dev/null
  done
}

ensure_secure_node_pool() {
  local project_id="$1"
  local location="$2"
  local cluster_name="$3"
  local sa_email="$4"

  log "Ensuring node pool '$NEW_NODE_POOL_NAME' exists in cluster '$cluster_name' ($project_id / $location) with SA '$sa_email'"

  # Determine if the cluster is regional or zonal based on the location string
  local loc_flag
  if [[ "$location" =~ ^[a-z]+-[a-z0-9]+[0-9]-[a-z]$ ]]; then
    loc_flag="--zone"
  else
    loc_flag="--region"
  fi

  # Check if node pool already exists
  if gcloud container node-pools describe "$NEW_NODE_POOL_NAME" \
      --cluster "$cluster_name" \
      $loc_flag "$location" \
      --project "$project_id" >/dev/null 2>&1; then
    log "  Node pool '$NEW_NODE_POOL_NAME' already exists; skipping creation."
    return 0
  fi

  # Create the new node pool with the specified service account
  log "  Creating node pool '$NEW_NODE_POOL_NAME' with service account '$sa_email'"
  gcloud container node-pools create "$NEW_NODE_POOL_NAME" \
    --cluster "$cluster_name" \
    $loc_flag "$location" \
    --project "$project_id" \
    --service-account="$sa_email" \
    --num-nodes=1 \
    --quiet

  log "  Node pool '$NEW_NODE_POOL_NAME' created."
}

get_node_pools_with_default_sa() {
  local project_id="$1"
  local location="$2"
  local cluster_name="$3"

  local loc_flag
  if [[ "$location" =~ ^[a-z]+-[a-z0-9]+[0-9]-[a-z]$ ]]; then
    loc_flag="--zone"
  else
    loc_flag="--region"
  fi

  # Returns newline-separated list of pools with config.serviceAccount == "default"
  gcloud container node-pools list \
    --cluster "$cluster_name" \
    $loc_flag "$location" \
    --project "$project_id" \
    --format='table[no-heading](name,config.serviceAccount)' \
    | awk '$2=="default"{print $1}'
}

cordon_and_drain_pool_nodes() {
  local project_id="$1"
  local location="$2"
  local cluster_name="$3"
  local node_pool="$4"

  # Requires kubectl and authentication to the cluster
  if ! command -v kubectl >/dev/null 2>&1; then
    log "  kubectl not found; skipping cordon/drain for pool '$node_pool'."
    return 0
  fi

  log "  Getting cluster credentials for cordon/drain"
  local loc_flag
  if [[ "$location" =~ ^[a-z]+-[a-z0-9]+[0-9]-[a]$ ]]; then
    loc_flag="--zone"
  else
    loc_flag="--region"
  fi

  gcloud container clusters get-credentials "$cluster_name" \
    $loc_flag "$location" \
    --project "$project_id" >/dev/null

  log "  Cordon and drain nodes in pool '$node_pool'"
  # Label selector matches the node pool name
  local selector="cloud.google.com/gke-nodepool=${node_pool}"

  # Cordon
  kubectl get nodes -l "$selector" -o name | xargs -r kubectl cordon

  # Drain (ignore daemonsets, delete local data)
  kubectl get nodes -l "$selector" -o name | \
    xargs -r -n1 kubectl drain --ignore-daemonsets --delete-emptydir-data --force
}

delete_default_sa_pools() {
  local project_id="$1"
  local location="$2"
  local cluster_name="$3"

  local pools
  pools="$(get_node_pools_with_default_sa "$project_id" "$location" "$cluster_name" || true)"
  [[ -z "$pools" ]] && { log "  No node pools using default SA found to delete."; return 0; }

  local loc_flag
  if [[ "$location" =~ ^[a-z]+-[a-z0-9]+[0-9]-[a-z]$ ]]; then
    loc_flag="--zone"
  else
    loc_flag="--region"
  fi

  while IFS= read -r pool; do
    [[ -z "$pool" ]] && continue
    log "  Handling default-SA node pool '$pool'"

    if [[ "$CORDON_AND_DRAIN_OLD_POOLS" == "true" ]]; then
      cordon_and_drain_pool_nodes "$project_id" "$location" "$cluster_name" "$pool"
    else
      log "  Skipping cordon/drain for '$pool' (CORDON_AND_DRAIN_OLD_POOLS=false)"
    fi

    log "  Deleting node pool '$pool'"
    gcloud container node-pools delete "$pool" \
      --cluster "$cluster_name" \
      $loc_flag "$location" \
      --project "$project_id" \
      --quiet
  done <<< "$pools"
}

run_audit_for_cluster() {
  local project_id="$1"
  local location="$2"
  local cluster_name="$3"

  log "Running CIS audit check for cluster '$cluster_name' ($project_id / $location)"

  local audit
  audit=$(PROJECT_ID="$project_id" CLUSTER_NAME="$cluster_name" LOCATION="$location" \
    /bin/bash -c "
      # Get NODE_POOL if not already set
      if [[ -z \$NODE_POOL ]]; then
        NODE_POOL=\$(gcloud container clusters describe \$CLUSTER_NAME --location \$LOCATION --project \$PROJECT_ID --format='value(nodePools[0].name)' 2>/dev/null)
        [[ -z \$NODE_POOL ]] && { echo 'FAILED_NODE_POOL'; exit 1; }
      fi

      # Get service account from node pool
      SA=\$(gcloud container node-pools describe \$NODE_POOL --cluster \$CLUSTER_NAME --location \$LOCATION --project \$PROJECT_ID --format='value(config.serviceAccount)' 2>/dev/null)
      [[ -z \$SA ]] && { echo 'FAILED_SERVICE_ACCOUNT'; exit 1; }

      # Check if service account is 'default'
      [[ \"\$SA\" == \"default\" ]] && echo 'DEFAULT_SA_IN_USE' || echo 'DEFAULT_SA_NOT_IN_USE'
    " || true)

  log "  Audit result: $audit"
  if [[ "$audit" == "DEFAULT_SA_IN_USE" ]]; then
    log "  WARNING: Primary node pool of cluster '$cluster_name' still using default SA."
    return 1
  fi
  return 0
}

############################################
# MAIN
############################################

main() {
  require_cmd gcloud awk

  [[ -z "$TARGET_CLUSTERS" ]] && die "TARGET_CLUSTERS is not set. Example: TARGET_CLUSTERS='proj-1:us-central1:prod-cluster,proj-1:us-central1-a:dev-cluster'"

  IFS=',' read -r -a clusters <<< "$TARGET_CLUSTERS"

  for spec in "${clusters[@]}"; do
    [[ -z "$spec" ]] && continue
    IFS=':' read -r project_id location cluster_name <<< "$spec" || die "Invalid cluster spec: '$spec' (expected project:location:cluster)"

    [[ -z "$project_id" || -z "$location" || -z "$cluster_name" ]] && die "Invalid cluster spec: '$spec' (expected project:location:cluster)"

    log "=== Processing cluster: project='$project_id', location='$location', name='$cluster_name' ==="

    # Ensure SA and roles
    sa_email="$(ensure_node_sa "$project_id")"
    ensure_sa_roles "$project_id" "$sa_email"

    # Ensure node pool with SA
    ensure_secure_node_pool "$project_id" "$location" "$cluster_name" "$sa_email"

    # Optionally delete old default-SA pools
    if [[ "$DELETE_OLD_DEFAULT_POOLS" == "true" ]]; then
      log "Deleting node pools using default service account in cluster '$cluster_name'"
      delete_default_sa_pools "$project_id" "$location" "$cluster_name"
    else
      log "Skipping deletion of default-SA node pools (DELETE_OLD_DEFAULT_POOLS=false)"
    fi

    # Verification
    run_audit_for_cluster "$project_id" "$location" "$cluster_name" || true
  done

  log "Completed remediation steps. Review audit output above for any remaining issues."
}

main "$@"

Additional Reading:​