Skip to main content

Ensure Container-Optimized OS Is Used For GKE Node Images

More Info:

Use the Container-Optimized OS (cos_containerd) image for GKE nodes, which is hardened and minimizes the attack surface. Other images carry more packages and larger exposure.

Risk Level

Medium

Address

Security

Compliance Standards

  • CIS GKE

Triage and Remediation

Remediation

Manual Steps
  1. List all node pools and their image types
    Run on any machine with gcloud configured:

    PROJECT_ID="<YOUR_PROJECT_ID>"
    LOCATION="<CLUSTER_LOCATION>"
    CLUSTER_NAME="<CLUSTER_NAME>"

    gcloud container node-pools list \
    --cluster "${CLUSTER_NAME}" \
    --location "${LOCATION}" \
    --project "${PROJECT_ID}" \
    --format="table(name,config.imageType)"
  2. Identify non‑compliant node pools
    From the output, note any node pool where config.imageType is not COS_CONTAINERD (e.g. UBUNTU, UBUNTU_CONTAINERD, COS) for further review.

  3. Assess workload and feature compatibility before changing images
    For each non‑COS_CONTAINERD node pool:

    • Confirm workloads don’t rely on OS‑specific packages, custom agents, or kernel modules only present on the current image.
    • Confirm any node‑level DaemonSets (monitoring, logging, security agents) support cos_containerd.
    • If unsure, create or use a test cluster/node pool with cos_containerd and validate workloads there first.
  4. Plan and execute node pool image migration
    For each node pool you decide to migrate, run on any machine with gcloud configured:

    PROJECT_ID="<YOUR_PROJECT_ID>"
    LOCATION="<CLUSTER_LOCATION>"
    CLUSTER_NAME="<CLUSTER_NAME>"
    NODE_POOL_NAME="<NODE_POOL_NAME>"

    gcloud container clusters upgrade "${CLUSTER_NAME}" \
    --location "${LOCATION}" \
    --node-pool "${NODE_POOL_NAME}" \
    --image-type "COS_CONTAINERD" \
    --project "${PROJECT_ID}"

    This recreates nodes in that pool with the new image and will cause pod evictions and rescheduling; perform during a maintenance window and ensure PodDisruptionBudgets and autoscaling are configured appropriately.

  5. Verify node pool image types after upgrade
    Re‑run, on any machine with gcloud configured:

    gcloud container node-pools list \
    --cluster "${CLUSTER_NAME}" \
    --location "${LOCATION}" \
    --project "${PROJECT_ID}" \
    --format="table(name,config.imageType)"

    Confirm all intended node pools now show COS_CONTAINERD as config.imageType.

  6. Spot‑check via the original audit command
    Optionally, for each node pool, run:

    NODE_POOL_NAME="<NODE_POOL_NAME>"

    gcloud container node-pools describe "${NODE_POOL_NAME}" \
    --cluster "${CLUSTER_NAME}" \
    --location "${LOCATION}" \
    --project "${PROJECT_ID}" \
    --format json | jq '.config.imageType'

    Ensure the value returned is "COS_CONTAINERD" for compliant node pools.

Using kubectl

kubectl cannot modify the node image type or node pool configuration; this setting is managed at the GKE control-plane / cloud-provider level (gcloud, console, or IaC). To address this finding, follow the guidance in the Manual Steps section using the appropriate GCP tools.

Automation
#!/usr/bin/env bash
#
# Report GKE node pool image types and flag any that are NOT cos_containerd.
# Run on: any machine with gcloud, jq, and access to the GCP project(s).
#
# Usage:
# PROJECT_ID=my-project \
# gcloud config set project "${PROJECT_ID}"
# ./check_gke_nodepool_images.sh
#
# Optional env vars:
# PROJECT_ID – if unset, uses current gcloud config project
# LOCATION – if set, limits to this region/zone (e.g. us-central1, us-central1-a)

set -euo pipefail

PROJECT_ID="${PROJECT_ID:-$(gcloud config get-value project 2>/dev/null)}"
if [[ -z "${PROJECT_ID}" || "${PROJECT_ID}" == "(unset)" ]]; then
echo "ERROR: PROJECT_ID not set and no default project in gcloud config." >&2
exit 1
fi

LOCATION_FILTER=""
if [[ -n "${LOCATION:-}" ]]; then
LOCATION_FILTER="--locations=${LOCATION}"
fi

echo "Project: ${PROJECT_ID}"
echo "Location filter: ${LOCATION:-<all locations>}"
echo

# List all clusters (regional and zonal)
clusters_json="$(gcloud container clusters list \
--project "${PROJECT_ID}" \
${LOCATION_FILTER} \
--format=json)"

if [[ "$(echo "${clusters_json}" | jq 'length')" -eq 0 ]]; then
echo "No GKE clusters found in project ${PROJECT_ID} (with given location filter)."
exit 0
fi

bad_found=0

echo "Cluster,nodePool,location,imageType,COMPLIANT"
echo "${clusters_json}" | jq -r '.[] | [.name, .location] | @tsv' | while IFS=$'\t' read -r CLUSTER LOCATION_VAL; do
# List node pools for each cluster
np_json="$(gcloud container node-pools list \
--cluster "${CLUSTER}" \
--location "${LOCATION_VAL}" \
--project "${PROJECT_ID}" \
--format=json)"

if [[ "$(echo "${np_json}" | jq 'length')" -eq 0 ]]; then
# Cluster with no node pools (unusual but handle it)
echo "${CLUSTER},<none>,${LOCATION_VAL},<none>,NON_COMPLIANT"
bad_found=1
continue
fi

echo "${np_json}" | jq -r '.[] | [.name, .config.imageType] | @tsv' | \
while IFS=$'\t' read -r NP_NAME IMAGE_TYPE; do
if [[ "${IMAGE_TYPE}" == "cos_containerd" ]]; then
echo "${CLUSTER},${NP_NAME},${LOCATION_VAL},${IMAGE_TYPE},COMPLIANT"
else
echo "${CLUSTER},${NP_NAME},${LOCATION_VAL},${IMAGE_TYPE},NON_COMPLIANT"
bad_found=1
fi
done
done

# Explanation of output:
cat <<'EOF'

Interpretation:
- Each line after the headers is:
Cluster,nodePool,location,imageType,COMPLIANT
- COMPLIANT – imageType == cos_containerd
- NON_COMPLIANT – imageType is anything else (e.g. UBUNTU, COS, UBUNTU_CONTAINERD)

Node pools marked NON_COMPLIANT are the ones that need manual review and
a potential upgrade, for example:
gcloud container clusters upgrade <cluster_name> \
--image-type cos_containerd \
--location <location> \
--node-pool <node_pool_name>
EOF