Prefer Using Container Optimized Os When Possible

More Info:

Container-Optimized OS is an operating system image that is designed for quick, secure deployment on Compute Engine VMs.

Risk Level

Medium

Address

Security

Compliance Standards

APRA CPS 234 (Australia)
BSI C5 (Germany)
Brazil LGPD
CCPA / CPRA (California)
CIS Critical Security Controls v8
CIS EKS
CMMC 2.0
CSA Cloud Controls Matrix v4
DPDPA
Digital Operational Resilience Act (EU)
Essential 8
ISO/IEC 27017
ISO/IEC 27018
ISO/IEC 27701
KSA PDPL
MAS Technology Risk Management (Singapore)
MITRE ATT&CK (Cloud)
NIS2 Directive
NIST SP 800-171
NYDFS 23 NYCRR 500
SWIFT Customer Security Controls Framework
Sarbanes-Oxley IT General Controls
UK NCSC Cyber Assessment Framework

Triage and Remediation

Remediation

Remediation

Manual Steps

Identify existing node OSes

Run on any machine with gcloud installed:

gcloud compute instances list \
  --format="table(name,zone,status,labels.*,machineType,creationTimestamp)"

For each node instance, check its OS:

gcloud compute instances describe NODE_INSTANCE_NAME \
  --zone=NODE_ZONE \
  --format="value(disks[0].licenses)"

Determine suitable Container-Optimized OS image
- Run on any machine with gcloud installed:
```
gcloud compute images list \
  --project cos-cloud \
  --no-standard-images \
  --filter="name~^cos-stable-" \
  --format="table(name,family)"
```
- Select the desired image family (for production, usually cos-stable).
Plan node replacement (drain workload from one node at a time)
- Run on any machine with kubectl access:
```
kubectl get nodes -o wide
kubectl drain NODE_NAME \
  --ignore-daemonsets \
  --delete-emptydir-data
```
- This cordons and evicts pods; ensure PodDisruptionBudgets are satisfied.

Recreate the drained node with Container-Optimized OS

Run on any machine with gcloud installed; first capture existing node settings:

gcloud compute instances describe NODE_INSTANCE_NAME \
  --zone=NODE_ZONE \
  --format="yaml(machineType,networkInterfaces,serviceAccounts,tags,metadata)"

Delete the old instance (the MIG / GKE node pool, if used, may manage this automatically; otherwise delete manually):
```
gcloud compute instances delete NODE_INSTANCE_NAME \
  --zone=NODE_ZONE
```

Create a replacement instance using Container-Optimized OS (adapt machine/network/SA/labels to match your cluster’s node-join mechanism, e.g., GKE node pool, kubeadm, etc.):

gcloud compute instances create NEW_NODE_INSTANCE_NAME \
  --zone=NODE_ZONE \
  --machine-type=MACHINE_TYPE \
  --image-family=cos-stable \
  --image-project=cos-cloud \
  --tags=TAG1,TAG2 \
  --service-account=SERVICE_ACCOUNT_EMAIL \
  --scopes=https://www.googleapis.com/auth/cloud-platform \
  --metadata-from-file startup-script=/path/to/your/node-join-script.sh

Wait for the node to join the cluster, then uncordon (if applicable):
```
kubectl uncordon NODE_NAME
```

Repeat for all remaining worker nodes
- Process each worker node one by one (drain → recreate with COS → uncordon) to avoid large-scale disruption.

Verification

Run on any machine with gcloud and kubectl to confirm all worker nodes are using Container-Optimized OS:

# Map k8s nodes to GCE instances
kubectl get nodes -o wide

# For each worker node's instance name/zone:
gcloud compute instances describe NODE_INSTANCE_NAME \
  --zone=NODE_ZONE \
  --format="value(disks[0].licenses)"

Ensure each worker node’s license string references cos-cloud and a cos- image (e.g., cos-stable).

Using kubectl

kubectl cannot change the underlying operating system image of worker nodes; this control is enforced at the host level via the node’s VM/OS configuration in your cloud provider or provisioning tooling. To address this finding, make changes where nodes are created and managed (e.g., instance templates, node pools, or IaC) as described in the Manual Steps section.

Automation

#!/usr/bin/env bash
#
# audit_gce_worker_os.sh
#
# Purpose:
#   MANUAL CHECK ONLY – this does NOT change anything.
#   It inventories all worker nodes in a GKE cluster and reports whether
#   their underlying GCE instances are using Container-Optimized OS
#   (COS / cos_container / cos_cloud / cos-arm64, etc.).
#
# Usage (run on any machine with kubectl and gcloud configured):
#   1. Ensure you are authenticated and have the correct project/cluster context:
#        gcloud config set project YOUR_PROJECT_ID
#        gcloud container clusters get-credentials YOUR_CLUSTER --zone YOUR_ZONE
#   2. Run:
#        bash audit_gce_worker_os.sh
#
# Notes:
#   - This script is read-only and idempotent; you can re-run it safely.
#   - Because this CIS control is MANUAL and has "No remediation",
#     migration to COS must be planned and executed separately (e.g., via
#     new node pools / instance templates / Terraform), not by this script.

set -euo pipefail

# Colors for readability (fall back gracefully if tput not available)
if command -v tput >/dev/null 2>&1; then
  GREEN="$(tput setaf 2)"
  RED="$(tput setaf 1)"
  YELLOW="$(tput setaf 3)"
  BLUE="$(tput setaf 4)"
  BOLD="$(tput bold)"
  RESET="$(tput sgr0)"
else
  GREEN=""; RED=""; YELLOW=""; BLUE=""; BOLD=""; RESET=""
fi

echo "${BOLD}==> Discovering worker nodes from Kubernetes...${RESET}"

# Get all non-control-plane nodes (worker nodes) by label
worker_nodes=$(kubectl get nodes -o jsonpath='{range .items[?(@.metadata.labels.node-role\.kubernetes\.io/control-plane!= "true" && @.metadata.labels.node-role\.kubernetes\.io/master!= "true")]}{.metadata.name}{"\n"}{end}' || true)

if [[ -z "${worker_nodes}" ]]; then
  echo "${YELLOW}No worker nodes found (or missing role labels).${RESET}"
  echo "You may need to adjust the node-role label selection in this script."
  exit 0
fi

echo "${BLUE}Worker nodes discovered:${RESET}"
echo "${worker_nodes}" | sed 's/^/  - /'

# Build a temporary map of nodeName -> GCE instance name / zone
echo
echo "${BOLD}==> Mapping Kubernetes nodes to GCE instances...${RESET}"

declare -A NODE_TO_INSTANCE
declare -A NODE_TO_ZONE

while read -r node; do
  [[ -z "${node}" ]] && continue

  # Extract providerID (for GKE on GCE it looks like: gce://PROJECT/ZONE/INSTANCE)
  provider_id=$(kubectl get node "${node}" -o jsonpath='{.spec.providerID}' 2>/dev/null || true)

  if [[ -z "${provider_id}" ]]; then
    echo "${YELLOW}  [WARN] Node ${node}: no providerID found; skipping GCE mapping.${RESET}"
    continue
  fi

  # Strip leading scheme "gce://"
  provider_id=${provider_id#gce://}

  # Split into PROJECT/ZONE/INSTANCE
  IFS='/' read -r project zone instance <<< "${provider_id}"

  if [[ -z "${instance}" || -z "${zone}" ]]; then
    echo "${YELLOW}  [WARN] Node ${node}: unexpected providerID format (${provider_id}); skipping.${RESET}"
    continue
  fi

  NODE_TO_INSTANCE["${node}"]="${instance}"
  NODE_TO_ZONE["${node}"]="${zone}"

  echo "  - ${node} -> instance ${instance} (zone ${zone})"
done <<< "${worker_nodes}"

echo
echo "${BOLD}==> Querying GCE instance OS images via gcloud...${RESET}"

# Header
printf "%-40s | %-40s | %-25s | %-10s\n" "K8s Node" "GCE Instance" "Image" "COS?"
printf -- "%-40s-+-%-40s-+-%-25s-+-%-10s\n" \
  "----------------------------------------" "----------------------------------------" "-------------------------" "----------"

non_cos_count=0
total_count=0

for node in "${!NODE_TO_INSTANCE[@]}"; do
  instance="${NODE_TO_INSTANCE[$node]}"
  zone="${NODE_TO_ZONE[$node]}"
  total_count=$((total_count + 1))

  # Query image used by the instance
  # Using 'properties.disks[0].licenses' or 'sourceImage', depending on instance type.
  image=$(gcloud compute instances describe "${instance}" \
            --zone="${zone}" \
            --format="value(disks[0].licenses.basename())" 2>/dev/null || true)

  if [[ -z "${image}" ]]; then
    # Fallback to sourceImage if licenses isn't populated
    image=$(gcloud compute instances describe "${instance}" \
              --zone="${zone}" \
              --format="value(disks[0].initializeParams.sourceImage.basename())" 2>/dev/null || echo "UNKNOWN")
  fi

  # Normalize image string
  image="${image:-UNKNOWN}"

  # Heuristic: check for common COS image identifiers
  # Examples: cos-stable, cos-beta, cos-123-456-789, cos_container, cos-cloud, cos-arm64, etc.
  if [[ "${image}" =~ (^|-)cos(|-[a-z0-9]+)*$ || "${image}" =~ (^|-)cos_ || "${image}" =~ cos-container || "${image}" =~ cos-cloud ]]; then
    cos_flag="${GREEN}YES${RESET}"
  else
    cos_flag="${RED}NO${RESET}"
    non_cos_count=$((non_cos_count + 1))
  fi

  printf "%-40s | %-40s | %-25s | %-10b\n" "${node}" "${instance}" "${image}" "${cos_flag}"
done

echo
echo "${BOLD}==> Summary:${RESET}"
echo "  Total worker nodes checked: ${total_count}"
echo "  Worker nodes NOT using a recognized Container-Optimized OS image: ${non_cos_count}"

if [[ "${non_cos_count}" -gt 0 ]]; then
  echo
  echo "${YELLOW}Manual review required (CIS EKS 3.3.1 – MANUAL control):${RESET}"
  cat <<'EOF'
- For nodes reported as "COS? NO", decide whether to:
    * Migrate workloads to new node pools using Container-Optimized OS images, or
    * Accept the risk and document the exception (e.g., for workloads requiring a non-COS base OS).
- Typical migration steps (outside the scope of this script):
    * Create a new GKE node pool with a COS-based image type.
    * Cordon and drain non-COS nodes.
    * Delete or repurpose the old non-COS node pool.
EOF
fi

echo
echo "${GREEN}Verification complete.${RESET} The table above is the evidence for this manual control."

Additional Reading:

https://aws.amazon.com/blogs/containers/bottlerocket-a-special-purpose-container-operating-system/

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​

Additional Reading:​