Ensure Latest Cni Version Is Used

More Info:

There are a variety of CNI plugins available for Kubernetes. If the CNI in use does not support Network Policies it may not be possible to effectively restrict traffic in the cluster.

Risk Level

Low

Address

Security

Compliance Standards

• APRA CPS 234 (Australia)
• BSI C5 (Germany)
• Brazil LGPD
• CCPA / CPRA (California)
• CIS Critical Security Controls v8
• CIS EKS
• CMMC 2.0
• CSA Cloud Controls Matrix v4
• DPDPA
• Digital Operational Resilience Act (EU)
• Essential 8
• ISO/IEC 27017
• ISO/IEC 27018
• ISO/IEC 27701
• KSA PDPL
• MAS Technology Risk Management (Singapore)
• MITRE ATT&CK (Cloud)
• NIS2 Directive
• NIST SP 800-171
• NYDFS 23 NYCRR 500
• SWIFT Customer Security Controls Framework
• Sarbanes-Oxley IT General Controls
• UK NCSC Cyber Assessment Framework

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Identify the CNI in use and relevant namespaces

   • On any machine with kubectl access:

     kubectl get pods -A -o wide | grep -Ei 'cni|calico|weave|cilium|aws-node'
     kubectl get ds -A | grep -Ei 'cni|aws-node|calico|cilium|weave'
     kubectl get pods -n kube-system -o wide | grep aws-node

   • Confirm that the AWS VPC CNI (aws-node) is the plugin in use.

2. Gather the currently running AWS CNI version

   • On any machine with kubectl access:

     # Get image version(s) in use by the DaemonSet
     kubectl -n kube-system get ds aws-node -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'
     
     # Confirm on running pods
     kubectl -n kube-system get pods -l k8s-app=aws-node -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[*].image}{"\n"}{end}'

   • Record the image tag(s) and digest(s) you see (e.g. 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.16.4).

3. Compare with the latest supported AWS CNI versions

   • In a browser, open the official AWS VPC CNI plugin documentation / release notes (for your region and EKS/Kubernetes version), for example:
     • “Amazon VPC CNI plugin for Kubernetes GitHub releases”
     • “Amazon EKS supported add-ons and versions”
   • Compare your recorded CNI image tag with the latest recommended version that is supported for your EKS/Kubernetes version and region.
   • Decide the target version (latest recommended, not just numerically latest) that is compatible with your cluster version.

4. Assess upgrade impact and plan

   • On any machine with kubectl access, review current CNI configuration for non‑default settings that might be affected by an upgrade:

     kubectl -n kube-system get cm aws-node -o yaml || echo "No aws-node ConfigMap found"
     kubectl -n kube-system describe ds aws-node

   • Note any custom environment variables, annotations, or tolerations; check the AWS documentation to ensure they are still valid in the target version.
   • Plan a maintenance window if required; upgrading the CNI can impact pod networking while DaemonSet pods roll.

5. Perform the upgrade using your chosen method (conceptual step)

   • If you manage AWS CNI via EKS add-ons or IaC (CloudFormation/Terraform/eksctl/console), follow the corresponding AWS documentation to update the CNI to the target version you selected in step 3.
   • If you manage manifests directly, obtain the updated DaemonSet manifest from the AWS documentation for the target version, review differences, then apply it with kubectl apply -f <manifest.yml> from any machine with kubectl access.
   • This step is intentionally high-level: the exact command and method depend on how your cluster is provisioned and must follow AWS’s documented procedure.

6. Verify the upgrade and network policy capability

   • On any machine with kubectl access:

     kubectl -n kube-system rollout status ds/aws-node
     kubectl -n kube-system get ds aws-node -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}'
     kubectl -n kube-system get pods -l k8s-app=aws-node -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[*].image}{"\n"}{end}'

   • Confirm that all aws-node pods are running the intended (latest supported) version.
   • Optionally, validate that network policies function as expected by creating or using an existing NetworkPolicy and confirming that traffic is allowed/denied according to the policy.

Using kubectl

# 1) Identify current CNI DaemonSet and version
# Run on: any machine with kubectl access

kubectl -n kube-system get daemonset \
  -o custom-columns='NAMESPACE:{metadata.namespace},NAME:{metadata.name},IMAGE:{.spec.template.spec.containers[*].image}' \
  | grep -iE 'cni|aws-node|calico|weave|cilium'

Review the IMAGE field for the AWS CNI plugin (typically amazon-k8s-cni or aws-node), for example:

• 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.8.0

A potential problem: the tag is clearly behind current upstream releases (compare manually with AWS CNI documentation/Release Notes).

---

# 2) Get detailed spec of the CNI DaemonSet
# Run on: any machine with kubectl access

kubectl -n kube-system get daemonset aws-node -o yaml

Review:

• spec.template.spec.containers[].image for an outdated tag.
• Any hard‑pinned image digests that do not match the latest recommended version. A problem exists if this image version is older than what AWS currently recommends for your EKS/Kubernetes version.

---

# 3) Check CNI pod versions actually running on nodes
# Run on: any machine with kubectl access

kubectl -n kube-system get pods -o wide | grep -i 'aws-node'
kubectl -n kube-system get pods -l k8s-app=aws-node -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'

If different pods show different image tags or obviously old tags, that indicates:

• Rolling update is incomplete, or
• Some nodes are running old CNI versions.

---

# 4) Correlate node EKS/Kubernetes version with CNI image
# Run on: any machine with kubectl access

kubectl get nodes -o wide

Use the VERSION column and compare manually against AWS documentation for:

• Minimum/required CNI version per Kubernetes/EKS version. A problem exists if your CNI image tag is below the minimum or recommended version for the node/Kubernetes version.

---

# 5) Inspect CNI network artifacts on nodes (evidence gathering only)
# Run on: any worker node via SSH (NOT via kubectl)

sudo ls -R /var/lib/cni/networks
sudo cat /var/log/aws-routed-eni/ipamd.log | head -n 50

Use these to:

• Confirm that AWS CNI is in use and functioning.
• Look for log messages indicating deprecated or outdated CNI behavior. A problem exists if logs show deprecation warnings or known issues tied to old CNI releases.

---

Once you have:

• The CNI DaemonSet image tag, and
• Your cluster/EKS/Kubernetes version,

manually compare them against the AWS VPC CNI documentation and release notes. If the running image is not the latest (or at least not the AWS‑recommended version for your cluster version), this finding should be considered a problem and you should plan an upgrade outside of kubectl (e.g., via EKS add‑on or your cluster provisioning/IaC tooling).

Automation

#!/usr/bin/env bash
# Purpose: Report AWS VPC CNI plugin version usage across the cluster
# Runs on: any machine with kubectl access and aws CLI (optional for latest lookup)
# Requires: kubectl configured with cluster-admin or equivalent RBAC

set -euo pipefail

echo "=== 1) Detecting AWS VPC CNI daemonset and image versions (kube-system, aws-node) ==="
kubectl get ds aws-node -n kube-system -o wide || {
  echo "aws-node DaemonSet not found in kube-system namespace."
  echo "If you use a different CNI, this specific check does not apply."
}

echo
echo "--- DaemonSet spec images (what should be running) ---"
kubectl get ds aws-node -n kube-system -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}' 2>/dev/null || true
echo

echo "--- Pods' actual images per node (what is running) ---"
kubectl get pods -n kube-system -l "k8s-app=aws-node" -o wide 2>/dev/null || true
echo
kubectl get pods -n kube-system -l "k8s-app=aws-node" -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.hostIP}{"\t"}{range .spec.containers[*]}{.name}{"="}{.image}{" "}{end}{"\n"}{end}' 2>/dev/null || true
echo

echo "=== 2) Extracting aws-node CNI image tags (from DaemonSet) ==="
DS_IMAGES=$(kubectl get ds aws-node -n kube-system -o jsonpath='{range .spec.template.spec.containers[*]}{.image}{"\n"}{end}' 2>/dev/null || true)
if [ -z "$DS_IMAGES" ]; then
  echo "No images found on aws-node DaemonSet. Check if another CNI is in use."
else
  echo "$DS_IMAGES" | sort -u | while read -r img; do
    [ -z "$img" ] && continue
    repo="${img%:*}"
    tag="${img##*:}"
    if [ "$repo" = "$tag" ]; then
      # no explicit tag -> implicitly 'latest' or digest-only
      echo "Image: $img"
      if [[ "$img" == *@* ]]; then
        echo "  -> Image uses a digest (immutable). Check EKS/AWS docs to identify its version."
      else
        echo "  -> Image has no tag; Kubernetes will treat it as ':latest'. Verify this is the desired latest CNI release."
      fi
    else
      echo "Image: $img"
      echo "  -> Parsed tag: $tag"
    fi
  done
fi
echo

echo "=== 3) Correlate CNI versions with EKS/Kubernetes versions (cluster-wide) ==="
echo "--- Cluster version info ---"
kubectl version -o yaml
echo

echo "=== 4) (Optional) Compare against latest AWS CNI release from GitHub ==="
if command -v curl >/dev/null 2>&1; then
  echo "Querying latest aws-node CNI release tag from GitHub..."
  LATEST_GH_TAG=$(curl -s https://api.github.com/repos/aws/amazon-vpc-cni-k8s/releases/latest | \
                  grep '"tag_name"' | head -n1 | sed -E 's/.*"tag_name": *"([^"]+)".*/\1/' || true)
  if [ -n "$LATEST_GH_TAG" ]; then
    echo "Latest GitHub release tag: $LATEST_GH_TAG"
    echo "Compare this tag with the image tags reported above."
  else
    echo "Could not determine latest tag from GitHub API. Review manually:"
    echo "  https://github.com/aws/amazon-vpc-cni-k8s/releases"
  fi
else
  echo "curl not installed; cannot auto-query GitHub releases."
  echo "Manually review latest version at:"
  echo "  https://github.com/aws/amazon-vpc-cni-k8s/releases"
fi
echo

echo "=== 5) Node-level CNI binaries/networks (for spot checks) ==="
echo "Note: Access to node shells is required for this part."
echo "Example commands to run on a worker node (not executed here):"
cat <<'EOF'
# On a worker node (SSH session):

# List CNI binaries and versions (if versioned or with --version support)
ls -l /opt/cni/bin
for bin in /opt/cni/bin/*; do
  "$bin" --version 2>/dev/null || true
done

# Check existing CNI network allocations
ls -R /var/lib/cni/networks
EOF
echo

echo "=== INTERPRETING RESULTS ==="
cat <<'EOF'
Potential problems indicated by this report:

1. aws-node DaemonSet missing:
   - If you expect to use AWS VPC CNI and the DaemonSet is absent, the cluster may be using a different CNI or is misconfigured.

2. Image tags clearly behind latest:
   - If the aws-node image tags (e.g. 1.11.x) are older than the latest release tag reported from GitHub or AWS EKS documentation for your cluster version,
     the CNI is not at the latest supported version.

3. Mixed CNI versions across nodes:
   - If 'kubectl get pods -n kube-system -l k8s-app=aws-node -o jsonpath=...' shows different image tags on different nodes,
     the cluster is in a partially upgraded state; ensure upgrade completes on all nodes.

4. Untagged or implicit ':latest' without governance:
   - If the image repo has no explicit version tag (or only 'latest'), you cannot reliably prove that the cluster is on the intended latest CNI version.
     This requires policy review: pin to a known-good version recommended for your EKS/Kubernetes version.

5. Node binary mismatch:
   - If node-level /opt/cni/bin contents or versions differ from the aws-node image tag and documentation,
     the installed binaries may not match the expected CNI version and require manual investigation.

This check is MANUAL:
- Use the image tags, cluster version, and AWS/EKS documentation to decide whether you are on the latest *supported* CNI version for your environment.
- Plan a controlled upgrade if you determine that the running version is behind the recommended release.
EOF

Additional Reading:

• https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/