Skip to main content

Container Images Should Not Use The latest Or Untagged Tag

More Info:​

Verifies images are pinned to an immutable tag or digest. :latest and untagged images make deployments non-reproducible and hard to audit.

Risk Level​

Low

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. Identify noncompliant Pods and their owning controllers (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | ((.spec.containers // []) + (.spec.initContainers // []))[]
      | .image as $img
      | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
      | select($ok | not)
      | "ns=\($m.namespace) pod=\($m.name) owner_kind=\($own.kind // "Pod") owner_name=\($own.name // $m.name) container=\(.name) image=\($img)"
      ][]'

  2. For each listed controller (e.g., Deployment, StatefulSet, DaemonSet) or standalone Pod, export the current manifest (run on any machine with kubectl access). Example for a Deployment:

    kubectl -n NAMESPACE get deployment OWNER_NAME -o yaml > /tmp/OWNER_NAME.yaml

    For a standalone Pod:

    kubectl -n NAMESPACE get pod POD_NAME -o yaml > /tmp/POD_NAME.yaml

  3. Edit the manifest locally to pin images to immutable tags or digests (run on the same machine where you saved the file):

    vi /tmp/OWNER_NAME.yaml

    In spec.template.spec.containers[] and spec.template.spec.initContainers[], change each image from:

    image: myregistry.azurecr.io/app:latest

    or

    image: myregistry.azurecr.io/app

    to a pinned reference, for example:

    image: myregistry.azurecr.io/app:v1.2.3

    or

    image: myregistry.azurecr.io/app@sha256:abcdef1234567890...

  4. Apply the updated manifest back to the cluster (run on any machine with kubectl access). Example:

    kubectl apply -f /tmp/OWNER_NAME.yaml

    For standalone Pods, delete and recreate them from the updated manifest:

    kubectl delete -f /tmp/POD_NAME.yaml
    kubectl apply  -f /tmp/POD_NAME.yaml

  5. Wait for new Pods to be ready using the pinned images (run on any machine with kubectl access). Example for a Deployment:

    kubectl -n NAMESPACE rollout status deployment/OWNER_NAME

  6. Verify that no remaining Pods use :latest or untagged images (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | .metadata as $m
      | (.spec.nodeName // "") as $node
      | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | ((.spec.containers // []) + (.spec.initContainers // []))[]
      | .image as $img
      | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
      | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
        + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
        + (if $node   == ""   then "" else " node=\($node)" end)
        + (if $labels == ""   then "" else " labels=\($labels)" end)
        + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
        + " container=\(.name) image=\($img)"
        + " is_compliant=\(if $ok then "true" else "false" end)"
      ] as $rows
      | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

    Confirm that all reported lines show is_compliant=true.

Using kubectl

On any machine with kubectl access:

  1. Identify non‑compliant Pods (for context only; same as audit):
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img)"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
  1. Export the owning workload manifest (Deployment/StatefulSet/Job, etc.), edit image tags, and re‑apply.

Example for a Pod owned by a Deployment:

# Replace NAMESPACE and DEPLOYMENT_NAME based on the "owner=" field from the audit
kubectl get deployment DEPLOYMENT_NAME \
  -n NAMESPACE \
  -o yaml > /tmp/deployment-DEPLOYMENT_NAME.yaml

Edit /tmp/deployment-DEPLOYMENT_NAME.yaml and, for each offending container under spec.template.spec.containers (and initContainers if present):

  • Replace an untagged image, e.g.:

    • image: myregistry.azurecr.io/app
      with an immutable tag or digest, e.g.:
    • image: myregistry.azurecr.io/app:v1.2.3 or
    • image: myregistry.azurecr.io/app@sha256:abcdef...

  • Replace :latest, e.g.:

    • image: myregistry.azurecr.io/app:latest
      with:
    • image: myregistry.azurecr.io/app:v1.2.3 or
    • image: myregistry.azurecr.io/app@sha256:abcdef...

Then apply:

kubectl apply -f /tmp/deployment-DEPLOYMENT_NAME.yaml

Repeat this export/edit/apply pattern for any other owning resources reported by the audit (e.g., StatefulSet, DaemonSet, Job, CronJob). For bare Pods without an owner reference, edit in place:

kubectl get pod POD_NAME -n NAMESPACE -o yaml > /tmp/pod-POD_NAME.yaml
# edit images in /tmp/pod-POD_NAME.yaml as above
kubectl delete pod POD_NAME -n NAMESPACE
kubectl apply -f /tmp/pod-POD_NAME.yaml
  1. Verification (on any machine with kubectl access):

After rollouts complete, re‑run the audit and confirm all lines show is_compliant=true (or only that single line):

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img)"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Automation
#!/usr/bin/env bash
#
# Fix CBP C1.10 for Pods on AKS:
# - Detect running Pods outside system namespaces using :latest or untagged images
# - Patch the owning workload to use a pinned tag supplied via annotation
#
# IMPORTANT:
# - This script CANNOT guess the correct immutable tag or digest.
# - For each container image that should be pinned, you MUST set an annotation
#   on the owning object with the desired immutable image.
#
#   Supported owners: Deployment, StatefulSet, DaemonSet, ReplicaSet, Job, CronJob
#
#   Annotation format (on the owner object):
#     c1-10.image-pins: |
#       containerName1=registry.example.com/app@sha256:deadbeef...
#       containerName2=myimage:1.2.3
#
#   The script reads this annotation and updates .spec.template.spec.containers[*].image
#   for containers listed there.
#
# Run on: any machine with kubectl access to the AKS cluster.
# Requirements: bash, kubectl, jq, yq (https://github.com/mikefarah/yq)

set -euo pipefail

command -v kubectl >/dev/null 2>&1 || { echo "kubectl not found in PATH"; exit 1; }
command -v jq >/dev/null 2>&1 || { echo "jq not found in PATH"; exit 1; }
command -v yq >/dev/null 2>&1 || { echo "yq not found in PATH"; exit 1; }

# Namespaces to exclude from the check
EXCLUDED_NAMESPACES_REGEX='^(kube-system|kube-public|kube-node-lease)$'

echo "Scanning Pods for non-pinned images (excluding kube-system, kube-public, kube-node-lease)..."

# Build a JSON list of offending pods and their owners/containers/images
offenders_json="$(kubectl get pods --all-namespaces -o json | \
  jq -c --arg re "$EXCLUDED_NAMESPACES_REGEX" '
    .items[]
    | select(.metadata.namespace | test($re) | not)
    | . as $pod
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | . as $c
    | .image as $img
    | (($img | contains("@"))
       or (($img | split("/") | last | contains(":"))
           and (($img | endswith(":latest")) | not))
      ) as $ok
    | select($ok | not)
    | {
        podNamespace: $pod.metadata.namespace,
        podName: $pod.metadata.name,
        containerName: .name,
        image: $img,
        owner: (
          [($pod.metadata.ownerReferences // [])[] | select(.controller)] | first
        )
      }
  ')"

if [[ -z "$offenders_json" ]]; then
  echo "No non-compliant Pod containers found."
  exit 0
fi

echo "Found non-compliant containers:"
echo "$offenders_json" | jq -r '. | "ns=\(.podNamespace) pod=\(.podName) container=\(.containerName) image=\(.image) ownerKind=\(.owner.kind // "Pod") ownerName=\(.owner.name // .podName)"' | sort -u

echo
echo "Attempting to patch owning workload objects using c1-10.image-pins annotation..."

# Helper: get image pin for a container from owner annotation
get_pin_from_annotation() {
  local ns="$1"
  local kind="$2"
  local name="$3"
  local container="$4"

  # Fetch annotation as multi-line "key=value"
  local pins
  pins="$(kubectl get "$kind" "$name" -n "$ns" -o jsonpath='{.metadata.annotations.c1-10\.image-pins}' 2>/dev/null || true)"
  [[ -z "$pins" ]] && return 1

  # Look for a line starting with "container="
  # Trim CRLF and spaces
  local line
  while IFS= read -r line; do
    line="${line%$'\r'}"
    line="$(echo "$line" | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')"
    [[ -z "$line" ]] && continue
    if [[ "$line" == "$container="* ]]; then
      echo "${line#*=}"
      return 0
    fi
  done <<< "$pins"

  return 1
}

# We will accumulate owners to patch to avoid multiple patches per owner
owners_to_patch_json="[]"

while IFS= read -r row; do
  ns="$(echo "$row" | jq -r '.podNamespace')"
  podName="$(echo "$row" | jq -r '.podName')"
  containerName="$(echo "$row" | jq -r '.containerName')"
  image="$(echo "$row" | jq -r '.image')"
  ownerKind="$(echo "$row" | jq -r '.owner.kind // "Pod"')"
  ownerName="$(echo "$row" | jq -r '.owner.name // .podName')"

  if [[ "$ownerKind" == "Pod" ]]; then
    echo "Skipping standalone Pod $ns/$podName (ownerKind=Pod). Pin image manually in its manifest or recreate from a pinned image."
    continue
  fi

  # Normalize ownerKind to match kubectl resource types
  case "$ownerKind" in
    Deployment|StatefulSet|DaemonSet|ReplicaSet|Job|CronJob) ;;
    *)
      echo "Unsupported ownerKind $ownerKind for $ns/$podName; handle manually."
      continue
      ;;
  esac

  # Try to get pin from annotation
  pin="$(get_pin_from_annotation "$ns" "$ownerKind" "$ownerName" "$containerName" || true)"
  if [[ -z "$pin" ]]; then
    echo "No c1-10.image-pins entry found for container '$containerName' on $ownerKind/$ns/$ownerName; leaving unchanged. Add annotation and re-run."
    continue
  fi

  echo "Will update $ownerKind/$ns/$ownerName container=$containerName image: $image -> $pin"

  # Add to owners_to_patch_json, merging multiple containers per owner
  owners_to_patch_json="$(jq -c \
    --arg ns "$ns" \
    --arg kind "$ownerKind" \
    --arg name "$ownerName" \
    --arg c "$containerName" \
    --arg img "$pin" '
    . as $root
    | (map(select(.namespace == $ns and .kind == $kind and .name == $name)) | length) as $exists
    | if $exists == 0 then
        $root + [{
          namespace: $ns,
          kind: $kind,
          name: $name,
          containers: [{name: $c, image: $img}]
        }]
      else
        map(
          if .namespace == $ns and .kind == $kind and .name == $name then
            .containers |= (
              . as $cs
              | (map(select(.name == $c)) | length) as $have
              | if $have == 0 then
                  $cs + [{name: $c, image: $img}]
                else
                  map(if .name == $c then .image = $img else . end)
                end
            )
          else .
          end
        )
      end
  ' <<< "$owners_to_patch_json")"

done <<< "$(echo "$offenders_json" | jq -c '.')"

if [[ "$owners_to_patch_json" == "[]" ]]; then
  echo
  echo "No owners had usable c1-10.image-pins annotations. Nothing patched."
  echo "Add c1-10.image-pins annotations with immutable tags/digests and re-run."
  exit 0
fi

echo
echo "Patching owning workload objects to use pinned images..."

echo "$owners_to_patch_json" | jq -c '.[]' | while IFS= read -r owner; do
  ns="$(echo "$owner" | jq -r '.namespace')"
  kind="$(echo "$owner" | jq -r '.kind')"
  name="$(echo "$owner" | jq -r '.name')"

  echo "Processing $kind/$ns/$name"

  # Get current YAML
  tmpfile="$(mktemp)"
  kubectl get "$kind" "$name" -n "$ns" -o yaml > "$tmpfile"

  # Build yq expression to update container images
  yq_expr=""
  while IFS= read -r c; do
    cname="$(echo "$c" | jq -r '.name')"
    cimg="$(echo "$c" | jq -r '.image')"
    # Update both containers and initContainers if present
    yq_expr+="
      (.spec.template.spec.containers[] | select(.name == \"$cname\").image) = \"$cimg\" |
      (.spec.template.spec.initContainers[]? | select(.name == \"$cname\").image) = \"$cimg\" |"
  done <<< "$(echo "$owner" | jq -c '.containers[]')"
  # Remove trailing pipe
  yq_expr="${yq_expr%|}"

  if [[ -z "$yq_expr" ]]; then
    echo "  No containers to update for $kind/$ns/$name"
    rm -f "$tmpfile"
    continue
  fi

  # Apply modifications
  yq -y "$yq_expr" "$tmpfile" > "${tmpfile}.patched"

  echo "  Applying patched manifest..."
  kubectl apply -f "${tmpfile}.patched"

  rm -f "$tmpfile" "${tmpfile}.patched"
done

echo
echo "Waiting for Pods to roll out with new images..."
# Simple wait: let the controllers reconcile; users can improve with rollout status
sleep 10

echo
echo "Re-running compliance audit to verify..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img)"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'