Skip to main content

Container Images Should Not Use The latest Or Untagged Tag

More Info:​

Verifies images are pinned to an immutable tag or digest. :latest and untagged images make deployments non-reproducible and hard to audit.

Risk Level​

Low

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. Identify all non-compliant pods (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.nodeName // "") as $node
      | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | ((.spec.containers // []) + (.spec.initContainers // []))[]
      | .image as $img
      | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
      | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
        + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
        + (if $node   == ""   then "" else " node=\($node)" end)
        + (if $labels == ""   then "" else " labels=\($labels)" end)
        + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
        + " container=\(.name) image=\($img)"
        + " is_compliant=\(if $ok then "true" else "false" end)"
      ] as $rows
      | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end' \
    | grep 'is_compliant=false'

  2. For each non-compliant pod, determine its managing controller (run on any machine with kubectl access). From the owner= field in the previous output, note the kind, namespace, and name. Then fetch the full manifest, for example:

    kubectl -n <namespace> get <kind-lowercase> <name> -o yaml > /tmp/<namespace>_<kind>_<name>.yaml

    Replace <kind-lowercase> with deployment, statefulset, daemonset, job, or cronjob as appropriate.

  3. In each saved manifest file (edit on any machine with kubectl access), update container images to use immutable tags or digests:

    • Find all image: fields under spec.template.spec.containers and spec.template.spec.initContainers.
    • Replace images like: 
      image: my-account.dkr.ecr.us-east-1.amazonaws.com/app:latest
      image: my-account.dkr.ecr.us-east-1.amazonaws.com/app
      with pinned values, for example: 
      image: my-account.dkr.ecr.us-east-1.amazonaws.com/app:1.4.3
      # or
      image: my-account.dkr.ecr.us-east-1.amazonaws.com/app@sha256:<immutable-digest>

    Obtain the exact tag or digest from your ECR repository or image build pipeline.

  4. Apply the updated manifests back to the cluster (run on any machine with kubectl access):

    kubectl apply -f /tmp/<namespace>_<kind>_<name>.yaml

    Repeat for each modified manifest. Kubernetes will roll out new pods using the pinned images.

  5. For non-controlled pods (no owner= in the audit output), edit the pod or its source manifest (run on any machine with kubectl access):

    • If created from a manifest, edit that file similarly to step 3 and re-apply: 
      kubectl apply -f <original-pod-manifest>.yaml
    • If created imperatively and ephemeral, recreate it with a pinned image, e.g.: 
      kubectl delete pod <pod-name> -n <namespace>
      kubectl run <pod-name> -n <namespace> --image=my-account.dkr.ecr.us-east-1.amazonaws.com/app:1.4.3

  6. Verify all pods now use pinned images (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.nodeName // "") as $node
      | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | ((.spec.containers // []) + (.spec.initContainers // []))[]
      | .image as $img
      | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
      | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
        + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
        + (if $node   == ""   then "" else " node=\($node)" end)
        + (if $labels == ""   then "" else " labels=\($labels)" end)
        + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
        + " container=\(.name) image=\($img)"
        + " is_compliant=\(if $ok then "true" else "false" end)"
      ] as $rows
      | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

    Confirm there are no lines with is_compliant=false.

Using kubectl

On any machine with kubectl access:

  1. Identify the non-compliant Pod and image(s)
kubectl get pod -n <NAMESPACE> <POD_NAME> -o jsonpath='{.spec.containers[*].name}{"\n"}{.spec.containers[*].image}{"\n"}{.spec.initContainers[*].name}{"\n"}{.spec.initContainers[*].image}{"\n"}'
  1. Get a digest (or immutable tag) for the image you want to pin
    (Example using ECR; adjust repository and original tag accordingly):
aws ecr describe-images \
  --region <AWS_REGION> \
  --repository-name <ECR_REPO_NAME> \
  --image-ids imageTag=<CURRENT_TAG_OR_LATEST> \
  --query 'imageDetails[0].{imageDigest:imageDigest,imageTags:imageTags}' \
  --output json

Note the imageDigest (e.g. sha256:abcd...) or a specific immutable tag you want to use.

  1. Export the current Pod spec and create a manifest file
kubectl get pod -n <NAMESPACE> <POD_NAME> -o yaml > pod-fixed.yaml

Edit pod-fixed.yaml:

  • Remove Pod status fields and cluster-assigned fields:

    • Delete status: section entirely.
    • Under metadata:, delete fields like uid, resourceVersion, selfLink, creationTimestamp, generateName, managedFields, ownerReferences, and annotations that are tool/cluster-generated (leave only what you need).

  • Under spec.containers and spec.initContainers, change each offending image:

From (examples):

image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/app:latest
# or
image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/app

To either a digest:

image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/app@sha256:abcd1234...

or a specific immutable tag:

image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/app:v1.2.3
  1. Recreate the Pod from the corrected manifest

Because Pods created by higher-level controllers (Deployment, ReplicaSet, DaemonSet, Job, etc.) will be recreated from the controller’s template, you must normally fix the controller instead of the Pod. If this Pod has an owner (see the audit output owner=...), edit that owner object instead (e.g. kubectl edit deployment ...) and update the images there.

Only if this is a standalone Pod (no owner):

kubectl delete pod -n <NAMESPACE> <POD_NAME>
kubectl apply -f pod-fixed.yaml
  1. Verification

Run the benchmark audit query again to confirm compliance:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img)"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Automation
#!/usr/bin/env bash
# Fix CBP C1.10 for Pods on Amazon EKS: ensure containers are not using :latest or untagged images.
# Runs from any machine with:
#   - kubectl configured for the cluster
#   - jq installed
#
# Behavior:
#   - Scans all pods in all namespaces (excluding kube-system, kube-public, kube-node-lease).
#   - Detects containers and initContainers with:
#       * :latest tag, or
#       * no tag at all
#   - For each OFFENDING container, it prompts for the pinned image to use
#     (immutable tag or digest, e.g. repo/app:v1.2.3 or repo/app@sha256:...).
#   - Patches the live Pod spec with kubectl.
#   - Safe to re-run; already-compliant containers are skipped.
#   - At the end, re-runs the benchmark audit jq to show remaining non-compliant Pods.

set -euo pipefail

if ! command -v kubectl >/dev/null 2>&1; then
  echo "kubectl is required but not found in PATH." >&2
  exit 1
fi

if ! command -v jq >/dev/null 2>&1; then
  echo "jq is required but not found in PATH." >&2
  exit 1
fi

# Check access
if ! kubectl auth can-i list pods --all-namespaces >/dev/null 2>&1; then
  echo "Current Kubernetes credentials cannot list pods in all namespaces." >&2
  exit 1
fi

echo "Scanning for Pods with containers using :latest or untagged images (excluding kube-system, kube-public, kube-node-lease)..."
echo

# Collect offending containers (both containers and initContainers)
offenders_json="$(kubectl get pods --all-namespaces -o json | jq -c '
  .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | {
      ns: .metadata.namespace,
      pod: .metadata.name,
      containers: (
        ((.spec.containers // []) | map({type:"container", name:.name, image:.image}))
        +
        ((.spec.initContainers // []) | map({type:"initContainer", name:.name, image:.image}))
      )
    }
  | . as $pod
  | $pod.containers[]
  | . as $c
  | ($c.image | tostring) as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
  | select($ok | not)
  | {
      namespace: $pod.ns,
      pod: $pod.pod,
      ctype: $c.type,
      cname: $c.name,
      image: $img
    }
')"

if [[ -z "${offenders_json}" ]]; then
  echo "No offending containers found. All Pods are compliant."
  exit 0
fi

# Use an associative array to avoid prompting multiple times for same original image
declare -A IMAGE_MAP

echo "The following containers are using :latest or untagged images and need to be pinned:"
echo "${offenders_json}" | jq -r '. | "ns=\(.namespace) pod=\(.pod) type=\(.ctype) container=\(.cname) image=\(.image)"' \
  | sort -u
echo

# Prompt for replacement images
while IFS= read -r line; do
  [[ -z "$line" ]] && continue
  ns="$(echo "$line" | jq -r '.namespace')"
  pod="$(echo "$line" | jq -r '.pod')"
  ctype="$(echo "$line" | jq -r '.ctype')"
  cname="$(echo "$line" | jq -r '.cname')"
  image="$(echo "$line" | jq -r '.image')"

  # Skip if we already have a mapping for this exact original image reference
  if [[ -n "${IMAGE_MAP[$image]+x}" ]]; then
    continue
  fi

  echo "Current image: ${image}"
  echo "  Found in: namespace=${ns} pod=${pod} type=${ctype} container=${cname}"
  echo "Enter pinned image (immutable tag or digest, e.g. repo/app:v1.2.3 or repo/app@sha256:...):"
  read -r new_image

  # Basic sanity check: must not be empty and must not be :latest or untagged
  if [[ -z "$new_image" ]]; then
    echo "Empty input; skipping mapping for ${image}. This container will remain non-compliant."
    continue
  fi
  if [[ "$new_image" =~ :latest$ ]]; then
    echo "New image must not use :latest. Skipping mapping for ${image}."
    continue
  fi
  # Ensure some tag or digest is specified
  if [[ "$new_image" != *"@"* && "$new_image" != *":"* ]]; then
    echo "New image must include a tag or digest. Skipping mapping for ${image}."
    continue
  fi

  IMAGE_MAP["$image"]="$new_image"
  echo "  Mapping set: ${image} -> ${new_image}"
  echo
done <<< "${offenders_json}"

if [[ "${#IMAGE_MAP[@]}" -eq 0 ]]; then
  echo "No mappings defined; nothing will be changed."
  exit 0
fi

echo "Applying image updates to Pods..."
echo

# For each pod, construct a strategic merge patch updating all matching containers/initContainers
# to the new pinned images (if mapped). This is idempotent and can be re-run.
pods_to_patch="$(echo "${offenders_json}" | jq -r '.namespace + "/" + .pod' | sort -u)"

while IFS= read -r ns_pod; do
  [[ -z "$ns_pod" ]] && continue
  ns="${ns_pod%%/*}"
  pod="${ns_pod##*/}"

  # Build patch only if there is at least one container in this pod with a mapping
  pod_json="$(kubectl get pod "${pod}" -n "${ns}" -o json 2>/dev/null || true)"
  if [[ -z "$pod_json" ]]; then
    echo "Pod ${ns}/${pod} no longer exists; skipping."
    continue
  fi

  # Build containers section
  containers_patch="$(echo "$pod_json" | jq -c --argjson nothing '{}' '
    {
      containers: (
        ((.spec.containers // []) | map(
          if (.image // "") as $img
             | $img != null
          then
            . # keep as-is; we will replace images later in bash
          else
            .
          end
        ))
      ),
      initContainers: (
        ((.spec.initContainers // []) | map(
          if (.image // "") as $img
             | $img != null
          then
            .
          else
            .
          end
        ))
      )
    }')"

  # Now in bash, replace image fields according to IMAGE_MAP
  tmp_patch="$(mktemp)"
  echo "${containers_patch}" > "${tmp_patch}"

  # Function to update images in-place within JSON file
  update_images() {
    local json_file="$1"
    local path="$2"  # .containers or .initContainers

    for orig in "${!IMAGE_MAP[@]}"; do
      new="${IMAGE_MAP[$orig]}"
      # Use jq to update matching images
      jq --arg orig "$orig" --arg new "$new" \
        --arg path "$path" '
        . as $root
        | if ($root[$path] | type) == "array" then
            $root
            | .[$path] = (.[ $path ] | map(
                if .image == $orig then
                  .image = $new
                else
                  .
                end
              ))
          else
            $root
          end
        ' "${json_file}" > "${json_file}.tmp" && mv "${json_file}.tmp" "${json_file}"
    done
  }

  update_images "${tmp_patch}" "containers"
  update_images "${tmp_patch}" "initContainers"

  # Remove empty keys if there are no arrays (to avoid unnecessary fields)
  final_patch="$(jq '
    if (.containers | length) == 0 then del(.containers) else . end
    | if (.initContainers | length) == 0 then del(.initContainers) else . end
  ' "${tmp_patch}")"

  # Check if patch actually changes anything (no-op if no images matched)
  current_spec="$(echo "$pod_json" | jq '{containers: .spec.containers, initContainers: .spec.initContainers}')"
  desired_spec="$(echo "$final_patch")"
  if [[ "$(echo "$current_spec" | jq -S '.')" == "$(echo "$desired_spec" | jq -S '.')" ]]; then
    echo "No mapped images found in ${ns}/${pod}; skipping."
    rm -f "${tmp_patch}"
    continue
  fi

  echo "Patching pod ${ns}/${pod}..."
  echo "$final_patch" | kubectl patch pod "${pod}" -n "${ns}" --type merge -p "$(cat)"
  rm -f "${tmp_patch}"
done <<< "${pods_to_patch}"

echo
echo "Re-running compliance check (same logic as benchmark audit)..."
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img)"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'