Container Images Should Not Use The latest Or Untagged Tag

More Info:

Verifies images are pinned to an immutable tag or digest. :latest and untagged images make deployments non-reproducible and hard to audit.

Risk Level

Low

Address

Security

Compliance Standards

• Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Identify non-compliant pods (run on any machine with kubectl access):

   kubectl get pods --all-namespaces -o json | jq -r '
     [ .items[]
     | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
     | .metadata as $m
     | (.spec.nodeName // "") as $node
     | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
     | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
     | ((.spec.containers // []) + (.spec.initContainers // []))[]
     | .image as $img
     | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
     | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
       + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
       + (if $node   == ""   then "" else " node=\($node)" end)
       + (if $labels == ""   then "" else " labels=\($labels)" end)
       + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
       + " container=\(.name) image=\($img)"
       + " is_compliant=\(if $ok then "true" else "false" end)"
     ] as $rows
     | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end' \
     | grep 'is_compliant=false'

2. For each non-compliant pod, determine the owning workload (run on any machine with kubectl access), using the owner= field from step 1. For example, if you see owner=Deployment/default/myapp/...:

   kubectl -n default get deployment myapp -o yaml > myapp-deployment.yaml

3. Edit the manifest locally to pin images to immutable tags or digests (run on any machine with kubectl access). In the downloaded YAML, update each offending container image, for example:

   # BEFORE
   image: gcr.io/my-project/myapp:latest
   # OR
   image: gcr.io/my-project/myapp
   
   # AFTER (example immutable references)
   image: gcr.io/my-project/myapp:v1.2.3
   # or
   image: gcr.io/my-project/myapp@sha256:0123456789abcdef...

   Repeat for spec.template.spec.containers and spec.template.spec.initContainers as needed.

4. Apply the updated manifest to the cluster (run on any machine with kubectl access):

   kubectl apply -f myapp-deployment.yaml

   GKE will roll out new pods using the pinned image; old pods will be terminated according to the controller’s update strategy.

5. For standalone Pods not managed by a controller (no owner= field), recreate them with a pinned image (run on any machine with kubectl access):

   # Export current pod spec
   kubectl -n default get pod mypod -o yaml > mypod.yaml
   
   # Edit mypod.yaml: remove status fields and metadata.fields like uid, resourceVersion, etc.,
   # and pin the image as in step 3, then recreate:
   kubectl -n default delete pod mypod
   kubectl -n default apply -f mypod.yaml

6. Verify all pods are now compliant (run on any machine with kubectl access):

   kubectl get pods --all-namespaces -o json | jq -r '
     [ .items[]
     | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
     | .metadata as $m
     | (.spec.nodeName // "") as $node
     | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
     | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
     | ((.spec.containers // []) + (.spec.initContainers // []))[]
     | .image as $img
     | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
     | "is_compliant=\(if $ok then "true" else "false" end)"
     ] as $rows
     | if ([.[]|select(.|test("is_compliant=false"))] | length) == 0
       then "is_compliant=true"
       else $rows[]
       end'

   Ensure the output is is_compliant=true and no lines contain is_compliant=false.

Using kubectl

On any machine with kubectl access:

1. Identify the non-compliant Pod and its owning controller (from the audit output’s owner= field). If the Pod is standalone (no owner=), fix the Pod spec directly; otherwise, fix the controller (Deployment, StatefulSet, Job, etc.).

2. Export the owning controller’s spec to a local manifest (example for a Deployment):

kubectl get deployment my-app-deployment \
  -n my-namespace \
  -o yaml > my-app-deployment.yaml

3. Edit the manifest locally and pin each affected container image to an immutable tag or digest:

spec:
  template:
    spec:
      containers:
        - name: my-app
          # BEFORE (non-compliant):
          # image: gcr.io/my-project/my-app:latest
          # OR
          # image: gcr.io/my-project/my-app
          # AFTER (compliant, using digest example):
          image: gcr.io/my-project/my-app@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
      # also fix any initContainers if present
      initContainers:
        - name: init-my-app
          image: gcr.io/my-project/init-my-app@sha256:fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210

4. Apply the updated manifest:

kubectl apply -f my-app-deployment.yaml

5. For a standalone Pod (no controller), export, edit, and re-create it:

kubectl get pod my-pod -n my-namespace -o yaml > my-pod.yaml
# edit my-pod.yaml: update .spec.containers[*].image and .spec.initContainers[*].image as above
# remove fields that prevent re-creation:
yq -i 'del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, .status)' my-pod.yaml
kubectl delete pod my-pod -n my-namespace
kubectl apply -f my-pod.yaml

6. Verification (on any machine with kubectl access):

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img)"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

Automation

#!/usr/bin/env bash
#
# Remediate pods using :latest or untagged images by patching their parent
# owner objects (Deployment, StatefulSet, DaemonSet, Job, CronJob) to use
# a pinned tag you specify via a mapping file.
#
# REQUIREMENTS:
#   - Run on any machine with kubectl, jq, and yq installed and configured
#     for the target GKE cluster/context.
#
# IMPORTANT:
#   - This script CANNOT guess safe immutable tags or digests.
#   - You MUST provide a mapping file that says which pinned image to use
#     for each offending image.
#
# MAPPING FILE FORMAT (YAML):
#   # mapping.yaml
#   mappings:
#     - match: "gcr.io/myproj/app:latest"
#       pinned: "gcr.io/myproj/app:v1.2.3"
#     - match: "gcr.io/myproj/sidecar"          # untagged
#       pinned: "gcr.io/myproj/sidecar@sha256:..."
#
# The `match` value must be EXACTLY the image string currently in the spec.
#
# USAGE:
#   ./fix-images.sh mapping.yaml
#
# Idempotency:
#   - If an image already equals the pinned value, it is left unchanged.
#   - If no mapping exists for a bad image, it is reported and skipped.
#   - Safe to re-run: patches reapply the same image values.

set -euo pipefail

MAPPING_FILE="${1:-}"
if [[ -z "${MAPPING_FILE}" ]]; then
  echo "Usage: $0 mapping.yaml" >&2
  exit 1
fi

if [[ ! -f "${MAPPING_FILE}" ]]; then
  echo "Mapping file not found: ${MAPPING_FILE}" >&2
  exit 1
fi

if ! command -v kubectl >/dev/null 2>&1; then
  echo "kubectl is required in PATH" >&2
  exit 1
fi

if ! command -v jq >/dev/null 2>&1; then
  echo "jq is required in PATH" >&2
  exit 1
fi

if ! command -v yq >/dev/null 2>&1; then
  echo "yq is required in PATH (https://mikefarah.gitbook.io/yq/)" >&2
  exit 1
fi

CONTEXT="$(kubectl config current-context 2>/dev/null || true)"
echo "Using kubectl context: ${CONTEXT}"

# Helper: get pinned replacement for a given image from mapping file
get_pinned_image() {
  local image="$1"
  yq -r --arg img "${image}" '
    .mappings[]? | select(.match == $img) | .pinned
  ' "${MAPPING_FILE}" 2>/dev/null | head -n1
}

# Find non-compliant pod containers (excluding control-plane namespaces)
echo "Discovering pods with :latest or untagged images (excluding kube-system, kube-public, kube-node-lease)..."
NON_COMPLIANT_JSON="$(kubectl get pods --all-namespaces -o json | jq '
  .items[]
  | select(.metadata.namespace as $n
           | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | select($own != null)
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or
     (($img | split("/") | last | contains(":")) and
      (($img | endswith(":latest")) | not))) as $ok
  | select($ok | not)
  | {
      ns: $m.namespace,
      pod: $m.name,
      ownerKind: $own.kind,
      ownerName: $own.name,
      image: $img,
      container: .name
    }
')"

if [[ -z "${NON_COMPLIANT_JSON}" ]]; then
  echo "No non-compliant images found."
else
  echo "Found non-compliant images:"
  echo "${NON_COMPLIANT_JSON}" | jq -r '. | "ns=\(.ns) pod=\(.pod) owner=\(.ownerKind)/\(.ownerName) container=\(.container) image=\(.image)"'
fi

# Build unique owner list to patch
OWNERS_JSON="$(printf '%s\n' "${NON_COMPLIANT_JSON}" | jq -s '
  group_by(.ownerKind + "|" + .ns + "|" + .ownerName)
  | map({
      ownerKind: .[0].ownerKind,
      ns: .[0].ns,
      ownerName: .[0].ownerName,
      containers: map({container: .container, image: .image})
    })
')"

if [[ -z "${OWNERS_JSON}" ]] || [[ "${OWNERS_JSON}" == "null" ]]; then
  echo "No owner resources to patch."
else
  echo
  echo "Patching owner workloads to use pinned images..."
fi

PATCH_TMP="$(mktemp)"
SKIPPED_IMAGES=()

# Iterate over each owner workload
echo "${OWNERS_JSON}" | jq -c '.[]' | while read -r OWNER; do
  OWNER_KIND="$(echo "${OWNER}" | jq -r '.ownerKind')"
  NS="$(echo "${OWNER}" | jq -r '.ns')"
  NAME="$(echo "${OWNER}" | jq -r '.ownerName')"

  # Only patch common workload kinds
  case "${OWNER_KIND}" in
    Deployment|StatefulSet|DaemonSet|Job|CronJob)
      ;;
    *)
      echo "Skipping unsupported owner kind ${OWNER_KIND} ${NS}/${NAME}"
      continue
      ;;
  esac

  echo
  echo "Processing ${OWNER_KIND} ${NS}/${NAME} ..."

  # Fetch full YAML of owner
  kubectl get "${OWNER_KIND}" "${NAME}" -n "${NS}" -o yaml > "${PATCH_TMP}.orig.yaml"

  # Prepare yq expression to update containers/initContainers image fields
  YQ_EXPR=". as \$root"

  # Containers
  CONTAINER_PATCHES="$(echo "${OWNER}" | jq -c '.containers[]' || true)"
  if [[ -z "${CONTAINER_PATCHES}" ]]; then
    echo "  No containers recorded for ${OWNER_KIND} ${NS}/${NAME}, skipping."
    continue
  fi

  while read -r C; do
    [[ -z "${C}" ]] && continue
    C_NAME="$(echo "${C}" | jq -r '.container')"
    C_IMAGE="$(echo "${C}" | jq -r '.image')"

    PINNED_IMAGE="$(get_pinned_image "${C_IMAGE}")"
    if [[ -z "${PINNED_IMAGE}" || "${PINNED_IMAGE}" == "null" ]]; then
      echo "  No mapping for image '${C_IMAGE}' (container=${C_NAME}); leaving unchanged. Add to ${MAPPING_FILE} if you want it fixed."
      SKIPPED_IMAGES+=("${C_IMAGE}")
      continue
    fi

    if [[ "${PINNED_IMAGE}" == "${C_IMAGE}" ]]; then
      echo "  ${C_NAME}: image already pinned as ${PINNED_IMAGE}, no change."
      continue
    fi

    echo "  ${C_NAME}: ${C_IMAGE} -> ${PINNED_IMAGE}"

    # Modify both containers and initContainers if present
    YQ_EXPR+="
      | ( .spec.template.spec.containers // [] | map(
            if .name == \"${C_NAME}\" and .image == \"${C_IMAGE}\" then
              .image = \"${PINNED_IMAGE}\"
            else .
            end
          )
        ) as \$c
      | ( .spec.template.spec.initContainers // [] | map(
            if .name == \"${C_NAME}\" and .image == \"${C_IMAGE}\" then
              .image = \"${PINNED_IMAGE}\"
            else .
            end
          )
        ) as \$ic
      | .spec.template.spec.containers = \$c
      | .spec.template.spec.initContainers = (if ( .spec.template.spec.initContainers // null ) != null then \$ic else .spec.template.spec.initContainers end)
    "
  done <<< "${CONTAINER_PATCHES}"

  # Generate patched YAML
  yq eval "${YQ_EXPR}" "${PATCH_TMP}.orig.yaml" > "${PATCH_TMP}.patched.yaml"

  # If no actual change, skip patch
  if diff -q "${PATCH_TMP}.orig.yaml" "${PATCH_TMP}.patched.yaml" >/dev/null 2>&1; then
    echo "  No effective changes for ${OWNER_KIND} ${NS}/${NAME}, skipping apply."
    continue
  fi

  echo "  Applying patched spec to ${OWNER_KIND} ${NS}/${NAME} ..."
  kubectl apply -f "${PATCH_TMP}.patched.yaml"
done

rm -f "${PATCH_TMP}" "${PATCH_TMP}.orig.yaml" "${PATCH_TMP}.patched.yaml" 2>/dev/null || true

if [[ "${#SKIPPED_IMAGES[@]}" -gt 0 ]]; then
  echo
  echo "The following images were found without mappings and were NOT changed:"
  printf '  %s\n' "${SKIPPED_IMAGES[@]}" | sort -u
  echo "Add them to ${MAPPING_FILE} if you want to pin them."
fi

echo
echo "Waiting for pods to roll out with updated images..."
# This will not block forever, but gives time for controllers to create new pods
sleep 10

echo
echo "Verification: re-running compliance check..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img)"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end
'