Mutable Image Tags Should Use imagePullPolicy Always

More Info:

Verifies containers on mutable tags set imagePullPolicy: Always so a node cannot silently run a stale cached image.

Risk Level

Low

Address

Security

Compliance Standards

• Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

1. On any machine with kubectl access, list the non-system pods that are non‑compliant so you know what to fix:

   kubectl get pods --all-namespaces -o json | jq -r '
     [ .items[]
     | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
     | .metadata as $m
     | (.spec.nodeName // "") as $node
     | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
     | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
     | ((.spec.containers // []) + (.spec.initContainers // []))[]
     | .image as $img
     | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
     | ($immutable or (.imagePullPolicy == "Always")) as $ok
     | select($ok | not)
     | "ns=\($m.namespace) pod=\($m.name) container=\(.name) image=\($img)"
   ][]'

2. For each listed pod, determine and edit its owning workload (Deployment, StatefulSet, DaemonSet, Job, CronJob, etc.) rather than the pod directly. For a known owner kind/name/namespace, patch all containers to set imagePullPolicy: Always. Example for a Deployment:

   kubectl -n <namespace> patch deployment <deployment-name> \
     --type='json' \
     -p='[{"op":"add","path":"/spec/template/spec/containers/0/imagePullPolicy","value":"Always"}]'

   If there are multiple containers, repeat with /containers/1, /containers/2, etc., adjusting indexes as needed.

3. If the pod has initContainers using mutable tags, also patch those to use imagePullPolicy: Always. Example:

   kubectl -n <namespace> patch deployment <deployment-name> \
     --type='json' \
     -p='[{"op":"add","path":"/spec/template/spec/initContainers/0/imagePullPolicy","value":"Always"}]'

4. If the pod is not controlled by a higher-level workload (no ownerReferences), edit the pod spec manifest used to create it (for example a YAML file in your source/IaC repository) and add imagePullPolicy: Always under each container and initContainer that uses a mutable tag, then re‑apply:

   kubectl apply -f <path-to-pod-manifest>.yaml

   If it was created imperatively and has no manifest, recreate it with --image-pull-policy=Always (for kubectl run) or by writing a manifest that sets imagePullPolicy: Always.

5. In AKS, system and add‑on pods in kube-system are managed by Azure and excluded by the check. Do not try to modify those via kubectl; for AKS add‑ons that you own via ARM/Bicep/Terraform/Helm, update those definitions there to include imagePullPolicy: Always for any mutable tags, then redeploy.

6. After updating all relevant workloads/manifests and allowing pods to be recreated, verify remediation from any machine with kubectl access:

   kubectl get pods --all-namespaces -o json | jq -r '
     [ .items[]
     | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
     | .metadata as $m
     | (.spec.nodeName // "") as $node
     | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
     | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
     | ((.spec.containers // []) + (.spec.initContainers // []))[]
     | .image as $img
     | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
     | ($immutable or (.imagePullPolicy == "Always")) as $ok
   ] | if (map(select($ok | not)) | length) == 0
       then "is_compliant=true"
       else "is_compliant=false"
     end'

Using kubectl

On any machine with kubectl access:

1. Identify non‑compliant Pods and their owning controllers

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
  | ($immutable or (.imagePullPolicy == "Always")) as $ok
  | select($ok | not)
  | "ns=\($m.namespace) pod=\($m.name) owner_kind=\($own.kind // "Pod") owner_name=\($own.name // $m.name)"
  ] | unique[]'

For any line where owner_kind=Pod, you must edit the Pod manifest source (YAML used to create it). For controller‑owned Pods (Deployment, StatefulSet, DaemonSet, Job, CronJob), update the controller spec and let it recreate Pods.

2. Patch a single controller to set imagePullPolicy: Always for mutable tags

Example for a Deployment (repeat similarly for other controllers):

kubectl -n <namespace> patch deployment <name> --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/imagePullPolicy",
    "value": "Always"
  }
]'

If there are multiple containers or initContainers, patch each index as needed, or edit instead:

kubectl -n <namespace> edit deployment <name>

In the opened YAML, under each spec.template.spec.containers[] or spec.template.spec.initContainers[] that uses a mutable image tag (e.g. :latest or no tag), add:

imagePullPolicy: Always

Save and exit; Kubernetes will roll out updated Pods.

3. For directly‑created Pods (no controller)

Edit and re‑apply from your original manifest:

kubectl -n <namespace> get pod <pod-name> -o yaml > /tmp/pod-fixed.yaml

Edit /tmp/pod-fixed.yaml:

• Remove Kubernetes‑managed fields such as status:, metadata.resourceVersion, metadata.uid, metadata.creationTimestamp, metadata.managedFields, metadata.ownerReferences.
• Under each spec.containers[] or spec.initContainers[] that uses a mutable image tag, ensure:

imagePullPolicy: Always

Apply:

kubectl apply -f /tmp/pod-fixed.yaml

If the Pod already exists, you may need to delete and recreate it:

kubectl -n <namespace> delete pod <pod-name>
kubectl apply -f /tmp/pod-fixed.yaml

4. Verification

Run the benchmark audit command again from any machine with kubectl access:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
  | ($immutable or (.imagePullPolicy == "Always")) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img) imagePullPolicy=\(.imagePullPolicy // "unset")"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

All reported lines should now show is_compliant=true or the final line is_compliant=true.

Automation

#!/usr/bin/env bash
# Purpose: For all Pods in the cluster using mutable image tags (:latest or untagged),
#          set imagePullPolicy: Always on each affected container.
#
# Scope:   Run on any machine with kubectl access to the AKS cluster.
# Re-run:  Safe to run multiple times (idempotent).

set -euo pipefail

# REQUIREMENTS:
# - kubectl configured to point to the target AKS cluster.
# - jq available in PATH.

echo "Scanning for Pods with mutable image tags and non-Always imagePullPolicy ..."

# Get all pods (excluding core infra namespaces) as JSON
ALL_PODS_JSON="$(kubectl get pods --all-namespaces -o json)"

# Build a list of Pod namespace/name pairs that have at least one non-compliant container
# We re-use the same mutable/immutable logic as the audit command.
mapfile -t TARGET_PODS < <(
  jq -r '
    .items[]
    | select(.metadata.namespace as $n
             | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | select(
        ((.spec.containers // []) + (.spec.initContainers // []))[]
        | .image as $img
        | (($img | contains("@")) or
           (($img | split("/") | last | contains(":")) and
            (($img | endswith(":latest")) | not))) as $immutable
        | ($immutable or (.imagePullPolicy == "Always")) as $ok
        | $ok | not
      )
    | "\(.metadata.namespace) \(.metadata.name)"
  ' <<<"$ALL_PODS_JSON" | sort -u
)

if [ "${#TARGET_PODS[@]}" -eq 0 ]; then
  echo "No Pods with mutable image tags and non-Always imagePullPolicy found."
  echo "Cluster is compliant with CBP C1.11 for existing Pods."
else
  echo "Found ${#TARGET_PODS[@]} non-compliant Pods. Patching imagePullPolicy to Always ..."

  for line in "${TARGET_PODS[@]}"; do
    ns="$(cut -d' ' -f1 <<<"$line")"
    name="$(cut -d' ' -f2 <<<"$line")"

    echo "Processing Pod ${ns}/${name} ..."

    # Fetch the live Pod JSON
    POD_JSON="$(kubectl -n "$ns" get pod "$name" -o json 2>/dev/null || true)"
    if [ -z "$POD_JSON" ]; then
      echo "  Skipped: Pod ${ns}/${name} no longer exists."
      continue
    fi

    # Build a JSON patch that sets imagePullPolicy: Always on all mutable-tag containers
    PATCH="$(jq '
      def mutable($img):
        # immutable if:
        #   - contains("@"), or
        #   - has explicit tag not equal to :latest
        ( ($img | contains("@")) or
          (($img | split("/") | last | contains(":")) and
           (($img | endswith(":latest")) | not))
        ) | not;

      # Helper to transform containers array
      def fix_containers:
        ( . // [] )
        | map(
            if mutable(.image)
            then
              .imagePullPolicy = "Always"
            else
              .
            end
          );

      {
        "spec": {
          "containers": (.spec.containers | fix_containers),
          "initContainers": (.spec.initContainers | fix_containers)
        }
      }
    ' <<<"$POD_JSON")"

    # If PATCH is empty or unchanged, skip patching
    if [ -z "$PATCH" ] || [ "$PATCH" = "{}" ]; then
      echo "  No mutable-tag containers found, skipping."
      continue
    fi

    # Apply strategic merge patch (idempotent)
    echo "$PATCH" | kubectl -n "$ns" patch pod "$name" --type merge -p "$(cat)" >/dev/null 2>&1 || {
      echo "  Warning: failed to patch Pod ${ns}/${name}. It may be controlled by a higher-level resource."
      echo "  Consider updating its controller manifest (Deployment/DaemonSet/StatefulSet/Job/CronJob)."
      continue
    }

    echo "  Patched Pod ${ns}/${name}."
  done
fi

echo
echo "Verifying compliance after patching ..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n
           | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or
     (($img | split("/") | last | contains(":")) and
      (($img | endswith(":latest")) | not))) as $immutable
  | ($immutable or (.imagePullPolicy == "Always")) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img) imagePullPolicy=\(.imagePullPolicy // "unset")"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | map(select(contains("is_compliant=false"))) | length) == 0
    then "is_compliant=true"
    else ($rows[] | select(contains("is_compliant=false")))
    end
'

echo "Automation run complete."