Skip to main content

Mutable Image Tags Should Use imagePullPolicy Always

More Info:​

Verifies containers on mutable tags set imagePullPolicy: Always so a node cannot silently run a stale cached image.

Risk Level​

Low

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. Identify noncompliant Pods

    • Run on any machine with kubectl access: 
      kubectl get pods --all-namespaces -o json | jq -r '
        [ .items[]
        | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
        | .metadata as $m
        | (.spec.nodeName // "") as $node
        | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
        | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
        | ((.spec.containers // []) + (.spec.initContainers // []))[]
        | .image as $img
        | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
        | ($immutable or (.imagePullPolicy == "Always")) as $ok
        | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
          + " container=\(.name) image=\($img) imagePullPolicy=\(.imagePullPolicy // "unset")"
          + " is_compliant=\(if $ok then "true" else "false" end)"
        ][]' | grep 'is_compliant=false'

  2. Decide how the Pod is managed

    • For each noncompliant line, note the owner= field (if present):
      • If owner= shows a controller (Deployment, StatefulSet, DaemonSet, Job, CronJob, ReplicaSet, etc.), you must fix that controller’s manifest, not the Pod directly.
      • If there is no owner= field, the Pod is standalone and can be edited directly.

  3. Edit the owning controller manifest to set imagePullPolicy: Always (preferred)

    • Run on any machine with kubectl access; example for a Deployment (adapt kind/namespace/name as needed): 
      kubectl -n <namespace> get deployment <name> -o yaml > /tmp/deploy-<name>.yaml
    • Open /tmp/deploy-<name>.yaml in an editor and, under every spec.template.spec.containers[] and spec.template.spec.initContainers[] entry that uses a mutable image tag (e.g. image: repo/app:latest or untagged), add or change: 
      imagePullPolicy: Always
    • Apply the updated manifest: 
      kubectl apply -f /tmp/deploy-<name>.yaml
    • Repeat for other controller types (StatefulSet, DaemonSet, Job, CronJob) using the corresponding kubectl get <kind> and kubectl apply -f commands.

  4. Edit standalone Pods (if any) to set imagePullPolicy: Always

    • Run on any machine with kubectl access: 
      kubectl -n <namespace> get pod <pod-name> -o yaml > /tmp/pod-<pod-name>.yaml
    • In /tmp/pod-<pod-name>.yaml, under each affected containers[] and initContainers[] entry using a mutable image tag, add or change: 
      imagePullPolicy: Always
    • Delete and recreate the Pod with the modified manifest: 
      kubectl -n <namespace> delete pod <pod-name>
      kubectl apply -f /tmp/pod-<pod-name>.yaml

  5. Confirm Pods are refreshed (for controllers)

    • After applying controller changes, ensure new Pods are running with updated templates: 
      kubectl -n <namespace> rollout status deployment/<name>
    • Use the analogous rollout status or describe commands for other controller types, as applicable.

  6. Verify compliance

    • Run on any machine with kubectl access: 
      kubectl get pods --all-namespaces -o json | jq -r '
        [ .items[]
        | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
        | .metadata as $m
        | (.spec.nodeName // "") as $node
        | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
        | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
        | ((.spec.containers // []) + (.spec.initContainers // []))[]
        | .image as $img
        | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
        | ($immutable or (.imagePullPolicy == "Always")) as $ok
        | "is_compliant=\(if $ok then "true" else "false" end)"
        ] as $rows
        | if ($rows | map(select(. == "is_compliant=false")) | length) == 0
          then "is_compliant=true"
          else $rows[]
          end'
    • Confirm the output is is_compliant=true and no lines show is_compliant=false.
Using kubectl

On any machine with kubectl access:

  1. Identify non‑compliant Pods (for reference)
kubectl get pods --all-namespaces -o wide
  1. For each affected Pod, edit its manifest to set imagePullPolicy: Always on any container using a mutable tag (e.g. :latest or untagged). This must be done on the owning workload (Deployment, StatefulSet, DaemonSet, Job, CronJob, etc.), not the Pod itself.

Example for a Deployment:

kubectl -n YOUR_NAMESPACE get deployment YOUR_DEPLOYMENT -o yaml > /tmp/deploy.yaml

Edit /tmp/deploy.yaml so each mutable image has imagePullPolicy: Always, for example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: YOUR_DEPLOYMENT
  namespace: YOUR_NAMESPACE
spec:
  template:
    spec:
      containers:
        - name: app
          image: your-registry/your-image:latest
          imagePullPolicy: Always
        - name: sidecar
          image: your-registry/another-image
          imagePullPolicy: Always
      initContainers:
        - name: init-app
          image: your-registry/init-image:latest
          imagePullPolicy: Always

Apply the updated manifest:

kubectl apply -f /tmp/deploy.yaml

Repeat the same pattern for other controller types:

  • StatefulSet:

    kubectl -n YOUR_NAMESPACE get statefulset YOUR_STATEFULSET -o yaml > /tmp/ss.yaml
    # edit /tmp/ss.yaml to add imagePullPolicy: Always where needed
    kubectl apply -f /tmp/ss.yaml

  • DaemonSet:

    kubectl -n YOUR_NAMESPACE get daemonset YOUR_DAEMONSET -o yaml > /tmp/ds.yaml
    # edit /tmp/ds.yaml to add imagePullPolicy: Always where needed
    kubectl apply -f /tmp/ds.yaml

  • Job:

    kubectl -n YOUR_NAMESPACE get job YOUR_JOB -o yaml > /tmp/job.yaml
    # edit /tmp/job.yaml to add imagePullPolicy: Always where needed
    kubectl apply -f /tmp/job.yaml

  • CronJob:

    kubectl -n YOUR_NAMESPACE get cronjob YOUR_CRONJOB -o yaml > /tmp/cj.yaml
    # edit /tmp/cj.yaml to add imagePullPolicy: Always where needed
    kubectl apply -f /tmp/cj.yaml

For Pods created directly (no controller ownerReference), patch them in place (they will not be recreated automatically if deleted):

kubectl -n YOUR_NAMESPACE edit pod YOUR_POD

and add imagePullPolicy: Always under each container/initContainer using a mutable tag.

  1. Verification

Run the same compliance check to confirm all remaining mutable-tag containers now have imagePullPolicy: Always:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
  | ($immutable or (.imagePullPolicy == "Always")) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img) imagePullPolicy=\(.imagePullPolicy // "unset")"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Automation
#!/usr/bin/env bash
set -euo pipefail

# Automation for: Mutable Image Tags Should Use imagePullPolicy Always
# Scope: any machine with kubectl access to the EKS cluster

# REQUIREMENTS:
# - kubectl configured with cluster-admin or equivalent rights
# - jq and yq (https://github.com/mikefarah/yq) installed
# - kubectl must have access to all namespaces you want to fix

# CONFIGURATION
DRY_RUN="${DRY_RUN:-false}"     # set DRY_RUN=true to preview changes
NAMESPACES_EXCLUDE_REGEX='^(kube-system|kube-public|kube-node-lease)$'

tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT

log() { printf '%s\n' "$*" >&2; }

if ! command -v kubectl >/dev/null 2>&1; then
  log "kubectl not found in PATH"
  exit 1
fi
if ! command -v jq >/dev/null 2>&1; then
  log "jq not found in PATH"
  exit 1
fi
if ! command -v yq >/dev/null 2>&1; then
  log "yq not found in PATH (https://github.com/mikefarah/yq)"
  exit 1
fi

# 1. Discover noncompliant Pods (mutable image + imagePullPolicy != Always)
log "Discovering noncompliant Pods..."
noncompliant_json="$tmpdir/noncompliant-pods.json"

kubectl get pods --all-namespaces -o json > "$tmpdir/all-pods.json"

jq '
  .items[]
  | select(.metadata.namespace | test("'"$NAMESPACES_EXCLUDE_REGEX"'") | not)
  | . as $pod
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | . as $c
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
  | select(($immutable | not) and (.imagePullPolicy != "Always"))
  | {
      namespace: $pod.metadata.namespace,
      name: $pod.metadata.name,
      container: .name,
      image: .image
    }
' "$tmpdir/all-pods.json" > "$noncompliant_json"

if ! [ -s "$noncompliant_json" ]; then
  log "No noncompliant Pods found. Cluster is already compliant."
  exit 0
fi

log "Noncompliant containers:"
jq -r '. | "ns=\(.namespace) pod=\(.name) container=\(.container) image=\(.image)"' "$noncompliant_json" | sort -u >&2

# 2. Group by Pod (ns/name)
pod_keys_file="$tmpdir/pod-keys.txt"
jq -r '.namespace + "/" + .name' "$noncompliant_json" | sort -u > "$pod_keys_file"

# 3. Patch each Pod manifest: set imagePullPolicy: Always on mutable-tag containers
while IFS=/ read -r ns pod; do
  [ -n "$ns" ] || continue

  log "Processing Pod ${ns}/${pod}..."

  pod_yaml="$tmpdir/${ns}_${pod}.yaml"
  kubectl get pod "$pod" -n "$ns" -o yaml > "$pod_yaml"

  # Build yq expression to update only containers flagged in noncompliant_json
  # For this Pod, build a JSON list of container names needing change
  containers_json="$tmpdir/${ns}_${pod}_containers.json"
  jq --arg ns "$ns" --arg pod "$pod" '
    select(.namespace == $ns and .name == $pod)
    | .container
  ' "$noncompliant_json" > "$containers_json"

  if ! [ -s "$containers_json" ]; then
    log "  No matching containers for ${ns}/${pod} in noncompliant list, skipping."
    continue
  fi

  # Turn container names into a CSV string for yq
  container_csv="$(tr '\n' ',' < "$containers_json" | sed 's/,$//')"

  # Patch spec.containers and spec.initContainers where container name is in list
  patched_yaml="$tmpdir/${ns}_${pod}_patched.yaml"
  yq '
    . as $root
    | (
        .spec.containers // [] |
        map(
          if (.name | split(",") | inside(['"$container_csv"'])) or (("'"$container_csv"'" | contains("," + .name + ","))) then
            .imagePullPolicy = "Always"
          else
            .
          end
        )
      ) as $c
    | (
        .spec.initContainers // [] |
        map(
          if (.name | split(",") | inside(['"$container_csv"'])) or (("'"$container_csv"'" | contains("," + .name + ","))) then
            .imagePullPolicy = "Always"
          else
            .
          end
        )
      ) as $ic
    | $root
    | .spec.containers = $c
    | .spec.initContainers = $ic
  ' "$pod_yaml" > "$patched_yaml"

  if [ "$DRY_RUN" = "true" ]; then
    log "  DRY_RUN=true, showing diff for ${ns}/${pod}:"
    diff -u "$pod_yaml" "$patched_yaml" || true
  else
    log "  Applying patched manifest for ${ns}/${pod}..."
    kubectl apply -n "$ns" -f "$patched_yaml"
  fi

done < "$pod_keys_file"

if [ "$DRY_RUN" = "true" ]; then
  log "DRY_RUN=true, no changes were applied."
  exit 0
fi

# 4. Verification (re-run audit logic, but only print remaining noncompliant items)
log "Verifying compliance after remediation..."
verify_output="$tmpdir/verify.txt"

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
  | ($immutable or (.imagePullPolicy == "Always")) as $ok
  | select($ok | not)
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img) imagePullPolicy=\(.imagePullPolicy // "unset")"
    + " is_compliant=false"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end
' > "$verify_output"

if grep -q 'is_compliant=false' "$verify_output"; then
  log "Some Pods are still noncompliant:"
  grep 'is_compliant=false' "$verify_output" >&2
  exit 2
else
  log "Verification successful: is_compliant=true"
  cat "$verify_output"
fi