Skip to main content

Mutable Image Tags Should Use imagePullPolicy Always

More Info:​

Verifies containers on mutable tags set imagePullPolicy: Always so a node cannot silently run a stale cached image.

Risk Level​

Low

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. Identify noncompliant Pods (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.nodeName // "") as $node
      | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | ((.spec.containers // []) + (.spec.initContainers // []))[]
      | .image as $img
      | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
      | ($immutable or (.imagePullPolicy == "Always")) as $ok
      | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
        + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
        + (if $node   == ""   then "" else " node=\($node)" end)
        + (if $labels == ""   then "" else " labels=\($labels)" end)
        + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
        + " container=\(.name) image=\($img) imagePullPolicy=\(.imagePullPolicy // "unset")"
        + " is_compliant=\(if $ok then "true" else "false" end)"
      ] as $rows
      | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end' \
    | grep 'is_compliant=false' || true

  2. For each noncompliant Pod that is not controlled by a higher‑level object (no owner= in the line), edit the Pod spec to set imagePullPolicy: Always (run on any machine with kubectl access):

    kubectl edit pod POD_NAME -n POD_NAMESPACE

    In the editor, under each affected containers: and initContainers: entry using a mutable image tag (e.g. :latest or untagged), add or change:

    imagePullPolicy: Always

  3. For Pods controlled by a higher-level object (Deployment, StatefulSet, DaemonSet, Job, CronJob), identify the controller kind/name from the owner= field in step 1 and edit that object instead (run on any machine with kubectl access). For example, for a Deployment:

    kubectl edit deployment DEPLOYMENT_NAME -n CONTROLLER_NAMESPACE

    Then, under .spec.template.spec.containers[] and .spec.template.spec.initContainers[] using mutable tags, set:

    imagePullPolicy: Always

  4. If you manage manifests as files (e.g. via Git), update them to match the live changes so they persist (run on your manifest authoring machine):

    • Open each YAML defining the affected Pod/Deployment/etc.
    • Under every container using a mutable image tag, set: 
      imagePullPolicy: Always
    • Apply the updated manifests: 
      kubectl apply -f PATH_TO_UPDATED_MANIFEST.yaml

  5. Allow the controllers to roll out updated Pods; monitor rollout status where applicable (run on any machine with kubectl access), for example:

    kubectl rollout status deployment DEPLOYMENT_NAME -n CONTROLLER_NAMESPACE

  6. Verify compliance (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.nodeName // "") as $node
      | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | ((.spec.containers // []) + (.spec.initContainers // []))[]
      | .image as $img
      | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
      | ($immutable or (.imagePullPolicy == "Always")) as $ok
      | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
        + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
        + (if $node   == ""   then "" else " node=\($node)" end)
        + (if $labels == ""   then "" else " labels=\($labels)" end)
        + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
        + " container=\(.name) image=\($img) imagePullPolicy=\(.imagePullPolicy // "unset")"
        + " is_compliant=\(if $ok then "true" else "false" end)"
      ] as $rows
      | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

    Confirm there are no lines with is_compliant=false.

Using kubectl

On any machine with kubectl access:

  1. Identify the non‑compliant Pod(s) (example uses one Pod in namespace default named example-pod):
kubectl get pod example-pod -n default -o yaml > /tmp/example-pod.yaml
  1. Edit the manifest to set imagePullPolicy: Always for every container using a mutable tag (e.g. :latest or untagged). For example, update /tmp/example-pod.yaml so relevant containers look like:
spec:
  containers:
    - name: app
      image: gcr.io/PROJECT/IMAGE:latest
      imagePullPolicy: Always
  initContainers:
    - name: init-app
      image: gcr.io/PROJECT/INIT-IMAGE:latest
      imagePullPolicy: Always
  1. Because Pods from higher-level controllers (Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob) will be recreated from their owner spec, you must usually patch the controller instead of the live Pod. For example, if the Pod is owned by a Deployment example-deploy in namespace default:
kubectl patch deployment example-deploy -n default --type=json -p='[
  {
    "op": "replace",
    "path": "/spec/template/spec/containers/0/imagePullPolicy",
    "value": "Always"
  }
]'

Repeat for each container index and any initContainers that use mutable tags, adjusting the path:

kubectl patch deployment example-deploy -n default --type=json -p='[
  {
    "op": "replace",
    "path": "/spec/template/spec/initContainers/0/imagePullPolicy",
    "value": "Always"
  }
]'

If the Pod is not managed by a controller (no ownerReferences), delete and recreate it from the fixed manifest:

kubectl delete pod example-pod -n default
kubectl apply -f /tmp/example-pod.yaml
  1. Verification (all namespaces, excluding system namespaces, matching the check):
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
  | ($immutable or (.imagePullPolicy == "Always")) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img) imagePullPolicy=\(.imagePullPolicy // "unset")"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Automation
#!/usr/bin/env bash
#
# Fix mutable image tags that do not use imagePullPolicy: Always
# Scope: any machine with kubectl access to the GKE cluster
# Requirements: kubectl, jq, and (for server-side apply) Kubernetes >= 1.18

set -euo pipefail

# 1. Discover non-compliant plain Pods (excluding system namespaces)
echo "[*] Discovering non-compliant Pods..."
non_compliant_json="$(kubectl get pods --all-namespaces -o json | jq '
  .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | . as $pod
  | ((.spec.containers // []) + (.spec.initContainers // [])) as $all
  | [ range(0; $all|length) ] as $idxs
  | $idxs[]
  | $all[.]
  | . as $c
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
  | ($immutable or (.imagePullPolicy == "Always")) as $ok
  | select($ok | not)
  | {
      namespace: $pod.metadata.namespace,
      name: $pod.metadata.name,
      uid: $pod.metadata.uid,
      containerName: .name
    }
' || true)"

if [[ -z "${non_compliant_json}" ]]; then
  echo "[*] No non-compliant Pods found."
else
  echo "[*] Non-compliant Pods found. Patching..."
fi

# 2. Patch each non-compliant Pod container with a JSONPatch to set imagePullPolicy: Always
#    Note: Pods created by controllers (Deployments, StatefulSets, etc.) will be recreated
#    from their controllers; you must also fix the controller manifests separately.
if [[ -n "${non_compliant_json}" ]]; then
  # Ensure we iterate over unique (ns,name,container) tuples
  echo "${non_compliant_json}" | jq -c '. ' | sort -u | while read -r item; do
    ns="$(echo "${item}" | jq -r '.namespace')"
    name="$(echo "${item}" | jq -r '.name')"
    cname="$(echo "${item}" | jq -r '.containerName')"

    echo "[*] Patching Pod ${ns}/${name}, container ${cname}..."
    # Build and apply a jsonpatch that adds/replaces imagePullPolicy on the specific container
    # We have to patch both containers and initContainers (if present) where names match.
    patch="$(kubectl get pod "${name}" -n "${ns}" -o json | jq --arg cname "${cname}" '
      def set_policy(path):
        (path // []) as $arr
        | [ range(0; ($arr|length)) ]
        | map(select($arr[.] .name == $cname))
        | map({ op:"replace", path:(path|tojson|fromjson + "/" + (tostring) + "/imagePullPolicy"), value:"Always" });
      (set_policy(.spec.containers) + set_policy(.spec.initContainers)) as $ops
      | if ($ops | length) == 0 then [] else $ops end
    ')"

    # If there is nothing to patch (container disappeared), skip
    if [[ "${patch}" == "[]" ]] || [[ -z "${patch}" ]]; then
      echo "    - Container not present anymore; skipping."
      continue
    fi

    echo "${patch}" | kubectl patch pod "${name}" -n "${ns}" --type=json -p "$(cat)"
  done
fi

# 3. Verification – re-run the audit and ensure all items are compliant
echo "[*] Verifying compliance..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | ((.spec.containers // []) + (.spec.initContainers // []))[]
  | .image as $img
  | (($img | contains("@")) or (($img | split("/") | last | contains(":")) and (($img | endswith(":latest")) | not))) as $immutable
  | ($immutable or (.imagePullPolicy == "Always")) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\($img) imagePullPolicy=\(.imagePullPolicy // "unset")"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | map(select(contains("is_compliant=false"))) | length) == 0
    then "is_compliant=true"
    else ($rows[] | select(contains("is_compliant=false")))
    end
'

echo "[*] Automation run complete."

Usage (run on any machine with kubectl access):

bash ./fix-imagepullpolicy-always.sh

This script is idempotent and can be safely re-run; only containers with mutable image tags that lack imagePullPolicy: Always are patched.