Configure Image Provenance Using ImagePolicyWebhook

More Info:

The ImagePolicyWebhook admission controller can enforce image provenance so that only trusted, verified images are admitted to the cluster.

Risk Level

High

Address

Security

Compliance Standards

• CIS Kubernetes

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Review current admission configuration for ImagePolicyWebhook

   • On any machine with kubectl access:

     # AdmissionConfiguration is usually passed via --admission-control-config-file or --admission-control-config
     # Try to discover it from the kube-apiserver static pod manifest
     sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep -E 'admission-control-config'

   • If you find a config file path (for example /etc/kubernetes/admission-config.yaml), inspect it:

     sudo cat /etc/kubernetes/admission-config.yaml

   • Confirm whether there is a ImagePolicyWebhook plugin section configured; if not, the feature is not in use.

2. Determine the desired image provenance mechanism and trust policy

   • Decide what constitutes a “trusted image” in your environment (e.g., signed by a particular key, coming from a specific registry/project, or verified by an external attestation service).
   • Choose or design the webhook backend that will enforce this policy (custom admission webhook service, in-cluster or external, or a supported third‑party image verification solution).
   • Document:
     • The verification method (signing/attestation technology).
     • The registries/namespaces allowed.
     • Failure policy (reject on verification error vs allow).

3. Design or validate the ImagePolicyWebhook AdmissionConfiguration

   • Draft an AdmissionConfiguration manifest including the ImagePolicyWebhook plugin that points to your webhook backend and enforces your policy decisions. For example (adjust URLs, timeouts, and CABundle to your environment):

     apiVersion: apiserver.k8s.io/v1alpha1
     kind: AdmissionConfiguration
     plugins:
       - name: ImagePolicyWebhook
         configuration:
           imagePolicy:
             kubeConfigFile: "/etc/kubernetes/image-policy-webhook.kubeconfig"
             allowTTL: 50
             denyTTL: 50
             retryBackoff: 500
             defaultAllow: false

   • Ensure the referenced kubeconfig (or other connection method your Kubernetes version supports) exists and points to your policy webhook service.
   • Do not apply yet; validate it against Kubernetes version compatibility and your webhook implementation docs.

4. Stage and test the webhook service and policy in a non‑production environment

   • Deploy the admission webhook service (if not already running) in a test or staging cluster using manifests:

     kubectl apply -f /path/to/image-policy-webhook-deployment.yaml
     kubectl apply -f /path/to/image-policy-webhook-service.yaml

   • Configure the apiserver in that environment to use your drafted AdmissionConfiguration and restart the apiserver as required by your distribution.
   • Test with sample workloads:

     kubectl run unsigned-test --image=UNTRUSTED_IMAGE
     kubectl run signed-test   --image=TRUSTED_IMAGE

   • Confirm that untrusted images are rejected and trusted ones are admitted, and refine policy if behavior is not as intended.

5. Promote the configuration to production and monitor

   • Copy the validated AdmissionConfiguration file and supporting artifacts (kubeconfig, CA certs) to the production control plane nodes, then update the kube-apiserver manifest (or equivalent) to reference it via the appropriate flag. This is done outside kubectl (host-level change) and will restart the apiserver.
   • After the apiserver restarts, verify that workloads using untrusted images are rejected and trusted images succeed:

     kubectl run provenance-test-bad --image=UNTRUSTED_IMAGE
     kubectl run provenance-test-good --image=TRUSTED_IMAGE

   • Review apiserver and webhook logs for errors or excessive denials and adjust policy if needed.

6. Document, periodically review, and re‑verify

   • Record: where the AdmissionConfiguration file lives, what the policy rules are, and who maintains the webhook service.
   • On a recurring basis, re‑check configuration and effectiveness:

     # Confirm the ImagePolicyWebhook plugin remains configured
     sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep -E 'admission-control-config'
     
     # Confirm the webhook endpoint is reachable from the cluster
     kubectl get pods,svc -n NAMESPACE_OF_WEBHOOK

   • Re-run functional tests (trusted vs untrusted images) after significant cluster or policy changes to ensure image provenance enforcement remains active.

Using kubectl

# 1) Check if any ImagePolicyWebhook configuration exists
# Run on: any machine with kubectl access

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
  -o wide

What to look for (problem indicators)

• No webhook configuration related to image policy or provenance (for example names not mentioning image-policy, imageprovenance, cosign, notary, policy, etc.).
• This usually means no image provenance enforcement is configured at all.

---

# 2) Inspect likely image-policy related ValidatingWebhookConfiguration(s)

# Adjust the name(s) based on the output of the previous command
kubectl get validatingwebhookconfigurations <webhook-name> -o yaml

What to look for (problem indicators)

• webhooks[].rules do not include apiGroups: [""] and resources: ["pods"] or other workload types (deployments, replicasets, statefulsets, etc.) — then pod/image creation may bypass the policy.
• failurePolicy is Ignore instead of Fail — unverified images may still be admitted.
• sideEffects is unset or incorrect (should typically be None).
• clientConfig.service points to a non‑existent namespace/service, or wrong path/port.
• namespaceSelector or objectSelector exclude important namespaces (e.g., default application namespaces), leaving workloads unenforced.
• No mention in annotations or configuration of trusted registries, signatures, or verification policy.

---

# 3) Check if any MutatingWebhookConfiguration is being used for image policy
kubectl get mutatingwebhookconfigurations -o yaml

What to look for (problem indicators)

• No mutating webhook that adjusts image references to a verified form (if your design expects this).
• Any mutating webhook that weakens image references (e.g., stripping digests or pinning to :latest) instead of strengthening provenance.

---

# 4) Discover and inspect policy engine components (if any)
# Common namespaces; adapt or add your org’s namespaces as needed
kubectl get pods -A | egrep -i 'image|policy|cosign|notary|kyverno|connaisseur|gatekeeper'

# Example: inspect a policy engine namespace
kubectl get all -n <policy-namespace>
kubectl describe deployment <policy-deployment> -n <policy-namespace>
kubectl get configmap,secret -n <policy-namespace> -o wide

What to look for (problem indicators)

• No policy engine or image-verification service deployed at all.
• Configuration (in ConfigMaps/Secrets) that does not reference trusted registries, keys, or signature verification settings.
• Policy engine pods in CrashLoopBackOff or Error state, indicating admission requests may fail open depending on failurePolicy.

---

# 5) Exercise the current configuration with a test pod
# Replace <untrusted/image:tag> and <trusted/image@sha256:...> with images appropriate for your environment

# Test A: attempt to create a pod using an image that should FAIL provenance checks
kubectl apply -f - << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: image-provenance-test-untrusted
spec:
  containers:
  - name: test
    image: untrusted/image:tag
    command: ["sleep", "3600"]
  restartPolicy: Never
EOF

# Test B: attempt to create a pod using an image that should PASS provenance checks
kubectl apply -f - << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: image-provenance-test-trusted
spec:
  containers:
  - name: test
    image: trusted/image@sha256:deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef
    command: ["sleep", "3600"]
  restartPolicy: Never
EOF

# Check results
kubectl get pods image-provenance-test-untrusted image-provenance-test-trusted
kubectl describe pod image-provenance-test-untrusted
kubectl describe pod image-provenance-test-trusted

What to look for (problem indicators)

• The “untrusted” pod is created and runs successfully; this strongly suggests image provenance is not enforced.
• No admission webhook denial messages in the pod events (kubectl describe pod ...) for the untrusted image.
• The “trusted” pod is rejected with provenance-related errors (misconfigured policy) or both pods are treated identically (no provenance enforcement).

---

# 6) Check API server flags for admission webhooks (if accessible via config in-cluster)
# Some managed clusters expose a ConfigMap or API object with apiserver flags.
# Example (adapt as needed):
kubectl get pods -n kube-system -l component=kube-apiserver -o yaml

What to look for (problem indicators)

• No reference to ImagePolicyWebhook in --enable-admission-plugins if your design relies on the legacy built‑in ImagePolicyWebhook admission plugin (on self‑managed control planes).
• For managed control planes (EKS/AKS/GKE/OKE), control plane flags are typically not visible; in that case, absence of any webhooks/configs as above is the main indicator.

---

All of the above commands only surface the current state. Any decision to introduce, tighten, or relax image provenance policies, and the exact implementation (ImagePolicyWebhook vs. external policy engines), requires human review against your organization’s threat model and Kubernetes documentation.

Automation

#!/usr/bin/env bash
#
# audit-image-policy-webhook.sh
#
# Purpose:
#   Report whether ImagePolicyWebhook-based image provenance is configured
#   and enforced across the cluster.
#
# Run on:
#   Any machine with kubectl access and correct kubeconfig context.
#
# Requirements:
#   - bash
#   - kubectl in PATH
#   - cluster-admin or equivalent access (to list webhooks, namespaces, and workloads)

set -euo pipefail

echo "=== Image Provenance / ImagePolicyWebhook Cluster Audit ==="
echo "Kubeconfig context: $(kubectl config current-context)"
echo

###############################################################################
# 1. Check for presence of ImagePolicyWebhook admission configuration object
###############################################################################
echo "1) AdmissionConfiguration objects that may define ImagePolicyWebhook:"
kubectl get cm -A \
  --field-selector metadata.name=admission-configuration \
  -o wide 2>/dev/null || true
echo

echo "   Attempting to locate AdmissionConfiguration in kube-system namespace:"
ADMISSION_CM_NAME=$(kubectl -n kube-system get cm \
  --no-headers 2>/dev/null | awk '/admission/{print $1}' || true)

if [ -n "${ADMISSION_CM_NAME:-}" ]; then
  echo "   Found potential AdmissionConfiguration ConfigMap(s) in kube-system:"
  kubectl -n kube-system get cm "${ADMISSION_CM_NAME}" -o yaml
else
  echo "   No obvious AdmissionConfiguration ConfigMap found in kube-system."
fi
echo

###############################################################################
# 2. Check for Mutating/ValidatingWebhookConfiguration that enforce image policy
###############################################################################
echo "2) ValidatingWebhookConfiguration objects related to image policy:"
kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io \
  -o custom-columns=NAME:.metadata.name \
  --no-headers 2>/dev/null | grep -i "image\|policy" || echo "   (none matching 'image' or 'policy' in name)"
echo

echo "   Full list of ValidatingWebhookConfiguration objects (for manual review):"
kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io -o wide || true
echo

echo "   Details for likely image-policy related webhooks:"
for w in $(kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io \
              -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null \
            | grep -i "image\|policy" || true); do
  echo "----- ValidatingWebhookConfiguration: $w -----"
  kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io "$w" -o yaml
  echo
done

echo "2b) MutatingWebhookConfiguration objects related to image policy (less common):"
kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io \
  -o custom-columns=NAME:.metadata.name \
  --no-headers 2>/dev/null | grep -i "image\|policy" || echo "   (none matching 'image' or 'policy' in name)"
echo

###############################################################################
# 3. Check for per-namespace admission configuration hints/annotations
###############################################################################
echo "3) Namespaces with annotations that may indicate image policy enforcement:"
kubectl get ns -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations}{"\n"}{end}' \
  2>/dev/null | grep -Ei 'image|policy|admission' || echo "   (no matching annotations found)"
echo

###############################################################################
# 4. Inventory workloads and their image registries (for provenance review)
###############################################################################
echo "4) Workloads and container images per namespace:"
echo "   NOTE: This does NOT prove image provenance is enforced; it supports review."

namespaces=$(kubectl get ns -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')
for ns in $namespaces; do
  echo "----- Namespace: $ns -----"
  kubectl get pods -n "$ns" -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.containers[*]}{.image}{" "}{end}{"\n"}{end}' 2>/dev/null \
    || echo "   (no pods or access denied)"
  echo
done

###############################################################################
# 5. High-level summary hints
###############################################################################
echo "=== Interpretation Guide (manual review required) ==="
cat <<'EOF'
This script does NOT determine compliance automatically. Use these indicators:

1) ImagePolicyWebhook configuration:
   - Problem indicators:
     * No AdmissionConfiguration object referencing 'ImagePolicyWebhook' or an
       external image policy service.
     * No cluster-wide documentation/config in ConfigMap or kube-apiserver
       flags describing ImagePolicyWebhook.
   - Healthy indicators:
     * AdmissionConfiguration (via ConfigMap or kube-apiserver config file)
       defines an 'imagePolicy' or 'pluginConfig' entry of type
       'ImagePolicyWebhook' with a reachable backend (e.g., HTTPS endpoint).

2) Webhook configurations:
   - Problem indicators:
     * No ValidatingWebhookConfiguration whose name, rules, or clientConfig
       clearly correspond to an image policy / provenance service.
     * Webhook exists but:
       - 'failurePolicy' is 'Ignore' (weak enforcement).
       - 'namespaceSelector' or 'objectSelector' excludes critical workloads.
       - 'rules' do NOT include create/update of core workload types
         (pods, deployments, statefulsets, daemonsets, jobs, cronjobs).
   - Healthy indicators:
     * A ValidatingWebhookConfiguration that:
       - Targets pod and workload creation/updates in relevant API groups.
       - Uses 'failurePolicy: Fail' for enforcement (as appropriate).
       - Points to a trusted image policy service.

3) Workload images:
   - Problem indicators:
     * Workloads pulling from untrusted or public registries without a clear
       corresponding image policy configuration.
   - Healthy indicators:
     * Images consistently sourced from approved registries, with a clearly
       documented policy on how provenance is validated by the webhook.

Next steps:
   - If you do not find any clear ImagePolicyWebhook or equivalent admission
     webhook enforcing image provenance, treat this as a gap and design a
     policy and webhook deployment following Kubernetes documentation.
EOF