Skip to main content

Verify That Admission Controllers Are Working Expected

More Info:​

Consider the use of an external secrets storage and management system, instead of using Kubernetes Secrets directly, if you have more complex secret management needs. Ensure the solution requires authentication to access secrets, has auditing of access to and use of secrets, and encrypts secrets. Some solutions also make it easier to rotate secrets.

Risk Level​

Low

Address​

Security

Compliance Standards​

  • CIS AKS

Triage and Remediation​

Remediation​

Manual Steps

  1. Identify which admission webhooks control secret usage

    • On any machine with kubectl access: 
      kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -A
    • Review names and annotations to identify webhooks that enforce policies around Secrets (e.g., disallowing plaintext secrets, enforcing external secret references, enforcing labels/annotations on secret-using pods).

  2. Inspect webhook configuration and scope

    • For each relevant webhook configuration from step 1: 
      kubectl get validatingwebhookconfiguration <NAME> -o yaml
      kubectl get mutatingwebhookconfiguration  <NAME> -o yaml
    • Manually verify:
      • rules include the API groups/resources you rely on for secrets or secret references (e.g., core/secrets, CRDs like ExternalSecret).
      • namespaceSelector/objectSelector match the namespaces and objects where you expect enforcement.
      • failurePolicy matches your risk appetite (e.g., Fail for strict control-plane environments).

  3. Confirm webhooks are reachable and healthy

    • Check the services and pods backing those webhooks: 
      # Example: extract service/namespace from webhookClientConfig and inspect
      kubectl get svc -A
      kubectl get pods -A
    • Manually confirm:
      • The webhook service exists in the configured namespace and name.
      • Pods for the webhook service are Running and Ready.
      • TLS config in clientConfig (CA bundle, service, port, path) corresponds to the actual service.

  4. Test enforcement using a controlled, non‑compliant object

    • On any machine with kubectl access, create a test manifest that should be denied or mutated by your policy (e.g., a Secret or Pod violating your secret policy): 
      cat << 'EOF' > /tmp/test-secret.yaml
      apiVersion: v1
      kind: Secret
      metadata:
        name: admission-test-secret
        namespace: default
      type: Opaque
      stringData:
        password: "plaintext-insecure"
      EOF

      kubectl apply -f /tmp/test-secret.yaml
    • Observe whether the webhook denies, mutates, or allows it, and confirm that the behavior matches your intended policy for secret management.

  5. Review audit logs to confirm traceability of secret-related operations

    • If API audit logging is enabled, use your log aggregation/search tool or, if logs are local on control-plane nodes, inspect them (example path, adjust to your environment): 
      # On a control-plane node, if API server writes local audit logs
      sudo ls -l /var/log/kubernetes/
      sudo grep -E '"resource":"secrets"' /var/log/kubernetes/audit.log | head
    • Manually verify that:
      • Secret create/update/read events are logged with user identity and source.
      • Webhook admission decisions appear as audit annotations if configured.

  6. If behavior is incorrect, update webhook manifests and re‑test

    • Edit the webhook configuration on any machine with kubectl access: 
      kubectl edit validatingwebhookconfiguration <NAME>
      # or
      kubectl edit mutatingwebhookconfiguration <NAME>
    • Adjust rules, selectors, failurePolicy, or clientConfig to align with your intended secret‑management and access‑control behavior.
    • Re-run steps 2–4 to verify that admission controllers now enforce your expected policies on secret usage.
Using kubectl
# 1) List all MutatingWebhookConfiguration objects
# Run on: any machine with kubectl access
kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io -A -o wide
  • Potential problems:
    • Webhooks you expect to be present are missing.
    • Webhooks unexpectedly reference non-existent services or CABundles (STATUS column may be empty or errors in events).
# 2) List all ValidatingWebhookConfiguration objects
kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io -A -o wide
  • Potential problems:
    • Critical validating webhooks (e.g., for security policies, secret controls) are missing.
    • Unexpected/unapproved webhooks exist (compare against your design/documents).
# 3) Inspect a specific webhook configuration in detail
# Replace <name> with an object from the get output above
kubectl get validatingwebhookconfiguration <name> -o yaml

Review in webhooks::

  • rules: – check that operations and resources match your intended coverage.
    • Problems: overly broad rules (e.g. resources: ["*"] in all groups) or missing coverage for critical resources (e.g., secrets, pods).
  • namespaceSelector / objectSelector – ensure protected namespaces/workloads are not unintentionally excluded.
  • failurePolicy – Ignore for security-critical checks is usually a risk; Fail may be required by your policy.
  • sideEffects – validate correctness for the implementation.
  • clientConfig.service – confirm name, namespace, and path point to an existing service (see next step).
  • timeoutSeconds – too high can impact API performance; too low may cause frequent timeouts.
# 4) Verify the webhook service targets exist
# Replace <namespace> and <service> using values seen in clientConfig.service
kubectl -n <namespace> get svc <service> -o wide
kubectl -n <namespace> get pods -o wide
  • Potential problems:
    • Service not found.
    • No backing pods, pods in CrashLoopBackOff, ImagePullBackOff, or NotReady.
    • Selector mismatch between service and pods.
# 5) Check events and logs around failed or ignored admissions
kubectl get events -A --sort-by=.lastTimestamp | grep -i webhook
  • Potential problems:
    • Frequent “calling webhook failed” or “timed out” messages.
    • Messages showing failurePolicy: Ignore leading to resources being admitted when the webhook is unavailable.
# 6) Probe that webhooks actually intercept requests
# Example: attempt an operation that SHOULD be blocked by policy
# (adjust to match your expected admission behavior)
kubectl run test-pod --image=nginx --dry-run=server -o yaml

Interpretation:

  • If you expect a validating webhook to reject this kind of pod and the command succeeds without an admission error, the webhook may not be working as intended (rules, selectors, or failurePolicy misconfigured).
  • If you see an explicit admission error from a webhook that matches your policy, that indicates it is active.
# 7) For webhooks managing secrets behavior, try a controlled test
# Adjust to an operation your policy should regulate, e.g., creating a Secret
kubectl create secret generic test-secret --from-literal=key=value --dry-run=server -o yaml
  • Potential problems:
    • Operation is admitted when your design requires a webhook decision (e.g., external secret manager integration, annotation enforcement).
    • Error messages indicate misconfigured URL/service or certificate issues.
# 8) Confirm API server is configured to use admission plugins (cluster-info only)
kubectl cluster-info dump | grep -i admission -n
  • Potential problems:
    • Missing or unexpected admission plugins compared to your cluster’s security design (e.g., PSP/PSS, external secrets-related controllers, or custom admission plugins).
    • Flags or config indicating disabled admission features you rely on.

Use these observations to decide whether current admission controllers and associated webhooks actually enforce your intended policies around secrets management and other security controls; any mismatch between expected and observed behavior indicates a problem that needs manual design and configuration changes.

Automation
#!/usr/bin/env bash
# Purpose: Cluster-wide report to help review that admission controllers and
#          secret-handling policies are behaving as expected.
# Scope:   Run on any machine with kubectl context to the target cluster.

set -euo pipefail

echo "=== 1) API Server admission configuration (Mutating/Validating configs) ==="
echo
echo "# MutatingWebhookConfiguration objects:"
kubectl get mutatingwebhookconfiguration -o wide || echo "  (none found)"
echo
echo "# ValidatingWebhookConfiguration objects:"
kubectl get validatingwebhookconfiguration -o wide || echo "  (none found)"
echo

echo "=== Detailed admission webhook configuration (for review) ==="
echo
kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration -o yaml | sed 's/^/  /' || \
  echo "  (no admission webhook configurations found)"
echo
cat <<'EOF'

[Review guidance: potential problems]
- No MutatingWebhookConfiguration/ValidatingWebhookConfiguration exist, even though
  your design expects policy enforcement (e.g. Pod Security admission via webhooks,
  custom policy engines, or secret-management webhooks).
- webhooks that *should* protect secret usage (e.g. reject plain Kubernetes Secrets)
  are missing, disabled, or have `failurePolicy: Ignore` or `sideEffects: Unknown`.
- `namespaceSelector` / `objectSelector` excludes namespaces that should be protected
  (e.g. workloads using external secret stores).
- `clientConfig` uses insecure endpoints (no TLS, or `insecureSkipTLSVerify: true`).

EOF

echo "=== 2) Workloads creating or using Kubernetes Secrets directly ==="
echo
echo "# Namespaces that contain Secret objects:"
kubectl get secret --all-namespaces --no-headers 2>/dev/null \
  | awk '{print $1}' | sort -u | sed 's/^/  /' || echo "  (no Secrets found)"
echo

echo "# Count of Secrets per namespace:"
kubectl get secret --all-namespaces --no-headers 2>/dev/null \
  | awk '{count[$1]++} END {for (ns in count) printf "%s\t%d\n", ns, count[ns]}' \
  | sort | sed 's/^/  /' || echo "  (no Secrets found)"
echo

echo "# Pods that mount or reference Secrets (envFrom/env):"
kubectl get pods --all-namespaces -o json \
  | jq -r '
    .items[]
    | {
        ns: .metadata.namespace,
        pod: .metadata.name,
        volumes: (.spec.volumes // [] | map(select(.secret != null) | .secret.secretName) | unique),
        envFrom: (
          .spec.containers // []
          | map(.envFrom // [] | map(select(.secretRef != null) | .secretRef.name) | .[])
        ),
        env: (
          .spec.containers // []
          | map(.env // [] | map(select(.valueFrom.secretKeyRef != null) | .valueFrom.secretKeyRef.name) | .[])
        )
      }
    | select((.volumes + .envFrom + .env) | length > 0)
    | "\(.ns)\t\(.pod)\tvolumes:\(.volumes|join(\",\"))\tenvFrom:\(.envFrom|join(\",\"))\tenv:\(.env|join(\",\"))"
  ' 2>/dev/null | sed 's/^/  /' || echo "  (no pods using Secrets found)"
echo
cat <<'EOF'

[Review guidance: potential problems]
- Business or security policy says workloads MUST use an external secret store,
  but many Secrets and Secret-consuming Pods are present.
- Sensitive workloads are referencing plain `Secret` objects instead of provider-
  specific CRDs for external secret managers (e.g. ExternalSecret, SecretProviderClass).

EOF

echo "=== 3) Evidence of external secret manager CRDs / controllers ==="
echo
echo "# Look for common external-secret CRDs:"
kubectl get crd 2>/dev/null | grep -iE 'secret|vault|external|provider' || \
  echo "  (no obvious secret-related CRDs found)"
echo

echo "# Controllers/Pods in namespaces that look like secret managers:"
kubectl get ns | grep -iE 'secret|vault|aws|azure|gcp|kms|hsm|key|crypto' || \
  echo "  (no obvious secret-related namespaces found)"
echo
for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do
  if echo "$ns" | grep -qiE 'secret|vault|aws|azure|gcp|kms|hsm|key|crypto'; then
    echo "## Pods in namespace: $ns"
    kubectl get pods -n "$ns" -o wide || true
    echo
  fi
done
cat <<'EOF'

[Review guidance: potential problems]
- Cluster is expected to use an external secrets system, but:
  - No CRDs or controllers for that system are visible, or
  - Relevant controller Pods are CrashLooping/NotReady.

EOF

echo "=== 4) Admission webhooks potentially involved in secrets/control ==="
echo
echo "# Webhooks whose names or rules reference secrets or policy:"
kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration -o json 2>/dev/null \
  | jq -r '
    .items[]
    | . as $cfg
    | .webhooks[]
    | select(
        (.name | test("secret|policy|vault|kms|compliance"; "i"))
        or ([.rules[].resources[]?] | join(",") | test("secret"; "i"))
      )
    | "\($cfg.kind)/\($cfg.metadata.name)\t\( .name )"
  ' | sed 's/^/  /' || echo "  (no obviously secret/policy-related webhooks found)"
echo

echo "# Full rules for those webhooks (for deeper review):"
kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration -o json 2>/dev/null \
  | jq '
    .items[]
    | . as $cfg
    | .webhooks[]
    | select(
        (.name | test("secret|policy|vault|kms|compliance"; "i"))
        or ([.rules[].resources[]?] | join(",") | test("secret"; "i"))
      )
    | {
        kind: $cfg.kind,
        configName: $cfg.metadata.name,
        webhookName: .name,
        failurePolicy: .failurePolicy,
        sideEffects: .sideEffects,
        admissionReviewVersions: .admissionReviewVersions,
        rules: .rules,
        namespaceSelector: .namespaceSelector,
        objectSelector: .objectSelector,
        clientConfig: .clientConfig
      }
  ' || echo "  (no matching webhooks; skip)"
echo
cat <<'EOF'

[Review guidance: potential problems]
- Secret/policy-related webhooks exist but:
  - `failurePolicy` is `Ignore` where `Fail` is required by your policy.
  - `namespaceSelector`/`objectSelector` unintentionally bypasses protection.
  - `clientConfig.service` points to non-existing Services/namespaces.

EOF

echo "=== 5) (Optional) Quick functional test of a basic admission path ==="
echo "# This creates a test Namespace and Pod, then deletes them."
TEST_NS="admission-test-$(date +%s)"
echo "  Creating test namespace: $TEST_NS"
kubectl create namespace "$TEST_NS"

cat <<'POD' | kubectl apply -n "$TEST_NS" -f -
apiVersion: v1
kind: Pod
metadata:
  name: admission-test-pod
spec:
  containers:
  - name: busybox
    image: busybox:1.36
    command: ["sh", "-c", "sleep 3600"]
POD

echo "  Pod created. Describe it to see if any mutating/validating webhooks acted on it:"
kubectl describe pod admission-test-pod -n "$TEST_NS" | sed 's/^/  /'
echo

echo "  Cleaning up test resources..."
kubectl delete namespace "$TEST_NS" --wait=false

cat <<'EOF'

[Review guidance: potential problems]
- Expected mutating/validating webhooks (e.g. security policies, sidecar injection,
  or secret-enforcement policies) do NOT appear in the Events/annotations of the
  test Pod, even though they should apply to all Pods/namespaces.

EOF

echo "=== 6) Summary: what indicates a potential problem? ==="
cat <<'EOF'
- No admission webhooks are configured where your design expects them.
- Secret-related or policy-enforcing webhooks exist but are:
  - Disabled, misconfigured, or set with failurePolicy=Ignore.
  - Not targeting the namespaces/resources they are supposed to protect.
- Workloads are storing or consuming Kubernetes Secrets directly, despite a
  policy to use an external secret manager.
- External secret manager components (CRDs/controllers) are missing or unhealthy.
- A simple test Pod shows no sign that expected admission controllers are invoked.

This script does NOT automatically fix anything. Use the above outputs to
decide whether admission controllers and secret-management policies are
behaving as designed, then adjust configurations manually as needed.
EOF