Sensitive Values Should Not Be Passed As Literal Env Vars

More Info:

Verifies secret-like env vars are not set as literal values. Literal values land in the pod manifest, logs and kubectl describe.

Risk Level

Medium

Address

Security

Compliance Standards

Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

Identify offending Pods and env vars (any machine with kubectl access)

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.containers // [] + .spec.initContainers // [])[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "ns=\($m.namespace) pod=\($m.name) container=\($c) env=\(.name)"
  ][]'

For one violating Pod, capture its manifest (any machine with kubectl access)
Replace NAMESPACE and POD_NAME with values from step 1.
```
kubectl get pod POD_NAME -n NAMESPACE -o yaml > /tmp/pod-POD_NAME.yaml
```
Create a Secret containing the sensitive value (any machine with kubectl access)
Choose a Secret name and key, then run:
```
kubectl create secret generic app-secret-POD_NAME \
  -n NAMESPACE \
  --from-literal=DB_PASSWORD='REPLACE_WITH_ACTUAL_PASSWORD'
```
Repeat with additional --from-literal=KEY='VALUE' flags for each sensitive env var.
Edit the Pod’s manifest to use valueFrom.secretKeyRef (any machine with kubectl access)
Open the file from step 2 and, under the relevant container’s env: section, replace:
```
- name: DB_PASSWORD
  value: "REPLACE_WITH_ACTUAL_PASSWORD"
```
with:
```
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: app-secret-POD_NAME
      key: DB_PASSWORD
```
Do this for each sensitive variable, ensuring key: matches what you stored in the Secret.
Recreate the Pod so it uses the Secret-based env vars (any machine with kubectl access)
For Pods managed by a higher-level controller (e.g., Deployment, StatefulSet), edit the controller instead of the live Pod. Example for a Deployment:
```
kubectl edit deployment DEPLOYMENT_NAME -n NAMESPACE
```
Apply the same env: changes to the Deployment spec, then let Kubernetes roll out new Pods.
If it is a standalone Pod (no controller), delete and recreate it from the edited manifest:
```
kubectl delete pod POD_NAME -n NAMESPACE
kubectl apply -f /tmp/pod-POD_NAME.yaml
```

Verify the cluster is compliant (any machine with kubectl access)

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.containers // [] + .spec.initContainers // [])[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
      + " container=\($c) env=\(.name) is_compliant=false"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

Confirm the output is is_compliant=true or that no lines for the fixed Pods remain.

Using kubectl

On any machine with kubectl access:

Identify the violating Pod and env var (from the audit output):

Example:
- Namespace: default
- Pod name: my-app-abc123
- Container: app
- Offending env: DB_PASSWORD

Create a Secret that will hold the sensitive value (one-time, per app/namespace):

kubectl -n default create secret generic my-app-secret \
  --from-literal=DB_PASSWORD='REPLACE_WITH_REAL_PASSWORD'

Export the current Pod manifest, edit it to use valueFrom.secretKeyRef, and apply it via its controller (Deployment, StatefulSet, etc.).
If the Pod is standalone (no controller), you must recreate it from a manifest.

a) Get the owning controller kind/name from the audit output (owner= field) or via:

kubectl -n default get pod my-app-abc123 -o jsonpath='{.metadata.ownerReferences[0].kind}{" "}{.metadata.ownerReferences[0].name}{"\n"}'

Assume it is a Deployment named my-app.

b) Export the Deployment manifest:

kubectl -n default get deploy my-app -o yaml > my-app-deploy.yaml

c) Edit my-app-deploy.yaml:

Locate the container and replace the literal value: with valueFrom.secretKeyRef:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
spec:
  template:
    spec:
      containers:
        - name: app
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: my-app-secret
                  key: DB_PASSWORD

Remove any previous value: ... line for DB_PASSWORD.

d) Apply the updated manifest:

kubectl apply -f my-app-deploy.yaml

This will roll out new Pods using the Secret-based env var.

Verification:

Run the audit command again and ensure there is no line for this Pod/env and that you see is_compliant=true when all violations are fixed:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.nodeName // "") as $node
    | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
      + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
      + (if $node   == ""   then "" else " node=\($node)" end)
      + (if $labels == ""   then "" else " labels=\($labels)" end)
      + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
      + " container=\($c) env=\(.name) is_compliant=false"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

Automation

#!/usr/bin/env bash
#
# Remediate CBP C4.1 for Pods in AKS:
# - Detect env vars with secret-like names that use a literal `value:`
# - Create/patch a Secret per Pod+container
# - Rewrite Pod spec to use valueFrom.secretKeyRef
#
# IMPORTANT:
# - Runs from any machine with kubectl access and jq installed.
# - Only touches namespaced Pods outside kube-system/kube-public/kube-node-lease.
# - Handles immutable Pods (e.g., controlled by Deployments) by patching the controller instead.
# - Safe to re-run: uses deterministic Secret names and key names; re-runs become no-ops.
#
# REQUIREMENTS:
# - kubectl configured for the AKS cluster
# - jq
#
# LIMITATIONS:
# - Cannot recover the original secret value from Git/IaC; it uses the currently running literal
#   value to build a Secret. You should later rotate these Secrets from a secure source of truth.

set -euo pipefail

# Configurable regex for "secret-like" env var names
SENSITIVE_ENV_REGEX='PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY'

# Namespace ignore list (system namespaces)
IGNORE_NAMESPACES=('kube-system' 'kube-public' 'kube-node-lease')

# Check prerequisites
command -v kubectl >/dev/null 2>&1 || { echo "kubectl is required"; exit 1; }
command -v jq >/dev/null 2>&1 || { echo "jq is required"; exit 1; }

is_ignored_ns() {
  local ns="$1"
  for n in "${IGNORE_NAMESPACES[@]}"; do
    if [[ "$n" == "$ns" ]]; then
      return 0
    fi
  done
  return 1
}

# Sanitize names for Secret/keys (DNS-1123 subdomain for Secret; key can be broader but we keep simple)
sanitize_name() {
  # Lowercase, replace invalid chars with '-', trim leading/trailing '-'
  echo "$1" \
    | tr '[:upper:]' '[:lower:]' \
    | sed -E 's/[^a-z0-9.-]+/-/g; s/^[^a-z0-9]+//; s/[^a-z0-9]+$//'
}

create_or_patch_secret() {
  local namespace="$1"
  local secret_name="$2"
  local key="$3"
  local value="$4"

  # Check if secret exists
  if kubectl -n "$namespace" get secret "$secret_name" >/dev/null 2>&1; then
    # Check if key exists with same value (to keep idempotent)
    local existing
    existing="$(kubectl -n "$namespace" get secret "$secret_name" -o json \
      | jq -r --arg k "$key" '.data[$k] // empty')"
    local new_b64
    new_b64="$(printf '%s' "$value" | base64 -w0 2>/dev/null || printf '%s' "$value" | base64)"

    if [[ "$existing" == "$new_b64" ]]; then
      echo "Secret ${namespace}/${secret_name} already has key ${key} with same value; skip update"
      return
    fi

    echo "Patching Secret ${namespace}/${secret_name} to add/update key ${key}"
    kubectl -n "$namespace" patch secret "$secret_name" \
      --type merge \
      -p "$(jq -n --arg k "$key" --arg v "$new_b64" '{data: {($k): $v}}')"
  else
    echo "Creating Secret ${namespace}/${secret_name} with key ${key}"
    kubectl -n "$namespace" create secret generic "$secret_name" \
      --from-literal "${key}=${value}"
  fi
}

patch_pod_env_to_secretref() {
  local namespace="$1"
  local pod="$2"
  local container="$3"
  local env_name="$4"
  local secret_name="$5"
  local secret_key="$6"

  echo "Patching Pod ${namespace}/${pod} container=${container} env=${env_name} -> secretKeyRef"
  # Use strategic merge patch on the Pod
  kubectl -n "$namespace" patch pod "$pod" --type merge -p "$(
    jq -n \
      --arg c "$container" \
      --arg en "$env_name" \
      --arg sn "$secret_name" \
      --arg sk "$secret_key" \
      '{
        spec: {
          containers: [
            {
              name: $c,
              env: [
                {
                  name: $en,
                  valueFrom: {
                    secretKeyRef: {
                      name: $sn,
                      key: $sk
                    }
                  }
                }
              ]
            }
          ]
        }
      }'
  )"
}

patch_controller_env_to_secretref() {
  local kind="$1"   # Deployment, StatefulSet, DaemonSet, Job, CronJob, ReplicaSet, ReplicationController
  local namespace="$2"
  local name="$3"
  local container="$4"
  local env_name="$5"
  local secret_name="$6"
  local secret_key="$7"

  echo "Patching ${kind} ${namespace}/${name} container=${container} env=${env_name} -> secretKeyRef"

  # Map kind to pod spec path
  local podspec_path
  case "$kind" in
    Deployment|StatefulSet|DaemonSet|ReplicaSet|ReplicationController)
      podspec_path='.spec.template.spec'
      ;;
    Job)
      podspec_path='.spec.template.spec'
      ;;
    CronJob)
      # For batch/v1 CronJob in AKS
      podspec_path='.spec.jobTemplate.spec.template.spec'
      ;;
    *)
      echo "Unsupported controller kind: $kind; skipping" >&2
      return
      ;;
  esac

  # Build patch dynamically to target the matching container
  # We use a JSON patch instead of strategic to surgically replace this env entry
  # 1. Fetch existing pod template
  local tmpl
  tmpl="$(kubectl -n "$namespace" get "$kind" "$name" -o json)"
  # 2. Build new containers array with this env changed
  local new_containers
  new_containers="$(echo "$tmpl" | jq \
    --arg c "$container" \
    --arg en "$env_name" \
    --arg sn "$secret_name" \
    --arg sk "$secret_key" \
    "$podspec_path.containers
     | map(
        if .name == \$c then
          .env |= (map(
            if .name == \$en then
              {name: \$en, valueFrom:{secretKeyRef:{name:\$sn,key:\$sk}}}
            else .
            end
          ))
        else .
        end
      )"
  )"

  # 3. Apply patch
  kubectl -n "$namespace" patch "$kind" "$name" --type merge -p "$(
    echo "$tmpl" | jq \
      --argjson nc "$new_containers" \
      "$podspec_path.containers = \$nc | {spec:.spec}"
  )"
}

# Main remediation
echo "Discovering Pods with secret-like env vars using literal values..."
pods_json="$(kubectl get pods --all-namespaces -o json)"

# Iterate over Pods
echo "$pods_json" | jq -c '.items[]' | while read -r pod; do
  ns="$(echo "$pod" | jq -r '.metadata.namespace')"
  if is_ignored_ns "$ns"; then
    continue
  fi

  pod_name="$(echo "$pod" | jq -r '.metadata.name')"

  # Check both containers and initContainers
  for field in containers initContainers; do
    echo "$pod" | jq -c --arg field "$field" ".spec[$field] // [] | .[]" | while read -r ctr; do
      container_name="$(echo "$ctr" | jq -r '.name')"
      echo "$ctr" | jq -c --arg re "$SENSITIVE_ENV_REGEX" '.env // [] | .[] | select((.value != null) and (.name | test($re;"i")))' | while read -r env; do
        env_name="$(echo "$env" | jq -r '.name')"
        env_val="$(echo "$env" | jq -r '.value')"

        # Determine controller owner (if any)
        owner_kind="$(echo "$pod" | jq -r '(.metadata.ownerReferences // []) | map(select(.controller)) | first?.kind // ""')"
        owner_name="$(echo "$pod" | jq -r '(.metadata.ownerReferences // []) | map(select(.controller)) | first?.name // ""')"

        # Derive Secret name and key
        # Secret name: secret-<pod>-<container> (sanitized)
        # Key: sanitized env name
        secret_name_raw="secret-${pod_name}-${container_name}"
        secret_name="$(sanitize_name "$secret_name_raw")"
        secret_key="$(sanitize_name "$env_name")"
        if [[ -z "$secret_name" || -z "$secret_key" ]]; then
          echo "Failed to derive Secret name/key for ${ns}/${pod_name} env=${env_name}; skipping" >&2
          continue
        fi

        create_or_patch_secret "$ns" "$secret_name" "$secret_key" "$env_val"

        if [[ -n "$owner_kind" && -n "$owner_name" ]]; then
          # Patch controller so future Pods are fixed
          patch_controller_env_to_secretref "$owner_kind" "$ns" "$owner_name" "$container_name" "$env_name" "$secret_name" "$secret_key"
        else
          # Patch the Pod directly (e.g., naked Pod)
          patch_pod_env_to_secretref "$ns" "$pod_name" "$container_name" "$env_name" "$secret_name" "$secret_key"
        fi
      done
    done
  done
done

echo
echo "Re-running compliance check to verify..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.nodeName // "") as $node
    | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
      + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
      + (if $node   == ""   then "" else " node=\($node)" end)
      + (if $labels == ""   then "" else " labels=\($labels)" end)
      + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
      + " container=\($c) env=\(.name) is_compliant=false"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

echo
echo "If the output above is 'is_compliant=true', the remediation is successful."

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​