Sensitive Values Should Not Be Passed As Literal Env Vars

More Info:

Verifies secret-like env vars are not set as literal values. Literal values land in the pod manifest, logs and kubectl describe.

Risk Level

Medium

Address

Security

Compliance Standards

Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

Identify offending Pods and env vars (any machine with kubectl access)

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "ns=\($m.namespace) pod=\($m.name) container=\($c) env=\(.name)"
  ][]'

For each offending Pod, determine the owning controller (any machine with kubectl access)

# Example: inspect one violating Pod
kubectl -n <namespace> get pod <pod-name> -o json | jq '
  {metadata:{name,namespace,ownerReferences},spec:{nodeName,containers:[.spec.containers[].name]}}'

If ownerReferences is set (e.g., Deployment, StatefulSet, Job), you must edit that controller, not the live Pod.

Create a Secret with the sensitive value (any machine with kubectl access)

For a single key:

kubectl -n <namespace> create secret generic <secret-name> \
  --from-literal=<env-var-name>=<sensitive-value>

Or from a file:

kubectl -n <namespace> create secret generic <secret-name> \
  --from-file=<env-var-name>=/absolute/path/to/file

Update the controller manifest to use valueFrom.secretKeyRef (any machine with kubectl access)

Get the current manifest:

kubectl -n <namespace> get deployment <deployment-name> -o yaml > /tmp/deployment.yaml

In /tmp/deployment.yaml, under the relevant container’s env: section, replace:

- name: MY_SECRET_PASSWORD
  value: "super-secret-value"

with:

- name: MY_SECRET_PASSWORD
  valueFrom:
    secretKeyRef:
      name: <secret-name>
      key: MY_SECRET_PASSWORD

Apply the change:
```
kubectl apply -f /tmp/deployment.yaml
```

This will roll the Pods for that controller and recreate them with the Secret reference.

For standalone Pods (no ownerReferences), recreate them using a manifest (any machine with kubectl access)
```
# Export current Pod spec to a file
kubectl -n <namespace> get pod <pod-name> -o yaml > /tmp/pod.yaml
```
Edit /tmp/pod.yaml:
- Remove fields under metadata such as uid, resourceVersion, creationTimestamp, managedFields, selfLink, generation.
- Remove status: entirely.
- In each offending container’s env: entry, change value: to valueFrom.secretKeyRef as in step 4. Then delete and recreate:
```
kubectl -n <namespace> delete pod <pod-name>
kubectl apply -f /tmp/pod.yaml
```

Verify the cluster is compliant (any machine with kubectl access)

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.nodeName // "") as $node
    | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
      + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
      + (if $node   == ""   then "" else " node=\($node)" end)
      + (if $labels == ""   then "" else " labels=\($labels)" end)
      + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
      + " container=\($c) env=\(.name) is_compliant=false"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

Confirm the output is is_compliant=true or that no offending env vars remain.

Using kubectl

On any machine with kubectl access:

Inspect the offending Pod and identify the literal sensitive env var(s):

kubectl get pod POD_NAME -n NAMESPACE -o yaml

Look under spec.containers[].env (and initContainers[].env) for entries like:

env:
  - name: DB_PASSWORD
    value: super-secret-password        # <-- needs to be moved to a Secret

Create (or update) a Secret containing the sensitive value(s). Example:

kubectl create secret generic app-secrets \
  -n NAMESPACE \
  --from-literal=DB_PASSWORD='super-secret-password'

If the Secret must be changed later, use:

kubectl delete secret app-secrets -n NAMESPACE
kubectl create secret generic app-secrets \
  -n NAMESPACE \
  --from-literal=DB_PASSWORD='new-secret-password'

Patch the Pod’s controller (Deployment/StatefulSet/Job/etc.) manifest so the Pod uses valueFrom.secretKeyRef instead of a literal value. First, export the owning controller manifest:

kubectl get deploy DEPLOYMENT_NAME -n NAMESPACE -o yaml > deployment.yaml

Edit deployment.yaml and change:

env:
  - name: DB_PASSWORD
    value: super-secret-password

to:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: app-secrets
        key: DB_PASSWORD

Repeat for every sensitive variable flagged by the check.

Apply the updated manifest:

kubectl apply -f deployment.yaml

This will roll out new Pods that reference the Secret.

If the Pod is not managed by a higher-level controller (a naked Pod), you must delete and recreate it from a corrected manifest:

kubectl get pod POD_NAME -n NAMESPACE -o yaml > pod.yaml
# edit pod.yaml as in step 3 to use valueFrom.secretKeyRef
kubectl delete pod POD_NAME -n NAMESPACE
kubectl apply -f pod.yaml

Verify that no literal sensitive env vars remain and that the check passes:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.nodeName // "") as $node
    | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
      + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
      + (if $node   == ""   then "" else " node=\($node)" end)
      + (if $labels == ""   then "" else " labels=\($labels)" end)
      + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
      + " container=\($c) env=\(.name) is_compliant=false"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

Automation

#!/usr/bin/env bash
#
# Remediates: Sensitive values should not be passed as literal env vars
# Scope: Any machine with kubectl access to the OKE cluster
#
# Strategy:
# - Discover all Pod specs that set secret-like env vars via literal `value:`
# - Export their owning controller manifests (Deployment/StatefulSet/DaemonSet/Job/CronJob)
# - For each offending env var, replace:
#       env:
#         - name: FOO_PASSWORD
#           value: literal
#   with:
#       env:
#         - name: FOO_PASSWORD
#           valueFrom:
#             secretKeyRef:
#               name: <auto-created Secret>
#               key: FOO_PASSWORD
#   where <auto-created Secret> is `<workload-name>-env-secrets`
# - Create/patch Secrets per-namespace with those keys and literal values
# - Apply the updated controller manifests
# - Verify via the audit query
#
# WARNING:
# - This script *changes* how your workloads receive sensitive values.
# - It must know the actual literal values; it reads them from the current
#   Pod specs, which still contain them.
# - If an env var already uses valueFrom, it is left untouched.
# - Controllers will roll Pods as they are updated.

set -euo pipefail

#-----------------------------
# Configuration / helpers
#-----------------------------

SENSITIVE_PATTERN='PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY'

# Only process these controller kinds; Pods directly created (no controller)
# will be skipped and must be handled manually.
CONTROLLER_KINDS='Deployment|StatefulSet|DaemonSet|Job|CronJob'

timestamp() { date -u +"%Y%m%dT%H%M%SZ"; }

workdir_base="${PWD}/env-secret-remediation-$(timestamp)"
mkdir -p "${workdir_base}"

echo "Working directory: ${workdir_base}"

# Require jq, yq (v4), kubectl
for bin in kubectl jq yq; do
  if ! command -v "${bin}" >/dev/null 2>&1; then
    echo "ERROR: ${bin} is required on this machine. Install it and re-run." >&2
    exit 1
  fi
done

#-----------------------------
# Step 1: Discover offending Pods
#-----------------------------

echo "Discovering Pods with secret-like env vars set via literal value..."

offenders_json="${workdir_base}/offending-pods.json"
kubectl get pods --all-namespaces -o json > "${offenders_json}"

# Build a compact list of offending containers + env vars + owning controller
offenders_list="${workdir_base}/offenders-list.json"
jq --arg re "${SENSITIVE_PATTERN}" '
  [
    .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | select($own != null and ($own.kind | test("'"${CONTROLLER_KINDS}"'")))
    | (.spec.containers // [] + .spec.initContainers // [])
      as $containers
    | reduce ($containers[]) as $c ([]; . + (
        ($c.env // [])
        | map(select((.value != null)
                     and (.name | test($re; "i"))))
        | map({
            namespace: $m.namespace,
            podName: $m.name,
            controllerKind: $own.kind,
            controllerName: $own.name,
            containerName: $c.name,
            envName: .name,
            envValue: .value
          })
      ))
  ] | add // []' "${offenders_json}" > "${offenders_list}"

if [[ ! -s "${offenders_list}" ]]; then
  echo "No offending literal secret-like env vars found. Cluster is compliant."
  exit 0
fi

echo "Found $(jq 'length' "${offenders_list}") offending env vars."

#-----------------------------
# Step 2: Prepare per-namespace Secret specs
#-----------------------------

# Map: namespace -> controller (kind/name) -> secretName -> keys
# We choose secret name: <controller-name>-env-secrets (dns-1123 trimmed)
echo "Preparing Secret manifests..."

secrets_dir="${workdir_base}/secrets"
mkdir -p "${secrets_dir}"

jq -c '.[]' "${offenders_list}" | while read -r row; do
  ns=$(jq -r '.namespace' <<<"${row}")
  ctrl_kind=$(jq -r '.controllerKind' <<<"${row}")
  ctrl_name=$(jq -r '.controllerName' <<<"${row}")
  env_name=$(jq -r '.envName' <<<"${row}")
  env_value=$(jq -r '.envValue' <<<"${row}")

  # Normalized secret name: <controller-name>-env-secrets
  base_name="${ctrl_name}-env-secrets"
  # Truncate to 253 chars to satisfy DNS-1123 label length, strip invalid chars
  secret_name=$(printf '%s' "${base_name}" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9-.' '-' | cut -c1-253)

  ns_dir="${secrets_dir}/${ns}"
  mkdir -p "${ns_dir}"
  secret_file="${ns_dir}/${secret_name}.yaml"

  if [[ ! -f "${secret_file}" ]]; then
    cat > "${secret_file}" <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ${secret_name}
  namespace: ${ns}
type: Opaque
data: {}
EOF
  fi

  # Add/overwrite the key with base64-encoded value (idempotent)
  b64=$(printf '%s' "${env_value}" | base64 | tr -d '\n')
  yq -i ".data.\"${env_name}\" = \"${b64}\"" "${secret_file}"
done

echo "Generated Secret manifests under ${secrets_dir}"

#-----------------------------
# Step 3: Export and patch controllers
#-----------------------------

controllers_dir="${workdir_base}/controllers"
mkdir -p "${controllers_dir}"

echo "Exporting and patching controller manifests..."

# Unique list of controller refs
controllers_list="${workdir_base}/controllers-unique.json"
jq '[ .[] | {namespace, controllerKind, controllerName} ] | unique' \
  "${offenders_list}" > "${controllers_list}"

jq -c '.[]' "${controllers_list}" | while read -r row; do
  ns=$(jq -r '.namespace' <<<"${row}")
  kind=$(jq -r '.controllerKind' <<<"${row}")
  name=$(jq -r '.controllerName' <<<"${row}")

  ctrl_file="${controllers_dir}/${ns}-${kind}-${name}.yaml"

  echo "  Processing ${kind}/${ns}/${name}"

  # Export current controller spec
  if ! kubectl get "${kind}" "${name}" -n "${ns}" -o yaml > "${ctrl_file}.orig" 2>/dev/null; then
    echo "    WARN: Unable to fetch ${kind}/${ns}/${name}, skipping."
    continue
  fi

  cp "${ctrl_file}.orig" "${ctrl_file}"

  # Determine corresponding Secret name
  base_name="${name}-env-secrets"
  secret_name=$(printf '%s' "${base_name}" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9-.' '-' | cut -c1-253)

  # Patch env entries for both containers and initContainers
  for field in spec.template.spec.containers spec.template.spec.initContainers; do
    # If field exists
    if ! yq e ".${field} == null" "${ctrl_file}" | grep -qi 'true'; then
      # For each container, rewrite offending env entries
      # Logic:
      #   env[].value != null and name matches sensitive pattern
      #   => delete .value, set .valueFrom.secretKeyRef
      yq -i "
        .${field}[] |= (
          .env |= (
            map(
              if (.value != null and (.name | test(\"${SENSITIVE_PATTERN}\"; \"i\"))) then
                .value = null
                | .valueFrom = {
                    secretKeyRef: {
                      name: \"${secret_name}\",
                      key: .name
                    }
                  }
              else .
              end
            )
          )
        )
      " "${ctrl_file}"
    fi
  done

  # Apply patched controller
  kubectl apply -f "${ctrl_file}"
done

#-----------------------------
# Step 4: Apply Secrets (create or update)
#-----------------------------

echo "Applying Secret manifests..."

find "${secrets_dir}" -type f -name '*.yaml' -print0 | while IFS= read -r -d '' f; do
  kubectl apply -f "${f}"
done

#-----------------------------
# Step 5: Verification
#-----------------------------

echo "Waiting for controller rollouts (Deployments/StatefulSets/DaemonSets)..."

# Best-effort rollout status for Deployments and StatefulSets
jq -c '.[]' "${controllers_list}" | while read -r row; do
  ns=$(jq -r '.namespace' <<<"${row}")
  kind=$(jq -r '.controllerKind' <<<"${row}")
  name=$(jq -r '.controllerName' <<<"${row}")

  case "${kind}" in
    Deployment|StatefulSet)
      echo "  Checking rollout for ${kind}/${ns}/${name}..."
      kubectl rollout status "${kind}/${name}" -n "${ns}" --timeout=180s || \
        echo "    WARN: rollout status timed out for ${kind}/${ns}/${name}"
      ;;
    DaemonSet)
      echo "  Checking DaemonSet availability for ${kind}/${ns}/${name}..."
      kubectl get ds "${name}" -n "${ns}"
      ;;
    Job|CronJob)
      echo "  Skipping rollout wait for ${kind}/${ns}/${name} (job-type)."
      ;;
  esac
done

echo "Re-running compliance query..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.containers // [] + .spec.initContainers // [])[]
    | (.env // [])[]
    | select((.value != null) and (.name | test("'"${SENSITIVE_PATTERN}"'"; "i")))
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else "is_compliant=false (remaining violations: \($rows | length))" end'

echo "Done. Review ${workdir_base} for backup manifests and applied changes."

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​