Sensitive Values Should Not Be Passed As Literal Env Vars

Manual Steps

On any machine with kubectl access, list offending Pods so you know what to fix (capture output for reference):

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.containers // [] + .spec.initContainers // [])[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "ns=\($m.namespace) pod=\($m.name) container=\($c) env=\(.name)"
  ] | .[]'

For one offending Pod, identify its controller (Deployment/StatefulSet/Job/etc.) and confirm you will edit the controller, not the live Pod:
NAMESPACE="example-namespace" POD_NAME="example-pod" kubectl get pod "${POD_NAME}" -n "${NAMESPACE}" -o jsonpath='{.metadata.ownerReferences[0].kind}{" "}{.metadata.ownerReferences[0].name}{"\n"}'
If there is no ownerReferences entry, the Pod is standalone and must be edited or recreated directly.

Create a Kubernetes Secret in the same namespace to hold the sensitive value (replace names and values appropriately):

NAMESPACE="example-namespace"
SECRET_NAME="app-credentials"
SECRET_KEY="DB_PASSWORD"
SECRET_VALUE="REPLACE_WITH_ACTUAL_PASSWORD"

kubectl create secret generic "${SECRET_NAME}" \
  -n "${NAMESPACE}" \
  --from-literal="${SECRET_KEY}=${SECRET_VALUE}"

Edit the owning controller manifest (or the standalone Pod) to replace the literal value: with valueFrom.secretKeyRef for each sensitive env var, using kubectl edit on any machine with kubectl access:

# Example for a Deployment; change Kind/name as discovered in step 2
kubectl -n "${NAMESPACE}" edit deployment/example-deployment

In the editor, change:

env:
  - name: DB_PASSWORD
    value: "REPLACE_WITH_ACTUAL_PASSWORD"

to:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: app-credentials
        key: DB_PASSWORD

Save and exit; Kubernetes will roll out updated Pods automatically.

For standalone Pods (no controller), export the manifest, modify it, delete the old Pod, and recreate it:

NAMESPACE="example-namespace"
POD_NAME="example-standalone-pod"

kubectl -n "${NAMESPACE}" get pod "${POD_NAME}" -o yaml > /tmp/pod-fixed.yaml
# Edit /tmp/pod-fixed.yaml: remove `status:`, change each sensitive env var to use valueFrom.secretKeyRef as in step 4.
sed -i '/^status:/,$d' /tmp/pod-fixed.yaml

kubectl -n "${NAMESPACE}" delete pod "${POD_NAME}"
kubectl -n "${NAMESPACE}" apply -f /tmp/pod-fixed.yaml

Verify that no sensitive values are still passed as literal env vars by rerunning the audit command from any machine with kubectl access; compliance is shown when it returns only is_compliant=true:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.nodeName // "") as $node
    | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
      + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
      + (if $node   == ""   then "" else " node=\($node)" end)
      + (if $labels == ""   then "" else " labels=\($labels)" end)
      + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
      + " container=\($c) env=\(.name) is_compliant=false"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

Using kubectl

On any machine with kubectl access:

Identify offending Pods and their env vars (example using the audit logic):

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "ns=\($m.namespace) pod=\($m.name) container=\($c) env=\(.name)"
  ][]'

For each Pod owned by a controller (Deployment/StatefulSet/Job/CronJob, etc.), edit the controller, not the Pod. Example: convert a literal DB_PASSWORD env var in a Deployment to use a Secret.

2.1. Create a Secret containing the sensitive value (replace placeholders with your real values):

kubectl -n your-namespace create secret generic db-password-secret \
  --from-literal=db-password='REPLACE_WITH_REAL_PASSWORD'

2.2. Edit the owning Deployment to use valueFrom.secretKeyRef:

kubectl -n your-namespace edit deployment your-deployment-name

In the container spec, replace:

env:
  - name: DB_PASSWORD
    value: "REPLACE_WITH_REAL_PASSWORD"

with:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-password-secret
        key: db-password

Save and exit; Kubernetes will roll out new Pods for that Deployment using the Secret.

If you prefer fully declarative changes, fetch, edit locally, and apply:

kubectl -n your-namespace get deployment your-deployment-name -o yaml > deployment.yaml

Edit deployment.yaml to remove literal secret values and reference Secrets via valueFrom.secretKeyRef as above, then apply:

kubectl apply -f deployment.yaml

For workloads using volume-mounted Secrets instead of env vars, create the Secret as in 2.1, then add:

volumes:
  - name: app-secrets
    secret:
      secretName: db-password-secret

containers:
  - name: your-container
    volumeMounts:
      - name: app-secrets
        mountPath: /etc/secrets
        readOnly: true

and update your app to read from the mounted file.

Verification (same audit command; a compliant cluster prints only is_compliant=true):

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.nodeName // "") as $node
    | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
      + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
      + (if $node   == ""   then "" else " node=\($node)" end)
      + (if $labels == ""   then "" else " labels=\($labels)" end)
      + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
      + " container=\($c) env=\(.name) is_compliant=false"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

Automation

#!/usr/bin/env bash
#
# Remediation: Move secret-like literal env vars in Pods to a Secret and
#              reference them via valueFrom.secretKeyRef.
#
# Scope: Any machine with kubectl access to the EKS cluster.
#
# Requirements:
#   - kubectl configured to point at the target cluster
#   - jq and yq (https://github.com/mikefarah/yq) installed and in PATH
#
# Notes:
#   - This script is idempotent: it only changes Pods with matching env vars.
#   - It creates/patches one Secret per Pod: <pod-name>-env-secrets in the
#     same namespace, with one key per offending env var.
#   - It deletes the original Pod so its controller (Deployment, ReplicaSet,
#     StatefulSet, Job, etc.) recreates it with the new env references.
#   - Standalone Pods (no controller ownerReference) will be re-created
#     directly by the script with the updated spec.

set -euo pipefail

if ! command -v kubectl >/dev/null 2>&1; then
  echo "kubectl not found in PATH" >&2
  exit 1
fi

if ! command -v jq >/dev/null 2>&1; then
  echo "jq not found in PATH" >&2
  exit 1
fi

if ! command -v yq >/dev/null 2>&1; then
  echo "yq not found in PATH (https://github.com/mikefarah/yq)" >&2
  exit 1
fi

SED_INPLACE=("-i")
if [[ "$(uname -s)" == "Darwin" ]]; then
  SED_INPLACE=("-i" "")
fi

TMPDIR="$(mktemp -d)"
trap 'rm -rf "${TMPDIR}"' EXIT

echo "Discovering Pods with sensitive literal env vars..."

# Reuse the provided audit logic to find offending pods and env vars
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.nodeName // "") as $node
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | {
        namespace: $m.namespace,
        pod: $m.name,
        container: $c,
        envName: .name,
        envValue: .value,
        ownerKind: ($own.kind // null),
        ownerName: ($own.name // null),
        ownerUid: ($own.uid // null)
      }
  ] | .[]' > "${TMPDIR}/offending-envs.json" || true

if [[ ! -s "${TMPDIR}/offending-envs.json" ]]; then
  echo "No offending env vars found. Cluster appears compliant."
  exit 0
fi

echo "Offending env vars detected. Grouping by pod..."

# Build a list of unique pod identifiers (namespace/pod)
jq -r '[.namespace + "/" + .pod] | unique[]' "${TMPDIR}/offending-envs.json" > "${TMPDIR}/pods.txt"

while IFS=/ read -r NS POD; do
  echo "Processing Pod ${NS}/${POD}..."

  # Extract all offending env vars for this pod
  jq --arg ns "${NS}" --arg pod "${POD}" '
    select(.namespace == $ns and .pod == $pod)
  ' "${TMPDIR}/offending-envs.json" > "${TMPDIR}/pod-envs.json"

  if [[ ! -s "${TMPDIR}/pod-envs.json" ]]; then
    echo "  No offending envs found for ${NS}/${POD} (skipping)."
    continue
  fi

  SECRET_NAME="${POD}-env-secrets"

  # Create or update Secret manifest
  SECRET_FILE="${TMPDIR}/${NS}-${SECRET_NAME}-secret.yaml"
  cat > "${SECRET_FILE}" <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ${SECRET_NAME}
  namespace: ${NS}
type: Opaque
data: {}
EOF

  # For each env var, add/overwrite key in Secret (base64-encoded)
  while read -r ENV_JSON; do
    ENV_NAME=$(jq -r '.envName' <<<"${ENV_JSON}")
    ENV_VALUE=$(jq -r '.envValue' <<<"${ENV_JSON}")
    # Base64 encode value
    B64_VALUE=$(printf '%s' "${ENV_VALUE}" | base64 | tr -d '\n')
    yq e "${SED_INPLACE[@]}" \
      ".data.\"${ENV_NAME}\" = \"${B64_VALUE}\"" \
      "${SECRET_FILE}"
  done < <(cat "${TMPDIR}/pod-envs.json")

  echo "  Applying Secret ${NS}/${SECRET_NAME}..."
  kubectl apply -f "${SECRET_FILE}"

  # Get full Pod manifest
  POD_FILE="${TMPDIR}/${NS}-${POD}-pod.yaml"
  kubectl get pod "${POD}" -n "${NS}" -o yaml > "${POD_FILE}"

  # Patch env vars in both containers and initContainers
  for PATH in "spec.containers" "spec.initContainers"; do
    COUNT=$(yq e ".${PATH} // [] | length" "${POD_FILE}")
    if [[ "${COUNT}" -eq 0 ]]; then
      continue
    fi

    for (( i=0; i<COUNT; i++ )); do
      # For each offending env in this pod, if present in this container, convert to valueFrom.secretKeyRef
      while read -r ENV_JSON; do
        ENV_NAME=$(jq -r '.envName' <<<"${ENV_JSON}")
        EXISTS=$(yq e ".${PATH}[${i}].env[]? | select(.name == \"${ENV_NAME}\") | length > 0" "${POD_FILE}" || echo "false")
        if [[ "${EXISTS}" != "true" ]]; then
          continue
        fi

        echo "  Updating ${PATH}[${i}] env ${ENV_NAME} to use Secret ${SECRET_NAME}..."

        # Remove literal value and replace with valueFrom.secretKeyRef
        yq e "${SED_INPLACE[@]}" "
          .${PATH}[${i}].env |=
            map(
              if .name == \"${ENV_NAME}\" then
                {name: .name, valueFrom: {secretKeyRef: {name: \"${SECRET_NAME}\", key: \"${ENV_NAME}\"}}}
              else .
              end
            )
        " "${POD_FILE}"
      done < <(cat "${TMPDIR}/pod-envs.json")
    done
  done

  # Remove Pod-specific runtime fields that prevent re-creation
  yq e "${SED_INPLACE[@]}" '
    del(.metadata.uid) |
    del(.metadata.resourceVersion) |
    del(.metadata.selfLink) |
    del(.metadata.creationTimestamp) |
    del(.metadata.generation) |
    del(.metadata.managedFields) |
    del(.status)
  ' "${POD_FILE}"

  # Determine if this Pod has a controller ownerReference
  OWNER_KIND=$(jq -r '.[0].ownerKind // ""' "${TMPDIR}/pod-envs.json")
  OWNER_NAME=$(jq -r '.[0].ownerName // ""' "${TMPDIR}/pod-envs.json")

  echo "  Deleting original Pod ${NS}/${POD}..."
  kubectl delete pod "${POD}" -n "${NS}" --wait=false || true

  if [[ -n "${OWNER_KIND}" && "${OWNER_KIND}" != "null" && -n "${OWNER_NAME}" && "${OWNER_NAME}" != "null" ]]; then
    echo "  Pod is managed by ${OWNER_KIND}/${OWNER_NAME}; controller will recreate it with updated Secret-based env."
    # We do NOT apply the pod manifest directly in this case, as the controller spec still has old env config.
    # To fully remediate, the owning controller spec should be updated via its manifest/IaC outside this script.
    echo "  NOTE: You must update the ${OWNER_KIND} ${OWNER_NAME} spec to reference Secret ${SECRET_NAME} instead of literal env values."
  else
    echo "  Pod appears standalone; recreating updated Pod from manifest..."
    kubectl apply -f "${POD_FILE}"
  fi

done < "${TMPDIR}/pods.txt"

echo
echo "Re-running verification..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.nodeName // "") as $node
    | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
      + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
      + (if $node   == ""   then "" else " node=\($node)" end)
      + (if $labels == ""   then "" else " labels=\($labels)" end)
      + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
      + " container=\($c) env=\(.name) is_compliant=false"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

AWS Introduction

AWS Pricing

AWS Threats

AWS Misconfigurations

Sensitive Values Should Not Be Passed As Literal Env Vars

More Info:

Risk Level

Address

Compliance Standards

Triage and Remediation

Remediation

​More Info:

​Risk Level

​Address

​Compliance Standards

​Triage and Remediation

​Remediation