Skip to main content

Sensitive Values Should Not Be Passed As Literal Env Vars

More Info:​

Verifies secret-like env vars are not set as literal values. Literal values land in the pod manifest, logs and kubectl describe.

Risk Level​

Medium

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. On any machine with kubectl access, list offending Pods so you know what to fix (capture output for reference):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
        | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
        | .metadata as $m
        | (.spec.containers // [] + .spec.initContainers // [])[]
        | .name as $c
        | (.env // [])[]
        | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
        | "ns=\($m.namespace) pod=\($m.name) container=\($c) env=\(.name)"
      ] | .[]'

  2. For one offending Pod, identify its controller (Deployment/StatefulSet/Job/etc.) and confirm you will edit the controller, not the live Pod:

    NAMESPACE="example-namespace"
    POD_NAME="example-pod"

    kubectl get pod "${POD_NAME}" -n "${NAMESPACE}" -o jsonpath='{.metadata.ownerReferences[0].kind}{" "}{.metadata.ownerReferences[0].name}{"\n"}'

    If there is no ownerReferences entry, the Pod is standalone and must be edited or recreated directly.

  3. Create a Kubernetes Secret in the same namespace to hold the sensitive value (replace names and values appropriately):

    NAMESPACE="example-namespace"
    SECRET_NAME="app-credentials"
    SECRET_KEY="DB_PASSWORD"
    SECRET_VALUE="REPLACE_WITH_ACTUAL_PASSWORD"

    kubectl create secret generic "${SECRET_NAME}" \
      -n "${NAMESPACE}" \
      --from-literal="${SECRET_KEY}=${SECRET_VALUE}"

  4. Edit the owning controller manifest (or the standalone Pod) to replace the literal value: with valueFrom.secretKeyRef for each sensitive env var, using kubectl edit on any machine with kubectl access:

    # Example for a Deployment; change Kind/name as discovered in step 2
    kubectl -n "${NAMESPACE}" edit deployment/example-deployment

    In the editor, change:

    env:
      - name: DB_PASSWORD
        value: "REPLACE_WITH_ACTUAL_PASSWORD"

    to:

    env:
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: app-credentials
            key: DB_PASSWORD

    Save and exit; Kubernetes will roll out updated Pods automatically.

  5. For standalone Pods (no controller), export the manifest, modify it, delete the old Pod, and recreate it:

    NAMESPACE="example-namespace"
    POD_NAME="example-standalone-pod"

    kubectl -n "${NAMESPACE}" get pod "${POD_NAME}" -o yaml > /tmp/pod-fixed.yaml
    # Edit /tmp/pod-fixed.yaml: remove `status:`, change each sensitive env var to use valueFrom.secretKeyRef as in step 4.
    sed -i '/^status:/,$d' /tmp/pod-fixed.yaml

    kubectl -n "${NAMESPACE}" delete pod "${POD_NAME}"
    kubectl -n "${NAMESPACE}" apply -f /tmp/pod-fixed.yaml

  6. Verify that no sensitive values are still passed as literal env vars by rerunning the audit command from any machine with kubectl access; compliance is shown when it returns only is_compliant=true:

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
        | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
        | .metadata as $m
        | (.spec.nodeName // "") as $node
        | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
        | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
        | ((.spec.containers // []) + (.spec.initContainers // []))[]
        | .name as $c
        | (.env // [])[]
        | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
        | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
          + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
          + (if $node   == ""   then "" else " node=\($node)" end)
          + (if $labels == ""   then "" else " labels=\($labels)" end)
          + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
          + " container=\($c) env=\(.name) is_compliant=false"
      ] as $rows
      | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Using kubectl

On any machine with kubectl access:

  1. Identify offending Pods and their env vars (example using the audit logic):
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "ns=\($m.namespace) pod=\($m.name) container=\($c) env=\(.name)"
  ][]'
  1. For each Pod owned by a controller (Deployment/StatefulSet/Job/CronJob, etc.), edit the controller, not the Pod. Example: convert a literal DB_PASSWORD env var in a Deployment to use a Secret.

2.1. Create a Secret containing the sensitive value (replace placeholders with your real values):

kubectl -n your-namespace create secret generic db-password-secret \
  --from-literal=db-password='REPLACE_WITH_REAL_PASSWORD'

2.2. Edit the owning Deployment to use valueFrom.secretKeyRef:

kubectl -n your-namespace edit deployment your-deployment-name

In the container spec, replace:

env:
  - name: DB_PASSWORD
    value: "REPLACE_WITH_REAL_PASSWORD"

with:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-password-secret
        key: db-password

Save and exit; Kubernetes will roll out new Pods for that Deployment using the Secret.

  1. If you prefer fully declarative changes, fetch, edit locally, and apply:
kubectl -n your-namespace get deployment your-deployment-name -o yaml > deployment.yaml

Edit deployment.yaml to remove literal secret values and reference Secrets via valueFrom.secretKeyRef as above, then apply:

kubectl apply -f deployment.yaml
  1. For workloads using volume-mounted Secrets instead of env vars, create the Secret as in 2.1, then add:
volumes:
  - name: app-secrets
    secret:
      secretName: db-password-secret

containers:
  - name: your-container
    volumeMounts:
      - name: app-secrets
        mountPath: /etc/secrets
        readOnly: true

and update your app to read from the mounted file.

  1. Verification (same audit command; a compliant cluster prints only is_compliant=true):
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.nodeName // "") as $node
    | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
      + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
      + (if $node   == ""   then "" else " node=\($node)" end)
      + (if $labels == ""   then "" else " labels=\($labels)" end)
      + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
      + " container=\($c) env=\(.name) is_compliant=false"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Automation
#!/usr/bin/env bash
#
# Remediation: Move secret-like literal env vars in Pods to a Secret and
#              reference them via valueFrom.secretKeyRef.
#
# Scope: Any machine with kubectl access to the EKS cluster.
#
# Requirements:
#   - kubectl configured to point at the target cluster
#   - jq and yq (https://github.com/mikefarah/yq) installed and in PATH
#
# Notes:
#   - This script is idempotent: it only changes Pods with matching env vars.
#   - It creates/patches one Secret per Pod: <pod-name>-env-secrets in the
#     same namespace, with one key per offending env var.
#   - It deletes the original Pod so its controller (Deployment, ReplicaSet,
#     StatefulSet, Job, etc.) recreates it with the new env references.
#   - Standalone Pods (no controller ownerReference) will be re-created
#     directly by the script with the updated spec.

set -euo pipefail

if ! command -v kubectl >/dev/null 2>&1; then
  echo "kubectl not found in PATH" >&2
  exit 1
fi

if ! command -v jq >/dev/null 2>&1; then
  echo "jq not found in PATH" >&2
  exit 1
fi

if ! command -v yq >/dev/null 2>&1; then
  echo "yq not found in PATH (https://github.com/mikefarah/yq)" >&2
  exit 1
fi

SED_INPLACE=("-i")
if [[ "$(uname -s)" == "Darwin" ]]; then
  SED_INPLACE=("-i" "")
fi

TMPDIR="$(mktemp -d)"
trap 'rm -rf "${TMPDIR}"' EXIT

echo "Discovering Pods with sensitive literal env vars..."

# Reuse the provided audit logic to find offending pods and env vars
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.nodeName // "") as $node
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | {
        namespace: $m.namespace,
        pod: $m.name,
        container: $c,
        envName: .name,
        envValue: .value,
        ownerKind: ($own.kind // null),
        ownerName: ($own.name // null),
        ownerUid: ($own.uid // null)
      }
  ] | .[]' > "${TMPDIR}/offending-envs.json" || true

if [[ ! -s "${TMPDIR}/offending-envs.json" ]]; then
  echo "No offending env vars found. Cluster appears compliant."
  exit 0
fi

echo "Offending env vars detected. Grouping by pod..."

# Build a list of unique pod identifiers (namespace/pod)
jq -r '[.namespace + "/" + .pod] | unique[]' "${TMPDIR}/offending-envs.json" > "${TMPDIR}/pods.txt"

while IFS=/ read -r NS POD; do
  echo "Processing Pod ${NS}/${POD}..."

  # Extract all offending env vars for this pod
  jq --arg ns "${NS}" --arg pod "${POD}" '
    select(.namespace == $ns and .pod == $pod)
  ' "${TMPDIR}/offending-envs.json" > "${TMPDIR}/pod-envs.json"

  if [[ ! -s "${TMPDIR}/pod-envs.json" ]]; then
    echo "  No offending envs found for ${NS}/${POD} (skipping)."
    continue
  fi

  SECRET_NAME="${POD}-env-secrets"

  # Create or update Secret manifest
  SECRET_FILE="${TMPDIR}/${NS}-${SECRET_NAME}-secret.yaml"
  cat > "${SECRET_FILE}" <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ${SECRET_NAME}
  namespace: ${NS}
type: Opaque
data: {}
EOF

  # For each env var, add/overwrite key in Secret (base64-encoded)
  while read -r ENV_JSON; do
    ENV_NAME=$(jq -r '.envName' <<<"${ENV_JSON}")
    ENV_VALUE=$(jq -r '.envValue' <<<"${ENV_JSON}")
    # Base64 encode value
    B64_VALUE=$(printf '%s' "${ENV_VALUE}" | base64 | tr -d '\n')
    yq e "${SED_INPLACE[@]}" \
      ".data.\"${ENV_NAME}\" = \"${B64_VALUE}\"" \
      "${SECRET_FILE}"
  done < <(cat "${TMPDIR}/pod-envs.json")

  echo "  Applying Secret ${NS}/${SECRET_NAME}..."
  kubectl apply -f "${SECRET_FILE}"

  # Get full Pod manifest
  POD_FILE="${TMPDIR}/${NS}-${POD}-pod.yaml"
  kubectl get pod "${POD}" -n "${NS}" -o yaml > "${POD_FILE}"

  # Patch env vars in both containers and initContainers
  for PATH in "spec.containers" "spec.initContainers"; do
    COUNT=$(yq e ".${PATH} // [] | length" "${POD_FILE}")
    if [[ "${COUNT}" -eq 0 ]]; then
      continue
    fi

    for (( i=0; i<COUNT; i++ )); do
      # For each offending env in this pod, if present in this container, convert to valueFrom.secretKeyRef
      while read -r ENV_JSON; do
        ENV_NAME=$(jq -r '.envName' <<<"${ENV_JSON}")
        EXISTS=$(yq e ".${PATH}[${i}].env[]? | select(.name == \"${ENV_NAME}\") | length > 0" "${POD_FILE}" || echo "false")
        if [[ "${EXISTS}" != "true" ]]; then
          continue
        fi

        echo "  Updating ${PATH}[${i}] env ${ENV_NAME} to use Secret ${SECRET_NAME}..."

        # Remove literal value and replace with valueFrom.secretKeyRef
        yq e "${SED_INPLACE[@]}" "
          .${PATH}[${i}].env |=
            map(
              if .name == \"${ENV_NAME}\" then
                {name: .name, valueFrom: {secretKeyRef: {name: \"${SECRET_NAME}\", key: \"${ENV_NAME}\"}}}
              else .
              end
            )
        " "${POD_FILE}"
      done < <(cat "${TMPDIR}/pod-envs.json")
    done
  done

  # Remove Pod-specific runtime fields that prevent re-creation
  yq e "${SED_INPLACE[@]}" '
    del(.metadata.uid) |
    del(.metadata.resourceVersion) |
    del(.metadata.selfLink) |
    del(.metadata.creationTimestamp) |
    del(.metadata.generation) |
    del(.metadata.managedFields) |
    del(.status)
  ' "${POD_FILE}"

  # Determine if this Pod has a controller ownerReference
  OWNER_KIND=$(jq -r '.[0].ownerKind // ""' "${TMPDIR}/pod-envs.json")
  OWNER_NAME=$(jq -r '.[0].ownerName // ""' "${TMPDIR}/pod-envs.json")

  echo "  Deleting original Pod ${NS}/${POD}..."
  kubectl delete pod "${POD}" -n "${NS}" --wait=false || true

  if [[ -n "${OWNER_KIND}" && "${OWNER_KIND}" != "null" && -n "${OWNER_NAME}" && "${OWNER_NAME}" != "null" ]]; then
    echo "  Pod is managed by ${OWNER_KIND}/${OWNER_NAME}; controller will recreate it with updated Secret-based env."
    # We do NOT apply the pod manifest directly in this case, as the controller spec still has old env config.
    # To fully remediate, the owning controller spec should be updated via its manifest/IaC outside this script.
    echo "  NOTE: You must update the ${OWNER_KIND} ${OWNER_NAME} spec to reference Secret ${SECRET_NAME} instead of literal env values."
  else
    echo "  Pod appears standalone; recreating updated Pod from manifest..."
    kubectl apply -f "${POD_FILE}"
  fi

done < "${TMPDIR}/pods.txt"

echo
echo "Re-running verification..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.nodeName // "") as $node
    | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | ((.spec.containers // []) + (.spec.initContainers // []))[]
    | .name as $c
    | (.env // [])[]
    | select((.value != null) and (.name | test("PASSWORD|SECRET|TOKEN|APIKEY|API_KEY|CREDENTIAL|PRIVATE_KEY"; "i")))
    | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
      + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
      + (if $node   == ""   then "" else " node=\($node)" end)
      + (if $labels == ""   then "" else " labels=\($labels)" end)
      + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
      + " container=\($c) env=\(.name) is_compliant=false"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'