Prefer Using Secrets As Files Over Secrets As Environment

More Info:

Secrets exposed as environment variables are more likely to be leaked through logs, child processes, or crash dumps. Prefer mounting secrets as files for improved confidentiality.

Risk Level

Medium

Address

Security

Compliance Standards

• CIS EKS

Triage and Remediation

Remediation

Remediation

Manual Steps

1. List all pods using secrets as environment variables (run on any machine with kubectl access):

   kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}'

2. For each affected workload (Deployment/StatefulSet/DaemonSet/Job/CronJob), export its manifest (run on any machine with kubectl access, example for a Deployment):

   NAMESPACE=default
   NAME=my-app
   kubectl -n "$NAMESPACE" get deploy "$NAME" -o yaml > "${NAME}.yaml"

3. In the saved manifest file, manually:

   • Under spec.template.spec, add or extend a volumes entry to mount the secret:

     volumes:
       - name: my-secret-vol
         secret:
           secretName: my-secret

   • Under the container spec, add or extend volumeMounts:

     volumeMounts:
       - name: my-secret-vol
         mountPath: /var/run/secrets/my-secret
         readOnly: true

   • Remove any env or envFrom entries that use secretKeyRef for that secret, and adjust your application code/config to read from the mounted file path instead.

4. Apply the updated manifest (this will roll pods; run on any machine with kubectl access):

   kubectl -n "$NAMESPACE" apply -f "${NAME}.yaml"

5. Repeat steps 2–4 for each listed resource kind (e.g., statefulset, daemonset, job, cronjob) that uses secretKeyRef in env or envFrom.

6. Verify no workloads are using secrets as environment variables (run on any machine with kubectl access):

   result=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}')
   if [ -z "$result" ]; then
     echo "NO_SECRETS_AS_ENV_VARS"
   else
     echo "SECRETS_AS_ENV_VARS_FOUND: $result"
   fi

Using kubectl

On any machine with kubectl access:

1. Identify pods using secrets as environment variables

kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}'

Pick one workload (example below uses a Deployment in namespace default named my-app).

2. Export its manifest for editing

kubectl get deploy my-app -n default -o yaml > my-app-deploy.yaml

3. Edit the manifest to replace env-based secrets with mounted secret files

In my-app-deploy.yaml, locate the container using secretKeyRef, for example:

      containers:
      - name: app
        image: my-image:tag
        env:
          - name: DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: db-secret
                key: password

Change it to mount the secret as a volume and have the app read from the file path instead of the env var. Remove the env entry and add volumeMounts and a volumes section like:

      containers:
      - name: app
        image: my-image:tag
        volumeMounts:
          - name: db-secret-vol
            mountPath: /etc/secrets/db
            readOnly: true
      volumes:
        - name: db-secret-vol
          secret:
            secretName: db-secret
            items:
              - key: password
                path: password

After this change, application code must read /etc/secrets/db/password instead of DB_PASSWORD. This code change is outside kubectl’s scope and must be done in the application.

4. Apply the updated manifest

kubectl apply -f my-app-deploy.yaml

Repeat steps 2–4 for each workload found in step 1.

5. Verification (cluster-wide)

result=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}')
if [ -z "$result" ]; then
  echo "NO_SECRETS_AS_ENV_VARS"
else
  echo "SECRETS_AS_ENV_VARS_FOUND: $result"
fi

Automation

#!/usr/bin/env bash
#
# Automation: Detect pods using secrets as environment variables and
# prepare manifest patches to migrate them to mounted secret volumes.
#
# IMPORTANT:
# - This script CANNOT safely rewrite application code or manifests automatically.
# - It only identifies affected workloads and generates skeleton patches
#   to help you switch from env vars to mounted secret files.
# - You must update the application code to read from files, then
#   review and apply the generated patches manually.
#
# Requirements:
# - Run on any machine with kubectl access and correct KUBECONFIG.
# - kubectl must be v1.18+ and configured with cluster-admin privileges.
#
# Usage:
#   ./fix-secrets-env-to-files.sh
#
# Outputs:
#   - ./secrets-env-findings.txt : list of workloads using secrets in env
#   - ./patches/                 : per-workload patch YAMLs (NOT applied)
#   - Final audit result printed at the end

set -euo pipefail

WORKDIR="$(pwd)"
PATCH_DIR="${WORKDIR}/patches"
FINDINGS_FILE="${WORKDIR}/secrets-env-findings.txt"

mkdir -p "${PATCH_DIR}"
: > "${FINDINGS_FILE}"

echo "Scanning for pods, deployments, statefulsets, daemonsets, and jobs using secrets as environment variables..."

# Helper function: list workloads of a given kind that have env[].valueFrom.secretKeyRef
list_workloads_with_secret_env() {
  local kind="$1"
  # jsonpath: select items where any container env has valueFrom.secretKeyRef
  kubectl get "${kind}" --all-namespaces -o json \
  | jq -r '
      .items[]
      | select(
          (
            (.spec.template.spec.containers[]?.env[]?.valueFrom.secretKeyRef? != null)
            or
            (.spec.template.spec.initContainers[]?.env[]?.valueFrom.secretKeyRef? != null)
          ) // false
        )
      | "\(.metadata.namespace) '"${kind}"' \(.metadata.name)"
    ' 2>/dev/null || true
}

# Collect all findings across key workload types
kinds=("pod" "deployment" "statefulset" "daemonset" "job" "cronjob")

for k in "${kinds[@]}"; do
  echo "Checking kind: ${k}..."
  if [[ "${k}" == "pod" ]]; then
    # pods have spec directly, not under .spec.template.spec
    kubectl get pod --all-namespaces -o json \
    | jq -r '
        .items[]
        | select(
            (
              (.spec.containers[]?.env[]?.valueFrom.secretKeyRef? != null)
              or
              (.spec.initContainers[]?.env[]?.valueFrom.secretKeyRef? != null)
            ) // false
          )
        | "\(.metadata.namespace) pod \(.metadata.name)"
      ' 2>/dev/null || true >> "${FINDINGS_FILE}"
  else
    list_workloads_with_secret_env "${k}" >> "${FINDINGS_FILE}"
  fi
done

sort -u -o "${FINDINGS_FILE}" "${FINDINGS_FILE}"

if [[ ! -s "${FINDINGS_FILE}" ]]; then
  echo "NO_SECRETS_AS_ENV_VARS"
  exit 0
fi

echo "SECRETS_AS_ENV_VARS_FOUND (details in ${FINDINGS_FILE}):"
cat "${FINDINGS_FILE}"

echo
echo "Generating skeleton patches for each affected workload into ${PATCH_DIR} ..."
echo "These patches DO NOT modify env vars automatically; they add example secret volume mounts you can customize."
echo

# For each finding, create a patch skeleton that:
# - Adds a volume referencing the secret
# - Adds a volumeMount to containers
# We must inspect existing env[].valueFrom.secretKeyRef to know which secret names are used.

while read -r ns kind name; do
  [[ -z "${ns}" ]] && continue
  patch_file="${PATCH_DIR}/${ns}__${kind}__${name}.patch.yaml"

  echo "  Preparing patch for ${ns} ${kind} ${name} -> ${patch_file}"

  # Extract unique secret names used in env[].valueFrom.secretKeyRef
  if [[ "${kind}" == "pod" ]]; then
    jsonpath='{.spec}'
  else
    jsonpath='{.spec.template.spec}'
  fi

  # Dump the spec portion for processing
  spec_json="$(kubectl get "${kind}" "${name}" -n "${ns}" -o jsonpath="${jsonpath}" 2>/dev/null || true)"
  if [[ -z "${spec_json}" ]]; then
    echo "    WARNING: could not fetch ${ns} ${kind} ${name}; skipping."
    continue
  fi

  # Collect secret names from env variables
  secret_names="$(printf '%s\n' "${spec_json}" \
    | jq -r '
        [
          (.containers[]?.env[]?.valueFrom.secretKeyRef.name? // empty),
          (.initContainers[]?.env[]?.valueFrom.secretKeyRef.name? // empty)
        ]
        | flatten
        | unique[]
      ' 2>/dev/null || true
  )"

  if [[ -z "${secret_names}" ]]; then
    echo "    No secretKeyRef names found (race condition?); skipping patch."
    continue
  fi

  # Build a simple patch skeleton.
  # NOTE: This is intentionally generic. You must:
  # - Update mountPath(s) to where your app expects files.
  # - Remove env entries that reference the secret AFTER updating the app.
  cat > "${patch_file}" <<EOF
# Patch for ${kind} ${ns}/${name}
# ACTION REQUIRED:
# 1. Update your application code to read these secrets from files under the mounted paths.
# 2. Adjust mountPath values to match your application expectations.
# 3. Remove the corresponding env[].valueFrom.secretKeyRef entries from the workload spec.
#
# Example: if the original env var was:
#   env:
#   - name: DB_PASSWORD
#     valueFrom:
#       secretKeyRef:
#         name: my-db-secret
#         key: password
# Then you might:
#   - Mount my-db-secret at /var/run/secrets/my-db-secret
#   - Read /var/run/secrets/my-db-secret/password from your code.
#
# After validating the patch, apply with:
#   kubectl patch ${kind} ${name} -n ${ns} --type merge --patch "\$(cat ${patch_file})"

spec:
EOF

  if [[ "${kind}" != "pod" ]]; then
    echo "  template:" >> "${patch_file}"
    echo "    spec:" >> "${patch_file}"
    indent="      "
  else
    indent="  "
  fi

  # Append volumes and volumeMounts example
  {
    echo "${indent}volumes:"
    for sname in ${secret_names}; do
      echo "${indent}- name: secret-${sname}"
      echo "${indent}  secret:"
      echo "${indent}    secretName: ${sname}"
    done

    echo
    echo "${indent}containers:"
    echo "${indent}- name: <REPLACE_WITH_CONTAINER_NAME>"
    echo "${indent}  volumeMounts:"
    for sname in ${secret_names}; do
      echo "${indent}  - name: secret-${sname}"
      echo "${indent}    mountPath: /var/run/secrets/${sname}"
      echo "${indent}    readOnly: true"
    done

    echo
    echo "${indent}# If you use initContainers, replicate similar volumeMounts there."
  } >> "${patch_file}"

done < "${FINDINGS_FILE}"

echo
echo "Patch skeletons generated under: ${PATCH_DIR}"
echo "Review and edit each patch, update your application code, then apply patches manually."
echo

echo "Re-running audit to verify current state (no changes have been applied by this script)..."
result="$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}' || true)"

if [[ -z "${result}" ]]; then
  echo "NO_SECRETS_AS_ENV_VARS"
else
  echo "SECRETS_AS_ENV_VARS_FOUND (unchanged until you update code and apply patches):"
  echo "${result}"
fi