Prefer Using Secrets Files Over Secrets Environment

More Info:

Kubernetes supports mounting secrets as data volumes or as environment variables. Minimize the use of environment variable secrets.

Risk Level

High

Address

Security

Compliance Standards

• APRA CPS 234 (Australia)
• AWS Startup Security Baseline
• BSI C5 (Germany)
• Brazil LGPD
• CCPA / CPRA (California)
• CIS Critical Security Controls v8
• CIS EKS
• CMMC 2.0
• CSA Cloud Controls Matrix v4
• DPDPA
• Digital Operational Resilience Act (EU)
• Essential 8
• ISO/IEC 27017
• ISO/IEC 27018
• ISO/IEC 27701
• KSA PDPL
• MAS Technology Risk Management (Singapore)
• MITRE ATT&CK (Cloud)
• NIST SP 800-171
• NYDFS 23 NYCRR 500
• SWIFT Customer Security Controls Framework
• Sarbanes-Oxley IT General Controls
• UK NCSC Cyber Assessment Framework

Triage and Remediation

Remediation

Remediation

Manual Steps

1. List all workloads using secrets as environment variables

   • Run on: any machine with kubectl access.

   kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}'

2. Inspect a specific workload’s env-based secret usage

   • Pick one namespace/kind/name from step 1 and inspect its manifest:

   # Example for a Deployment
   kubectl -n NAMESPACE get deploy DEPLOYMENT_NAME -o yaml > /tmp/deploy-NAMESPACE-DEPLOYMENT_NAME.yaml

   • In the downloaded YAML, locate env: entries that use valueFrom.secretKeyRef.

3. Refactor the manifest to mount secrets as files instead of env vars

   • Edit the saved manifest file on your local machine:
   • Under spec.template.spec, add or update a volumes entry referencing the secret:

   spec:
     template:
       spec:
         volumes:
           - name: app-secret
             secret:
               secretName: SECRET_NAME

   • For each container that currently uses env.valueFrom.secretKeyRef, remove those env entries and mount the secret as a volume:

   containers:
     - name: app-container
       volumeMounts:
         - name: app-secret
           mountPath: /var/run/secrets/app
           readOnly: true

   • Ensure the application code/config is updated to read the secret from the mounted file path (for example, /var/run/secrets/app/KEY_NAME). This requires application changes outside of Kubernetes and must be coordinated with the application owner.

4. Apply the updated manifest to the cluster

   • Run on: any machine with kubectl access.

   kubectl apply -f /tmp/deploy-NAMESPACE-DEPLOYMENT_NAME.yaml

   • Repeat steps 2–4 for each resource reported in step 1 (Deployment, StatefulSet, DaemonSet, Job, CronJob, etc.), adapting kind and filenames as needed.

5. Optionally remove any remaining env-based secret refs programmatically (review required)

   • To review all manifests currently using secretKeyRef before final cleanup:

   kubectl get all --all-namespaces -o yaml | grep -n "secretKeyRef" -n

   • Manually confirm each usage is safe to remove and that the application reads from files; then adjust the corresponding manifests as in steps 2–4. Do not delete env entries without confirming application behavior.

6. Verify no secrets are used as environment variables

   • Run on: any machine with kubectl access.

   result=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}')
   if [ -z "$result" ]; then
     echo "NO_SECRETS_AS_ENV_VARS"
   else
     echo "SECRETS_AS_ENV_VARS_FOUND: $result"
   fi

Using kubectl

On any machine with kubectl access:

1. Identify workloads using secretKeyRef (same as audit)

kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}'

2. For each listed object, switch from env-var secrets to volume-mounted secrets.

Example: convert a Deployment using env.valueFrom.secretKeyRef to use a Secret volume.

Before (env-var based):

kubectl get deployment my-app -n my-namespace -o yaml > /tmp/my-app-deployment.yaml

Edit /tmp/my-app-deployment.yaml and replace:

        env:
          - name: DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: db-secret
                key: password

with volume-based access:

        volumeMounts:
          - name: db-secret-vol
            mountPath: /var/run/secrets/db
            readOnly: true
        env:
          - name: DB_PASSWORD_FILE
            value: /var/run/secrets/db/password
      volumes:
        - name: db-secret-vol
          secret:
            secretName: db-secret

Make sure volumes: is under spec.template.spec and volumeMounts: is under the container spec. Adjust mountPath, name, and secretName as needed. Your application code must read from the file path (e.g., /var/run/secrets/db/password) instead of the environment variable.

3. Apply the updated manifest (this will roll the workload)

kubectl apply -f /tmp/my-app-deployment.yaml

Repeat step 2–3 for each Deployment/StatefulSet/DaemonSet/Pod reported in step 1.

4. Verification (no remaining secretKeyRef in running workloads)

result=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}')
if [ -z "$result" ]; then
  echo "NO_SECRETS_AS_ENV_VARS"
else
  echo "SECRETS_AS_ENV_VARS_FOUND: $result"
fi

Automation

#!/usr/bin/env bash
#
# Automation: Prefer using secrets as files over secrets as environment variables
#
# Scope: any machine with kubectl access to the cluster
#
# IMPORTANT:
# - This script CANNOT safely rewrite application code or pod specs automatically.
# - It is designed to:
#   * Detect all workloads using secretKeyRef in env or envFrom
#   * Generate patched manifest templates that mount the same secrets as volumes
#     under /var/run/secrets/<name>/<key>, with comments guiding manual spec changes
#   * Allow you to manually adjust container commands/code to read from files
# - It is safe to re-run; it overwrites only its own output directory and re-runs audits.
#
# You must manually:
# - Update application code/images to read from the mounted files instead of env vars
# - Remove the env/secretKeyRef entries once applications are updated and tested
#
# Requirements:
# - bash, kubectl, jq, yq (v4+), and mkdir/cat/sed
# - Cluster context configured (kubectl works without extra flags)
#
# NOTE:
# - There is no deterministic, fully automated fix because the benchmark requires
#   application-level changes. This script prepares and guides those changes.


set -euo pipefail

OUT_DIR="./secret-env-to-file-plans"
TIMESTAMP="$(date -u +%Y%m%dT%H%M%SZ)"

mkdir -p "${OUT_DIR}"

echo "=== [1/4] Detecting workloads using secrets as environment variables ==="

# Reuse the authoritative audit logic, but also collect JSON for processing.
AUDIT_RAW_FILE="${OUT_DIR}/audit_raw_${TIMESTAMP}.txt"
AUDIT_JSON_FILE="${OUT_DIR}/audit_workloads_${TIMESTAMP}.json"

result=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}') || true
echo "${result}" > "${AUDIT_RAW_FILE}"

if [ -z "${result}" ]; then
  echo "NO_SECRETS_AS_ENV_VARS"
  echo "No workloads using secretKeyRef in env were found. Nothing to prepare."
  exit 0
fi

echo "SECRETS_AS_ENV_VARS_FOUND:"
cat "${AUDIT_RAW_FILE}"

# Collect full JSON of affected resources (idempotent snapshot)
# We deliberately limit to objects that 'get all' may include, but cover common controllers.
kubectl get all --all-namespaces -o json > "${AUDIT_JSON_FILE}"

echo "Raw audit list saved to: ${AUDIT_RAW_FILE}"
echo "Full JSON snapshot saved to: ${AUDIT_JSON_FILE}"

echo "=== [2/4] Generating migration plans with volume-mounted secrets ==="

PLAN_DIR="${OUT_DIR}/plans_${TIMESTAMP}"
mkdir -p "${PLAN_DIR}"

# Function to generate a plan for a single named object
generate_plan_for_object() {
  local ns="$1"
  local kind="$2"
  local name="$3"

  local safe_ns safe_kind safe_name
  safe_ns="${ns//\//_}"
  safe_kind="${kind//\//_}"
  safe_name="${name//\//_}"

  local out_file="${PLAN_DIR}/${safe_ns}__${safe_kind}__${safe_name}.yaml"

  echo "  - Processing ${ns} ${kind} ${name}"

  # Fetch live object as YAML
  if ! kubectl get "${kind}" "${name}" -n "${ns}" -o yaml > "${out_file}.orig" 2>/dev/null; then
    echo "    ! Skipping (unable to fetch YAML for ${ns}/${kind}/${name})"
    rm -f "${out_file}.orig"
    return
  fi

  # Copy original to working file
  cp "${out_file}.orig" "${out_file}"

  # Add comments at the top of the file to guide manual changes
  {
    echo "# ============================================================="
    echo "# PLAN for ${ns} ${kind} ${name}"
    echo "# Generated: ${TIMESTAMP}"
    echo "#"
    echo "# This file is a COPY of the live object and is NOT applied automatically."
    echo "# Actions you should take:"
    echo "# 1) For each container using env.secretKeyRef or envFrom.secretRef,"
    echo "#    this plan adds a volume and volumeMount for the referenced Secret"
    echo "#    under: /var/run/secrets/<secret-name>/<key>"
    echo "# 2) Modify container code/command/args to read secrets from these files"
    echo "#    instead of from environment variables."
    echo "# 3) Once the application works with files, REMOVE the corresponding"
    echo "#    env and envFrom.secretRef entries, then apply this manifest:"
    echo "#      kubectl apply -f ${out_file}"
    echo "# 4) After rollout, re-run the verification step (see end of script)."
    echo "#"
    echo "# NOTE: This plan does not alter the live cluster until you apply it."
    echo "# ============================================================="
    echo ""
  } | cat - "${out_file}" > "${out_file}.tmp" && mv "${out_file}.tmp" "${out_file}"

  # Using yq to append volumes and volumeMounts for each secretRef used in env or envFrom.
  #
  # Strategy:
  # - For each spec.template.spec.containers[*]:
  #     * Collect unique secret names from:
  #         - env[].valueFrom.secretKeyRef.name
  #         - envFrom[].secretRef.name
  #     * For each unique secret name:
  #         - Ensure a pod spec volume named "secret-<secret-name>" exists:
  #           spec.template.spec.volumes += { name: "secret-<secret-name>", secret: { secretName: <secret-name> } }
  #         - Ensure the container has a volumeMount:
  #           mountPath: "/var/run/secrets/<secret-name>"
  #           name: "secret-<secret-name>"
  #
  # The keys will be present as files inside that directory.
  #

  # Add a helper annotation to mark that a plan was generated
  yq -i '
    .metadata.annotations."secret-env-to-file-plan/generated" = "'"${TIMESTAMP}"'"' \
    "${out_file}"

  # Process top-level pod specs and template-based specs
  # Handle objects with spec.template.spec (Deployments, DaemonSets, etc.)
  yq -i '
    (.. | select(has("template") and .template.spec? != null) | .template.spec) |=
      (
        . as $spec |
        ($spec.containers // []) as $cts |
        ($cts | to_entries | map(
          . as $ci |
          (
            $ci.value.env // [] |
            map(select(.valueFrom.secretKeyRef.name != null) | .valueFrom.secretKeyRef.name)
          ) + (
            $ci.value.envFrom // [] |
            map(select(.secretRef.name != null) | .secretRef.name)
          )
          | unique
          | . as $secretNames
          |
          # ensure volumes
          ($spec.volumes // []) as $vols |
          $secretNames as $sns |
          (
            $sns | reduce .[] as $s (
              $vols;
              if any(.[]; .name == ("secret-" + $s)) then .
              else . + [{ name: ("secret-" + $s), secret: { secretName: $s } }]
              end
            )
          ) as $newVols
          |
          # ensure volumeMounts per container
          ($ci.value.volumeMounts // []) as $vms |
          $sns as $sns2 |
          (
            $sns2 | reduce .[] as $s2 (
              $vms;
              if any(.[]; .name == ("secret-" + $s2)) then .
              else . + [{
                name: ("secret-" + $s2),
                mountPath: ("/var/run/secrets/" + $s2)
              }]
              end
            )
          ) as $newVms
          |
          # write back mounts to this container entry
          $spec.containers[$ci.key].volumeMounts = $newVms
        )) |
        # write back updated volumes
        .volumes = (
          ($spec.volumes // []) +
          ([] | .) # no-op for structure
        ) |
        .volumes = ($spec | .volumes? // [])
      )
  ' "${out_file}"

  # Handle Pod objects (without template) similarly
  yq -i '
    (select(.kind == "Pod") | .spec) |=
      (
        . as $spec |
        ($spec.containers // []) as $cts |
        ($cts | to_entries | map(
          . as $ci |
          (
            $ci.value.env // [] |
            map(select(.valueFrom.secretKeyRef.name != null) | .valueFrom.secretKeyRef.name)
          ) + (
            $ci.value.envFrom // [] |
            map(select(.secretRef.name != null) | .secretRef.name)
          )
          | unique
          | . as $secretNames
          |
          ($spec.volumes // []) as $vols |
          $secretNames as $sns |
          (
            $sns | reduce .[] as $s (
              $vols;
              if any(.[]; .name == ("secret-" + $s)) then .
              else . + [{ name: ("secret-" + $s), secret: { secretName: $s } }]
              end
            )
          ) as $newVols
          |
          ($ci.value.volumeMounts // []) as $vms |
          $sns as $sns2 |
          (
            $sns2 | reduce .[] as $s2 (
              $vms;
              if any(.[]; .name == ("secret-" + $s2)) then .
              else . + [{
                name: ("secret-" + $s2),
                mountPath: ("/var/run/secrets/" + $s2)
              }]
              end
            )
          ) as $newVms
          |
          $spec.containers[$ci.key].volumeMounts = $newVms
        )) |
        .volumes = (
          ($spec.volumes // []) +
          ([] | .)
        ) |
        .volumes = ($spec | .volumes? // [])
      )
  ' "${out_file}"

  echo "    Plan written to: ${out_file}"
  echo "    Original saved as: ${out_file}.orig"
}

# Iterate over the audit results and generate plans
while read -r ns kind name; do
  [ -z "${ns}" ] && continue
  generate_plan_for_object "${ns}" "${kind}" "${name}"
done <<< "${result}"

echo "=== [3/4] Manual steps required (read carefully) ==="
cat <<'EOF'

For each generated PLAN file under the directory printed above:

1) Open the plan file and review added volume and volumeMount entries.
   - Secrets referenced by env.secretKeyRef/envFrom.secretRef are now:
       mounted under /var/run/secrets/<secret-name> in each affected container.

2) Update application behavior to read secrets from files:
   - If using command/args: change them to read from file paths.
   - Prefer updating application code/images so that:
       - They load secrets from the mounted file paths.
       - They no longer depend on the corresponding environment variables.

3) Remove secretKeyRef-based environment variables from the plan:
   - Delete specific env entries that use valueFrom.secretKeyRef.
   - Delete envFrom entries that use secretRef.
   - Be careful not to remove non-secret env vars.

4) Apply and roll out the modified manifests:
   - kubectl apply -f <PLAN_FILE>
   - Monitor rollout and logs to confirm correct behavior.

5) Once all workloads have been migrated and are stable, re-run this script
   to confirm that no env.secretKeyRef usages remain.

EOF

echo "Plans directory: ${PLAN_DIR}"

echo "=== [4/4] Verification (post-migration) ==="
echo "When you believe you have fully migrated workloads to file-based secrets,"
echo "run the following verification command on any machine with kubectl access:"

cat <<'EOF'
result=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}')
if [ -z "$result" ]; then
  echo "NO_SECRETS_AS_ENV_VARS"
else
  echo "SECRETS_AS_ENV_VARS_FOUND: $result"
fi
EOF

echo "This script has completed generating migration plans and guidance."

Additional Reading:

• https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets