Skip to main content

Prefer Using Secrets Files Over Secrets As Environment

More Info:​

Kubernetes supports mounting secrets as data volumes or as environment variables. Minimize the use of environment variable secrets.

Risk Level​

Low

Address​

Security

Compliance Standards​

  • APRA CPS 234 (Australia)
  • BSI C5 (Germany)
  • Brazil LGPD
  • CCPA / CPRA (California)
  • CIS AKS
  • CIS Critical Security Controls v8
  • CMMC 2.0
  • CSA Cloud Controls Matrix v4
  • DPDPA
  • Digital Operational Resilience Act (EU)
  • Essential 8
  • ISO/IEC 27017
  • ISO/IEC 27018
  • ISO/IEC 27701
  • KSA PDPL
  • MAS Technology Risk Management (Singapore)
  • MITRE ATT&CK (Cloud)
  • NIST SP 800-171
  • NYDFS 23 NYCRR 500
  • SWIFT Customer Security Controls Framework
  • Sarbanes-Oxley IT General Controls
  • UK NCSC Cyber Assessment Framework

Triage and Remediation​

Remediation​

Manual Steps

  1. Identify workloads using secrets as environment variables

    • On any machine with kubectl access: 
      kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}' \
        | sed '/^$/d' | sort -u
    • For each listed resource, get its full manifest: 
      kubectl get -n <NAMESPACE> <KIND>/<NAME> -o yaml > /tmp/<NAMESPACE>_<KIND>_<NAME>.yaml

  2. Edit pod specs to replace env/envFrom secret refs with projected secret volumes

    • On any machine with kubectl access, edit each saved manifest file:
      • Remove env entries that use valueFrom.secretKeyRef and any envFrom entries with secretRef.
      • Add a volumes entry to reference the same Secret: 
        spec:
          volumes:
            - name: app-secret
              secret:
                secretName: <SECRET_NAME>
      • Under each container that previously used the secret as env vars, mount the volume: 
        spec:
          containers:
            - name: <CONTAINER_NAME>
              volumeMounts:
                - name: app-secret
                  mountPath: /etc/secrets
                  readOnly: true
    • Save each modified file.

  3. Update application code/configuration to read secrets from files

    • Coordinate with application owners to change configuration so the app reads from mounted files (for example /etc/secrets/<key>) instead of environment variables.
    • Do not apply manifest changes until the application has been updated to use file-based secrets to avoid runtime failures.

  4. Apply the updated manifests

    • On any machine with kubectl access, for each edited file: 
      kubectl apply -f /tmp/<NAMESPACE>_<KIND>_<NAME>.yaml
    • If needed, restart Deployments/StatefulSets/DaemonSets to pick up changes: 
      kubectl rollout restart -n <NAMESPACE> deployment/<NAME>

  5. Clean up legacy secret environment variable usage (if any remain)

    • Re-check individual resources that you expect to be fixed: 
      kubectl get -n <NAMESPACE> <KIND>/<NAME> -o jsonpath='{..env[?(@.valueFrom.secretKeyRef)]}' ; echo
      kubectl get -n <NAMESPACE> <KIND>/<NAME> -o jsonpath='{..envFrom[?(@.secretRef)]}' ; echo
    • If output is not empty, repeat step 2 for that resource until both commands return empty output.

  6. Verification

    • On any machine with kubectl access: 
      output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
      if [ -z "$output" ]; then echo "NO_ENV_SECRET_REFERENCES"; else echo "ENV_SECRET_REFERENCES_FOUND"; fi
Using kubectl

On any machine with kubectl access:

  1. Identify workloads using secrets as environment variables
kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.namespace} {.metadata.name} {"\n"}{end}'

Optionally inspect a specific deployment (example):

kubectl -n default get deploy my-app -o yaml > /tmp/my-app-deploy.yaml
  1. Refactor a workload to mount secrets as files (example pattern)

Edit the manifest locally:

nano /tmp/my-app-deploy.yaml

In each container spec:

  • Remove any env entries using secretKeyRef, e.g.:
env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-secret
        key: password
  • Remove any envFrom that reference secrets, e.g.:
envFrom:
  - secretRef:
      name: app-secrets
  • Add a volumeMounts entry:
volumeMounts:
  - name: db-secret-volume
    mountPath: /var/run/secrets/db
    readOnly: true
  • Add a volumes entry at the pod spec level:
volumes:
  - name: db-secret-volume
    secret:
      secretName: db-secret

If you previously used envFrom with multiple keys, mount that secret similarly and update the application to read from the files in the mount path (one file per key).

  1. Apply the updated manifest
kubectl apply -f /tmp/my-app-deploy.yaml

Repeat this edit/apply process for each workload listed by the identification command until no env/envFrom+secretKeyRef references remain.

  1. Verification
output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
if [ -z "$output" ]; then echo "NO_ENV_SECRET_REFERENCES"; else echo "ENV_SECRET_REFERENCES_FOUND"; echo "$output"; fi
Automation
#!/usr/bin/env bash
#
# Refactor workloads that use Secrets as environment variables
# to instead mount those Secrets as files.
#
# LIMITATION:
# This script cannot safely rewrite application code to read from files
# instead of env vars. It only:
#   - Detects affected resources
#   - Produces modified manifests with volume mounts for secrets
#   - Applies those manifests (opt‑in per resource)
#
# You MUST ensure each application can read its secrets from the mounted
# files before approving and applying the generated patches.
#
# Run on: any machine with kubectl access and current context set.
#
# Requirements: bash, kubectl, jq, yq (https://github.com/mikefarah/yq)

set -euo pipefail

TMP_DIR="/tmp/k8s-secret-env-to-volume"
mkdir -p "${TMP_DIR}"

echo "Detecting resources using secrets as environment variables..."

# Get all namespaced resources that have .spec.template (Deployments, DaemonSets, StatefulSets, Jobs, CronJobs, etc.)
# and filter for those with secretKeyRef under containers/env or envFrom.
kubectl get deploy,ds,sts,job,cronjob --all-namespaces -o json \
  | jq -c '
    .items[]
    | select(..|has("secretKeyRef")?)
    | {kind, namespace: .metadata.namespace, name: .metadata.name}
  ' > "${TMP_DIR}/affected-resources.json"

if [[ ! -s "${TMP_DIR}/affected-resources.json" ]]; then
  echo "No resources with secretKeyRef in env/envFrom detected."
  exit 0
fi

echo "The following resources reference Secrets via environment variables:"
cat "${TMP_DIR}/affected-resources.json" | jq -r '.kind + " " + .namespace + "/" + .name'

echo
echo "For each resource, this script will:"
echo "  - Fetch the full manifest"
echo "  - Add Secret volumes and mounts for each referenced Secret"
echo "  - Leave env/envFrom entries in place (no removal, no behavior change)"
echo
echo "You can then manually edit workloads to read secrets from files and later"
echo "remove the env/envFrom secret references yourself."
echo

read -r -p "Continue and generate/apply volume-mount patches? [y/N]: " CONT
if [[ "${CONT}" != "y" && "${CONT}" != "Y" ]]; then
  echo "Aborted by user."
  exit 1
fi

# Helper: ensure yq exists
if ! command -v yq >/dev/null 2>&1; then
  echo "ERROR: yq is required but not found in PATH."
  echo "Install yq from https://github.com/mikefarah/yq and re-run."
  exit 1
fi

# Process each affected resource
while IFS= read -r line; do
  kind=$(echo "${line}" | jq -r '.kind')
  ns=$(echo "${line}"   | jq -r '.namespace')
  name=$(echo "${line}" | jq -r '.name')

  echo
  echo "Processing ${kind} ${ns}/${name} ..."

  manifest_file="${TMP_DIR}/${kind,,}-${ns}-${name}.yaml"

  # Export the single resource as YAML
  kubectl get "${kind}" "${name}" -n "${ns}" -o yaml > "${manifest_file}"

  # Extract all secret names used via env or envFrom.secretRef
  secret_list=$(yq '
    .. | select(has("env") or has("envFrom")) | (
      (.env[]? | select(.valueFrom.secretKeyRef).valueFrom.secretKeyRef.name),
      (.envFrom[]? | select(has("secretRef")).secretRef.name)
    )' "${manifest_file}" 2>/dev/null | sort -u || true)

  if [[ -z "${secret_list}" ]]; then
    echo "  No secretKeyRef actually found in detailed scan; skipping."
    continue
  fi

  echo "  Secrets referenced as env/envFrom:"
  echo "${secret_list}" | sed 's/^/    - /'

  # Build yq expression to add volumes and mounts
  # For each secret name S:
  #   - Add/ensure a volume named "secret-S" with secret.name=S
  #   - For each container and initContainer, add/ensure a volumeMount
  #     with name "secret-S" and mountPath "/var/run/secrets/S"
  yq_expr=""
  while IFS= read -r sname; do
    [[ -z "${sname}" ]] && continue
    vol_name="secret-${sname}"

    # Volume addition
    yq_expr+="
      .spec.template.spec.volumes |=
        ( . // [] ) |
        ( any(.[]; .name == \"${vol_name}\") | not ) as \$missing
        | if \$missing then
            . += [{name: \"${vol_name}\", secret: {secretName: \"${sname}\"}}]
          else
            .
          end
      |
    "

    # Container mounts
    yq_expr+="
      (.spec.template.spec.containers // []) |=
        map(
          (.volumeMounts |=
            ( . // [] ) |
            ( any(.[]; .name == \"${vol_name}\") | not ) as \$cmissing
            | if \$cmissing then
                . += [{name: \"${vol_name}\", mountPath: \"/var/run/secrets/${sname}\"}]
              else
                .
              end
          )
        )
      |
    "

    # initContainer mounts
    yq_expr+="
      (.spec.template.spec.initContainers // []) |=
        map(
          (.volumeMounts |=
            ( . // [] ) |
            ( any(.[]; .name == \"${vol_name}\") | not ) as \$icmissing
            | if \$icmissing then
                . += [{name: \"${vol_name}\", mountPath: \"/var/run/secrets/${sname}\"}]
              else
                .
              end
          )
        )
      |
    "
  done <<< "${secret_list}"

  # Remove trailing pipe at the end of yq_expr
  yq_expr="${yq_expr%|}"

  patched_manifest="${TMP_DIR}/${kind,,}-${ns}-${name}-patched.yaml"

  # Apply the transformation
  yq "${yq_expr}" "${manifest_file}" > "${patched_manifest}"

  echo "  Generated patched manifest: ${patched_manifest}"
  echo "  Preview diff (patched vs live):"
  echo "------------------------------------------------------------"
  diff -u "${manifest_file}" "${patched_manifest}" || true
  echo "------------------------------------------------------------"

  read -r -p "Apply patched manifest to ${kind} ${ns}/${name}? [y/N]: " APPLY
  if [[ "${APPLY}" != "y" && "${APPLY}" != "Y" ]]; then
    echo "  Skipping apply for ${kind} ${ns}/${name}."
    continue
  fi

  kubectl apply -n "${ns}" -f "${patched_manifest}"
  echo "  Applied patched manifest for ${kind} ${ns}/${name}."

done < "${TMP_DIR}/affected-resources.json"

echo
echo "Verification: re-running environment-secret audit..."

output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
if [ -z "$output" ]; then
  echo "NO_ENV_SECRET_REFERENCES"
else
  echo "ENV_SECRET_REFERENCES_FOUND"
  echo "Remaining resources still using secrets as env vars:"
  echo "${output}"
  echo
  echo "To fully comply, you must:"
  echo "  - Update applications to consume secrets from mounted files"
  echo "  - Remove env/envFrom secretKeyRef usages from their manifests"
fi

Additional Reading:​