Prefer Using Secrets As Files Over Secrets As Environment

More Info:

Secrets injected as environment variables are more likely to be exposed through logs, child processes or crash dumps than secrets mounted as files. Applications should read secrets from mounted secret files.

Risk Level

Medium

Address

Security

Compliance Standards

• CIS GKE

Triage and Remediation

Remediation

Remediation

Manual Steps

1. List all workloads that reference secrets via environment variables

   • Run on: any machine with kubectl access

   kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.namespace} {.metadata.name} {"\n"}{end}' | sort

2. Inspect a specific Pod/Deployment to identify which env vars use secrets

   • Run on: any machine with kubectl access
     Replace NAMESPACE and NAME as needed.

   kubectl get deployment NAME -n NAMESPACE -o yaml > /tmp/deployment-NAMESPACE-NAME.yaml
   grep -n "secretKeyRef" -n /tmp/deployment-NAMESPACE-NAME.yaml -n

3. Design the file-based secret mount (no change yet)

   • Run on: any machine with kubectl access
     In /tmp/deployment-NAMESPACE-NAME.yaml, note:
   • The secretKeyRef.name (the Secret object name)
   • Each secretKeyRef.key used as an environment variable
     Decide a mount path, e.g. /var/run/secrets/APPNAME, and expected filenames, e.g. matching the key names.

4. Modify the workload manifest to mount the secret as files and stop using env vars

   • Run on: any machine with kubectl access
     Edit /tmp/deployment-NAMESPACE-NAME.yaml:
   • In spec.template.spec.volumes, add:

     - name: app-secret-volume
       secret:
         secretName: SECRET_NAME

   • In the relevant container under spec.template.spec.containers[]:
     • Add a volumeMounts entry:

       volumeMounts:
         - name: app-secret-volume
           mountPath: /var/run/secrets/APPNAME
           readOnly: true

     • Remove or comment out the env entries that use secretKeyRef. Save the file, then apply:

   kubectl apply -f /tmp/deployment-NAMESPACE-NAME.yaml

5. Update application code/configuration to read secrets from files

   • Run on: application source/deployment pipeline (outside cluster)
     Change the application so it reads secret values from the mounted files (for example, /var/run/secrets/APPNAME/SECRET_KEY) instead of environment variables. Rebuild and redeploy the image if necessary, then ensure the Deployment uses the updated image:

   kubectl set image deployment/NAME -n NAMESPACE CONTAINER_NAME=IMAGE:TAG

6. Verify no remaining environment-variable secret references

   • Run on: any machine with kubectl access

   output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
   if [ -z "$output" ]; then echo "NO_ENV_SECRET_REFERENCES"; else echo "ENV_SECRET_REFERENCES_FOUND"; echo "$output"; fi

Using kubectl

On any machine with kubectl access:

1. Identify pods using secrets as environment variables

kubectl get pods --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}'

2. For each affected workload, get the controller manifest (Deployment, StatefulSet, etc.)

Example for a Deployment:

kubectl -n <NAMESPACE> get deploy <DEPLOYMENT_NAME> -o yaml > /tmp/deploy-with-env-secrets.yaml

3. Edit the manifest locally to:

• Remove env entries that use secretKeyRef
• Add a volume that references the Secret
• Mount that volume into the container

Minimal pattern (replace placeholders):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-namespace
spec:
  template:
    spec:
      volumes:
        - name: my-secret-vol
          secret:
            secretName: my-secret
      containers:
        - name: my-app
          image: my-image:tag
          volumeMounts:
            - name: my-secret-vol
              mountPath: /var/run/secrets/my-secret
              readOnly: true
          # Remove this style:
          # env:
          #   - name: MY_PASSWORD
          #     valueFrom:
          #       secretKeyRef:
          #         name: my-secret
          #         key: password
          #
          # Update app code/config to read from:
          #   /var/run/secrets/my-secret/password

4. Apply the updated manifest

kubectl apply -f /tmp/deploy-with-env-secrets.yaml

Repeat for all controllers (Deployment/StatefulSet/DaemonSet/CronJob/Job) using env-based secrets.

5. Verification

output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
if [ -z "$output" ]; then echo "NO_ENV_SECRET_REFERENCES"; else echo "ENV_SECRET_REFERENCES_FOUND"; fi

Automation

#!/usr/bin/env bash
#
# Automation: Prefer using secrets as files over secrets as environment variables
#
# Scope: any machine with kubectl access
#
# WARNING:
# - This script CANNOT automatically rewrite application code to read secrets from files.
# - It performs discovery, creates candidate patched manifests that mount secrets as files,
#   and shows diffs. A human must review, adjust application code, and then apply.
#
# Requirements:
# - kubectl configured with sufficient RBAC to get/list Pods/Deployments/ReplicaSets/StatefulSets/DaemonSets/Jobs/CronJobs
# - jq, yq (v4) installed for JSON/YAML processing
#
# Idempotence:
# - Re-running will re-generate the same work directory content and re-run the verification.
#

set -euo pipefail

WORKDIR="${PWD}/env-secret-remediation"
mkdir -p "${WORKDIR}/original" "${WORKDIR}/patched" "${WORKDIR}/diffs"

echo "[1/5] Discovering objects that use env vars sourced from Secrets..."

# Capture all objects that have secretKeyRef in their spec
# (Pods and higher-level controllers)
kubectl get all --all-namespaces -o json > "${WORKDIR}/all.json"

# Extract namespaced objects with secretKeyRef
# We will focus on Pod, Deployment, StatefulSet, DaemonSet, Job, CronJob, ReplicaSet
jq '
  .items[]
  | select(.. | has("secretKeyRef")?)
  | select(.kind == "Pod"
        or .kind == "Deployment"
        or .kind == "StatefulSet"
        or .kind == "DaemonSet"
        or .kind == "Job"
        or .kind == "CronJob"
        or .kind == "ReplicaSet")
  | {kind, apiVersion, ns: .metadata.namespace, name: .metadata.name}
' "${WORKDIR}/all.json" > "${WORKDIR}/affected.json"

if [ ! -s "${WORKDIR}/affected.json" ]; then
  echo "NO_ENV_SECRET_REFERENCES"
  exit 0
fi

echo "ENV_SECRET_REFERENCES_FOUND"
echo "Details will be written under: ${WORKDIR}"
echo

echo "[2/5] Exporting original manifests for editing..."

# Helper to export one object
export_object() {
  local ns="$1" kind="$2" name="$3"
  local file="${WORKDIR}/original/${ns}___${kind,,}___${name}.yaml"
  kubectl get "${kind,,}.${kind,,}.k8s.io" -n "${ns}" "${name}" -o yaml > "${file}" 2>/dev/null || \
  kubectl get "${kind}" -n "${ns}" "${name}" -o yaml > "${file}"
}

# Export each affected object once
jq -r '.ns+" "+.kind+" "+.name' "${WORKDIR}/affected.json" | sort -u | while read -r ns kind name; do
  echo "  - Exporting ${kind}/${name} in namespace ${ns}"
  export_object "${ns}" "${kind}" "${name}" || {
    echo "    WARNING: Failed to export ${kind}/${name} in ${ns}, skipping."
    continue
  }
done

echo
echo "[3/5] Creating PATCH CANDIDATES that mount secrets as files (NOT applied automatically)..."

cat > "${WORKDIR}/README-HUMAN-ACTION.txt" <<'EOF'
These "patched" manifests are CANDIDATES ONLY.

What has been done:
- For each container env entry like:
    env:
    - name: MY_SECRET
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: password
  a corresponding Secret volume and mount have been proposed:
    spec:
      volumes:
      - name: my-secret
        secret:
          secretName: my-secret

    containers:
    - name: ...
      volumeMounts:
      - name: my-secret
        mountPath: /var/run/secrets/my-secret
        readOnly: true

- The env entries referencing that secret are LEFT IN PLACE.
  A human must:
  - Update application code to read from the mounted file instead of the env var.
  - Then remove the env var(s) once safe.

How to use:
1. Review the diffs in ./diffs to understand proposed volume/volumeMount changes.
2. Adjust the patched YAMLs under ./patched as needed:
   - Consolidate multiple keys/paths.
   - Ensure no mountPath conflicts.
3. Update your application code to read secrets from files.
4. When ready, apply the patched manifests, for example:
   kubectl apply -f ./patched/<file>.yaml
5. Rerun this script to verify that no remaining env->secretKeyRef usages are present.

NOTE:
- This transformation is heuristic; it does NOT understand your app semantics.
- You may decide some env-based secrets are acceptable risk; in that case, document the decision.
EOF

# Function to process one YAML file with yq:
# - For each container/env with .valueFrom.secretKeyRef, ensure:
#   * spec.volumes has a secret volume named after the Secret (deduplicated)
#   * container.volumeMounts has a mount pointing to /var/run/secrets/<secretName>
process_yaml() {
  local in_file="$1"
  local out_file="$2"

  yq eval '
    # Build a map of secret names used in env -> to ensure volumes exist
    def secretNames:
      (.. | objects | select(has("env")) | .env[]? | select(has("valueFrom"))
       | select(.valueFrom.secretKeyRef.name != null)
       | .valueFrom.secretKeyRef.name) as $name
      | {$name};

    # Collect all unique secret names referenced by env vars
    . as $root
    | ($root
       | .. | objects
       | select(has("env"))
       | .env[]?
       | select(has("valueFrom"))
       | select(.valueFrom.secretKeyRef.name != null)
       | .valueFrom.secretKeyRef.name) as $sec
    | $root
    | .metadata.annotations."env-secret-remediation/last-processed" = "'"$(date -u +%FT%TZ)"'"
    # Ensure volumes list exists
    | ( .spec.volumes // [] ) as $vols
    | .spec.volumes =
        ( $vols +
          ( [ ($sec | select(. != null)) ] | unique | map(
              if ($vols[]? | select(.name == .) ) then empty
              else
                {
                  name: .,
                  secret: { secretName: . }
                }
              end
          ))
        )
    # For each container/ephemeralContainer/initContainer, ensure volumeMounts for each secret
    | (
        (.spec.containers // []) as $containers
        | .spec.containers = (
            $containers
            | map(
                . as $c
                | ($c.env // []) as $envs
                | ($envs
                   | map(select(has("valueFrom"))
                         | select(.valueFrom.secretKeyRef.name != null)
                         | .valueFrom.secretKeyRef.name)
                  ) as $used
                | ($c.volumeMounts // []) as $vm
                | .volumeMounts =
                    ($vm +
                     ( [ $used[]? ] | unique | map(
                         if ($vm[]? | select(.name == .)) then empty
                         else
                           {
                             name: .,
                             mountPath: ("/var/run/secrets/" + .),
                             readOnly: true
                           }
                         end
                       ))
                    )
            )
        )
      )
    # Do the same for initContainers
    | (
        (.spec.initContainers // []) as $containers
        | .spec.initContainers = (
            $containers
            | map(
                . as $c
                | ($c.env // []) as $envs
                | ($envs
                   | map(select(has("valueFrom"))
                         | select(.valueFrom.secretKeyRef.name != null)
                         | .valueFrom.secretKeyRef.name)
                  ) as $used
                | ($c.volumeMounts // []) as $vm
                | .volumeMounts =
                    ($vm +
                     ( [ $used[]? ] | unique | map(
                         if ($vm[]? | select(.name == .)) then empty
                         else
                           {
                             name: .,
                             mountPath: ("/var/run/secrets/" + .),
                             readOnly: true
                           }
                         end
                       ))
                    )
            )
        )
      )
    # And for ephemeralContainers if present
    | (
        (.spec.ephemeralContainers // []) as $containers
        | .spec.ephemeralContainers = (
            $containers
            | map(
                . as $c
                | ($c.env // []) as $envs
                | ($envs
                   | map(select(has("valueFrom"))
                         | select(.valueFrom.secretKeyRef.name != null)
                         | .valueFrom.secretKeyRef.name)
                  ) as $used
                | ($c.volumeMounts // []) as $vm
                | .volumeMounts =
                    ($vm +
                     ( [ $used[]? ] | unique | map(
                         if ($vm[]? | select(.name == .)) then empty
                         else
                           {
                             name: .,
                             mountPath: ("/var/run/secrets/" + .),
                             readOnly: true
                           }
                         end
                       ))
                    )
            )
        )
      )
  ' "${in_file}" > "${out_file}"
}

for f in "${WORKDIR}/original"/*.yaml; do
  [ -e "$f" ] || continue
  base="$(basename "$f")"
  out="${WORKDIR}/patched/${base}"
  echo "  - Generating candidate patch for ${base}"
  process_yaml "${f}" "${out}" || {
    echo "    WARNING: Failed to process ${f}, skipping."
    continue
  }
  # Generate a diff for human review
  diff_file="${WORKDIR}/diffs/${base}.diff"
  diff -u "${f}" "${out}" > "${diff_file}" || true
done

echo
echo "[4/5] NOTE: No changes have been applied to the cluster."

cat <<EOF

Next steps (manual, REQUIRED):
1) Review candidate patches under:
   - Original: ${WORKDIR}/original
   - Patched:  ${WORKDIR}/patched
   - Diffs:    ${WORKDIR}/diffs

2) For each object:
   - Confirm the proposed Secret volumes and volumeMounts are correct.
   - Update application code to read secrets from the mounted files at:
       /var/run/secrets/<secretName>/
     (or adjust mountPath as needed).
   - Once code is updated and deployed, remove the corresponding env variables
     that use valueFrom.secretKeyRef from your manifests.
   - Apply your finalized manifests with:
       kubectl apply -f <your-reviewed-file>.yaml

3) After you have deployed your changes, re-run this script to verify.

EOF

echo "[5/5] Verification: checking for remaining env-based Secret references..."

output="$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')"

if [ -z "${output}" ]; then
  echo "Verification result: NO_ENV_SECRET_REFERENCES"
else
  echo "Verification result: ENV_SECRET_REFERENCES_FOUND"
  echo "Remaining objects with env->Secret references:"
  echo "${output}"
fi