Prefer Using Secrets As Files Over Secrets As Environment Variables

Manual Steps

List all workloads that reference secrets via environment variables

Run on: any machine with kubectl access

kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.namespace} {.metadata.name} {"\n"}{end}' | sort

Inspect a specific Pod/Deployment to identify which env vars use secrets

Run on: any machine with kubectl access
Replace NAMESPACE and NAME as needed.

kubectl get deployment NAME -n NAMESPACE -o yaml > /tmp/deployment-NAMESPACE-NAME.yaml
grep -n "secretKeyRef" -n /tmp/deployment-NAMESPACE-NAME.yaml -n

Design the file-based secret mount (no change yet)
- Run on: any machine with kubectl access
  In /tmp/deployment-NAMESPACE-NAME.yaml, note:
- The secretKeyRef.name (the Secret object name)
- Each secretKeyRef.key used as an environment variable
  Decide a mount path, e.g. /var/run/secrets/APPNAME, and expected filenames, e.g. matching the key names.
Modify the workload manifest to mount the secret as files and stop using env vars
- Run on: any machine with kubectl access
  Edit /tmp/deployment-NAMESPACE-NAME.yaml:
- In spec.template.spec.volumes, add:
  - name: app-secret-volume secret: secretName: SECRET_NAME
- In the relevant container under spec.template.spec.containers[]:
  - Add a volumeMounts entry:
    volumeMounts: - name: app-secret-volume mountPath: /var/run/secrets/APPNAME readOnly: true
  - Remove or comment out the env entries that use secretKeyRef. Save the file, then apply:
kubectl apply -f /tmp/deployment-NAMESPACE-NAME.yaml
Update application code/configuration to read secrets from files
- Run on: application source/deployment pipeline (outside cluster)
  Change the application so it reads secret values from the mounted files (for example, /var/run/secrets/APPNAME/SECRET_KEY) instead of environment variables. Rebuild and redeploy the image if necessary, then ensure the Deployment uses the updated image:
kubectl set image deployment/NAME -n NAMESPACE CONTAINER_NAME=IMAGE:TAG

Verify no remaining environment-variable secret references

Run on: any machine with kubectl access

output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
if [ -z "$output" ]; then echo "NO_ENV_SECRET_REFERENCES"; else echo "ENV_SECRET_REFERENCES_FOUND"; echo "$output"; fi

Using kubectl

On any machine with kubectl access:

Identify pods using secrets as environment variables

kubectl get pods --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}'

For each affected workload, get the controller manifest (Deployment, StatefulSet, etc.)

Example for a Deployment:

kubectl -n <NAMESPACE> get deploy <DEPLOYMENT_NAME> -o yaml > /tmp/deploy-with-env-secrets.yaml

Edit the manifest locally to:

Remove env entries that use secretKeyRef
Add a volume that references the Secret
Mount that volume into the container

Minimal pattern (replace placeholders):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-namespace
spec:
  template:
    spec:
      volumes:
        - name: my-secret-vol
          secret:
            secretName: my-secret
      containers:
        - name: my-app
          image: my-image:tag
          volumeMounts:
            - name: my-secret-vol
              mountPath: /var/run/secrets/my-secret
              readOnly: true
          # Remove this style:
          # env:
          #   - name: MY_PASSWORD
          #     valueFrom:
          #       secretKeyRef:
          #         name: my-secret
          #         key: password
          #
          # Update app code/config to read from:
          #   /var/run/secrets/my-secret/password

Apply the updated manifest

kubectl apply -f /tmp/deploy-with-env-secrets.yaml

Repeat for all controllers (Deployment/StatefulSet/DaemonSet/CronJob/Job) using env-based secrets.

Verification

output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
if [ -z "$output" ]; then echo "NO_ENV_SECRET_REFERENCES"; else echo "ENV_SECRET_REFERENCES_FOUND"; fi

Automation

#!/usr/bin/env bash
#
# Automation: Prefer using secrets as files over secrets as environment variables
#
# Scope: any machine with kubectl access
#
# WARNING:
# - This script CANNOT automatically rewrite application code to read secrets from files.
# - It performs discovery, creates candidate patched manifests that mount secrets as files,
#   and shows diffs. A human must review, adjust application code, and then apply.
#
# Requirements:
# - kubectl configured with sufficient RBAC to get/list Pods/Deployments/ReplicaSets/StatefulSets/DaemonSets/Jobs/CronJobs
# - jq, yq (v4) installed for JSON/YAML processing
#
# Idempotence:
# - Re-running will re-generate the same work directory content and re-run the verification.
#

set -euo pipefail

WORKDIR="${PWD}/env-secret-remediation"
mkdir -p "${WORKDIR}/original" "${WORKDIR}/patched" "${WORKDIR}/diffs"

echo "[1/5] Discovering objects that use env vars sourced from Secrets..."

# Capture all objects that have secretKeyRef in their spec
# (Pods and higher-level controllers)
kubectl get all --all-namespaces -o json > "${WORKDIR}/all.json"

# Extract namespaced objects with secretKeyRef
# We will focus on Pod, Deployment, StatefulSet, DaemonSet, Job, CronJob, ReplicaSet
jq '
  .items[]
  | select(.. | has("secretKeyRef")?)
  | select(.kind == "Pod"
        or .kind == "Deployment"
        or .kind == "StatefulSet"
        or .kind == "DaemonSet"
        or .kind == "Job"
        or .kind == "CronJob"
        or .kind == "ReplicaSet")
  | {kind, apiVersion, ns: .metadata.namespace, name: .metadata.name}
' "${WORKDIR}/all.json" > "${WORKDIR}/affected.json"

if [ ! -s "${WORKDIR}/affected.json" ]; then
  echo "NO_ENV_SECRET_REFERENCES"
  exit 0
fi

echo "ENV_SECRET_REFERENCES_FOUND"
echo "Details will be written under: ${WORKDIR}"
echo

echo "[2/5] Exporting original manifests for editing..."

# Helper to export one object
export_object() {
  local ns="$1" kind="$2" name="$3"
  local file="${WORKDIR}/original/${ns}___${kind,,}___${name}.yaml"
  kubectl get "${kind,,}.${kind,,}.k8s.io" -n "${ns}" "${name}" -o yaml > "${file}" 2>/dev/null || \
  kubectl get "${kind}" -n "${ns}" "${name}" -o yaml > "${file}"
}

# Export each affected object once
jq -r '.ns+" "+.kind+" "+.name' "${WORKDIR}/affected.json" | sort -u | while read -r ns kind name; do
  echo "  - Exporting ${kind}/${name} in namespace ${ns}"
  export_object "${ns}" "${kind}" "${name}" || {
    echo "    WARNING: Failed to export ${kind}/${name} in ${ns}, skipping."
    continue
  }
done

echo
echo "[3/5] Creating PATCH CANDIDATES that mount secrets as files (NOT applied automatically)..."

cat > "${WORKDIR}/README-HUMAN-ACTION.txt" <<'EOF'
These "patched" manifests are CANDIDATES ONLY.

What has been done:
- For each container env entry like:
    env:
    - name: MY_SECRET
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: password
  a corresponding Secret volume and mount have been proposed:
    spec:
      volumes:
      - name: my-secret
        secret:
          secretName: my-secret

    containers:
    - name: ...
      volumeMounts:
      - name: my-secret
        mountPath: /var/run/secrets/my-secret
        readOnly: true

- The env entries referencing that secret are LEFT IN PLACE.
  A human must:
  - Update application code to read from the mounted file instead of the env var.
  - Then remove the env var(s) once safe.

How to use:
1. Review the diffs in ./diffs to understand proposed volume/volumeMount changes.
2. Adjust the patched YAMLs under ./patched as needed:
   - Consolidate multiple keys/paths.
   - Ensure no mountPath conflicts.
3. Update your application code to read secrets from files.
4. When ready, apply the patched manifests, for example:
   kubectl apply -f ./patched/<file>.yaml
5. Rerun this script to verify that no remaining env->secretKeyRef usages are present.

NOTE:
- This transformation is heuristic; it does NOT understand your app semantics.
- You may decide some env-based secrets are acceptable risk; in that case, document the decision.
EOF

# Function to process one YAML file with yq:
# - For each container/env with .valueFrom.secretKeyRef, ensure:
#   * spec.volumes has a secret volume named after the Secret (deduplicated)
#   * container.volumeMounts has a mount pointing to /var/run/secrets/<secretName>
process_yaml() {
  local in_file="$1"
  local out_file="$2"

  yq eval '
    # Build a map of secret names used in env -> to ensure volumes exist
    def secretNames:
      (.. | objects | select(has("env")) | .env[]? | select(has("valueFrom"))
       | select(.valueFrom.secretKeyRef.name != null)
       | .valueFrom.secretKeyRef.name) as $name
      | {$name};

    # Collect all unique secret names referenced by env vars
    . as $root
    | ($root
       | .. | objects
       | select(has("env"))
       | .env[]?
       | select(has("valueFrom"))
       | select(.valueFrom.secretKeyRef.name != null)
       | .valueFrom.secretKeyRef.name) as $sec
    | $root
    | .metadata.annotations."env-secret-remediation/last-processed" = "'"$(date -u +%FT%TZ)"'"
    # Ensure volumes list exists
    | ( .spec.volumes // [] ) as $vols
    | .spec.volumes =
        ( $vols +
          ( [ ($sec | select(. != null)) ] | unique | map(
              if ($vols[]? | select(.name == .) ) then empty
              else
                {
                  name: .,
                  secret: { secretName: . }
                }
              end
          ))
        )
    # For each container/ephemeralContainer/initContainer, ensure volumeMounts for each secret
    | (
        (.spec.containers // []) as $containers
        | .spec.containers = (
            $containers
            | map(
                . as $c
                | ($c.env // []) as $envs
                | ($envs
                   | map(select(has("valueFrom"))
                         | select(.valueFrom.secretKeyRef.name != null)
                         | .valueFrom.secretKeyRef.name)
                  ) as $used
                | ($c.volumeMounts // []) as $vm
                | .volumeMounts =
                    ($vm +
                     ( [ $used[]? ] | unique | map(
                         if ($vm[]? | select(.name == .)) then empty
                         else
                           {
                             name: .,
                             mountPath: ("/var/run/secrets/" + .),
                             readOnly: true
                           }
                         end
                       ))
                    )
            )
        )
      )
    # Do the same for initContainers
    | (
        (.spec.initContainers // []) as $containers
        | .spec.initContainers = (
            $containers
            | map(
                . as $c
                | ($c.env // []) as $envs
                | ($envs
                   | map(select(has("valueFrom"))
                         | select(.valueFrom.secretKeyRef.name != null)
                         | .valueFrom.secretKeyRef.name)
                  ) as $used
                | ($c.volumeMounts // []) as $vm
                | .volumeMounts =
                    ($vm +
                     ( [ $used[]? ] | unique | map(
                         if ($vm[]? | select(.name == .)) then empty
                         else
                           {
                             name: .,
                             mountPath: ("/var/run/secrets/" + .),
                             readOnly: true
                           }
                         end
                       ))
                    )
            )
        )
      )
    # And for ephemeralContainers if present
    | (
        (.spec.ephemeralContainers // []) as $containers
        | .spec.ephemeralContainers = (
            $containers
            | map(
                . as $c
                | ($c.env // []) as $envs
                | ($envs
                   | map(select(has("valueFrom"))
                         | select(.valueFrom.secretKeyRef.name != null)
                         | .valueFrom.secretKeyRef.name)
                  ) as $used
                | ($c.volumeMounts // []) as $vm
                | .volumeMounts =
                    ($vm +
                     ( [ $used[]? ] | unique | map(
                         if ($vm[]? | select(.name == .)) then empty
                         else
                           {
                             name: .,
                             mountPath: ("/var/run/secrets/" + .),
                             readOnly: true
                           }
                         end
                       ))
                    )
            )
        )
      )
  ' "${in_file}" > "${out_file}"
}

for f in "${WORKDIR}/original"/*.yaml; do
  [ -e "$f" ] || continue
  base="$(basename "$f")"
  out="${WORKDIR}/patched/${base}"
  echo "  - Generating candidate patch for ${base}"
  process_yaml "${f}" "${out}" || {
    echo "    WARNING: Failed to process ${f}, skipping."
    continue
  }
  # Generate a diff for human review
  diff_file="${WORKDIR}/diffs/${base}.diff"
  diff -u "${f}" "${out}" > "${diff_file}" || true
done

echo
echo "[4/5] NOTE: No changes have been applied to the cluster."

cat <<EOF

Next steps (manual, REQUIRED):
1) Review candidate patches under:
   - Original: ${WORKDIR}/original
   - Patched:  ${WORKDIR}/patched
   - Diffs:    ${WORKDIR}/diffs

2) For each object:
   - Confirm the proposed Secret volumes and volumeMounts are correct.
   - Update application code to read secrets from the mounted files at:
       /var/run/secrets/<secretName>/
     (or adjust mountPath as needed).
   - Once code is updated and deployed, remove the corresponding env variables
     that use valueFrom.secretKeyRef from your manifests.
   - Apply your finalized manifests with:
       kubectl apply -f <your-reviewed-file>.yaml

3) After you have deployed your changes, re-run this script to verify.

EOF

echo "[5/5] Verification: checking for remaining env-based Secret references..."

output="$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')"

if [ -z "${output}" ]; then
  echo "Verification result: NO_ENV_SECRET_REFERENCES"
else
  echo "Verification result: ENV_SECRET_REFERENCES_FOUND"
  echo "Remaining objects with env->Secret references:"
  echo "${output}"
fi

GCP Introduction

GCP Pricing

GCP Threats

GCP Misconfigurations

Resources

GCP Threats

Prefer Using Secrets As Files Over Secrets As Environment Variables

More Info:

Risk Level

Address

Compliance Standards

Triage and Remediation

Remediation

​More Info:

​Risk Level

​Address

​Compliance Standards

​Triage and Remediation

​Remediation