Prefer Secrets Files Over Secrets As Environment Variables

More Info:

Kubernetes supports mounting secrets as data volumes or as environment variables. Minimize the use of environment variable secrets.

Risk Level

High

Address

Security

Compliance Standards

• APRA CPS 234 (Australia)
• BSI C5 (Germany)
• Brazil LGPD
• CCPA / CPRA (California)
• CIS Critical Security Controls v8
• CIS GKE
• CMMC 2.0
• CSA Cloud Controls Matrix v4
• DPDPA
• Digital Operational Resilience Act (EU)
• Essential 8
• ISO/IEC 27017
• ISO/IEC 27018
• ISO/IEC 27701
• KSA PDPL
• MAS Technology Risk Management (Singapore)
• MITRE ATT&CK (Cloud)
• NIST SP 800-171
• NYDFS 23 NYCRR 500
• SWIFT Customer Security Controls Framework
• Sarbanes-Oxley IT General Controls
• UK NCSC Cyber Assessment Framework

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Identify workloads using secrets as environment variables

   • Run on any machine with kubectl access:

     kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.namespace} {.metadata.name} {"\n"}{end}'

   • Save the output and decide which Deployments/DaemonSets/StatefulSets/Pods/Jobs/CronJobs you will change.

2. Inspect how the secret is used in each workload

   • For each item from step 1, describe it and review the env section:

     kubectl -n NAMESPACE get KIND NAME -o yaml

   • Confirm the application reads the secret from environment variables and determine if it can instead read from a file path.

3. Update application code/configuration to read secrets from files

   • Modify application code or configuration outside the cluster so it reads secrets from a file path (for example /var/run/secrets/<app>/<key>) instead of an environment variable.
   • Build and push the updated image or adjust configuration so that the application expects the secret at the chosen file path.

4. Modify the workload manifest to mount the secret as a volume

   • Edit the manifest on any machine with kubectl access (either via file + apply, or kubectl edit):
     • Remove env entries that use secretKeyRef.
     • Add a volumes entry at the pod spec level and a volumeMounts entry for the container. Example pattern:

     spec:
       volumes:
         - name: my-secret-volume
           secret:
             secretName: my-secret
       containers:
         - name: app
           image: your-registry/your-image:tag
           volumeMounts:
             - name: my-secret-volume
               mountPath: /var/run/secrets/my-app
               readOnly: true

   • Apply updated manifests if editing locally:

     kubectl apply -f UPDATED_MANIFEST.yaml

5. Validate application behavior and remove remaining env-secret usage

   • Confirm the updated pods are running and healthy:

     kubectl -n NAMESPACE get pods
     kubectl -n NAMESPACE logs POD_NAME

   • If the application works with file-based secrets, ensure no remaining env entries with secretKeyRef exist in that workload:

     kubectl -n NAMESPACE get KIND NAME -o json | jq '..|objects|select(has("secretKeyRef"))'

6. Verification (cluster-wide)

   • Re-run the audit command from any machine with kubectl access:

     output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
     if [ -z "$output" ]; then echo "NO_ENV_SECRET_REFERENCES"; else echo "ENV_SECRET_REFERENCES_FOUND"; fi

   • Repeat steps 2–5 until NO_ENV_SECRET_REFERENCES is printed or only the explicitly accepted exceptions remain.

Using kubectl

On any machine with kubectl access:

1. Identify workloads using env/envFrom secret references

kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.namespace} {.metadata.name} {"\n"}{end}'

2. For each affected Pod controller (Deployment/StatefulSet/DaemonSet/Job/CronJob), fetch the manifest, edit it locally, and re-apply.

Example for a Deployment:

# Export current manifest
kubectl get deployment <deployment-name> -n <namespace> -o yaml > deployment-with-env-secrets.yaml

Edit deployment-with-env-secrets.yaml:

• Remove env entries that use secretKeyRef, or entire envFrom entries that reference a Secret.
• Add a volume sourced from the Secret.
• Mount that volume into the container at a path your application can read.

Example transformation:

Before:

spec:
  template:
    spec:
      containers:
      - name: app
        image: your-image
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-secret
              key: password

After:

spec:
  template:
    spec:
      containers:
      - name: app
        image: your-image
        volumeMounts:
        - name: db-secret-vol
          mountPath: /etc/secrets/db
          readOnly: true
        env: []   # or remove the specific secret env entries
      volumes:
      - name: db-secret-vol
        secret:
          secretName: db-secret

Your application must be updated to read /etc/secrets/db/password instead of the environment variable.

Apply the updated manifest:

kubectl apply -f deployment-with-env-secrets.yaml

Repeat this export/edit/apply pattern for each affected controller kind:

kubectl get statefulset <name> -n <namespace> -o yaml > sts-with-env-secrets.yaml
kubectl apply -f sts-with-env-secrets.yaml

kubectl get daemonset <name> -n <namespace> -o yaml > ds-with-env-secrets.yaml
kubectl apply -f ds-with-env-secrets.yaml

kubectl get job <name> -n <namespace> -o yaml > job-with-env-secrets.yaml
kubectl apply -f job-with-env-secrets.yaml

kubectl get cronjob <name> -n <namespace> -o yaml > cj-with-env-secrets.yaml
kubectl apply -f cj-with-env-secrets.yaml

3. Verification

After updating all workloads and redeploying the application code to read mounted files, confirm there are no remaining secretKeyRef usages:

output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
if [ -z "$output" ]; then echo "NO_ENV_SECRET_REFERENCES"; else echo "ENV_SECRET_REFERENCES_FOUND"; fi

Automation

#!/usr/bin/env bash
#
# Automation: Identify workloads using Secret environment variables and
#             generate file-based Secret mounts plus updated manifests.
#
# NOTE: This script CANNOT safely and generically rewrite application code
#       or update all manifests automatically, because that requires
#       application-specific changes (code/config reading from files instead
#       of env vars) and deployment pipeline coordination.
#
#       It instead:
#         - Detects all env-var Secret references
#         - Generates suggested pod spec patches that mount the same Secrets
#           as files under /var/run/secrets/<secret-name>
#         - Optionally (opt-in) applies those patches to add volume mounts
#           while leaving env vars intact (for a migration period)
#
#       You must then:
#         - Update application code/config to read from files
#         - Remove the env-var Secret usage from the manifests
#
# REQUIREMENTS:
#   - Run on: any machine with kubectl access to the cluster
#   - Tools: bash, kubectl, jq, yq (go-yq v4 style: yq eval)
#
# USAGE:
#   chmod +x automate_secret_env_to_files.sh
#   ./automate_secret_env_to_files.sh
#
#   Environment flags:
#     DRY_RUN=true   # (default) only print patches; do not apply
#     DRY_RUN=false  # apply patches that add volumes/volumeMounts
#
set -euo pipefail

DRY_RUN="${DRY_RUN:-true}"

if ! command -v kubectl >/dev/null 2>&1; then
  echo "ERROR: kubectl not found in PATH" >&2
  exit 1
fi

if ! command -v jq >/dev/null 2>&1; then
  echo "ERROR: jq not found in PATH" >&2
  exit 1
fi

if ! command -v yq >/dev/null 2>&1; then
  echo "ERROR: yq not found in PATH (expected go-yq v4: 'yq eval')" >&2
  exit 1
fi

echo "=== Detecting Kubernetes resources that use Secret env vars ==="
ENV_SECRET_JSON=$(kubectl get all --all-namespaces -o json | jq '
  .items[]
  | select(..|has("secretKeyRef")? )
  | {kind, metadata: {name: .metadata.name, namespace: .metadata.namespace}}
')

if [ -z "$ENV_SECRET_JSON" ]; then
  echo "NO_ENV_SECRET_REFERENCES (no resources with env secretKeyRef found)."
  exit 0
fi

echo "$ENV_SECRET_JSON" | jq -r '"- \(.kind) \(.metadata.namespace)/\(.metadata.name)"'

cat <<'EOF'

INFO:
  - The remediation requires changing applications to read Secrets from files
    instead of environment variables.
  - This script cannot change application code; it only helps add file-based
    Secret mounts to the affected workloads.

EOF

# Work on workload controllers, not individual Pods:
#   Supported kinds: Deployment, StatefulSet, DaemonSet, Job, CronJob, ReplicaSet, ReplicationController
SUPPORTED_KINDS='["Deployment","StatefulSet","DaemonSet","Job","CronJob","ReplicaSet","ReplicationController"]'

# Get unique (namespace, kind, name) triplets that reference secretKeyRef
echo "=== Building unique list of controllers to process ==="
CONTROLLERS_JSON=$(kubectl get all --all-namespaces -o json | jq -r --argjson kinds "$SUPPORTED_KINDS" '
  .items[]
  | select(..|has("secretKeyRef")? )
  | select(.kind as $k | $k | IN($kinds[]) )
  | {"kind": .kind, "name": .metadata.name, "namespace": .metadata.namespace}
  | @base64
')

if [ -z "$CONTROLLERS_JSON" ]; then
  echo "No supported controller types with env-based Secrets were found."
  exit 0
fi

TMPDIR=$(mktemp -d)
trap 'rm -rf "$TMPDIR"' EXIT

process_controller() {
  local enc="$1"

  local obj
  obj=$(echo "$enc" | base64 --decode)
  local kind ns name
  kind=$(echo "$obj" | jq -r '.kind')
  ns=$(echo "$obj" | jq -r '.namespace')
  name=$(echo "$obj" | jq -r '.name')

  echo "=== Processing $kind $ns/$name ==="

  local yaml
  if ! yaml=$(kubectl -n "$ns" get "$kind" "$name" -o yaml 2>/dev/null); then
    echo "WARN: Failed to fetch $kind $ns/$name, skipping." >&2
    return
  fi

  local podspec_path
  case "$kind" in
    Deployment|StatefulSet|DaemonSet|ReplicaSet|ReplicationController)
      podspec_path=".spec.template.spec"
      ;;
    Job)
      podspec_path=".spec.template.spec"
      ;;
    CronJob)
      podspec_path=".spec.jobTemplate.spec.template.spec"
      ;;
    *)
      echo "WARN: Unsupported kind $kind, skipping." >&2
      return
      ;;
  esac

  # Identify all secretKeyRef.secretName used in env or envFrom
  local secrets_json
  secrets_json=$(echo "$yaml" | yq eval -o=json "$podspec_path" - | jq '
    def walk(f):
      . as $in
      | if type == "object" then
          reduce keys[] as $key
          ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f
        elif type == "array" then
          map( walk(f) ) | f
        else
          f
        end;

    walk(.) as $node
    | if ($node|type) == "object" and ($node.env? or $node.envFrom?) then
        [
          ($node.env[]?|select(.valueFrom.secretKeyRef?).valueFrom.secretKeyRef.secretName),
          ($node.envFrom[]?|select(.secretRef?).secretRef.name)
        ]
      else
        empty
      end
  ')

  if [ -z "$secrets_json" ]; then
    echo "  No secretKeyRef found in pod spec (likely only in status), skipping."
    return
  fi

  local unique_secrets
  unique_secrets=$(echo "$secrets_json" | jq -r '.[]' | sed '/^null$/d' | sort -u)

  if [ -z "$unique_secrets" ]; then
    echo "  No env-based Secret names resolved, skipping."
    return
  fi

  echo "  Env-based Secrets referenced:"
  echo "$unique_secrets" | sed 's/^/    - /'

  # Build a patch that:
  #   - adds volumes for each Secret (if not already present)
  #   - adds volumeMounts for each container (readOnly, mountPath /var/run/secrets/<secret>)
  # Existing volumes/volumeMounts are preserved.
  local patch_file="$TMPDIR/${ns}-${kind}-${name}-patch.yaml"

  # Start with current spec for manipulation in yq
  echo "$yaml" > "$TMPDIR/${ns}-${kind}-${name}-orig.yaml"

  # Build yq expression to update podSpec
  local yq_expr=""

  # Ensure .spec.template.spec.volumes or jobTemplate variant exists and add any missing Secret volumes
  echo "$unique_secrets" | while read -r secret; do
    [ -n "$secret" ] || continue
    # sanitize for volume name
    vol_name="secret-${secret//./-}"
    yq_expr+="
      $podspec_path.volumes |= ( . // [] ) |
      $podspec_path.volumes |=
        ( if map(.name) | index(\"$vol_name\") == null then
            . + [{\"name\":\"$vol_name\",\"secret\":{\"secretName\":\"$secret\"}}]
          else
            .
          end ) |
"
  done

  # For each container & initContainer: ensure volumeMounts and add mount for each secret
  yq_expr+="
    ($podspec_path.containers[]?, $podspec_path.initContainers[]?) as \$c ireduce (.; .) |
    $podspec_path.containers |= ( . // [] ) |
    $podspec_path.initContainers |= ( . // [] ) |
    $podspec_path.containers |=
      ( map(
          . as \$ct |
          .volumeMounts |= ( . // [] )
          | .volumeMounts |= (
              . +
              ["
  first_secret=1
  echo "$unique_secrets" | while read -r secret; do
    [ -n "$secret" ] || continue
    vol_name="secret-${secret//./-}"
    mount_path="/var/run/secrets/${secret}"
    if [ $first_secret -eq 0 ]; then
      yq_expr+=","
    fi
    first_secret=0
    yq_expr+="
                select( [ .[].name ] | index(\"$vol_name\") == null ) |
                {\"name\":\"$vol_name\",\"mountPath\":\"$mount_path\",\"readOnly\":true}
    "
  done
  yq_expr+="]
            )
        ) ) |
    $podspec_path.initContainers |=
      ( map(
          . as \$ct |
          .volumeMounts |= ( . // [] )
          | .volumeMounts |= (
              . +
              ["
  first_secret=1
  echo "$unique_secrets" | while read -r secret; do
    [ -n "$secret" ] || continue
    vol_name="secret-${secret//./-}"
    mount_path="/var/run/secrets/${secret}"
    if [ $first_secret -eq 0 ]; then
      yq_expr+=","
    fi
    first_secret=0
    yq_expr+="
                select( [ .[].name ] | index(\"$vol_name\") == null ) |
                {\"name\":\"$vol_name\",\"mountPath\":\"$mount_path\",\"readOnly\":true}
    "
  done
  yq_expr+="]
            )
        ) )
  "

  # Apply yq expression and build final patch
  yq eval "$yq_expr" "$TMPDIR/${ns}-${kind}-${name}-orig.yaml" > "$patch_file"

  echo "  Generated patch with Secret file mounts at: $patch_file"

  if [ "$DRY_RUN" = "true" ]; then
    echo "  DRY_RUN=true: showing diff (no changes applied):"
    diff -u "$TMPDIR/${ns}-${kind}-${name}-orig.yaml" "$patch_file" || true
  else
    echo "  Applying patch with kubectl apply -f:"
    kubectl apply -f "$patch_file"
  fi
}

echo "=== Generating patches to add file-based secret mounts ==="
while read -r enc; do
  [ -n "$enc" ] || continue
  process_controller "$enc"
done <<< "$CONTROLLERS_JSON"

echo
echo "=== Verification: re-running audit for env-based Secret usage ==="
output=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]} {.kind} {.metadata.name} {"\n"}{end}')
if [ -z "$output" ]; then
  echo "NO_ENV_SECRET_REFERENCES"
else
  echo "ENV_SECRET_REFERENCES_FOUND (env-based secrets still present in these resources):"
  echo "$output"
  echo
  echo "NOTE:"
  echo "  - The benchmark remediation ultimately requires removing env-based Secret usage."
  echo "  - After you update application code to read Secrets from files and clean up"
  echo "    env vars in your manifests, re-run this script to confirm."
fi

Additional Reading:

• https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets