Skip to main content

Consider External Secret Storage

More Info:​

Kubernetes supports mounting secrets as data volumes or as environment variables. Minimize the use of environment variable secrets.

Risk Level​

High

Address​

Security

Compliance Standards​

  • APRA CPS 234 (Australia)
  • BSI C5 (Germany)
  • Brazil LGPD
  • CCPA / CPRA (California)
  • CIS AKS
  • CIS Critical Security Controls v8
  • CMMC 2.0
  • CSA Cloud Controls Matrix v4
  • DPDPA
  • Digital Operational Resilience Act (EU)
  • Essential 8
  • ISO/IEC 27017
  • ISO/IEC 27018
  • ISO/IEC 27701
  • KSA PDPL
  • MAS Technology Risk Management (Singapore)
  • MITRE ATT&CK (Cloud)
  • NIS2 Directive
  • NIST SP 800-171
  • NYDFS 23 NYCRR 500
  • SWIFT Customer Security Controls Framework
  • Sarbanes-Oxley IT General Controls
  • UK NCSC Cyber Assessment Framework

Triage and Remediation​

Remediation​

Manual Steps

  1. Inventory all Secrets in use (any machine with kubectl access)

    kubectl get secrets -A -o wide

    Export details for review (base64-decoded data omitted in this step):

    kubectl get secrets -A -o yaml > /tmp/all-secrets.yaml

  2. Identify workloads consuming Secrets as environment variables (any machine with kubectl access)

    kubectl get deploy,sts,ds,job,cronjob -A -o yaml > /tmp/all-workloads.yaml

    Search for env-based secret usage:

    grep -nE 'valueFrom:[[:space:]]*$' -n /tmp/all-workloads.yaml -A5 | grep -n 'secretKeyRef' -A3

  3. Review and classify secret usage (manual review on any machine)

    • From /tmp/all-workloads.yaml, list containers using env/envFrom with secretKeyRef or secretRef.
    • For each, determine:
      • Is the secret long‑lived or highly sensitive (DB passwords, API keys, TLS private keys)?
      • Is the secret needed only at startup (e.g., DB migration) or throughout runtime?
    • Prefer candidates that are long‑lived and runtime‑sensitive for external secret storage.

  4. Evaluate external secret storage options and integration pattern (design/decision step)

    • Choose provider: e.g., Azure Key Vault, HashiCorp Vault, SOPS + KMS, or cloud‑native secret manager.
    • Decide access pattern per workload:
      • App fetches secrets directly via SDK/API at runtime, or
      • Use an operator (e.g., external-secrets) to sync from external store to Kubernetes Secret, or
      • Sidecar/agent that mounts secrets from external store into the container filesystem.
    • Document the chosen pattern and access controls (identity, RBAC, network).

  5. Refactor one workload as a pilot away from env var secrets (any machine with kubectl access)

    • For the selected workload:
      • Remove env/envFrom entries that reference sensitive secrets.
      • Add configuration needed for the external store pattern (e.g., volume mount from sidecar, SDK config, or ExternalSecret CRD).
    • Apply changes: 
      kubectl apply -f <updated-manifest>.yaml
    • Coordinate with application owners to update code/config to read secrets from the new mechanism (not environment variables).

  6. Verify reduction of environment-variable secrets and repeat iteratively (any machine with kubectl access)

    • Re-run workload inspection: 
      kubectl get deploy,sts,ds,job,cronjob -A -o yaml > /tmp/all-workloads-post.yaml
      grep -n 'secretKeyRef' -n /tmp/all-workloads-post.yaml -A3
    • Confirm the pilot workload no longer pulls sensitive secrets via env/envFrom.
    • Expand the same process to additional workloads, prioritizing the most sensitive secrets first.
Using kubectl
# 1) List all namespaces
# Run on: any machine with kubectl access
kubectl get ns
# 2) For each namespace, list pods and show env and envFrom sections
# (replace <namespace> with each namespace name)
kubectl get pods -n <namespace> -o yaml | \
  sed -n '/env:/,/^ *[^- ]/p;/envFrom:/,/^ *[^- ]/p' | \
  sed 's/^/  /'

What to look for as a problem

In the output, inspect env: and envFrom: blocks under each container:

  • Direct secret keys in env: 
    env:
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: db-credentials
            key: password
  • Whole-secret imports into env: 
    envFrom:
      - secretRef:
          name: app-secrets

These patterns indicate secrets are exposed as environment variables instead of via mounted volumes or external secret management.

# 3) Show where each Secret is referenced by Pods (cluster-wide)
kubectl get secrets --all-namespaces -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' | \
while read ns name; do
  echo "=== Secret: $ns/$name ==="
  kubectl get pods -n "$ns" -o yaml | \
    grep -nE "secretKeyRef:|secretRef:" -n || echo "  (no direct pod references found)"
done

What to look for as a problem

For secrets that contain high-value data (DB credentials, API keys, tokens), any use via secretKeyRef / secretRef in env or envFrom should be flagged for review as a candidate to move to:

  • projected/volume-mounted secrets, or
  • an external secret management solution.
# 4) Inspect containers that use many env vars from secrets
kubectl get pods --all-namespaces -o yaml | \
  awk '/name: / {cname=$2} /envFrom:/,/^ *[^- ]/ {print "Container: " cname " => " $0}'

What to look for as a problem

Containers that pull entire secrets into their environment via envFrom.secretRef are higher risk and stronger candidates for redesign.

# 5) Optional: focus on pods in critical namespaces (kube-system, production, etc.)
kubectl get pods -n kube-system -o yaml | \
  sed -n '/env:/,/^ *[^- ]/p;/envFrom:/,/^ *[^- ]/p'

What to look for as a problem

Environment-based secrets in critical infrastructure or production workloads should be prioritized for migration to external secret storage or more secure patterns.

These commands only surface current usage; a human must decide which secrets should be moved to external storage and how to refactor each workload.

Automation
#!/usr/bin/env bash
#
# Report Kubernetes workload use of Secrets as environment variables vs volumes.
# Run on: any machine with kubectl access and appropriate RBAC.
# Requires: kubectl, jq
#
# This DOES NOT change anything; it only reports for human review.

set -euo pipefail

# Helper: print a section header
hdr() {
  printf '\n===== %s =====\n' "$1"
}

# 1) Cluster‑wide inventory of Secrets (for reference)
hdr "All Secrets by type (cluster-wide)"
kubectl get secrets --all-namespaces -o json \
  | jq -r '
    .items[]
    | [.metadata.namespace, .metadata.name, .type]
    | @tsv' \
  | column -t

# 2) Workloads using Secrets as environment variables
#
# These are the primary cases to review and potentially migrate
# to volume-mounted secrets or external secret stores.

workload_kinds=(
  Deployment
  StatefulSet
  DaemonSet
  ReplicaSet
  Job
  CronJob
)

for kind in "${workload_kinds[@]}"; do
  hdr "Secrets referenced as ENV VARS in ${kind}s"
  # We look for:
  # - env[].valueFrom.secretKeyRef
  # - envFrom[].secretRef
  kubectl get "$kind" --all-namespaces -o json 2>/dev/null \
    | jq -r '
      .items[]
      | . as $w
      | (
          .spec.template.spec.containers[]
          | {
              ns: $w.metadata.namespace,
              kind: "'$kind'",
              name: $w.metadata.name,
              c: .name,
              env: ( .env // [] | map(select(.valueFrom.secretKeyRef)) ),
              envFrom: ( .envFrom // [] | map(select(.secretRef)) )
            }
        ) // empty
      | select((.env|length) > 0 or (.envFrom|length) > 0)
      | (
          .env[]
          | {
              ns,kind,name,c,
              ref_type:"env",
              secret: .valueFrom.secretKeyRef.name,
              key: .valueFrom.secretKeyRef.key
            }
        )?,(
          .envFrom[]
          | {
              ns,kind,name,c,
              ref_type:"envFrom",
              secret: .secretRef.name,
              key:"*"
            }
        )?
      ' 2>/dev/null \
    | jq -r '[.ns,.kind,.name,.c,.ref_type,.secret,.key] | @tsv' \
    | sort -u \
    | column -t || echo "No ${kind}s found or no env-based secret usage."
done

# 3) Workloads using Secrets as mounted volumes
#
# These typically align better with the recommendation to avoid
# environment-variable secrets, but still require review.

for kind in "${workload_kinds[@]}"; do
  hdr "Secrets used as VOLUME MOUNTS in ${kind}s"
  kubectl get "$kind" --all-namespaces -o json 2>/dev/null \
    | jq -r '
      .items[]
      | . as $w
      | {
          ns: .metadata.namespace,
          kind: "'$kind'",
          name: .metadata.name,
          vols: (.spec.template.spec.volumes // [])
        }
      | select(.vols | length > 0)
      | .vols[]
      | select(.secret)
      | [ .ns, .kind, .name, .name as $vname | ., "secret", .secret.secretName ]
      | @tsv
    ' 2>/dev/null \
    | sort -u \
    | column -t || echo "No ${kind}s found or no secret volumes."
done

# 4) Pods directly (for completeness; may include static pods or bare pods)
hdr "Standalone Pods using Secrets as ENV VARS"
kubectl get pods --all-namespaces -o json 2>/dev/null \
  | jq -r '
    .items[]
    | . as $p
    | (
        .spec.containers[]
        | {
            ns: $p.metadata.namespace,
            kind: "Pod",
            name: $p.metadata.name,
            c: .name,
            env: ( .env // [] | map(select(.valueFrom.secretKeyRef)) ),
            envFrom: ( .envFrom // [] | map(select(.secretRef)) )
          }
      ) // empty
    | select((.env|length) > 0 or (.envFrom|length) > 0)
    | (
        .env[]
        | {
            ns,kind,name,c,
            ref_type:"env",
            secret: .valueFrom.secretKeyRef.name,
            key: .valueFrom.secretKeyRef.key
          }
      )?,(
        .envFrom[]
        | {
            ns,kind,name,c,
            ref_type:"envFrom",
            secret: .secretRef.name,
            key:"*"
          }
      )?
  ' \
  | jq -r '[.ns,.kind,.name,.c,.ref_type,.secret,.key] | @tsv' \
  | sort -u \
  | column -t || echo "No pods or no env-based secret usage."

hdr "Standalone Pods using Secrets as VOLUME MOUNTS"
kubectl get pods --all-namespaces -o json 2>/dev/null \
  | jq -r '
    .items[]
    | . as $p
    | {
        ns: .metadata.namespace,
        kind: "Pod",
        name: .metadata.name,
        vols: (.spec.volumes // [])
      }
    | select(.vols | length > 0)
    | .vols[]
    | select(.secret)
    | [ .ns, .kind, .name, .name as $vname | ., "secret", .secret.secretName ]
    | @tsv
  ' \
  | sort -u \
  | column -t || echo "No pods or no secret volumes."

echo
echo "Review guidance:"
echo "  - ENV VAR problems: any lines labeled 'env' or 'envFrom' above indicate"
echo "    secrets exposed as environment variables, which the benchmark says to"
echo "    minimize. Prioritize these for migration to:"
echo "      * volume-mounted secrets, and/or"
echo "      * an external secret manager (cloud-native or third party)."
echo "  - VOLUME usage: lines in the VOLUME sections show where secrets are"
echo "    mounted as files. These are generally preferred to env-based secrets,"
echo "    but still should be evaluated for potential migration to external"
echo "    secret storage where appropriate."

What output indicates a problem

  • Any rows in the Secrets referenced as ENV VARS in ... sections (including the standalone Pods section) show containers that are using Kubernetes Secrets via:
    • ref_type = env (per-variable valueFrom.secretKeyRef), or
    • ref_type = envFrom (all keys from a secret imported as environment variables).

Those rows are the candidates you should review and, where feasible, migrate away from environment-variable secrets toward volume-mounted secrets and/or an external secret manager.

Additional Reading:​