Consider External Secret Storage

More Info:

Storing secrets in an external, dedicated secrets manager reduces exposure compared to native Kubernetes secrets. Evaluate cloud provider or third-party secrets management solutions.

Risk Level

Medium

Address

Security

Compliance Standards

• APRA CPS 234 (Australia)
• AWS Startup Security Baseline
• BSI C5 (Germany)
• Brazil LGPD
• CCPA / CPRA (California)
• CIS Critical Security Controls v8
• CIS EKS
• CMMC 2.0
• CSA Cloud Controls Matrix v4
• DPDPA
• Digital Operational Resilience Act (EU)
• Essential 8
• ISO/IEC 27017
• ISO/IEC 27018
• ISO/IEC 27701
• KSA PDPL
• MAS Technology Risk Management (Singapore)
• MITRE ATT&CK (Cloud)
• NIS2 Directive
• NIST SP 800-171
• NYDFS 23 NYCRR 500
• SWIFT Customer Security Controls Framework
• Sarbanes-Oxley IT General Controls
• UK NCSC Cyber Assessment Framework

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Inventory current Kubernetes Secrets usage
   Run these on any machine with kubectl access:

   # List all namespaces
   kubectl get ns
   
   # List all Secrets per namespace (type + age only)
   for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
     echo "=== Namespace: $ns ==="
     kubectl get secrets -n "$ns" -o wide
   done

   If allowed in your environment, spot-check contents (never export broadly in shared terminals/logs):

   # Inspect a specific Secret (base64-encoded data)
   kubectl get secret -n <namespace> <secret-name> -o yaml

2. Identify workloads depending on Kubernetes Secrets

   # Find Pods that reference Secrets in env / volume mounts
   for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
     echo "=== Namespace: $ns ==="
     kubectl get pods -n "$ns" -o json | \
       jq -r '.items[] |
         {ns: .metadata.namespace,
          pod: .metadata.name,
          sa: .spec.serviceAccountName,
          envSecrets: ([.spec.containers[].env[]? | select(.valueFrom.secretKeyRef).valueFrom.secretKeyRef.name] | unique),
          volSecrets: ([.spec.volumes[]? | select(.secret).secret.secretName] | unique)} |
         select((.envSecrets|length>0) or (.volSecrets|length>0))'
   done

   Record which applications, namespaces, and service accounts rely on Secrets, and which specific secrets are high sensitivity (DB creds, API keys, encryption keys, etc.).

3. Evaluate current secret protection controls
   On any machine with kubectl access, check if secret data at rest is protected (for managed EKS, review in cloud console/IaC if envelope encryption for secrets is enabled and with which KMS key). For cluster-scoped view:

   # Check if Secrets are encrypted at rest via EncryptionConfiguration (for self-managed control planes)
   kubectl get pods -n kube-system -l component=kube-apiserver -o yaml | \
     grep -E -- '--encryption-provider-config|--encryption-provider-config-file' || echo "Check in control-plane config / cloud console"

   In your cloud console / IaC, review:

   • Whether Kubernetes Secrets are encrypted with a customer-managed KMS key.
   • Who can access that KMS key (IAM roles/policies).
   • Who can list/get Secrets via cluster RBAC.

4. Assess need and options for external secret storage
   Based on steps 1–3, decide which secrets should move to an external manager (e.g., all production / highly sensitive secrets). For your environment, review in cloud console/IaC:

   • Availability of cloud-native secret managers (e.g., AWS Secrets Manager / Parameter Store, GCP Secret Manager, Azure Key Vault, HashiCorp Vault).
   • Native integrations or operators for Kubernetes (e.g., External Secrets Operator, Secrets Store CSI Driver).
   • Compliance or audit requirements that may mandate external secret storage or HSM-backed keys.

5. Design the integration pattern and update manifests
   Choose one pattern (examples, adapt to your platform and tooling):

   • Secrets Store CSI Driver / external secrets operator: pods mount secrets from external store as volumes or sync to Kubernetes Secrets.
   • Sidecar / init container that fetches secrets from the external manager and writes them to a tmpfs volume.
     For the chosen pattern, update deployment manifests to stop embedding static credentials and instead reference the external source. Example (CSI driver style – conceptual, not a one-size-fits-all fix):

   # Example Pod volume using a secret store CSI driver
   volumes:
     - name: app-secrets
       csi:
         driver: secrets-store.csi.k8s.io
         readOnly: true
         volumeAttributes:
           secretProviderClass: app-secrets-spc

   Apply updated manifests from a machine with kubectl access:

   kubectl apply -f <updated-manifest>.yaml

6. Verify and decommission in-cluster secrets where appropriate
   a. Verify workloads now retrieve secrets from the external store and still function:

   # Check Pod status and events
   kubectl get pods -A
   kubectl describe pod <pod-name> -n <namespace>

   b. Confirm that application pods no longer list the previous Kubernetes Secrets in env or volume mounts:

   kubectl get pod <pod-name> -n <namespace> -o json | \
     jq '.spec.containers[].env?[], .spec.volumes?[]' | grep -i secret || echo "No direct Secret refs found (verify manually if needed)"

   c. After confirming safe cutover and rollback plan, carefully delete or reduce high-sensitivity Kubernetes Secrets that are no longer needed:

   kubectl delete secret -n <namespace> <secret-name>

   Document which applications now depend on the external secrets manager and ensure access to the external secrets is governed via cloud/IAM policies instead of (or in addition to) in-cluster RBAC.

Using kubectl

# 1) List all namespaces
# Run on: any machine with kubectl access
kubectl get namespaces

Review all non-system namespaces (ignore kube-system, kube-public, kube-node-lease).

# 2) For each relevant namespace, list all Secrets and how they are used
# Run on: any machine with kubectl access

NAMESPACE=default

# List secrets (type and age give quick hints)
kubectl get secrets -n "${NAMESPACE}" -o wide

# Show detailed info for each secret (excluding the value)
kubectl get secrets -n "${NAMESPACE}" -o yaml

# Show which workloads reference secrets via env / envFrom / volumes
kubectl get deploy,sts,ds,job,cronjob -n "${NAMESPACE}" -o yaml | \
  grep -nE 'secretKeyRef|secretRef|secretName' -A3 -B3

Indicators of higher risk from this output:

• Large numbers of application-owned Opaque secrets (not just service account or TLS certs).
• Secrets holding long‑lived credentials (database users, API keys, cloud access keys).
• Same secret reused across many workloads or namespaces.
• Secrets embedded as env vars in many pods (easy to leak via logs, debug, etc.).

# 3) Identify secrets that look like external system credentials
# Run on: any machine with kubectl access

NAMESPACE=default

# Show secret names only, to focus the review
kubectl get secrets -n "${NAMESPACE}" -o custom-columns=NAME:.metadata.name,TYPE:.type

# For specific secrets that might be external creds, inspect metadata only
SECRET_NAME=my-db-credentials
kubectl get secret "${SECRET_NAME}" -n "${NAMESPACE}" -o yaml | \
  sed '/^data:/q'

Human review guidance:

• Names like *-db-password, *-api-key, *-aws-credentials, *-token usually indicate external system secrets that are good candidates for an external manager.
• If those secrets are long‑lived and manually rotated, that strengthens the case for an external secrets manager.

# 4) Check for cluster‑wide patterns (where external manager would help most)
# Run on: any machine with kubectl access

# Count secrets by namespace
kubectl get secrets --all-namespaces \
  -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.type | \
  sort | uniq -c | sort -nr | head

# List all Opaque secrets across all namespaces (excluding well-known system ones)
kubectl get secrets --all-namespaces \
  -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.type | \
  grep -vE 'kube-system|kube-public|kube-node-lease' | \
  grep 'Opaque'

Problem indicators in this cluster-wide view:

• Many Opaque secrets in application namespaces storing business‑critical or external credentials.
• Same‑name secrets across namespaces that obviously mirror the same external system (e.g., db-credentials in many namespaces) instead of being sourced from a dedicated secrets manager.
• No evidence of any integration with an external secrets operator (e.g., no CRDs like externalsecret), suggesting everything is native Kubernetes secrets.

# 5) Check if an external secrets solution is already in use
# Run on: any machine with kubectl access

# Look for common CRDs used by external secret managers
kubectl get crds | grep -iE 'secret|vault|externalsecret|external-secrets|secrets-store'

# Look for known operators in all namespaces
kubectl get pods --all-namespaces | grep -iE 'secret|vault|external|secrets-store' || true

Interpretation:

• If there are no CRDs or controllers related to external secrets and you see many high‑sensitivity Opaque secrets as described above, that is a strong signal that external secret storage should be considered.
• If external secret tooling is present, verify whether the high‑sensitivity secrets identified earlier are actually sourced from it or still managed as plain Kubernetes secrets.

Automation

#!/usr/bin/env bash
#
# cis-eks-4.4.2-secrets-usage-report.sh
#
# Run on: any machine with kubectl access and correct context
#
# This script does NOT fix anything. It summarizes how secrets are used so you
# can decide whether external secret storage should be adopted or expanded.
#
# It focuses on:
#   - Count and age of native Kubernetes Secrets
#   - Which types are used (Opaque, kubernetes.io/dockerconfigjson, tls, etc.)
#   - Workloads that project secrets as env vars or volumes
#   - Namespaces with heavy secret use
#
# Interpretation guidance is at the end of this file.

set -euo pipefail

# Ensure kubectl works
if ! kubectl version --short >/dev/null 2>&1; then
  echo "ERROR: kubectl is not configured or cannot reach the cluster" >&2
  exit 1
fi

echo "=== [1] Cluster-wide summary of Kubernetes Secret objects ==="
kubectl get secrets --all-namespaces -o json \
  | jq -r '
    .items
    | ( "NAMESPACE,NAME,TYPE,CREATED,AGE_DAYS,DATA_KEYS" ),
      ( .[] |
        . as $s
        | .metadata.namespace as $ns
        | .metadata.name as $name
        | .type as $type
        | .metadata.creationTimestamp as $created
        | ( (now - ($created | fromdate)) / 86400 | floor ) as $age_days
        | ($s.data // {} | keys | join("|")) as $keys
        | [$ns, $name, $type, $created, ($age_days|tostring), $keys]
        | @csv
      )
  ' | column -s, -t

echo
echo "=== [2] Secret counts per namespace and type ==="
kubectl get secrets --all-namespaces -o json \
  | jq -r '
    .items
    | group_by(.metadata.namespace)[] as $nsGroup
    | ($nsGroup[0].metadata.namespace) as $ns
    | ($nsGroup
        | group_by(.type)[]
        | [ $ns,
            .[0].type,
            (length | tostring)
          ] | @tsv
      )
  ' | awk 'BEGIN { printf "%-30s %-40s %s\n", "NAMESPACE", "TYPE", "COUNT" }
           { printf "%-30s %-40s %s\n", $1, $2, $3 }'

echo
echo "=== [3] Top namespaces by number of Secret objects ==="
kubectl get secrets --all-namespaces \
  | awk 'NR>1 {count[$1]++} END { for (ns in count) print count[ns], ns }' \
  | sort -nr \
  | head -20 \
  | awk 'BEGIN { printf "%-30s %s\n", "NAMESPACE", "SECRET_COUNT" }
               { printf "%-30s %s\n", $2, $1 }'

echo
echo "=== [4] Workloads using Secrets as environment variables ==="
echo "--- Deployments ---"
kubectl get deploy --all-namespaces -o json \
  | jq -r '
    .items[]
    | .metadata.namespace as $ns
    | .metadata.name as $name
    | (.spec.template.spec.containers // [])[]? as $c
    | ($c.env // [])[]? as $e
    | select($e.valueFrom.secretKeyRef != null)
    | [ $ns,
        $name,
        $c.name,
        $e.name,
        $e.valueFrom.secretKeyRef.name,
        $e.valueFrom.secretKeyRef.key
      ] | @tsv
  ' 2>/dev/null \
  | awk 'BEGIN { printf "%-30s %-40s %-30s %-25s %-40s %s\n", "NAMESPACE", "DEPLOYMENT", "CONTAINER", "ENV_VAR", "SECRET_NAME", "SECRET_KEY" }
               { printf "%-30s %-40s %-30s %-25s %-40s %s\n", $1, $2, $3, $4, $5, $6 }'

echo
echo "--- StatefulSets ---"
kubectl get statefulset --all-namespaces -o json \
  | jq -r '
    .items[]
    | .metadata.namespace as $ns
    | .metadata.name as $name
    | (.spec.template.spec.containers // [])[]? as $c
    | ($c.env // [])[]? as $e
    | select($e.valueFrom.secretKeyRef != null)
    | [ $ns,
        $name,
        $c.name,
        $e.name,
        $e.valueFrom.secretKeyRef.name,
        $e.valueFrom.secretKeyRef.key
      ] | @tsv
  ' 2>/dev/null \
  | awk 'BEGIN { printf "%-30s %-40s %-30s %-25s %-40s %s\n", "NAMESPACE", "STATEFULSET", "CONTAINER", "ENV_VAR", "SECRET_NAME", "SECRET_KEY" }
               { printf "%-30s %-40s %-30s %-25s %-40s %s\n", $1, $2, $3, $4, $5, $6 }'

echo
echo "--- DaemonSets ---"
kubectl get daemonset --all-namespaces -o json \
  | jq -r '
    .items[]
    | .metadata.namespace as $ns
    | .metadata.name as $name
    | (.spec.template.spec.containers // [])[]? as $c
    | ($c.env // [])[]? as $e
    | select($e.valueFrom.secretKeyRef != null)
    | [ $ns,
        $name,
        $c.name,
        $e.name,
        $e.valueFrom.secretKeyRef.name,
        $e.valueFrom.secretKeyRef.key
      ] | @tsv
  ' 2>/dev/null \
  | awk 'BEGIN { printf "%-30s %-40s %-30s %-25s %-40s %s\n", "NAMESPACE", "DAEMONSET", "CONTAINER", "ENV_VAR", "SECRET_NAME", "SECRET_KEY" }
               { printf "%-30s %-40s %-30s %-25s %-40s %s\n", $1, $2, $3, $4, $5, $6 }'

echo
echo "=== [5] Workloads mounting Secrets as volumes ==="
for kind in deployment statefulset daemonset; do
  kind_uc=$(echo "$kind" | tr '[:lower:]' '[:upper:]')
  echo "--- ${kind_uc}s ---"
  kubectl get "$kind" --all-namespaces -o json \
    | jq -r '
      .items[]
      | .metadata.namespace as $ns
      | .metadata.name as $name
      | (.spec.template.spec.volumes // [])[]
      | select(.secret != null)
      | [ $ns,
          $name,
          .name,
          .secret.secretName
        ] | @tsv
    ' 2>/dev/null \
    | awk -v KIND="$kind_uc" 'BEGIN { printf "%-30s %-40s %-30s %s\n", "NAMESPACE", KIND, "VOLUME_NAME", "SECRET_NAME" }
                              { printf "%-30s %-40s %-30s %s\n", $1, $2, $3, $4 }'
  echo
done

echo "=== [6] Pods directly referencing Secrets (non-controller-managed) ==="
kubectl get pods --all-namespaces -o json \
  | jq -r '
    .items[]
    | select(.metadata.ownerReferences == null or (.metadata.ownerReferences | length == 0))
    | .metadata.namespace as $ns
    | .metadata.name as $name
    | (
        [(.spec.volumes // [])[]? | select(.secret != null) | ["volume", .name, .secret.secretName]],
        [(.spec.containers // [])[]? as $c
          | ($c.env // [])[]?
          | select(.valueFrom.secretKeyRef != null)
          | ["env", $c.name, .valueFrom.secretKeyRef.name]
        ]
      )[]
    | [ $ns, $name, .[0], .[1], .[2] ] | @tsv
  ' 2>/dev/null \
  | awk 'BEGIN { printf "%-30s %-40s %-10s %-30s %s\n", "NAMESPACE", "POD", "REF_TYPE", "CONTAINER/VOLUME", "SECRET_NAME" }
               { printf "%-30s %-40s %-10s %-30s %s\n", $1, $2, $3, $4, $5 }'

echo
echo "================================================================================"
echo "INTERPRETING THIS OUTPUT (WHAT MAY INDICATE A PROBLEM)"
echo
cat <<'EOF'
This control is MANUAL: there is no one correct target state. Use the data above to
decide if you should adopt or expand an external secrets manager (cloud-native or
third-party):

Signs you may want to move more to external secret storage:
- [1]/[2]/[3]: Large numbers of Opaque secrets, especially:
  - With very old AGE_DAYS (stale or long-lived credentials).
  - In many different namespaces without consistent naming or ownership.
- [4]: Applications using many env vars from secrets:
  - Env vars are easily exposed via "kubectl describe pod" and logs.
  - Consider replacing sensitive values with references to an external
    secrets manager via a sidecar, CSI driver, or operator.
- [5]/[6]: Workloads mounting large or generic secrets (e.g. "app-config", "credentials")
  that likely mix different secret values instead of using a dedicated external store.

This script only reports usage. To remediate, you must:
- Select and configure a cloud provider or third-party secrets manager.
- Redesign workloads to fetch or project secrets from that system.
EOF