Consider External Secret Storage

More Info:

Native Kubernetes Secrets are only base64-encoded and stored in etcd. Consider a dedicated external secrets management solution to improve secret protection and lifecycle management.

Risk Level

Medium

Address

Security

Compliance Standards

• APRA CPS 234 (Australia)
• BSI C5 (Germany)
• Brazil LGPD
• CCPA / CPRA (California)
• CIS Critical Security Controls v8
• CIS GKE
• CMMC 2.0
• CSA Cloud Controls Matrix v4
• DPDPA
• Digital Operational Resilience Act (EU)
• Essential 8
• ISO/IEC 27017
• ISO/IEC 27018
• ISO/IEC 27701
• KSA PDPL
• MAS Technology Risk Management (Singapore)
• MITRE ATT&CK (Cloud)
• NIS2 Directive
• NIST SP 800-171
• NYDFS 23 NYCRR 500
• SWIFT Customer Security Controls Framework
• Sarbanes-Oxley IT General Controls
• UK NCSC Cyber Assessment Framework

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Inventory how Kubernetes Secrets are currently used

   • On any machine with kubectl access, list all Secrets and their types:

     kubectl get secrets -A -o wide

   • Inspect a few representative secrets (workload-related, not service-account tokens):

     kubectl get secret -n <namespace> <secret-name> -o yaml

   • Note where secrets are consumed (Pods, Deployments, StatefulSets, Jobs) via env.valueFrom.secretKeyRef and volumes.secret.

2. Check for existing external secret management integrations

   • Look for operators/controllers that integrate with external stores (for example, External Secrets Operator, Secrets Store CSI Driver, or cloud‑specific controllers):

     kubectl get pods -A | egrep -i 'secret|vault|csi'
     kubectl get crds | egrep -i 'secret|vault|externalsecret|csi'

   • If such components exist, inspect their configuration to see which backend is in use (e.g., AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault).

3. Review cloud provider secret-management options and policies

   • In your cloud console and/or IaC, identify whether a managed secret store is already configured for this cluster’s workloads (e.g., IAM roles, Workload Identity, KMS keys, Secret Manager/Key Vault/Secrets Manager resources).
   • Confirm if organizational policy requires use of that store instead of (or in addition to) Kubernetes Secrets for application credentials.

4. Decide which secrets must move to external storage

   • Based on the inventory, classify secrets by sensitivity (database passwords, API keys, encryption keys vs. low‑risk config).
   • For highly sensitive or regulated data, decide to migrate them to a supported external secret store, following your cloud provider’s recommended integration pattern (operator, CSI driver, or sidecar).

5. Plan and implement the migration pattern

   • For each selected workload, design how it will obtain secrets from the external store (environment variables injected by a controller, mounted files via CSI, or app‑level API calls).
   • Update or create manifests (via kubectl and IaC) to consume secrets from the chosen integration (for example, defining ExternalSecret or CSI SecretProviderClass resources), and remove or deprecate direct reliance on native Kubernetes Secrets for those values.

6. Verify reduced reliance on native Kubernetes Secrets

   • After changes, confirm that sensitive values are now sourced via external mechanisms and that corresponding Kubernetes Secrets are no longer needed or only store references/tokens as designed:

     kubectl get secrets -A
     kubectl describe pod -n <namespace> <pod-name>

   • Ensure applications start successfully and retrieve expected secret values via the external store before removing any legacy Kubernetes Secrets.

Using kubectl

# 1) List all namespaces that may contain Secrets
# Run on: any machine with kubectl access
kubectl get ns

Review which namespaces are used for applications (beyond kube-system, kube-public, etc.); these are likely to contain sensitive Secrets.

# 2) List all Secrets in all namespaces (names and types only)
kubectl get secrets --all-namespaces

Potential problems indicated by:

• Many Opaque or kubernetes.io/dockerconfigjson secrets in app namespaces.
• Secrets present in non-critical namespaces where config maps would suffice.

# 3) Inspect specific Secrets for sensitive payloads
# Replace <namespace> and <secret-name> based on previous output
kubectl get secret <secret-name> -n <namespace> -o yaml

Look at the .data entries (base64-encoded). Problems indicated by:

• Keys like password, token, apikey, ssh-privatekey, tls.key, db_url, etc.
• Large numbers of such sensitive values stored as native Secrets instead of a dedicated external store.

# 4) Find Pods that mount or reference Secrets
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}' \
  | while read ns pod; do
      echo "=== $ns/$pod ==="
      kubectl get pod "$pod" -n "$ns" -o jsonpath='Volumes: {range .spec.volumes[*]}{.name}{"=>"}{.secret.secretName}{"\n"}{end}EnvFrom: {range .spec.containers[*].envFrom[*]}{.secretRef.name}{"\n"}{end}Env: {range .spec.containers[*].env[*]}{.name}{"=>"}{.valueFrom.secretKeyRef.name}{"\n"}{end}' 2>/dev/null
      echo
    done

Problems indicated by:

• Critical workloads (auth, payments, PII, production databases) consuming many Secrets via env vars or volumes.
• Secrets used across many Pods/namespaces, suggesting centralized secrets would be better managed in an external vault.

# 5) Check for any use of external secret managers already (for context)
# Common CRDs used by external secret operators
kubectl get crd | egrep -i 'secret|vault|externalsecret'

Interpretation:

• If you see CRDs like externalsecrets.external-secrets.io, clusterssecrets, vaultsecrets, the cluster may already integrate with an external secrets manager.
• If no such resources exist and Steps 2–4 show widespread in-cluster storage of high-value credentials, that suggests a stronger case for adopting an external secrets management solution as recommended by the benchmark.

Automation

#!/usr/bin/env bash
# Purpose: Cluster-wide secrets usage report to support review of external secret storage adoption.
# Runs on: any machine with kubectl access and cluster-wide read permissions.

set -euo pipefail

# 1) Basic cluster context
echo "=== Cluster Context ==="
kubectl config current-context
echo

# 2) Count and list native Kubernetes Secret objects
echo "=== Native Kubernetes Secrets (all namespaces) ==="
kubectl get secrets --all-namespaces | wc -l | awk '{print "Total Secret objects (including header): "$1}'
echo

# Detailed summary by type (helps spot high‑risk uses like generic Opaque)
echo "--- Secret count by type ---"
kubectl get secrets --all-namespaces -o json \
  | jq -r '.items[] | [.metadata.namespace, .type] | @tsv' \
  | sort | uniq -c | sort -nr \
  | awk '{printf "%7s  ns=%s  type=%s\n", $1, $2, $3}'
echo

# 3) Show a sample of Secrets by type (names only, no data)
echo "--- Sample Secrets by type (names only, per namespace) ---"
kubectl get secrets --all-namespaces -o json \
  | jq -r '.items[] | [.metadata.namespace, .metadata.name, .type] | @tsv' \
  | column -t | head -100
echo

# 4) Detect common external-secrets controllers / CRDs in use
echo "=== External Secret Integrations Detected ==="

echo "--- Checking for ExternalSecrets.io (external-secrets) ---"
kubectl api-resources --api-group=external-secrets.io 2>/dev/null || echo "No external-secrets.io API group found"
echo
kubectl get crd | grep -i externalsecret || echo "No ExternalSecret CRD found"
echo
kubectl get ns | grep -Ei 'secret|vault|external' || echo "No namespaces with obvious secret-related names"
echo

echo "--- Checking for Secrets Store CSI Driver ---"
kubectl api-resources --api-group=secrets-store.csi.x-k8s.io 2>/dev/null || echo "No secrets-store.csi.x-k8s.io API group found"
echo
kubectl get crd | grep -i secretproviderclass || echo "No SecretProviderClass CRD found"
echo

# 5) Workload references to Secret objects (indicates reliance on native Secrets)
echo "=== Workloads referencing native Kubernetes Secrets ==="

# helper: list resources that mount secrets as volumes or environment variables
for kind in deployment statefulset daemonset job cronjob pod replicaSet replicaset; do
  echo "--- ${kind^}s using secrets ---"
  kubectl get "$kind" --all-namespaces -o json 2>/dev/null \
    | jq -r '
      .items[]
      | . as $obj
      | ($obj.spec.template // $obj)  # handle Pod vs controllers
      | [
          $obj.metadata.namespace,
          $obj.metadata.name,
          "'"$kind"'",
          ([
             (.spec.volumes[]? | select(.secret != null) | "vol:"+.secret.secretName),
             (.spec.containers[]?.env[]? | select(.valueFrom.secretKeyRef != null) | "env:"+.valueFrom.secretKeyRef.name),
             (.spec.initContainers[]?.env[]? | select(.valueFrom.secretKeyRef != null) | "env:"+.valueFrom.secretKeyRef.name)
           ] | unique | join(","))
        ]
      | @tsv
    ' 2>/dev/null \
    | awk -F'\t' 'NF==4 && $4!="" {printf "ns=%s  kind=%s  name=%s  secrets=[%s]\n", $1, $3, $2, $4}' \
    || echo "No $kind resources found or no secret usage detected"
  echo
done

# 6) Cluster-wide summary: namespaces with most native Secrets referenced by workloads
echo "=== Summary: Workload Secret Usage by Namespace ==="
kubectl get deploy,statefulset,daemonset,job,cronjob,pod --all-namespaces -o json 2>/dev/null \
  | jq -r '
    .items[]
    | . as $obj
    | ($obj.spec.template // $obj)
    | [
        $obj.metadata.namespace,
        ([
           (.spec.volumes[]? | select(.secret != null) | .secret.secretName),
           (.spec.containers[]?.env[]? | select(.valueFrom.secretKeyRef != null) | .valueFrom.secretKeyRef.name),
           (.spec.initContainers[]?.env[]? | select(.valueFrom.secretKeyRef != null) | .valueFrom.secretKeyRef.name)
         ] | unique | length)
      ]
    | @tsv
  ' 2>/dev/null \
  | awk -F'\t' 'NF==2 && $2>0 {print $1}' \
  | sort | uniq -c | sort -nr \
  | awk '{printf "%7s namespaces-with-secrets: %s\n", $1, $2}'
echo

echo "=== Interpretation Guidance ==="
cat <<'EOF'
This script does NOT decide compliance automatically. Use the output as follows:

- High native Secret counts and many workloads referencing Opaque or generic Secrets:
  * Indicates strong reliance on built-in Kubernetes Secrets and etcd storage.
  * From a CIS perspective, this is where you should consider migrating to or integrating with
    an external secrets management solution (cloud KMS-based, vault, or similar).

- Absence of external-secrets-related CRDs or CSI drivers:
  * Suggests no cluster-integrated external secret manager is currently in use.
  * This is a gap if your policy is to store application secrets externally.

- Presence of ExternalSecret / SecretProviderClass or similar CRDs and controllers:
  * Indicates that external secret management exists.
  * Review:
    - Which namespaces and apps are using these external mechanisms vs. native Secrets.
    - Whether particularly sensitive apps (payments, auth, PII) still rely on Opaque Secrets.

Use this report to:
- Identify namespaces and workloads that should be prioritized for external secret integration.
- Confirm where external secret controllers are installed and actually used in manifests.
EOF