Consider External Secret Storage

More Info:

Storing secrets in an external, dedicated secrets manager can improve protection over native Kubernetes Secrets. Evaluate such options.

Risk Level

Informational

Address

Security

Compliance Standards

CIS Kubernetes

Triage and Remediation

Remediation

Remediation

Manual Steps

Inventory how Kubernetes Secrets are currently used
- On any machine with kubectl access, list all Secret objects and their types:
```
kubectl get secrets -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.type' | sort
```
- Identify workloads that mount or reference these Secrets:
```
kubectl get pods -A -o yaml | grep -nE 'secretName:|valueFrom:' -n
```
- Note any use of type kubernetes.io/basic-auth, kubernetes.io/ssh-auth, database creds, API keys, tokens, etc., which are strong candidates for external storage.
Assess in-cluster Secret protection and sensitivity
- Check if Encryption at Rest is enabled for Secrets (control plane–specific; if you do not manage it, retrieve documentation/config from your provider or IaC):
  - For self-managed clusters, on every control plane node:
    sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep -n 'encryption-provider-config'
    sudo cat /etc/kubernetes/encryption-config.yaml
- Classify secrets by sensitivity (e.g., “high” for long-lived credentials to production systems, “medium” for non-production, “low” for ephemeral tokens) and record which namespaces and apps depend on them.
Review available external secrets managers and cluster integration options
- From your cloud console or IaC, identify supported secrets managers and their Kubernetes integrations, for example (depending on environment):
  - AWS: AWS Secrets Manager / SSM Parameter Store + External Secrets Operator or Secrets Store CSI Driver.
  - GCP: Secret Manager + Secret Manager CSI driver / external-secrets.
  - Azure: Key Vault + Secrets Store CSI Driver / external-secrets.
- Confirm organizational standards or regulatory requirements that may mandate a particular secrets manager or key management scheme (e.g., use of HSM-backed keys, centralized audit, rotation policies).
Decide scope and pattern for migration to external secret storage
- Choose a preferred integration pattern (e.g., External Secrets Operator with CRDs like ExternalSecret, or CSI driver volume mounts) based on your platform.
- Select a pilot scope: one or a few high-sensitivity applications and their Secrets (from step 1) to migrate first.
- For those pilot apps, map each existing Secret to a candidate external secret path/name in the chosen manager.
Implement and test external secrets for a pilot workload
- Using cloud console or IaC, create corresponding secrets in the external manager for the pilot application, ensuring:
  - Access control (IAM/RBAC) only allows the Kubernetes cluster/namespace/service account to read them.
  - Rotation policy is defined where supported.
- On any machine with kubectl access, deploy the chosen integration (operator or CSI driver) according to its vendor documentation, then define a test manifest that consumes the external secret (example pattern only, adjust to your tool):
```
kubectl apply -f external-secret-test.yaml
```
- Verify the pod(s) receive correct secret data and the original Kubernetes Secret is no longer required for that workload.
Plan progressive rollout and deprecation of native Secrets where appropriate
- For remaining high- and medium-sensitivity Secrets, repeat the mapping and migration process, updating manifests in your IaC to source data from the external store instead of embedding values into Kubernetes Secrets.
- Verify after each migration that no workloads still depend on the old Secrets:
```
kubectl get pods -A -o yaml | grep -n 'secretName:' | grep '<old-secret-name>' || echo "No remaining references"
```
- Once confirmed, delete or minimize high-sensitivity native Secrets and document the new standard: which classes of secrets must be stored in the external manager, and which (if any) may remain as Kubernetes Secrets.

Using kubectl

# 1) List all namespaces
# Run on: any machine with kubectl access
kubectl get ns

# 2) List secrets per namespace (excluding service-account token secrets)
# Replace <namespace> with each namespace from the previous command
kubectl get secrets -n <namespace> \
  --field-selector='type!=kubernetes.io/service-account-token'

# 3) Inspect individual secrets for sensitive data patterns
# Replace <secret-name> and <namespace>
kubectl get secret <secret-name> -n <namespace> -o yaml

What to look for in the output (indicates a problem / candidate for external storage):

type: Opaque secrets containing:
- Long-lived application credentials (database users/passwords, API keys, TLS private keys not associated with short-lived certs).
- Cloud provider access keys, tokens, or passwords to external services.
Secrets with generic names but clearly sensitive values, for example:
- db-credentials, payment-gateway-key, smtp-password, aws-credentials, gcp-service-account, azure-sp-credentials.
Secrets referenced broadly across many workloads (suggesting high blast radius).

# 4) See how secrets are consumed by workloads (env vars and volumes)
# Replace <namespace>
kubectl get deploy,sts,ds -n <namespace> -o yaml

In the above output, potential issues:

env: or envFrom: entries that reference sensitive secrets via valueFrom.secretKeyRef.
volumes: with secret: sources that mount sensitive secrets to many pods.
Same secret name used across many different applications/namespaces (central, critical credential).

# 5) Identify secrets likely holding cloud or external service credentials
# Common keyword search in secret names (namespace by namespace)
kubectl get secrets -n <namespace> | grep -Ei 'aws|gcp|azure|db|mysql|postgres|mongo|redis|api|token|key|cert|tls'

If this surfaces secrets whose values (from step 3) are:

Long-lived,
Grant access to external or critical systems,
Needed by many different workloads,

then they are strong candidates to be moved to an external secrets manager, with Kubernetes only holding references or short-lived material.

# 6) Check for cluster-wide TLS / CA material stored as regular secrets
kubectl get secrets -A | grep -Ei 'tls|ca|certificate'

Potential concern:

TLS private keys or CA roots that are long-lived and reused across multiple services or ingresses; those are often better managed in a dedicated secrets/PKI system.

Next step (manual decision):

Based on the above inspection, decide which secrets should remain as native Kubernetes Secrets (low-risk, internal-only, non-critical), and which should instead be sourced from an external secrets manager per your cloud provider or chosen third-party solution.

Automation

#!/usr/bin/env bash
# Run on: any machine with kubectl access and current context set to the target cluster
# Purpose: Report how Kubernetes Secrets are used so you can decide
# whether to move to an external secrets manager.

set -euo pipefail

echo "=== Cluster-wide secret inventory (by namespace and type) ==="
kubectl get secrets --all-namespaces -o json \
  | jq -r '
    .items[]
    | {ns: .metadata.namespace, name: .metadata.name, type: .type}
    | [.ns, .name, .type]
    | @tsv' \
  | column -t

echo
echo "=== Count of secrets per namespace (excluding service-account tokens) ==="
kubectl get secrets --all-namespaces -o json \
  | jq -r '
    .items[]
    | select(.type != "kubernetes.io/service-account-token")
    | .metadata.namespace' \
  | sort \
  | uniq -c \
  | sort -nr

echo
echo "=== Workloads that mount or reference native Kubernetes Secrets ==="
echo "--- Pods (includes Deployments, StatefulSets, etc. via their Pods) ---"
kubectl get pods --all-namespaces -o json \
  | jq -r '
    .items[]
    | {
        ns: .metadata.namespace,
        pod: .metadata.name,
        containers: ([.spec.containers[], (.spec.initContainers[]?)]),
        volumes: .spec.volumes
      }
    | {
        ns,
        pod,
        envSecrets: (
          [ .containers[]
            | .env[]? 
            | select(.valueFrom.secretKeyRef?)
            | .valueFrom.secretKeyRef.name
          ] | unique
        ),
        envFromSecrets: (
          [ .containers[]
            | .envFrom[]?
            | select(.secretRef?)
            | .secretRef.name
          ] | unique
        ),
        volumeSecrets: (
          [ .volumes[]
            | select(.secret?)
            | .secret.secretName
          ] | unique
        )
      }
    | select((.envSecrets + .envFromSecrets + .volumeSecrets) | length > 0)
    | [
        .ns,
        .pod,
        ( ( .envSecrets // [] ) | join(",") | if . == "" then "-" else . end ),
        ( ( .envFromSecrets // [] ) | join(",") | if . == "" then "-" else . end ),
        ( ( .volumeSecrets // [] ) | join(",") | if . == "" then "-" else . end )
      ]
    | @tsv' \
  | column -t \
  | sed '1iNAMESPACE  POD  ENV_SECRETREFS  ENVFROM_SECRETREFS  VOLUME_SECRETS'

echo
echo "=== ConfigMaps that embed data with names suggesting secrets/credentials ==="
kubectl get configmaps --all-namespaces -o json \
  | jq -r '
    .items[]
    | {
        ns: .metadata.namespace,
        name: .metadata.name,
        keys: (.data | keys? // [])
      }
    | select(.keys | length > 0)
    | select(
        ([.keys[]]
          | map( ascii_downcase
                 | test("password|passwd|pass|secret|token|apikey|api_key|key|cert|certificate")
               )
          | any
        )
      )
    | [.ns, .name, (.keys | join(","))]
    | @tsv' \
  | column -t \
  || echo "No suspicious ConfigMaps detected."

echo
echo "=== Namespaces with signs of external secret managers already in use ==="
echo "--- Looking for common CRDs and controllers (external-secrets.io, secrets-store.csi.x-k8s.io, etc.) ---"

echo "- CustomResourceDefinitions related to external secrets:"
kubectl get crd 2>/dev/null | grep -E 'externalsecret|secrets-store|vault|onepassword' || \
  echo "  None of the common external-secret CRDs found."

echo
echo "- Pods in namespaces commonly used by external secret operators:"
kubectl get pods -A \
  | grep -Ei 'external-secret|external.secrets|secrets-store|vault|hashicorp-vault|1password|onepassword' \
  || echo "  No obvious external secret controllers found."

echo
echo "=== SUMMARY: manual interpretation guidance ==="
cat <<'EOF'
Interpretation:

1) High native secret usage
   - Many non-service-account Secrets (see "Count of secrets per namespace") and
     many workloads in the "Workloads that mount or reference native Kubernetes Secrets"
     section indicate heavy reliance on Kubernetes Secrets.
   - That is a SIGNAL to review whether these should instead come from an external
     secrets manager per CIS 5.4.2.

2) Riskier patterns
   Treat the following as *higher concern* and candidates for externalization:
   - Namespaces with:
     - Large counts of Secrets holding long-lived app credentials (DB passwords,
       API keys, tokens, private keys).
   - Workloads where:
     - Secrets are mounted widely across many pods or namespaces.
     - The same sensitive Secret is reused across multiple apps.
   - ConfigMaps that appear to hold passwords/tokens (see the ConfigMap section);
     those should almost always be moved to a secrets manager.

3) Signs external secret storage is already in use
   - If you see CRDs like "externalsecrets.external-secrets.io" or
     "secretproviderclasses.secrets-store.csi.x-k8s.io", or controllers such as
     external-secrets, secrets-store-csi-driver, or vault agents, your cluster
     may already be integrated with an external secrets system.
   - In that case, focus review on namespaces and workloads that still rely
     primarily on native Kubernetes Secrets.

This script does NOT implement an automatic fix.
Use the output to:
- Identify high-value / high-risk secrets.
- Decide which should move to an external secrets manager offered by your cloud
  provider or a third party, in line with CIS 5.4.2.
EOF

What output indicates a problem (for review):

Large counts of non-service-account Secrets in critical namespaces (e.g., default, app namespaces) suggest widespread storage of app credentials in native Secrets.
Many pods showing populated ENV_SECRETREFS, ENVFROM_SECRETREFS, or VOLUME_SECRETS columns mean extensive direct use of Kubernetes Secrets instead of an external manager.
ConfigMaps listed in the “ConfigMaps that embed data…” section likely contain secrets in plain text and are strong candidates for migration to an external secrets manager.
Absence of any external-secrets-related CRDs or controllers plus heavy native secret use suggests the cluster relies solely on native Secrets and should be evaluated for adoption of an external secrets manager.

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​