Skip to main content

More Info:

Storing secrets in an external, dedicated secrets manager can improve protection over native Kubernetes Secrets. Evaluate such options.

Risk Level

Informational

Address

Security

Compliance Standards

  • CIS Kubernetes

Triage and Remediation

Remediation

Manual Steps

  1. Inventory how Kubernetes Secrets are currently used
    • On any machine with kubectl access, list all Secret objects and their types:
    • Identify workloads that mount or reference these Secrets:
    • Note any use of type kubernetes.io/basic-auth, kubernetes.io/ssh-auth, database creds, API keys, tokens, etc., which are strong candidates for external storage.
  2. Assess in-cluster Secret protection and sensitivity
    • Check if Encryption at Rest is enabled for Secrets (control plane–specific; if you do not manage it, retrieve documentation/config from your provider or IaC):
      • For self-managed clusters, on every control plane node:
    • Classify secrets by sensitivity (e.g., “high” for long-lived credentials to production systems, “medium” for non-production, “low” for ephemeral tokens) and record which namespaces and apps depend on them.
  3. Review available external secrets managers and cluster integration options
    • From your cloud console or IaC, identify supported secrets managers and their Kubernetes integrations, for example (depending on environment):
      • AWS: AWS Secrets Manager / SSM Parameter Store + External Secrets Operator or Secrets Store CSI Driver.
      • GCP: Secret Manager + Secret Manager CSI driver / external-secrets.
      • Azure: Key Vault + Secrets Store CSI Driver / external-secrets.
    • Confirm organizational standards or regulatory requirements that may mandate a particular secrets manager or key management scheme (e.g., use of HSM-backed keys, centralized audit, rotation policies).
  4. Decide scope and pattern for migration to external secret storage
    • Choose a preferred integration pattern (e.g., External Secrets Operator with CRDs like ExternalSecret, or CSI driver volume mounts) based on your platform.
    • Select a pilot scope: one or a few high-sensitivity applications and their Secrets (from step 1) to migrate first.
    • For those pilot apps, map each existing Secret to a candidate external secret path/name in the chosen manager.
  5. Implement and test external secrets for a pilot workload
    • Using cloud console or IaC, create corresponding secrets in the external manager for the pilot application, ensuring:
      • Access control (IAM/RBAC) only allows the Kubernetes cluster/namespace/service account to read them.
      • Rotation policy is defined where supported.
    • On any machine with kubectl access, deploy the chosen integration (operator or CSI driver) according to its vendor documentation, then define a test manifest that consumes the external secret (example pattern only, adjust to your tool):
    • Verify the pod(s) receive correct secret data and the original Kubernetes Secret is no longer required for that workload.
  6. Plan progressive rollout and deprecation of native Secrets where appropriate
    • For remaining high- and medium-sensitivity Secrets, repeat the mapping and migration process, updating manifests in your IaC to source data from the external store instead of embedding values into Kubernetes Secrets.
    • Verify after each migration that no workloads still depend on the old Secrets:
    • Once confirmed, delete or minimize high-sensitivity native Secrets and document the new standard: which classes of secrets must be stored in the external manager, and which (if any) may remain as Kubernetes Secrets.
What to look for in the output (indicates a problem / candidate for external storage):
  • type: Opaque secrets containing:
    • Long-lived application credentials (database users/passwords, API keys, TLS private keys not associated with short-lived certs).
    • Cloud provider access keys, tokens, or passwords to external services.
  • Secrets with generic names but clearly sensitive values, for example:
    • db-credentials, payment-gateway-key, smtp-password, aws-credentials, gcp-service-account, azure-sp-credentials.
  • Secrets referenced broadly across many workloads (suggesting high blast radius).
In the above output, potential issues:
  • env: or envFrom: entries that reference sensitive secrets via valueFrom.secretKeyRef.
  • volumes: with secret: sources that mount sensitive secrets to many pods.
  • Same secret name used across many different applications/namespaces (central, critical credential).
If this surfaces secrets whose values (from step 3) are:
  • Long-lived,
  • Grant access to external or critical systems,
  • Needed by many different workloads,
then they are strong candidates to be moved to an external secrets manager, with Kubernetes only holding references or short-lived material.
Potential concern:
  • TLS private keys or CA roots that are long-lived and reused across multiple services or ingresses; those are often better managed in a dedicated secrets/PKI system.
Next step (manual decision):
  • Based on the above inspection, decide which secrets should remain as native Kubernetes Secrets (low-risk, internal-only, non-critical), and which should instead be sourced from an external secrets manager per your cloud provider or chosen third-party solution.
What output indicates a problem (for review):
  • Large counts of non-service-account Secrets in critical namespaces (e.g., default, app namespaces) suggest widespread storage of app credentials in native Secrets.
  • Many pods showing populated ENV_SECRETREFS, ENVFROM_SECRETREFS, or VOLUME_SECRETS columns mean extensive direct use of Kubernetes Secrets instead of an external manager.
  • ConfigMaps listed in the “ConfigMaps that embed data…” section likely contain secrets in plain text and are strong candidates for migration to an external secrets manager.
  • Absence of any external-secrets-related CRDs or controllers plus heavy native secret use suggests the cluster relies solely on native Secrets and should be evaluated for adoption of an external secrets manager.