> ## Documentation Index > Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Consider External Secret Storage ### More Info: Storing secrets in an external, dedicated secrets manager reduces exposure compared to native Kubernetes secrets. Evaluate cloud provider or third-party secrets management solutions. ### Risk Level Medium ### Address Security ### Compliance Standards * APRA CPS 234 (Australia) * AWS Startup Security Baseline * BSI C5 (Germany) * Brazil LGPD * CCPA / CPRA (California) * CIS Critical Security Controls v8 * CIS EKS * CMMC 2.0 * CSA Cloud Controls Matrix v4 * DPDPA * Digital Operational Resilience Act (EU) * Essential 8 * ISO/IEC 27017 * ISO/IEC 27018 * ISO/IEC 27701 * KSA PDPL * MAS Technology Risk Management (Singapore) * MITRE ATT\&CK (Cloud) * NIS2 Directive * NIST SP 800-171 * NYDFS 23 NYCRR 500 * SWIFT Customer Security Controls Framework * Sarbanes-Oxley IT General Controls * UK NCSC Cyber Assessment Framework ### Triage and Remediation ### Remediation 1. **Inventory current Kubernetes Secrets usage**\ Run these on any machine with kubectl access: ```bash theme={null} # List all namespaces kubectl get ns # List all Secrets per namespace (type + age only) for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do echo "=== Namespace: $ns ===" kubectl get secrets -n "$ns" -o wide done ``` If allowed in your environment, spot-check contents (never export broadly in shared terminals/logs): ```bash theme={null} # Inspect a specific Secret (base64-encoded data) kubectl get secret -n -o yaml ``` 2. **Identify workloads depending on Kubernetes Secrets** ```bash theme={null} # Find Pods that reference Secrets in env / volume mounts for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do echo "=== Namespace: $ns ===" kubectl get pods -n "$ns" -o json | \ jq -r '.items[] | {ns: .metadata.namespace, pod: .metadata.name, sa: .spec.serviceAccountName, envSecrets: ([.spec.containers[].env[]? | select(.valueFrom.secretKeyRef).valueFrom.secretKeyRef.name] | unique), volSecrets: ([.spec.volumes[]? | select(.secret).secret.secretName] | unique)} | select((.envSecrets|length>0) or (.volSecrets|length>0))' done ``` Record which applications, namespaces, and service accounts rely on Secrets, and which specific secrets are high sensitivity (DB creds, API keys, encryption keys, etc.). 3. **Evaluate current secret protection controls**\ On any machine with kubectl access, check if secret data at rest is protected (for managed EKS, review in cloud console/IaC if envelope encryption for secrets is enabled and with which KMS key). For cluster-scoped view: ```bash theme={null} # Check if Secrets are encrypted at rest via EncryptionConfiguration (for self-managed control planes) kubectl get pods -n kube-system -l component=kube-apiserver -o yaml | \ grep -E -- '--encryption-provider-config|--encryption-provider-config-file' || echo "Check in control-plane config / cloud console" ``` In your cloud console / IaC, review: * Whether Kubernetes Secrets are encrypted with a customer-managed KMS key. * Who can access that KMS key (IAM roles/policies). * Who can list/get Secrets via cluster RBAC. 4. **Assess need and options for external secret storage**\ Based on steps 1–3, decide which secrets should move to an external manager (e.g., all production / highly sensitive secrets). For your environment, review in cloud console/IaC: * Availability of cloud-native secret managers (e.g., AWS Secrets Manager / Parameter Store, GCP Secret Manager, Azure Key Vault, HashiCorp Vault). * Native integrations or operators for Kubernetes (e.g., External Secrets Operator, Secrets Store CSI Driver). * Compliance or audit requirements that may mandate external secret storage or HSM-backed keys. 5. **Design the integration pattern and update manifests**\ Choose one pattern (examples, adapt to your platform and tooling): * **Secrets Store CSI Driver / external secrets operator**: pods mount secrets from external store as volumes or sync to Kubernetes Secrets. * **Sidecar / init container** that fetches secrets from the external manager and writes them to a tmpfs volume.\ For the chosen pattern, update deployment manifests to stop embedding static credentials and instead reference the external source. Example (CSI driver style – conceptual, not a one-size-fits-all fix): ```yaml theme={null} # Example Pod volume using a secret store CSI driver volumes: - name: app-secrets csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: app-secrets-spc ``` Apply updated manifests from a machine with kubectl access: ```bash theme={null} kubectl apply -f .yaml ``` 6. **Verify and decommission in-cluster secrets where appropriate**\ a. Verify workloads now retrieve secrets from the external store and still function: ```bash theme={null} # Check Pod status and events kubectl get pods -A kubectl describe pod -n ``` b. Confirm that application pods no longer list the previous Kubernetes Secrets in env or volume mounts: ```bash theme={null} kubectl get pod -n -o json | \ jq '.spec.containers[].env?[], .spec.volumes?[]' | grep -i secret || echo "No direct Secret refs found (verify manually if needed)" ``` c. After confirming safe cutover and rollback plan, carefully delete or reduce high-sensitivity Kubernetes Secrets that are no longer needed: ```bash theme={null} kubectl delete secret -n ``` Document which applications now depend on the external secrets manager and ensure access to the external secrets is governed via cloud/IAM policies instead of (or in addition to) in-cluster RBAC. ```bash theme={null} # 1) List all namespaces # Run on: any machine with kubectl access kubectl get namespaces ``` Review all non-system namespaces (ignore kube-system, kube-public, kube-node-lease). ```bash theme={null} # 2) For each relevant namespace, list all Secrets and how they are used # Run on: any machine with kubectl access NAMESPACE=default # List secrets (type and age give quick hints) kubectl get secrets -n "${NAMESPACE}" -o wide # Show detailed info for each secret (excluding the value) kubectl get secrets -n "${NAMESPACE}" -o yaml # Show which workloads reference secrets via env / envFrom / volumes kubectl get deploy,sts,ds,job,cronjob -n "${NAMESPACE}" -o yaml | \ grep -nE 'secretKeyRef|secretRef|secretName' -A3 -B3 ``` Indicators of higher risk from this output: * Large numbers of application-owned `Opaque` secrets (not just service account or TLS certs). * Secrets holding long‑lived credentials (database users, API keys, cloud access keys). * Same secret reused across many workloads or namespaces. * Secrets embedded as env vars in many pods (easy to leak via logs, debug, etc.). ```bash theme={null} # 3) Identify secrets that look like external system credentials # Run on: any machine with kubectl access NAMESPACE=default # Show secret names only, to focus the review kubectl get secrets -n "${NAMESPACE}" -o custom-columns=NAME:.metadata.name,TYPE:.type # For specific secrets that might be external creds, inspect metadata only SECRET_NAME=my-db-credentials kubectl get secret "${SECRET_NAME}" -n "${NAMESPACE}" -o yaml | \ sed '/^data:/q' ``` Human review guidance: * Names like `*-db-password`, `*-api-key`, `*-aws-credentials`, `*-token` usually indicate external system secrets that are good candidates for an external manager. * If those secrets are long‑lived and manually rotated, that strengthens the case for an external secrets manager. ```bash theme={null} # 4) Check for cluster‑wide patterns (where external manager would help most) # Run on: any machine with kubectl access # Count secrets by namespace kubectl get secrets --all-namespaces \ -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.type | \ sort | uniq -c | sort -nr | head # List all Opaque secrets across all namespaces (excluding well-known system ones) kubectl get secrets --all-namespaces \ -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.type | \ grep -vE 'kube-system|kube-public|kube-node-lease' | \ grep 'Opaque' ``` Problem indicators in this cluster-wide view: * Many `Opaque` secrets in application namespaces storing business‑critical or external credentials. * Same‑name secrets across namespaces that obviously mirror the same external system (e.g., `db-credentials` in many namespaces) instead of being sourced from a dedicated secrets manager. * No evidence of any integration with an external secrets operator (e.g., no CRDs like `externalsecret`), suggesting everything is native Kubernetes secrets. ```bash theme={null} # 5) Check if an external secrets solution is already in use # Run on: any machine with kubectl access # Look for common CRDs used by external secret managers kubectl get crds | grep -iE 'secret|vault|externalsecret|external-secrets|secrets-store' # Look for known operators in all namespaces kubectl get pods --all-namespaces | grep -iE 'secret|vault|external|secrets-store' || true ``` Interpretation: * If there are **no** CRDs or controllers related to external secrets and you see many high‑sensitivity `Opaque` secrets as described above, that is a strong signal that external secret storage should be considered. * If external secret tooling is present, verify whether the high‑sensitivity secrets identified earlier are actually sourced from it or still managed as plain Kubernetes secrets. ```bash theme={null} #!/usr/bin/env bash # # cis-eks-4.4.2-secrets-usage-report.sh # # Run on: any machine with kubectl access and correct context # # This script does NOT fix anything. It summarizes how secrets are used so you # can decide whether external secret storage should be adopted or expanded. # # It focuses on: # - Count and age of native Kubernetes Secrets # - Which types are used (Opaque, kubernetes.io/dockerconfigjson, tls, etc.) # - Workloads that project secrets as env vars or volumes # - Namespaces with heavy secret use # # Interpretation guidance is at the end of this file. set -euo pipefail # Ensure kubectl works if ! kubectl version --short >/dev/null 2>&1; then echo "ERROR: kubectl is not configured or cannot reach the cluster" >&2 exit 1 fi echo "=== [1] Cluster-wide summary of Kubernetes Secret objects ===" kubectl get secrets --all-namespaces -o json \ | jq -r ' .items | ( "NAMESPACE,NAME,TYPE,CREATED,AGE_DAYS,DATA_KEYS" ), ( .[] | . as $s | .metadata.namespace as $ns | .metadata.name as $name | .type as $type | .metadata.creationTimestamp as $created | ( (now - ($created | fromdate)) / 86400 | floor ) as $age_days | ($s.data // {} | keys | join("|")) as $keys | [$ns, $name, $type, $created, ($age_days|tostring), $keys] | @csv ) ' | column -s, -t echo echo "=== [2] Secret counts per namespace and type ===" kubectl get secrets --all-namespaces -o json \ | jq -r ' .items | group_by(.metadata.namespace)[] as $nsGroup | ($nsGroup[0].metadata.namespace) as $ns | ($nsGroup | group_by(.type)[] | [ $ns, .[0].type, (length | tostring) ] | @tsv ) ' | awk 'BEGIN { printf "%-30s %-40s %s\n", "NAMESPACE", "TYPE", "COUNT" } { printf "%-30s %-40s %s\n", $1, $2, $3 }' echo echo "=== [3] Top namespaces by number of Secret objects ===" kubectl get secrets --all-namespaces \ | awk 'NR>1 {count[$1]++} END { for (ns in count) print count[ns], ns }' \ | sort -nr \ | head -20 \ | awk 'BEGIN { printf "%-30s %s\n", "NAMESPACE", "SECRET_COUNT" } { printf "%-30s %s\n", $2, $1 }' echo echo "=== [4] Workloads using Secrets as environment variables ===" echo "--- Deployments ---" kubectl get deploy --all-namespaces -o json \ | jq -r ' .items[] | .metadata.namespace as $ns | .metadata.name as $name | (.spec.template.spec.containers // [])[]? as $c | ($c.env // [])[]? as $e | select($e.valueFrom.secretKeyRef != null) | [ $ns, $name, $c.name, $e.name, $e.valueFrom.secretKeyRef.name, $e.valueFrom.secretKeyRef.key ] | @tsv ' 2>/dev/null \ | awk 'BEGIN { printf "%-30s %-40s %-30s %-25s %-40s %s\n", "NAMESPACE", "DEPLOYMENT", "CONTAINER", "ENV_VAR", "SECRET_NAME", "SECRET_KEY" } { printf "%-30s %-40s %-30s %-25s %-40s %s\n", $1, $2, $3, $4, $5, $6 }' echo echo "--- StatefulSets ---" kubectl get statefulset --all-namespaces -o json \ | jq -r ' .items[] | .metadata.namespace as $ns | .metadata.name as $name | (.spec.template.spec.containers // [])[]? as $c | ($c.env // [])[]? as $e | select($e.valueFrom.secretKeyRef != null) | [ $ns, $name, $c.name, $e.name, $e.valueFrom.secretKeyRef.name, $e.valueFrom.secretKeyRef.key ] | @tsv ' 2>/dev/null \ | awk 'BEGIN { printf "%-30s %-40s %-30s %-25s %-40s %s\n", "NAMESPACE", "STATEFULSET", "CONTAINER", "ENV_VAR", "SECRET_NAME", "SECRET_KEY" } { printf "%-30s %-40s %-30s %-25s %-40s %s\n", $1, $2, $3, $4, $5, $6 }' echo echo "--- DaemonSets ---" kubectl get daemonset --all-namespaces -o json \ | jq -r ' .items[] | .metadata.namespace as $ns | .metadata.name as $name | (.spec.template.spec.containers // [])[]? as $c | ($c.env // [])[]? as $e | select($e.valueFrom.secretKeyRef != null) | [ $ns, $name, $c.name, $e.name, $e.valueFrom.secretKeyRef.name, $e.valueFrom.secretKeyRef.key ] | @tsv ' 2>/dev/null \ | awk 'BEGIN { printf "%-30s %-40s %-30s %-25s %-40s %s\n", "NAMESPACE", "DAEMONSET", "CONTAINER", "ENV_VAR", "SECRET_NAME", "SECRET_KEY" } { printf "%-30s %-40s %-30s %-25s %-40s %s\n", $1, $2, $3, $4, $5, $6 }' echo echo "=== [5] Workloads mounting Secrets as volumes ===" for kind in deployment statefulset daemonset; do kind_uc=$(echo "$kind" | tr '[:lower:]' '[:upper:]') echo "--- ${kind_uc}s ---" kubectl get "$kind" --all-namespaces -o json \ | jq -r ' .items[] | .metadata.namespace as $ns | .metadata.name as $name | (.spec.template.spec.volumes // [])[] | select(.secret != null) | [ $ns, $name, .name, .secret.secretName ] | @tsv ' 2>/dev/null \ | awk -v KIND="$kind_uc" 'BEGIN { printf "%-30s %-40s %-30s %s\n", "NAMESPACE", KIND, "VOLUME_NAME", "SECRET_NAME" } { printf "%-30s %-40s %-30s %s\n", $1, $2, $3, $4 }' echo done echo "=== [6] Pods directly referencing Secrets (non-controller-managed) ===" kubectl get pods --all-namespaces -o json \ | jq -r ' .items[] | select(.metadata.ownerReferences == null or (.metadata.ownerReferences | length == 0)) | .metadata.namespace as $ns | .metadata.name as $name | ( [(.spec.volumes // [])[]? | select(.secret != null) | ["volume", .name, .secret.secretName]], [(.spec.containers // [])[]? as $c | ($c.env // [])[]? | select(.valueFrom.secretKeyRef != null) | ["env", $c.name, .valueFrom.secretKeyRef.name] ] )[] | [ $ns, $name, .[0], .[1], .[2] ] | @tsv ' 2>/dev/null \ | awk 'BEGIN { printf "%-30s %-40s %-10s %-30s %s\n", "NAMESPACE", "POD", "REF_TYPE", "CONTAINER/VOLUME", "SECRET_NAME" } { printf "%-30s %-40s %-10s %-30s %s\n", $1, $2, $3, $4, $5 }' echo echo "================================================================================" echo "INTERPRETING THIS OUTPUT (WHAT MAY INDICATE A PROBLEM)" echo cat <<'EOF' This control is MANUAL: there is no one correct target state. Use the data above to decide if you should adopt or expand an external secrets manager (cloud-native or third-party): Signs you may want to move more to external secret storage: - [1]/[2]/[3]: Large numbers of Opaque secrets, especially: - With very old AGE_DAYS (stale or long-lived credentials). - In many different namespaces without consistent naming or ownership. - [4]: Applications using many env vars from secrets: - Env vars are easily exposed via "kubectl describe pod" and logs. - Consider replacing sensitive values with references to an external secrets manager via a sidecar, CSI driver, or operator. - [5]/[6]: Workloads mounting large or generic secrets (e.g. "app-config", "credentials") that likely mix different secret values instead of using a dedicated external store. This script only reports usage. To remediate, you must: - Select and configure a cloud provider or third-party secrets manager. - Redesign workloads to fetch or project secrets from that system. EOF ```