> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Consider External Secret Storage

### More Info:

Consider the use of an external secrets storage and management system, instead of using Kubernetes Secrets directly, if you have more complex secret management needs. Ensure the solution requires authentication to access secrets, has auditing of access to and use of secrets, and encrypts secrets. Some solutions also make it easier to rotate secrets.

### Risk Level

Medium

### Address

Security

### Compliance Standards

* CIS Kubernetes

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Manual Steps" defaultOpen="true">
        1. **Inventory current Secret usage**\
           Run on any machine with kubectl access:
           ```bash theme={null}
           kubectl get secrets -A
           kubectl get secrets -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.type}{"\n"}{end}' | sort
           ```
           Identify which applications store sensitive data (passwords, tokens, keys) directly in Kubernetes Secrets.

        2. **Identify how applications consume Secrets**\
           Run on any machine with kubectl access:
           ```bash theme={null}
           kubectl get deploy,sts,ds -A -o yaml > /tmp/workloads.yaml
           grep -nE 'secretKeyRef|secretRef|secretName' /tmp/workloads.yaml
           ```
           For workloads using Secrets, note whether they mount them as environment variables or volumes and how many distinct secrets each app depends on.

        3. **Assess complexity and compliance requirements**\
           For a representative set of applications (e.g., by namespace or criticality), review:
           * Frequency of secret changes/rotations (ask app owners or check change history in Git/IaC).
           * Regulatory or internal requirements for:
             * Strong auditing of secret access
             * Centralized key management
             * Automated or frequent rotation\
               Decide whether built‑in Kubernetes Secrets (with at-rest encryption enabled) are sufficient or whether centralized external management is required.

        4. **Evaluate existing external secret solutions in your environment**\
           Check if an external manager is already in use (examples: AWS Secrets Manager, AWS KMS + Parameter Store, GCP Secret Manager, Azure Key Vault, HashiCorp Vault):
           ```bash theme={null}
           # Look for common external-secrets operators or integrations
           kubectl get pods -A | grep -Ei 'external-secret|externalsecret|vault|secrets-manager|secret-manager|akeyless|doppler'
           kubectl get crds | grep -Ei 'externalsecret|secretstore|clustersecretstore'
           ```
           If such components exist, review their configuration and confirm they provide: authentication, encryption, and auditable access logs.

        5. **Decide and design the target pattern**\
           Based on steps 1–4, choose per application (or namespace):
           * Continue using Kubernetes Secrets (if complexity/compliance needs are low), ensuring:
             * Encryption at rest is enabled at the cluster/etcd level.
             * Access is restricted via RBAC.
           * Or adopt an external secret manager, typically by:
             * Configuring a SecretStore/ClusterSecretStore or similar CRD for your provider.
             * Mapping external secret entries to Kubernetes Secrets consumed by workloads.\
               Document the chosen approach and migration plan (which apps, in which order).

        6. **Implement and verify the chosen solution**\
           For one pilot application, implement your chosen pattern (for external manager, deploy the integration operator/agent and define the mapping manifests). Then verify:
           ```bash theme={null}
           # Confirm secrets are now sourced via the external mechanism (if used)
           kubectl get externalsecrets.secret-store.io -A 2>/dev/null || echo "No ExternalSecret CRs detected"
           # Confirm workloads still receive the necessary Secrets
           kubectl describe pod -n <namespace> <pod-name> | grep -A5 -E 'Mounts:|Environment:'
           ```
           Separately, in your cloud provider console or external secrets system, confirm: authentication is required, access is logged/audited, and secrets are encrypted and can be rotated.
      </Accordion>

      <Accordion title="Using kubectl">
        ```bash theme={null}
        # 1) List all namespaces
        # Run on: any machine with kubectl access
        kubectl get namespaces
        ```

        Review which namespaces are in use for production or sensitive workloads.

        ```bash theme={null}
        # 2) List all Secrets in each namespace
        # Run once per namespace of interest
        kubectl get secrets -n default
        kubectl get secrets -n kube-system
        kubectl get secrets -n production
        ```

        **Problem indication:** Many `Opaque` or `kubernetes.io/dockerconfigjson` secrets in app namespaces may mean application credentials are stored only in Kubernetes Secrets.

        ```bash theme={null}
        # 3) Inspect individual Secrets (metadata only)
        kubectl get secret my-secret -n production -o yaml
        ```

        Review:

        * `type:` (commonly `Opaque` for app secrets)
        * `metadata.annotations` and `labels` (look for external-secrets operators, e.g. `external-secrets.io/*`, `vault.hashicorp.com/*`, `secrets-store.csi.x-k8s.io/*`)

        **Problem indication:** Sensitive app secrets of type `Opaque` with no annotations/labels tying them to an external manager likely come only from in-cluster storage.

        ```bash theme={null}
        # 4) Check for use of external secret controllers
        kubectl get crd | grep -iE 'secret|vault|external|csi'
        ```

        Common indicators:

        * `externalsecrets.external-secrets.io`
        * `secretproviderclasses.secrets-store.csi.x-k8s.io`
        * `vaultsecrets.*`, etc.

        **Problem indication:** No CRDs or resources related to external secrets providers may indicate the cluster relies entirely on native Kubernetes Secrets.

        ```bash theme={null}
        # 5) List known external secret resources (if present)
        kubectl get externalsecrets -A 2>/dev/null
        kubectl get secretproviderclasses -A 2>/dev/null
        ```

        **Problem indication:** Application namespaces with Secrets but no `ExternalSecret` or `SecretProviderClass` resources suggest no external system is backing those secrets.

        ```bash theme={null}
        # 6) Identify Secrets likely used for application credentials
        kubectl get secrets -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.type}{"\n"}{end}' \
          | grep -E 'Opaque|kubernetes.io/basic-auth|kubernetes.io/ssh-auth'
        ```

        **Problem indication:** Large numbers of such secrets in app namespaces, without any external-secret mechanism in the cluster, indicate a stronger case to consider external secret storage.

        ```bash theme={null}
        # 7) Spot-check Pods that mount/use Secrets
        kubectl get pods -n production -o yaml | grep -nE 'secretRef|secretName'
        ```

        **Problem indication:** Many Pods directly consuming `Opaque` Secrets that are not clearly sourced from an external manager points to a need to evaluate external secret storage.

        Use these outputs to decide:

        * Whether secrets are currently managed only as native Kubernetes Secrets.
        * Whether that is acceptable for your risk profile, or whether you should adopt an external secrets system as the benchmark recommends.
      </Accordion>

      <Accordion title="Automation">
        ```bash theme={null}
        #!/usr/bin/env bash
        set -euo pipefail

        # This script runs on: any machine with kubectl access and appropriate RBAC.
        # It reports where and how native Kubernetes Secrets are used, so you can
        # review what should be migrated to an external secrets manager.

        # 1) List all Secrets (namespaced and cluster-scoped fields)
        echo "=== ALL SECRETS (namespace, type, name, age, size-of-data-keys) ==="
        kubectl get secrets --all-namespaces -o json \
          | jq -r '
            .items[]
            | [
                .metadata.namespace,
                .type,
                .metadata.name,
                (now - (.metadata.creationTimestamp | fromdate)) as $age,
                ($age/86400|floor|tostring + "d"),
                ( .data | keys | length ) as $keys,
                ($keys|tostring)
              ]
            | @tsv
          ' 2>/dev/null \
          | awk 'BEGIN{OFS="\t"; print "NAMESPACE","TYPE","NAME","AGE","DATA_KEY_COUNT"} {print $1,$2,$3,$5,$6}'
        echo

        # 2) Highlight Secrets that are very likely to hold sensitive credentials
        # (these warrant special review for moving to an external manager)
        echo "=== POTENTIALLY SENSITIVE SECRETS (by common name patterns) ==="
        kubectl get secrets --all-namespaces -o json \
          | jq -r '
            .items[]
            | select(
                (
                  .metadata.name
                  | test("(?i)(password|passwd|secret|token|key|cert|tls|ssh|api|auth)")
                )
                or
                (
                  (.type|tostring)
                  | test("(?i)(tls|dockercfg|dockerconfigjson|basic|ssh)")
                )
              )
            | [.metadata.namespace, .type, .metadata.name]
            | @tsv
          ' 2>/dev/null \
          | awk 'BEGIN{OFS="\t"; print "NAMESPACE","TYPE","NAME"} {print $1,$2,$3}'
        echo

        # 3) Find workloads that mount or reference Secrets directly
        echo "=== WORKLOADS USING SECRETS (envFrom, env.valueFrom, volumes.secret) ==="
        for kind in deployments statefulsets daemonsets jobs cronjobs replicasets pods; do
          kubectl get "$kind" --all-namespaces -o json 2>/dev/null \
            | jq -r --arg KIND "$kind" '
              .items[]?
              | . as $obj
              | ($obj.spec.template // $obj) as $tpl
              | [
                  $obj.metadata.namespace,
                  $KIND,
                  $obj.metadata.name,
                  (
                    ($tpl.spec.containers // []) as $cs
                    | ($tpl.spec.initContainers // []) as $ics
                    | ($cs + $ics)
                    | map(
                        [
                          .name,
                          (
                            ((.envFrom // [])[]?.secretRef.name? // empty),
                            ((.env // [])[]?.valueFrom.secretKeyRef.name? // empty)
                          ) | unique | join(",")
                        ] | @tsv
                      )
                    | unique
                    | join(";")
                  ),
                  (
                    ($tpl.spec.volumes // [])
                    | map(select(.secret != null) | .secret.secretName)
                    | unique
                    | join(",")
                  )
                ]
              | @tsv
            ' 2>/dev/null
        done \
          | awk 'BEGIN{
              OFS="\t";
              print "NAMESPACE","KIND","WORKLOAD","CONTAINERS->SECRET_ENV","VOLUME_SECRETS";
            } {print $1,$2,$3,$4,$5}'
        echo

        # 4) Summarize Secrets usage by namespace to show “hot spots”
        echo "=== SUMMARY: SECRETS COUNT BY NAMESPACE ==="
        kubectl get secrets --all-namespaces \
          | awk 'NR>1 {cnt[$1]++} END {OFS="\t"; print "NAMESPACE","SECRET_COUNT"; for (ns in cnt) print ns,cnt[ns]}' \
          | sort -k2,2nr
        echo

        # 5) Optional: show if EncryptionConfiguration is configured for Secrets (best-effort)
        # NOTE: This only works if you can access the API server manifest on control plane nodes.
        # Here we only check the API server config seen by kubectl (may be empty on managed control planes).
        echo "=== NOTE: Encryption at rest for Secrets must be verified on control-plane hosts ==="
        echo "Run on every control plane node (over SSH), for example:"
        echo "  sudo grep -E \"encryption-provider-config\" /etc/kubernetes/manifests/kube-apiserver.yaml || true"
        echo

        echo "=== INTERPRETING OUTPUT ==="
        cat <<'EOF'
        Output indicating higher priority for external secret storage review:

        1) ALL SECRETS:
           - Namespaces with a high SECRET_COUNT (in the summary) or many Secrets of type:
               * Opaque
               * kubernetes.io/basic-auth
               * kubernetes.io/ssh-auth
               * kubernetes.io/tls
               * kubernetes.io/dockerconfigjson
           suggest heavier reliance on native Secrets.

        2) POTENTIALLY SENSITIVE SECRETS:
           - Any Secret listed here likely contains credentials or keys.
           - These are prime candidates for moving to an external secrets manager that:
               * Requires strong authentication
               * Audits access and use
               * Encrypts data and supports rotation

        3) WORKLOADS USING SECRETS:
           - Workloads with many SECRET_ENV or VOLUME_SECRETS entries are tightly coupled
             to Kubernetes Secrets.
           - Namespaces where most workloads consume such Secrets should be evaluated for
             integration with an external secrets solution (e.g., syncing or injecting
             from a cloud/third-party secrets manager instead of manual Secret objects).

        This script does not apply any changes. Use it to:
           - Identify which secrets and namespaces should move first.
           - Scope the impact of introducing an external secrets manager.
        EOF
        ```
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>