Consider External Secret Storage

More Info:

Consider the use of an external secrets storage and management system, instead of using Kubernetes Secrets directly, if you have more complex secret management needs.

Risk Level

Medium

Address

Security

Compliance Standards

APRA CPS 234 (Australia)
BSI C5 (Germany)
Brazil LGPD
CCPA / CPRA (California)
CIS Critical Security Controls v8
CIS OKE
CMMC 2.0
CSA Cloud Controls Matrix v4
DPDPA
Digital Operational Resilience Act (EU)
Essential 8
ISO/IEC 27017
ISO/IEC 27018
ISO/IEC 27701
KSA PDPL
MAS Technology Risk Management (Singapore)
MITRE ATT&CK (Cloud)
NIS2 Directive
NIST SP 800-171
NYDFS 23 NYCRR 500
SWIFT Customer Security Controls Framework
Sarbanes-Oxley IT General Controls
UK NCSC Cyber Assessment Framework

Triage and Remediation

Remediation

Remediation

Manual Steps

Inventory current Kubernetes Secrets usage
- Machine: any machine with kubectl access
- Command:
```
kubectl get secrets -A
```
- Identify which namespaces and applications store sensitive data (passwords, API keys, tokens, private keys) in native Kubernetes Secrets.
Check if OCI Vault secret encryption is enabled for the cluster
- This is configured at cluster creation/through the Oracle Cloud Console/CLI, not via kubectl.
- In the Oracle Cloud Console, navigate to: Developer Services → Kubernetes Clusters (OKE) → your cluster → Security / Encryption settings, and confirm whether “Encrypt Kubernetes Secrets with OCI Vault” (or equivalent) is enabled and which Vault/Key is used.
- If you manage the cluster via Terraform/CLI, review the cluster resource for kms_key_id or equivalent Vault configuration.
Assess sensitivity and complexity of current secrets
- Machine: any machine with kubectl access
- Sample inspection command (avoid printing values in shared terminals):
```
kubectl get secret -n <namespace> <secret-name> -o yaml
```
- Determine whether you need: centralized rotation, audit trails, per-secret access controls, cross-cluster sharing, or integration with other OCI services—these are indicators you should use an external secrets manager (e.g., OCI Vault) instead of, or in addition to, native Secrets.
Decide on your external secrets strategy
- If OKE secret encryption with OCI Vault is not enabled and you require stronger at-rest protection, plan to:
  - Enable Kubernetes secrets encryption with OCI Vault at the cluster level (via OCI Console/CLI/IaC).
- If you need more advanced workflows (dynamic secrets, app-driven retrieval), plan to:
  - Use OCI Vault directly from applications, or
  - Deploy a secrets integration solution (e.g., external-secrets-style operator for OCI Vault) and reference external secrets from Pods instead of storing raw values in Kubernetes Secrets.
Implement the chosen external secrets approach and migrate sensitive data
- For OCI Vault–backed encryption of Kubernetes Secrets:
  - Enable Vault integration for the cluster in the OCI Console/CLI/IaC, then ensure new secrets are created after encryption is enabled, or re-create existing secrets so they are written encrypted in etcd.
- For application-level or operator-based use of external secrets:
  - Create secrets in OCI Vault.
  - Update application manifests to:
    - Either fetch secrets at runtime from OCI Vault, or
    - Reference Kubernetes Secret objects that are kept in sync from OCI Vault by your chosen integration, instead of embedding cleartext values in manifests.
Verify the configuration and ongoing use of external secret storage
- Machine: any machine with kubectl access
- Confirm that sensitive data is no longer defined directly in manifests as literals:
```
grep -R "password\|secret\|token\|apiKey" ./manifests || echo "No obvious inline secrets found"
```
- Confirm that Kubernetes Secrets in use are either:
  - Known to be encrypted at rest via OCI Vault (from step 2), and/or
  - Populated from an external secrets system according to your design.
- Periodically re-run:
```
kubectl get secrets -A
```
  and review new or changed secrets to ensure they align with your chosen external secrets strategy.

Using kubectl

On any machine with kubectl access:

List all Secrets and their types (cluster-wide overview)

kubectl get secrets --all-namespaces -o custom-columns=NS:metadata.namespace,NAME:metadata.name,TYPE:type

Review guidance:

A large number of Opaque secrets, especially in many namespaces, suggests broad use of raw Kubernetes Secrets.
Secrets used for long‑lived credentials (DB passwords, API keys, SSH keys) are candidates for external secret management.

Inspect sample Secret contents (to understand sensitivity)

Pick a few representative secrets (application, CI/CD, infra):

kubectl -n <namespace> get secret <secret-name> -o yaml

Then locally decode values (do NOT paste into shared terminals/logs):

kubectl -n <namespace> get secret <secret-name> -o jsonpath='{.data}' | jq -r 'to_entries[] | "\(.key)=\(.value|@base64d)"'

Review guidance:

If you see high‑sensitivity values (DB root passwords, cloud provider keys, tokens with broad scope), relying solely on in‑cluster Secrets may be insufficient.
Widespread use of such secrets across many apps indicates a stronger case for an external secrets system.

Identify Secrets likely created manually (no external controller labels)

kubectl get secrets --all-namespaces --show-labels

Review guidance:

Secrets lacking labels/annotations that indicate an external manager (for example, anything like secrets-manager=external, kubernetes.io/service-account-token is native) are probably maintained manually.
A pattern of manually managed high‑value secrets is a signal to consider a centralized external secrets system.

Check for any existing integration with an external secrets manager

kubectl get pods -A -o wide | egrep -i 'secret|vault|sm|kms|external'
kubectl get crd | egrep -i 'secret|vault|external|asc'
kubectl get ns --show-labels | egrep -i 'vault|secret'

Review guidance:

If you do NOT see controllers/CRDs such as “external-secrets”, “vault-operator”, or cloud‑specific secret-store integrations, then you likely do not have an external secrets solution in place.
Absence of such components plus heavy use of sensitive Secrets makes the benchmark recommendation more relevant.

Check for at-rest encryption of Secrets in etcd (cluster‑level posture)

If your environment supports it, look for encryption configuration (this may be exposed differently per managed service; for standard clusters with access to encryption config):

kubectl get secrets -n kube-system | grep -i encryption || true

(For many managed control planes you cannot see the encryption config via kubectl; you must check the cloud console/CLI.)

Review guidance:

If the provider’s documentation or console indicates that etcd secret encryption is DISABLED, and you see many high‑sensitivity Secrets as above, an external secrets manager (and/or enabling etcd encryption) should be actively considered.
If etcd encryption is enabled but Secrets hold very high‑value long‑lived credentials, you may still opt for an external manager that supports rotation, auditing, and tight access control.

Verification step (re‑run after your review/decisions):

Once you’ve made a decision (for or against adopting external secret storage), you can always re‑run:

kubectl get secrets --all-namespaces -o custom-columns=NS:metadata.namespace,NAME:metadata.name,TYPE:type --sort-by=metadata.namespace
kubectl get pods -A | egrep -i 'secret|vault|external' || true

Your manual assessment is complete when:

You understand what types of secrets are stored as Kubernetes Secrets.
You know whether any external secrets system is already in use.
You have consciously decided, based on sensitivity and complexity, whether to adopt or expand an external secrets storage/management solution.

Automation

#!/usr/bin/env bash
#
# cisoke-4.4.2-external-secrets-report.sh
#
# Run from: any machine with kubectl access and correct context set.
# Requires: kubectl, jq
#
# This script does NOT change anything. It gathers evidence to help you decide
# whether to move to an external secrets management system and whether etcd
# encryption with OCI Vault is enabled.

set -euo pipefail

# Helper: check dependencies
for bin in kubectl jq; do
  if ! command -v "$bin" >/dev/null 2>&1; then
    echo "ERROR: $bin is required but not installed or not in PATH" >&2
    exit 1
  fi
done

echo "==== CIS OKE 4.4.2: External Secrets Storage Evidence Report ===="
echo

###############################################################################
# 1. Cluster-level: check if encryption at rest is configured for secrets
###############################################################################
echo "== 1. Kube-apiserver encryption configuration (secrets at rest) =="

# Try to find kube-apiserver Pod and its arguments
# This works for clusters where kube-apiserver is exposed as a Pod/Deployment.
# On fully managed control planes, this may return nothing.
if kubectl get pods -A >/dev/null 2>&1; then
  API_PODS_JSON=$(kubectl get pods -A -o json 2>/dev/null || echo '')
else
  API_PODS_JSON=''
fi

if [ -n "$API_PODS_JSON" ]; then
  echo "Scanning for kube-apiserver Pods and --encryption-provider-config flag..."
  echo

  echo "$API_PODS_JSON" | jq -r '
    .items[]
    | select(.metadata.name|test("kube-apiserver"))
    | "Namespace: \(.metadata.namespace)\nPod: \(.metadata.name)\n"
      + ( .spec.containers[]
          | select(.name|test("kube-apiserver"))
          | "  Command: \(.command // [])\n  Args: \(.args // [])\n"
        )
  '

  echo
  echo "Summary: presence of --encryption-provider-config for kube-apiserver:"
  echo "$API_PODS_JSON" | jq -r '
    .items[]
    | select(.metadata.name|test("kube-apiserver"))
    | .metadata.namespace + "/" + .metadata.name + ": " +
      (if ([.spec.containers[]
            | select(.name|test("kube-apiserver"))
            | (.args // [] + .command // [])
            | map(select(startswith("--encryption-provider-config=")))
           ] | add // empty) != null
       then "HAS encryption-provider-config"
       else "NO encryption-provider-config"
       end)
  ' 2>/dev/null || true

  echo
  cat <<'EOF'
Interpretation:
- "HAS encryption-provider-config":
    Indicates that encryption at rest for Kubernetes secrets may be enabled.
    You still need to verify that the provider config uses OCI Vault.
- "NO encryption-provider-config":
    Indicates that secrets stored in etcd are likely NOT encrypted at rest
    using an external key management system. This is a concern for this control.

On fully managed OKE control planes, kube-apiserver Pods may not be visible
via kubectl. In that case, use the OCI Console/CLI to verify cluster-level
encryption with OCI Vault.
EOF
else
  echo "Could not list Pods (no permissions or managed control plane)."
  echo "Use OCI Console/CLI to confirm if Kubernetes secrets at rest are"
  echo "encrypted with OCI Vault for this cluster."
fi

echo
###############################################################################
# 2. Inventory of Kubernetes Secret usage in the cluster
###############################################################################
echo "== 2. Inventory of Secrets and their usage across namespaces =="

echo "Listing all Secrets (name, type, namespace, number of data keys)..."
kubectl get secrets --all-namespaces -o json \
  | jq -r '
    .items[]
    | [.metadata.namespace,
       .metadata.name,
       .type,
       ( .data | if . == null then 0 else (keys | length) end)
      ]
    | @tsv
  ' \
  | awk 'BEGIN { OFS="\t"; print "NAMESPACE","NAME","TYPE","DATA_KEY_COUNT" } { print }'

cat <<'EOF'

Interpretation:
- Large numbers of generic Opaque secrets, especially with many data keys,
  can indicate complex secret management needs better served by an external
  vault (e.g., OCI Vault) or a secrets operator.
- Highly sensitive items (DB passwords, API keys, SSH keys) stored as
  Kubernetes Secrets are a higher-risk area if etcd encryption is not
  configured.
EOF

echo
###############################################################################
# 3. Workload references to Secrets (Pods/Deployments/StatefulSets/CronJobs)
###############################################################################
echo "== 3. Workloads referencing Secrets =="

echo "Collecting Pods that reference Secrets..."
kubectl get pods --all-namespaces -o json \
  | jq -r '
    .items[]
    | {
        ns: .metadata.namespace,
        pod: .metadata.name,
        sa: .spec.serviceAccountName,
        secretRefs:
          (
            # Volumes
            (.spec.volumes // [])
            | map(select(.secret != null)
                  | {type:"volume", name:.name, secretName:.secret.secretName})
          )
          +
          (
            # EnvFrom
            ([.spec.containers[]?, .spec.initContainers[]?] // [])
            | map(.envFrom[]? | select(.secretRef != null)
                  | {type:"envFrom", name:.name, secretName:.secretRef.name})
          )
          +
          (
            # Env
            ([.spec.containers[]?, .spec.initContainers[]?] // [])
            | map(.env[]? | select(.valueFrom != null and .valueFrom.secretKeyRef != null)
                  | {type:"env", name:.name, secretName:.valueFrom.secretKeyRef.name})
          )
      }
    | select(.secretRefs | length > 0)
    | .secretRefs[]
    | [ .ns, .pod, .sa, .type, .name, .secretName ]
    | @tsv
  ' \
  | awk 'BEGIN { OFS="\t"; print "NAMESPACE","POD","SERVICE_ACCOUNT","REF_TYPE","REF_NAME","SECRET_NAME" } { print }'

cat <<'EOF'

Interpretation:
- Pods heavily reliant on Secrets for configuration (multiple SECRET_NAME
  entries) may benefit from integration with an external secret store:
  - OCI Vault + CSI driver or secret sync controller
  - Third-party solutions that pull secrets at runtime
- This report does NOT indicate non-compliance by itself; it highlights where
  a migration to an external secrets system would have the most impact.

EOF

echo
###############################################################################
# 4. (If visible) Inspect encryption provider config from kube-apiserver args
###############################################################################
echo "== 4. Attempt to locate and display encryption-provider-config paths =="

if [ -n "${API_PODS_JSON:-}" ]; then
  ENC_CONFIG_PATHS=$(echo "$API_PODS_JSON" | jq -r '
    .items[]
    | select(.metadata.name|test("kube-apiserver"))
    | .spec.containers[]
    | select(.name|test("kube-apiserver"))
    | (.args // [] + .command // [])
    | map(select(startswith("--encryption-provider-config=")))
    | .[]
    | sub("^--encryption-provider-config=";"")
  ' 2>/dev/null || true)

  if [ -n "$ENC_CONFIG_PATHS" ]; then
    echo "Found encryption-provider-config references in kube-apiserver:"
    echo "$ENC_CONFIG_PATHS" | sed 's/^/  - /'
    cat <<'EOF'

Next manual step (cannot be automated with kubectl alone):
- Log into the control plane node(s) over SSH.
- Inspect the referenced encryption-provider-config file(s).
- Verify that:
  * Resources include 'secrets'
  * Provider type uses KMS/OCI Vault and not just local keys
  * Policies and keys are managed according to your requirements

If you are on a fully managed OKE control plane, perform an equivalent check
via the OCI Console/CLI for the cluster's secret encryption configuration.
EOF
  else
    echo "No --encryption-provider-config argument found in kube-apiserver containers."
    echo "This suggests that secrets in etcd are NOT encrypted with an external KMS."
  fi
else
  echo "kube-apiserver Pods not visible via kubectl; skip provider config path discovery."
fi

echo
echo "==== End of CIS OKE 4.4.2 evidence report ===="

Explanation of what indicates a problem when you review the script’s output:

Section 1:
- Any kube-apiserver entry showing NO encryption-provider-config suggests Kubernetes Secrets are stored unencrypted in etcd; you should strongly consider enabling encryption with OCI Vault for new or reconfigured clusters.
Section 2:
- Many Opaque secrets, or secrets with many DATA_KEY_COUNT values, especially in production namespaces, indicate complex secret usage that may justify moving to an external secret store.
Section 3:
- Pods with numerous SECRET_NAME references or highly sensitive workloads (DB, payments, identity services) relying on Kubernetes Secrets are prime candidates for migration to an external secrets manager.
Section 4:
- Missing or non-KMS/OCI-Vault-based encryption provider config means secrets are not protected by an external key management system at rest in etcd.

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​

More Info:

Risk Level

Address

Compliance Standards

Triage and Remediation

Remediation