Encrypting Kubernetes Secrets At Rest In Etcd

More Info:

Encrypt Kubernetes secrets at rest in etcd using the Oracle Cloud Infrastructure Vault service.

Risk Level

Medium

Address

Security

Compliance Standards

• APRA CPS 234 (Australia)
• BSI C5 (Germany)
• Brazil LGPD
• CCPA / CPRA (California)
• CIS Critical Security Controls v8
• CIS OKE
• CMMC 2.0
• CSA Cloud Controls Matrix v4
• DPDPA
• Digital Operational Resilience Act (EU)
• ISO/IEC 27017
• ISO/IEC 27018
• ISO/IEC 27701
• KSA PDPL
• MAS Technology Risk Management (Singapore)
• MITRE ATT&CK (Cloud)
• NIS2 Directive
• NIST SP 800-171
• NYDFS 23 NYCRR 500
• SWIFT Customer Security Controls Framework
• Sarbanes-Oxley IT General Controls
• UK NCSC Cyber Assessment Framework

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Confirm current cluster encryption configuration

   • On any machine with OCI CLI access and permissions in the cluster’s tenancy, list clusters and inspect the target one:

     oci ce cluster list \
       --compartment-id ocid1.compartment.oc1..<cluster_compartment_ocid> \
       --all

     oci ce cluster get \
       --cluster-id ocid1.cluster.oc1..<cluster_ocid> \
       --query "data.{name:name,\"kms-key-id\":kms-key-id}" \
       --output table

   • If kms-key-id is null/empty, Kubernetes secrets at rest in etcd are not using OCI Vault encryption.

2. Decide tenancy model: same-tenancy vs cross-tenancy key

   • If the master encryption key will be in the same tenancy as the cluster (“same-tenancy”): you only need policies in that tenancy.
   • If the master encryption key will be in a different tenancy (“cross-tenancy” as in the remediation text): identify:
     • ClusterTenancy (where the OKE cluster runs) – its tenancy OCID
     • KeyTenancy (where the Vault master key lives) – its tenancy OCID
     • The compartment OCID where clusters are created in ClusterTenancy
     • The admin IAM group in ClusterTenancy that manages OKE (e.g. OKEAdminGroup) and confirm it exists.

3. Create or identify OCI Vault master key

   • In the tenancy where the key will live (KeyTenancy or same tenancy as cluster), using Console or CLI, ensure a Vault and master key exist and record the key OCID:

     # List keys in the compartment that will hold the master key
     oci kms management key list \
       --compartment-id ocid1.compartment.oc1..<key_compartment_ocid> \
       --all \
       --query "data[].{name:\"display-name\",id:id}" \
       --output table

   • If no suitable key exists, create one via Console or CLI, then note ocid1.key.oc1..<unique_ID> to use as kms-key-id.

4. Review and configure IAM policies (same- or cross-tenancy)

   • For cross-tenancy (exactly as in the remediation):
     • In KeyTenancy, in the root compartment, create/update policies:

       Define tenancy OKE_Tenancy as ocid1.tenancy.oc1..<ClusterTenancy_ocid>
       Define dynamic-group RemoteOKEClusterDynGroup as ocid1.dynamicgroup.oc1..<OKEAdminDynGroup_ocid>
       Define group RemoteOKEAdminGroup as ocid1.group.oc1..<OKEAdminGroup_ocid>
       Admit dynamic-group RemoteOKEClusterDynGroup of tenancy OKE_Tenancy to use keys in tenancy where target.key.id = 'ocid1.key.oc1..<key_ocid>'
       Admit group RemoteOKEAdminGroup of tenancy OKE_Tenancy to use keys in tenancy where target.key.id = 'ocid1.key.oc1..<key_ocid>'

     • In ClusterTenancy, in the root compartment, create/update policies:

       Define tenancy KMS_Tenancy as ocid1.tenancy.oc1..<KeyTenancy_ocid>
       Endorse group OKEAdminGroup to use keys in tenancy KMS_Tenancy
       Endorse dynamic-group OKEAdminDynGroup to use keys in tenancy KMS_Tenancy
       Allow dynamic-group OKEAdminDynGroup to use keys in tenancy where target.key.id = 'ocid1.key.oc1..<key_ocid>'

   • For same-tenancy, ensure at minimum that the OKE admin group and the OKE dynamic group can use the selected key in that tenancy’s Vault compartment. Adjust and save policies as required.

5. Create (or re-create) the cluster using the KMS key

   • Enabling etcd encryption with a master key is applied at cluster creation time for OKE. If the existing cluster has kms-key-id unset, plan to create a new cluster that uses the key and then migrate workloads.
   • To create a new encrypted cluster in the ClusterTenancy:

     oci ce cluster create \
       --name oke-with-kms \
       --kubernetes-version v1.16.8 \
       --vcn-id ocid1.vcn.oc1.iad.<vcn_ocid> \
       --service-lb-subnet-ids '["ocid1.subnet.oc1.iad.<subnet_ocid>"]' \
       --compartment-id ocid1.compartment.oc1..<cluster_compartment_ocid> \
       --kms-key-id ocid1.key.oc1.iad.<key_ocid>

   • Migrate applications and data from the old cluster to the new one as per your operational procedures, then decommission the non-encrypted cluster if appropriate.

6. Verify encryption configuration is in effect

   • On any machine with OCI CLI access, confirm the new (or target) cluster is associated with the master key:

     oci ce cluster get \
       --cluster-id ocid1.cluster.oc1..<new_cluster_ocid> \
       --query "data.{name:name,\"kms-key-id\":kms-key-id}" \
       --output table

   • Ensure kms-key-id shows the expected ocid1.key.oc1..<key_ocid>. This is the determinable evidence that Kubernetes secrets at rest in etcd are configured to use OCI Vault encryption for this cluster.

Using kubectl

kubectl cannot configure encryption of Kubernetes secrets at rest in etcd, because this is controlled by the managed OKE control plane and cluster creation / cloud policies (for example, via --kms-key-id and OCI Vault/IAM configuration), not by Kubernetes API objects. Use the cloud provider console/CLI/IaC to apply the changes described in the Manual Steps section.

Automation

#!/usr/bin/env bash
#
# Purpose:
#   Report whether an Oracle OKE cluster is configured to encrypt Kubernetes
#   Secrets at rest in etcd using OCI KMS (Vault), so it can be reviewed at scale.
#
# Requirements:
#   - oci CLI installed and configured (tenancy, region, auth)
#   - jq installed
#
# Usage:
#   ./check_oke_secrets_encryption.sh
#
# Scope:
#   - Runs from any machine with oci + jq configured.
#   - Uses cloud-provider APIs only (managed control plane; does NOT touch nodes).

set -euo pipefail

if ! command -v oci >/dev/null 2>&1; then
  echo "ERROR: oci CLI not found in PATH" >&2
  exit 1
fi

if ! command -v jq >/dev/null 2>&1; then
  echo "ERROR: jq not found in PATH" >&2
  exit 1
fi

echo "Collecting OKE cluster encryption configuration..."
echo

# Get all clusters in all compartments visible to the configured principal.
# Adjust --compartment-id-in-subtree / root compartment as needed for your tenancy.
CLUSTERS_JSON=$(oci ce cluster list \
  --all \
  --compartment-id-in-subtree true \
  --output json)

echo "${CLUSTERS_JSON}" | jq -r '
  .data[]
  | {
      "id": .id,
      "name": .name,
      "lifecycle_state": .lifecycle-state,
      "compartment_id": .compartment-id,
      "endpoint_config": .endpoint-config,
      "kms_key_id": .kms-key-id
    }
  ' > /tmp/oke_clusters_raw.json

if ! [ -s /tmp/oke_clusters_raw.json ]; then
  echo "No clusters returned by oci ce cluster list."
  exit 0
fi

echo "Per-cluster encryption status:"
echo "---------------------------------------------------------------------"
echo "FIELDS:"
echo "  - cluster_name: OKE cluster name"
echo "  - cluster_ocid: OKE cluster OCID"
echo "  - lifecycle_state: cluster lifecycle (ACTIVE, CREATING, etc.)"
echo "  - kms_key_id: OCID of KMS key associated to the cluster (if any)"
echo "  - encryption_config: current master key and status (if retrievable)"
echo

# Iterate clusters and print relevant encryption info
jq -c '.' /tmp/oke_clusters_raw.json | while read -r CLUSTER; do
  CLUSTER_ID=$(echo "${CLUSTER}" | jq -r '.id')
  CLUSTER_NAME=$(echo "${CLUSTER}" | jq -r '.name')
  LIFECYCLE_STATE=$(echo "${CLUSTER}" | jq -r '.lifecycle_state')
  KMS_KEY_ID=$(echo "${CLUSTER}" | jq -r '.kms_key_id // "null"')

  echo "cluster_name=${CLUSTER_NAME}"
  echo "cluster_ocid=${CLUSTER_ID}"
  echo "lifecycle_state=${LIFECYCLE_STATE}"
  echo "kms_key_id=${KMS_KEY_ID}"

  # If kms_key_id is set, try to fetch KMS key details for basic validation
  if [ "${KMS_KEY_ID}" != "null" ]; then
    echo "  -> Attempting to describe KMS key (basic sanity check)..."
    # KMS key OCID encodes the region; we rely on currently configured region.
    # If cross-region / cross-tenancy is used, ensure oci CLI profile matches.
    if KEY_JSON=$(oci kms management key get --key-id "${KMS_KEY_ID}" --output json 2>/dev/null); then
      KEY_STATE=$(echo "${KEY_JSON}" | jq -r '.data."lifecycle-state"')
      KEY_COMPARTMENT=$(echo "${KEY_JSON}" | jq -r '.data."compartment-id"')
      echo "    key_lifecycle_state=${KEY_STATE}"
      echo "    key_compartment_id=${KEY_COMPARTMENT}"
    else
      echo "    WARNING: Unable to retrieve KMS key details with current oci credentials / region."
      echo "    This may indicate cross-tenancy or cross-region key usage. Review IAM policies."
    fi
  else
    echo "  -> No kms_key_id configured for this cluster."
  fi

  echo "---------------------------------------------------------------------"
done

cat <<'EOF'

INTERPRETING RESULTS (what indicates a problem for CIS OKE 5.3.1):

1) Cluster with kms_key_id = null
   - Indicates the cluster was created without specifying a KMS master key.
   - For such clusters, Kubernetes Secrets stored in etcd are NOT encrypted
     with OCI Vault (KMS) and likely fail CIS OKE 5.3.1.
   - Action: consider recreating the cluster with --kms-key-id, or migrating
     workloads to a compliant cluster.

2) Cluster with kms_key_id set but oci kms management key get fails
   - Possible reasons:
     * Current oci principal/region does not have permissions to view the key.
     * The key resides in a different tenancy or region and cross-tenancy /
       cross-region policies are not accessible from this profile.
   - This is not automatically non-compliant, but it requires manual review:
     * Confirm the key OCID is correct.
     * Confirm cross-tenancy policies as described in the benchmark remediation.
     * Confirm the key is in an ACTIVE state and permitted for cryptographic use.

3) Cluster with kms_key_id set and key_lifecycle_state = DISABLED or DELETED
   - Indicates a misconfigured or unusable master key.
   - Even though kms_key_id is present, secrets encryption may not function
     properly.
   - Action: investigate the KMS key state and update cluster / keys per
     operational requirements and CIS guidance.

NOTE:
- This script reports control-plane configuration (presence and basic status
  of kms_key_id and its KMS key). It does NOT directly prove that every
  Kubernetes Secret in etcd is encrypted, nor can it auto-remediate.
- Use these results to drive manual review and policy validation, especially
  for cross-tenancy setups as described in the CIS remediation text.

EOF

Additional Reading:

• https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
• https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengencryptingdata.htm