Secrets Should Be Encrypted At Rest

More Info:

Advisory: EncryptionConfiguration with a KMS provider should be enabled for Secret resources so etcd does not store secrets in plaintext.

Risk Level

High

Address

Security

Compliance Standards

Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

Identify the cluster(s) to review
- On any machine with gcloud installed, set the project and list clusters:
```
gcloud config set project PROJECT_ID
gcloud container clusters list
```
- Note each cluster’s NAME, LOCATION, and whether it is Autopilot or Standard.
Check if secrets encryption is already enabled for each cluster
- For each cluster, run:
```
gcloud container clusters describe CLUSTER_NAME \
  --location=CLUSTER_LOCATION \
  --format='yaml(databaseEncryption)'
```
- Review the output:
  - state: ENABLED and a non-empty keyName means KMS envelope encryption is enabled.
  - state: DECRYPTED or missing databaseEncryption means this control is not satisfied.
Review the KMS key configuration and status (if enabled)
- From the databaseEncryption.keyName output, identify the KMS key:
  projects/PROJECT_ID/locations/LOCATION/keyRings/RING/cryptoKeys/KEY
- Verify the key exists and is enabled:
```
gcloud kms keys describe KEY \
  --keyring=RING \
  --location=LOCATION
```
- Confirm an appropriate rotation policy and IAM permissions for GKE’s service account.
Decide on enabling encryption for clusters where it is not enabled
- Understand impact: this is a control-plane configuration change; enabling database encryption on an existing GKE cluster is supported but may be irreversible and can have performance implications.
- Decide per cluster whether to:
  - Leave it as-is (with documented risk acceptance), or
  - Enable KMS database encryption using either a new or existing CMEK key that meets your org’s key-management and IAM policies.

Enable database (secrets) encryption for a cluster (if required)

Ensure you have or create a CMEK key:

gcloud kms keyrings create RING \
  --location=LOCATION

gcloud kms keys create KEY \
  --keyring=RING \
  --location=LOCATION \
  --purpose=encryption

Enable encryption on the target cluster (Standard or Autopilot):

gcloud container clusters update CLUSTER_NAME \
  --location=CLUSTER_LOCATION \
  --database-encryption-key=projects/PROJECT_ID/locations/LOCATION/keyRings/RING/cryptoKeys/KEY

Follow any prompts; note this may take several minutes and affects how data (including Secrets) is stored in etcd.

Verify and document the final state
- Re-run the describe command to confirm encryption is now enabled and points to the intended key:
```
gcloud container clusters describe CLUSTER_NAME \
  --location=CLUSTER_LOCATION \
  --format='yaml(databaseEncryption)'
```
- Record for each cluster: state, keyName, KMS key rotation policy, and the date of the change or the explicit decision to accept the risk if encryption remains disabled.

Using kubectl

kubectl cannot enable or configure envelope/KMS encryption for Secrets in GKE; this setting is part of the GKE cluster’s control-plane configuration managed via the Google Cloud Console, gcloud CLI, or IaC. To address this finding, make the change at the cloud provider level as described in the Manual Steps section.

Automation

#!/usr/bin/env bash
#
# GKE Secret encryption-at-rest check (MANUAL REVIEW)
#
# Requirements:
#   - gcloud installed and authenticated
#   - jq installed
#
# This script DOES NOT change anything. It only reports whether
# GKE Secret payloads are encrypted at rest with a CMEK KMS key.
#
# Run location:
#   - Any machine with gcloud access to the relevant projects.
#
# Usage examples:
#   Check all clusters in a single project:
#     PROJECT_ID="my-project" ./check_gke_secret_encryption.sh
#
#   Check all clusters in multiple projects:
#     PROJECT_ID_LIST="proj-a proj-b" ./check_gke_secret_encryption.sh
#
# Notes:
#   - On GKE, enabling a customer-managed KMS key for application-layer
#     secret encryption is done at cluster create/upgrade time and
#     cannot be fully inferred from kubectl output alone.
#   - This script uses the GKE API to inspect the databaseEncryption
#     field of each cluster and then does a light kubectl sampling
#     check, purely for operator confidence.

set -euo pipefail

PROJECT_ID_LIST="${PROJECT_ID_LIST:-}"
SINGLE_PROJECT="${PROJECT_ID:-}"

if [[ -z "${PROJECT_ID_LIST}" && -z "${SINGLE_PROJECT}" ]]; then
  echo "ERROR: Set PROJECT_ID or PROJECT_ID_LIST before running."
  echo "  PROJECT_ID=\"my-project\" ./check_gke_secret_encryption.sh"
  echo "  PROJECT_ID_LIST=\"proj-a proj-b\" ./check_gke_secret_encryption.sh"
  exit 1
fi

if [[ -n "${SINGLE_PROJECT}" && -n "${PROJECT_ID_LIST}" ]]; then
  echo "ERROR: Use either PROJECT_ID or PROJECT_ID_LIST, not both."
  exit 1
fi

if [[ -n "${SINGLE_PROJECT}" ]]; then
  PROJECTS=("${SINGLE_PROJECT}")
else
  read -r -a PROJECTS <<< "${PROJECT_ID_LIST}"
fi

check_dependencies() {
  command -v gcloud >/dev/null 2>&1 || {
    echo "ERROR: gcloud not found in PATH." >&2
    exit 1
  }
  command -v jq >/dev/null 2>&1 || {
    echo "ERROR: jq not found in PATH." >&2
    exit 1
  }
  command -v kubectl >/dev/null 2>&1 || {
    echo "WARNING: kubectl not found. Will skip in-cluster sampling checks." >&2
  }
}

print_header() {
  echo "====================================================================="
  echo "GKE Secret Encryption-at-Rest Status (KMS/CMEK for databaseEncryption)"
  echo "====================================================================="
  echo
  echo "INTERPRETATION:"
  echo "  - databaseEncryption.state = 'ENCRYPTED' and kmsKeyName set:"
  echo "      => GKE cluster is configured to encrypt Kubernetes Secrets"
  echo "         at the application layer using a KMS key (expected)."
  echo "  - databaseEncryption.state = 'DECRYPTED' or missing:"
  echo "      => etcd stores Secrets without application-layer KMS"
  echo "         encryption. This is a finding and requires remediation."
  echo
  echo "NOTE: This script reports configuration only. It does NOT verify"
  echo "      historical storage or migration details."
  echo
}

check_cluster_api_encryption() {
  local project="$1"

  echo
  echo "---- Project: ${project} ----"

  # List all clusters (zonal and regional) in the project
  local clusters
  clusters=$(gcloud container clusters list \
    --project "${project}" \
    --format=json)

  if [[ "${clusters}" == "[]" ]]; then
    echo "No GKE clusters found in project ${project}."
    return
  fi

  echo "${clusters}" | jq -c '.[]' | while read -r cluster; do
    local name location endpoint
    name=$(echo "${cluster}" | jq -r '.name')
    location=$(echo "${cluster}" | jq -r '.location')
    endpoint=$(echo "${cluster}" | jq -r '.endpoint')

    # Re-fetch full cluster description to be sure databaseEncryption is present
    local full
    full=$(gcloud container clusters describe "${name}" \
      --project "${project}" \
      --region "${location}" \
      --format=json 2>/dev/null || \
      gcloud container clusters describe "${name}" \
        --project "${project}" \
        --zone "${location}" \
        --format=json 2>/dev/null || echo "{}")

    local db_state kms_key_name
    db_state=$(echo "${full}" | jq -r '.databaseEncryption.state // "UNSPECIFIED"')
    kms_key_name=$(echo "${full}" | jq -r '.databaseEncryption.keyName // ""')

    echo
    echo "Cluster: ${name}"
    echo "  Location:            ${location}"
    echo "  API Endpoint:        ${endpoint}"
    echo "  databaseEncryption:"

    if [[ "${db_state}" == "ENCRYPTED" && -n "${kms_key_name}" && "${kms_key_name}" != "null" ]]; then
      echo "    state:             ENCRYPTED"
      echo "    keyName (CMEK):    ${kms_key_name}"
      echo "    STATUS:            OK (Secrets configured for KMS encryption at rest)"
    elif [[ "${db_state}" == "ENCRYPTED" && ( -z "${kms_key_name}" || "${kms_key_name}" == "null" ) ]]; then
      echo "    state:             ENCRYPTED"
      echo "    keyName:           <missing>"
      echo "    STATUS:            REVIEW (encrypted but keyName not surfaced; confirm in console)"
    else
      echo "    state:             ${db_state}"
      echo "    keyName:           ${kms_key_name}"
      echo "    STATUS:            FINDING (Secrets NOT using KMS envelope encryption)"
      echo "    ACTION NEEDED:     Plan cluster recreate or upgrade with 'Customer-managed"
      echo "                       encryption key' for application-layer Secret encryption."
    fi

    # Optional: lightweight kubectl sampling if context can be obtained
    if command -v kubectl >/dev/null 2>&1; then
      echo
      echo "  Attempting in-cluster sampling (non-deterministic, for review only)..."
      echo "    - Requires a kubeconfig/context for this cluster."

      # Try to get current context that matches this cluster, if any
      # (User may need to run: gcloud container clusters get-credentials ...)
      local matched_context
      matched_context=$(kubectl config get-contexts -o name 2>/dev/null | \
        grep -E "(^|/)${name}$" || true)

      if [[ -z "${matched_context}" ]]; then
        echo "    Skipped: No kubectl context found for cluster '${name}'."
        echo "    Hint:    Run:"
        echo "             gcloud container clusters get-credentials ${name} --project ${project} --region ${location}"
      else
        # Use the first matching context
        matched_context=$(echo "${matched_context}" | head -n1)
        echo "    Using kubectl context: ${matched_context}"
        # Sample a few Secrets (names only) to confirm existence
        set +e
        local sample_output
        sample_output=$(kubectl --context "${matched_context}" get secrets --all-namespaces \
          --no-headers 2>/dev/null | head -n 5)
        local rc=$?
        set -e
        if [[ ${rc} -ne 0 ]]; then
          echo "    Could not list Secrets with kubectl; check RBAC/credentials."
        else
          echo "    Sampled Secrets (for operator review only):"
          if [[ -z "${sample_output}" ]]; then
            echo "      <no Secrets listed>"
          else
            echo "${sample_output}" | sed 's/^/      /'
          fi
          echo "    NOTE: This sampling does NOT prove encryption state; rely mainly"
          echo "          on 'databaseEncryption.state' above."
        fi
      fi
    fi

  done
}

check_dependencies
print_header

for proj in "${PROJECTS[@]}"; do
  check_cluster_api_encryption "${proj}"
done

echo
echo "====================================================================="
echo "REVIEW GUIDANCE"
echo "====================================================================="
echo "Flag as a PROBLEM (requires remediation) when:"
echo "  - 'databaseEncryption.state' is 'DECRYPTED', 'UNSPECIFIED', or 'null', OR"
echo "  - the 'databaseEncryption' block is completely absent."
echo
echo "Consider COMPLIANT when:"
echo "  - 'databaseEncryption.state' is 'ENCRYPTED' AND"
echo "  - 'databaseEncryption.keyName' is a valid Cloud KMS key resource path."
echo
echo "Because this is a MANUAL check, final compliance determination should be"
echo "made by reviewing:"
echo "  - The cluster configuration in the GCP console (Security -> Encryption)"
echo "  - Your provisioning IaC (Terraform, Deployment Manager, etc.)."

What output indicates a problem

For any cluster where the script prints:
- STATUS: FINDING (Secrets NOT using KMS envelope encryption)
  or
- databaseEncryption.state: DECRYPTED, UNSPECIFIED, or missing,

that cluster does not have GKE Secret KMS encryption at rest configured and should be treated as non-compliant with “Secrets Should Be Encrypted At Rest” until reviewed and remediated.

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​