> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Secrets Should Be Encrypted At Rest

### More Info:

Advisory: EncryptionConfiguration with a KMS provider should be enabled for Secret resources so etcd does not store secrets in plaintext.

### Risk Level

High

### Address

Security

### Compliance Standards

* Cloudanix Best Practice

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Manual Steps" defaultOpen="true">
        1. **Identify the cluster(s) to review**
           * On any machine with `gcloud` installed, set the project and list clusters:
             ```bash theme={null}
             gcloud config set project PROJECT_ID
             gcloud container clusters list
             ```
           * Note each cluster’s `NAME`, `LOCATION`, and whether it is **Autopilot** or **Standard**.

        2. **Check if secrets encryption is already enabled for each cluster**
           * For each cluster, run:
             ```bash theme={null}
             gcloud container clusters describe CLUSTER_NAME \
               --location=CLUSTER_LOCATION \
               --format='yaml(databaseEncryption)'
             ```
           * Review the output:
             * `state: ENABLED` and a non-empty `keyName` means KMS envelope encryption is enabled.
             * `state: DECRYPTED` or missing `databaseEncryption` means this control is **not** satisfied.

        3. **Review the KMS key configuration and status (if enabled)**
           * From the `databaseEncryption.keyName` output, identify the KMS key:\
             `projects/PROJECT_ID/locations/LOCATION/keyRings/RING/cryptoKeys/KEY`
           * Verify the key exists and is enabled:
             ```bash theme={null}
             gcloud kms keys describe KEY \
               --keyring=RING \
               --location=LOCATION
             ```
           * Confirm an appropriate rotation policy and IAM permissions for GKE’s service account.

        4. **Decide on enabling encryption for clusters where it is not enabled**
           * Understand impact: this is a **control-plane configuration change**; enabling database encryption on an existing GKE cluster is supported but may be irreversible and can have performance implications.
           * Decide per cluster whether to:
             * Leave it as-is (with documented risk acceptance), or
             * Enable KMS database encryption using either a **new** or **existing** CMEK key that meets your org’s key-management and IAM policies.

        5. **Enable database (secrets) encryption for a cluster (if required)**
           * Ensure you have or create a CMEK key:
             ```bash theme={null}
             gcloud kms keyrings create RING \
               --location=LOCATION

             gcloud kms keys create KEY \
               --keyring=RING \
               --location=LOCATION \
               --purpose=encryption
             ```
           * Enable encryption on the target cluster (Standard or Autopilot):
             ```bash theme={null}
             gcloud container clusters update CLUSTER_NAME \
               --location=CLUSTER_LOCATION \
               --database-encryption-key=projects/PROJECT_ID/locations/LOCATION/keyRings/RING/cryptoKeys/KEY
             ```
           * Follow any prompts; note this may take several minutes and affects how data (including Secrets) is stored in etcd.

        6. **Verify and document the final state**
           * Re-run the describe command to confirm encryption is now enabled and points to the intended key:
             ```bash theme={null}
             gcloud container clusters describe CLUSTER_NAME \
               --location=CLUSTER_LOCATION \
               --format='yaml(databaseEncryption)'
             ```
           * Record for each cluster: `state`, `keyName`, KMS key rotation policy, and the date of the change or the explicit decision to accept the risk if encryption remains disabled.
      </Accordion>

      <Accordion title="Using kubectl">
        kubectl cannot enable or configure envelope/KMS encryption for Secrets in GKE; this setting is part of the GKE cluster’s control-plane configuration managed via the Google Cloud Console, gcloud CLI, or IaC. To address this finding, make the change at the cloud provider level as described in the Manual Steps section.
      </Accordion>

      <Accordion title="Automation">
        ```bash theme={null}
        #!/usr/bin/env bash
        #
        # GKE Secret encryption-at-rest check (MANUAL REVIEW)
        #
        # Requirements:
        #   - gcloud installed and authenticated
        #   - jq installed
        #
        # This script DOES NOT change anything. It only reports whether
        # GKE Secret payloads are encrypted at rest with a CMEK KMS key.
        #
        # Run location:
        #   - Any machine with gcloud access to the relevant projects.
        #
        # Usage examples:
        #   Check all clusters in a single project:
        #     PROJECT_ID="my-project" ./check_gke_secret_encryption.sh
        #
        #   Check all clusters in multiple projects:
        #     PROJECT_ID_LIST="proj-a proj-b" ./check_gke_secret_encryption.sh
        #
        # Notes:
        #   - On GKE, enabling a customer-managed KMS key for application-layer
        #     secret encryption is done at cluster create/upgrade time and
        #     cannot be fully inferred from kubectl output alone.
        #   - This script uses the GKE API to inspect the databaseEncryption
        #     field of each cluster and then does a light kubectl sampling
        #     check, purely for operator confidence.

        set -euo pipefail

        PROJECT_ID_LIST="${PROJECT_ID_LIST:-}"
        SINGLE_PROJECT="${PROJECT_ID:-}"

        if [[ -z "${PROJECT_ID_LIST}" && -z "${SINGLE_PROJECT}" ]]; then
          echo "ERROR: Set PROJECT_ID or PROJECT_ID_LIST before running."
          echo "  PROJECT_ID=\"my-project\" ./check_gke_secret_encryption.sh"
          echo "  PROJECT_ID_LIST=\"proj-a proj-b\" ./check_gke_secret_encryption.sh"
          exit 1
        fi

        if [[ -n "${SINGLE_PROJECT}" && -n "${PROJECT_ID_LIST}" ]]; then
          echo "ERROR: Use either PROJECT_ID or PROJECT_ID_LIST, not both."
          exit 1
        fi

        if [[ -n "${SINGLE_PROJECT}" ]]; then
          PROJECTS=("${SINGLE_PROJECT}")
        else
          read -r -a PROJECTS <<< "${PROJECT_ID_LIST}"
        fi

        check_dependencies() {
          command -v gcloud >/dev/null 2>&1 || {
            echo "ERROR: gcloud not found in PATH." >&2
            exit 1
          }
          command -v jq >/dev/null 2>&1 || {
            echo "ERROR: jq not found in PATH." >&2
            exit 1
          }
          command -v kubectl >/dev/null 2>&1 || {
            echo "WARNING: kubectl not found. Will skip in-cluster sampling checks." >&2
          }
        }

        print_header() {
          echo "====================================================================="
          echo "GKE Secret Encryption-at-Rest Status (KMS/CMEK for databaseEncryption)"
          echo "====================================================================="
          echo
          echo "INTERPRETATION:"
          echo "  - databaseEncryption.state = 'ENCRYPTED' and kmsKeyName set:"
          echo "      => GKE cluster is configured to encrypt Kubernetes Secrets"
          echo "         at the application layer using a KMS key (expected)."
          echo "  - databaseEncryption.state = 'DECRYPTED' or missing:"
          echo "      => etcd stores Secrets without application-layer KMS"
          echo "         encryption. This is a finding and requires remediation."
          echo
          echo "NOTE: This script reports configuration only. It does NOT verify"
          echo "      historical storage or migration details."
          echo
        }

        check_cluster_api_encryption() {
          local project="$1"

          echo
          echo "---- Project: ${project} ----"

          # List all clusters (zonal and regional) in the project
          local clusters
          clusters=$(gcloud container clusters list \
            --project "${project}" \
            --format=json)

          if [[ "${clusters}" == "[]" ]]; then
            echo "No GKE clusters found in project ${project}."
            return
          fi

          echo "${clusters}" | jq -c '.[]' | while read -r cluster; do
            local name location endpoint
            name=$(echo "${cluster}" | jq -r '.name')
            location=$(echo "${cluster}" | jq -r '.location')
            endpoint=$(echo "${cluster}" | jq -r '.endpoint')

            # Re-fetch full cluster description to be sure databaseEncryption is present
            local full
            full=$(gcloud container clusters describe "${name}" \
              --project "${project}" \
              --region "${location}" \
              --format=json 2>/dev/null || \
              gcloud container clusters describe "${name}" \
                --project "${project}" \
                --zone "${location}" \
                --format=json 2>/dev/null || echo "{}")

            local db_state kms_key_name
            db_state=$(echo "${full}" | jq -r '.databaseEncryption.state // "UNSPECIFIED"')
            kms_key_name=$(echo "${full}" | jq -r '.databaseEncryption.keyName // ""')

            echo
            echo "Cluster: ${name}"
            echo "  Location:            ${location}"
            echo "  API Endpoint:        ${endpoint}"
            echo "  databaseEncryption:"

            if [[ "${db_state}" == "ENCRYPTED" && -n "${kms_key_name}" && "${kms_key_name}" != "null" ]]; then
              echo "    state:             ENCRYPTED"
              echo "    keyName (CMEK):    ${kms_key_name}"
              echo "    STATUS:            OK (Secrets configured for KMS encryption at rest)"
            elif [[ "${db_state}" == "ENCRYPTED" && ( -z "${kms_key_name}" || "${kms_key_name}" == "null" ) ]]; then
              echo "    state:             ENCRYPTED"
              echo "    keyName:           <missing>"
              echo "    STATUS:            REVIEW (encrypted but keyName not surfaced; confirm in console)"
            else
              echo "    state:             ${db_state}"
              echo "    keyName:           ${kms_key_name}"
              echo "    STATUS:            FINDING (Secrets NOT using KMS envelope encryption)"
              echo "    ACTION NEEDED:     Plan cluster recreate or upgrade with 'Customer-managed"
              echo "                       encryption key' for application-layer Secret encryption."
            fi

            # Optional: lightweight kubectl sampling if context can be obtained
            if command -v kubectl >/dev/null 2>&1; then
              echo
              echo "  Attempting in-cluster sampling (non-deterministic, for review only)..."
              echo "    - Requires a kubeconfig/context for this cluster."

              # Try to get current context that matches this cluster, if any
              # (User may need to run: gcloud container clusters get-credentials ...)
              local matched_context
              matched_context=$(kubectl config get-contexts -o name 2>/dev/null | \
                grep -E "(^|/)${name}$" || true)

              if [[ -z "${matched_context}" ]]; then
                echo "    Skipped: No kubectl context found for cluster '${name}'."
                echo "    Hint:    Run:"
                echo "             gcloud container clusters get-credentials ${name} --project ${project} --region ${location}"
              else
                # Use the first matching context
                matched_context=$(echo "${matched_context}" | head -n1)
                echo "    Using kubectl context: ${matched_context}"
                # Sample a few Secrets (names only) to confirm existence
                set +e
                local sample_output
                sample_output=$(kubectl --context "${matched_context}" get secrets --all-namespaces \
                  --no-headers 2>/dev/null | head -n 5)
                local rc=$?
                set -e
                if [[ ${rc} -ne 0 ]]; then
                  echo "    Could not list Secrets with kubectl; check RBAC/credentials."
                else
                  echo "    Sampled Secrets (for operator review only):"
                  if [[ -z "${sample_output}" ]]; then
                    echo "      <no Secrets listed>"
                  else
                    echo "${sample_output}" | sed 's/^/      /'
                  fi
                  echo "    NOTE: This sampling does NOT prove encryption state; rely mainly"
                  echo "          on 'databaseEncryption.state' above."
                fi
              fi
            fi

          done
        }

        check_dependencies
        print_header

        for proj in "${PROJECTS[@]}"; do
          check_cluster_api_encryption "${proj}"
        done

        echo
        echo "====================================================================="
        echo "REVIEW GUIDANCE"
        echo "====================================================================="
        echo "Flag as a PROBLEM (requires remediation) when:"
        echo "  - 'databaseEncryption.state' is 'DECRYPTED', 'UNSPECIFIED', or 'null', OR"
        echo "  - the 'databaseEncryption' block is completely absent."
        echo
        echo "Consider COMPLIANT when:"
        echo "  - 'databaseEncryption.state' is 'ENCRYPTED' AND"
        echo "  - 'databaseEncryption.keyName' is a valid Cloud KMS key resource path."
        echo
        echo "Because this is a MANUAL check, final compliance determination should be"
        echo "made by reviewing:"
        echo "  - The cluster configuration in the GCP console (Security -> Encryption)"
        echo "  - Your provisioning IaC (Terraform, Deployment Manager, etc.)."
        ```

        **What output indicates a problem**

        * For any cluster where the script prints:

          * `STATUS: FINDING (Secrets NOT using KMS envelope encryption)`\
            or
          * `databaseEncryption.state: DECRYPTED`, `UNSPECIFIED`, or missing,

        that cluster does not have GKE Secret KMS encryption at rest configured and should be treated as non-compliant with “Secrets Should Be Encrypted At Rest” until reviewed and remediated.
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>