Skip to main content

API Server Should Set Encryption Provider Config

More Info:​

Verifies that --encryption-provider-config is set so secrets are encrypted at rest in etcd rather than stored in plaintext.

Risk Level​

High

Address​

Security

Compliance Standards​

  • CIS Kubernetes

Triage and Remediation​

Remediation​

Manual Steps
  1. On every control plane node, create a secure directory and EncryptionConfig file (adjust key value if you generate your own):
sudo mkdir -p /etc/kubernetes/encryption
sudo chmod 700 /etc/kubernetes/encryption

cat << 'EOF' | sudo tee /etc/kubernetes/encryption/encryption-config.yaml >/dev/null
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: dGhpc2lzMzJieXRlbG9uZ2VuY3J5cHRpb25rZXk=
      - identity: {}
EOF

sudo chmod 600 /etc/kubernetes/encryption/encryption-config.yaml
sudo chown root:root /etc/kubernetes/encryption/encryption-config.yaml
  1. On every control plane node, edit the API server static pod manifest to add the encryption-provider-config flag:
sudo sed -i '/- kube-apiserver/a\    - --encryption-provider-config=/etc/kubernetes/encryption/encryption-config.yaml' /etc/kubernetes/manifests/kube-apiserver.yaml

(If the flag already exists with a different path, edit that line instead to set it to /etc/kubernetes/encryption/encryption-config.yaml.)

  1. On every control plane node, ensure the EncryptionConfig file is mounted into the kube-apiserver pod by adding a volume and volumeMount if they are not present. Edit /etc/kubernetes/manifests/kube-apiserver.yaml with a root editor:
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Under spec.containers[0].volumeMounts add:

        - mountPath: /etc/kubernetes/encryption
          name: encryption-config
          readOnly: true

Under spec.volumes add:

      - name: encryption-config
        hostPath:
          path: /etc/kubernetes/encryption
          type: DirectoryOrCreate

Save and exit; kubelet will automatically restart the kube-apiserver static pod when the manifest changes.

  1. On any machine with kubectl access, after the apiserver pods are running again, re-encrypt existing Secret resources so they are stored encrypted in etcd. First label all namespaces that should be processed (example: all non-system namespaces):
for ns in $(kubectl get ns --no-headers | awk '!/kube-system|kube-public|kube-node-lease|default/ {print $1}'); do
  kubectl get secrets -n "$ns" -o json | kubectl replace -f -
done
  1. On every control plane node, verify that the kube-apiserver process is now using the encryption-provider-config flag:
/bin/ps -ef | grep kube-apiserver | grep -v grep | grep -- '--encryption-provider-config=/etc/kubernetes/encryption/encryption-config.yaml'

A matching line in the output confirms the setting is active.

Using kubectl

kubectl cannot modify the API server’s host-level static pod manifest or its process flags. This finding must be remediated directly on every control plane node by editing /etc/kubernetes/manifests/kube-apiserver.yaml and configuring the encryption provider config there; see the Manual Steps section for the full procedure.

Automation
#!/usr/bin/env bash
#
# automate-kube-apiserver-encryption.sh
#
# Idempotently ensure kube-apiserver uses an EncryptionConfig file and
# has --encryption-provider-config set in /etc/kubernetes/manifests/kube-apiserver.yaml.
#
# Run on: every control plane node (as root).
#
# Assumptions:
# - kube-apiserver is deployed as a static Pod from /etc/kubernetes/manifests/kube-apiserver.yaml
# - etcd data directory is /var/lib/etcd (adjust ETCD_PREFIX/dir as needed)
#
# This script:
#   1. Creates /etc/kubernetes/encryption-config.yaml if missing.
#   2. Ensures kube-apiserver manifest has --encryption-provider-config flag pointing to it.
#   3. Triggers static pod restart automatically by touching the manifest.
#   4. Verifies the process has the flag set.
#
# NOTE: This config enables AES-CBC encryption for secrets with a single key.
#       Review and customize per your security requirements.

set -euo pipefail

MANIFEST_PATH="/etc/kubernetes/manifests/kube-apiserver.yaml"
ENC_CONFIG_DIR="/etc/kubernetes"
ENC_CONFIG_PATH="${ENC_CONFIG_DIR}/encryption-config.yaml"
BACKUP_SUFFIX=".pre-encryption.bak"
# etcd encryption only affects new/updated secrets; existing ones remain in plaintext
# until rewritten. This script does NOT rewrite existing secrets.

require_root() {
  if [[ "$(id -u)" -ne 0 ]]; then
    echo "ERROR: This script must be run as root on a control plane node." >&2
    exit 1
  fi
}

check_files_exist() {
  if [[ ! -f "${MANIFEST_PATH}" ]]; then
    echo "ERROR: kube-apiserver manifest not found at ${MANIFEST_PATH}." >&2
    echo "This script supports static-pod control planes only." >&2
    exit 1
  fi
}

backup_file_once() {
  local src="$1"
  local backup="${src}${BACKUP_SUFFIX}"
  if [[ -f "${src}" && ! -f "${backup}" ]]; then
    cp -p "${src}" "${backup}"
    echo "Backup created: ${backup}"
  fi
}

generate_encryption_key() {
  # Generate a 32-byte base64 key suitable for AES-CBC
  openssl rand -base64 32
}

ensure_encryption_config() {
  mkdir -p "${ENC_CONFIG_DIR}"
  chmod 700 "${ENC_CONFIG_DIR}"

  if [[ -f "${ENC_CONFIG_PATH}" ]]; then
    echo "EncryptionConfig already exists at ${ENC_CONFIG_PATH}; leaving as-is."
    return
  fi

  echo "Creating new EncryptionConfig at ${ENC_CONFIG_PATH} ..."
  local key
  key="$(generate_encryption_key)"

  cat > "${ENC_CONFIG_PATH}" <<EOF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: ${key}
      - identity: {}
EOF

  chmod 600 "${ENC_CONFIG_PATH}"
  echo "EncryptionConfig created."
}

ensure_manifest_has_flag() {
  local manifest="${MANIFEST_PATH}"
  local flag="--encryption-provider-config=${ENC_CONFIG_PATH}"

  if grep -q -- "${flag}" "${manifest}"; then
    echo "kube-apiserver manifest already contains ${flag}."
    return
  fi

  backup_file_once "${manifest}"

  echo "Updating kube-apiserver manifest to include ${flag} ..."

  # Insert the flag in the command section under spec.containers[0]
  # This uses a conservative YAML edit with awk/sed and is idempotent.
  tmpfile="$(mktemp)"
  trap 'rm -f "${tmpfile}"' EXIT

  awk -v flag="${flag}" '
    BEGIN { in_cmd=0; inserted=0 }
    {
      if ($0 ~ /^[[:space:]]*command:[[:space:]]*$/) {
        in_cmd=1
      } else if (in_cmd && $0 !~ /^[[:space:]]*-/) {
        # End of command list
        if (!inserted) {
          print "    - " flag
          inserted=1
        }
        in_cmd=0
      }
      print $0
    }
    END {
      if (in_cmd && !inserted) {
        # command list ran to EOF
        print "    - " flag
      }
    }
  ' "${manifest}" > "${tmpfile}"

  mv "${tmpfile}" "${manifest}"

  echo "Manifest updated. kubelet will restart kube-apiserver static pod automatically."
}

verify_process_flag() {
  echo "Waiting for kube-apiserver process to reflect new flag ..."
  # Wait up to 120 seconds for restart
  local timeout=120
  local interval=5
  local elapsed=0

  while (( elapsed < timeout )); do
    if /bin/ps -ef | grep kube-apiserver | grep -v grep | grep -q -- "--encryption-provider-config=${ENC_CONFIG_PATH}"; then
      echo "VERIFICATION SUCCESS: kube-apiserver is running with --encryption-provider-config=${ENC_CONFIG_PATH}"
      /bin/ps -ef | grep kube-apiserver | grep -v grep
      return 0
    fi
    sleep "${interval}"
    elapsed=$((elapsed + interval))
  done

  echo "VERIFICATION WARNING: kube-apiserver process does not show --encryption-provider-config=${ENC_CONFIG_PATH} after ${timeout}s." >&2
  echo "Please check kubelet and kube-apiserver logs for issues." >&2
  /bin/ps -ef | grep kube-apiserver | grep -v grep || true
  return 1
}

main() {
  require_root
  check_files_exist
  ensure_encryption_config
  ensure_manifest_has_flag
  verify_process_flag
}

main "$@"