API Server Should Set Kubelet Client Certificate And Key

More Info:

Verifies that --kubelet-client-certificate and --kubelet-client-key are set so the API server authenticates to kubelets over TLS. Without them the apiserver-kubelet connection is not mutually authenticated.

Risk Level

High

Address

Security

Compliance Standards

CIS Kubernetes

Triage and Remediation

Remediation

Remediation

Manual Steps

Generate or obtain a dedicated client certificate and key for the apiserver to authenticate to kubelets (run on any secure machine, then copy to each control plane node):

# Example: create a key and CSR (adjust CN/O/hosts per your PKI policy)
openssl genrsa -out kubelet-client.key 2048
openssl req -new -key kubelet-client.key -out kubelet-client.csr -subj "/CN=kube-apiserver-kubelet-client"
# Sign kubelet-client.csr with your cluster CA, producing kubelet-client.crt
# Then securely copy kubelet-client.crt and kubelet-client.key to each control plane node:
scp kubelet-client.crt kubelet-client.key root@CONTROL_PLANE_IP:/etc/kubernetes/pki/

On every control plane node, back up the existing API server manifest:

sudo cp -a /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml.bak

On every control plane node, edit the API server static pod manifest to add the kubelet client certificate and key flags (this edit will cause the kube-apiserver static pod to restart):

sudo sed -i '/- kube-apiserver/a\  - --kubelet-client-certificate=/etc/kubernetes/pki/kubelet-client.crt\n  - --kubelet-client-key=/etc/kubernetes/pki/kubelet-client.key' /etc/kubernetes/manifests/kube-apiserver.yaml

Confirm the resulting kube-apiserver.yaml includes:

    - kube-apiserver
    - --kubelet-client-certificate=/etc/kubernetes/pki/kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/kubelet-client.key

Ensure file permissions on the certificate and key are appropriately restricted on every control plane node:

sudo chown root:root /etc/kubernetes/pki/kubelet-client.crt /etc/kubernetes/pki/kubelet-client.key
sudo chmod 600 /etc/kubernetes/pki/kubelet-client.key
sudo chmod 644 /etc/kubernetes/pki/kubelet-client.crt

Wait for the kube-apiserver static pod to be recreated and become Ready (run on any machine with kubectl configured):

kubectl get pods -n kube-system -l component=kube-apiserver -o wide

Do not proceed until all kube-apiserver pods show STATUS as Running and READY as 1/1.

Verify on every control plane node that the kube-apiserver process is now running with the required flags:

/bin/ps -ef | grep kube-apiserver | grep -v grep

Confirm the output includes both --kubelet-client-certificate=/etc/kubernetes/pki/kubelet-client.crt and --kubelet-client-key=/etc/kubernetes/pki/kubelet-client.key.

Using kubectl

kubectl cannot modify the API server’s host-level configuration or the static pod manifest at /etc/kubernetes/manifests/kube-apiserver.yaml on control plane nodes. To remediate this finding, you must edit that file directly on every control plane node; see the Manual Steps section for the exact procedure.

Automation

#!/usr/bin/env bash
#
# automate_kubelet_client_tls.sh
#
# Idempotently ensure kube-apiserver is configured with
# --kubelet-client-certificate and --kubelet-client-key on every control plane node.
#
# RUN THIS ON: every control plane node (with root privileges).
#
# Requirements:
#   - /etc/kubernetes/manifests/kube-apiserver.yaml (static pod)
#   - OpenSSL available
#
# Behavior:
#   - Creates a dedicated kubelet client cert/key if missing
#   - Updates kube-apiserver manifest to use them
#   - Kubelet will auto-restart the kube-apiserver static pod
#   - Safe to re-run

set -euo pipefail

APISERVER_MANIFEST="/etc/kubernetes/manifests/kube-apiserver.yaml"
CERT_DIR="/etc/kubernetes/pki"
CLIENT_KEY="${CERT_DIR}/apiserver-kubelet-client.key"
CLIENT_CSR="${CERT_DIR}/apiserver-kubelet-client.csr"
CLIENT_CERT="${CERT_DIR}/apiserver-kubelet-client.crt"
CA_CERT="${CERT_DIR}/ca.crt"
CA_KEY="${CERT_DIR}/ca.key"
BACKUP_SUFFIX="$(date +%Y%m%d%H%M%S)"

if [[ $EUID -ne 0 ]]; then
  echo "ERROR: This script must be run as root." >&2
  exit 1
fi

if [[ ! -f "${APISERVER_MANIFEST}" ]]; then
  echo "ERROR: kube-apiserver manifest not found at ${APISERVER_MANIFEST}" >&2
  exit 1
fi

if [[ ! -f "${CA_CERT}" || ! -f "${CA_KEY}" ]]; then
  echo "ERROR: CA files ${CA_CERT} and/or ${CA_KEY} not found." >&2
  echo "       Ensure cluster CA is present before running this script." >&2
  exit 1
fi

mkdir -p "${CERT_DIR}"
chmod 700 "${CERT_DIR}"

# 1) Create kubelet client key/cert if missing
if [[ ! -f "${CLIENT_KEY}" || ! -f "${CLIENT_CERT}" ]]; then
  echo "Generating kubelet client key and certificate in ${CERT_DIR}..."

  openssl genrsa -out "${CLIENT_KEY}" 2048
  chmod 600 "${CLIENT_KEY}"

  cat > "${CLIENT_CSR}.cnf" <<EOF
[ req ]
default_bits       = 2048
prompt             = no
distinguished_name = dn
req_extensions     = v3_req

[ dn ]
CN = kube-apiserver-kubelet-client

[ v3_req ]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
EOF

  openssl req -new -key "${CLIENT_KEY}" -out "${CLIENT_CSR}" -config "${CLIENT_CSR}.cnf"

  openssl x509 -req -in "${CLIENT_CSR}" \
    -CA "${CA_CERT}" -CAkey "${CA_KEY}" -CAcreateserial \
    -out "${CLIENT_CERT}" -days 3650 -extensions v3_req -extfile "${CLIENT_CSR}.cnf"

  chmod 644 "${CLIENT_CERT}"
  rm -f "${CLIENT_CSR}" "${CLIENT_CSR}.cnf"
fi

# 2) Backup existing manifest once per run
BACKUP_FILE="${APISERVER_MANIFEST}.${BACKUP_SUFFIX}.bak"
cp "${APISERVER_MANIFEST}" "${BACKUP_FILE}"
chmod --reference="${APISERVER_MANIFEST}" "${BACKUP_FILE}"
echo "Backup of kube-apiserver manifest created at ${BACKUP_FILE}"

# 3) Ensure flags exist or are updated
#   Handles both "--flag=value" and "--flag value" styles idempotently.

ensure_flag() {
  local flag="$1"
  local value="$2"
  local file="$3"
  local tmp
  tmp="$(mktemp)"

  # Remove any existing occurrences (both styles)
  sed -E "/${flag}(=|\s)/d" "${file}" > "${tmp}"
  mv "${tmp}" "${file}"

  # Add new flag under the 'command:' or argument list
  # We simply append it as a new line with proper YAML list item.
  # This is safe for manifests using standard 'command: [...]' args.
  if grep -qE '^\s*- kube-apiserver' "${file}"; then
    # Append directly after the kube-apiserver command entry
    tmp="$(mktemp)"
    awk -v flag="${flag}" -v value="${value}" '
      {
        print $0
        if ($0 ~ /^\s*- kube-apiserver/) {
          print "    - " flag "=" value
        }
      }' "${file}" > "${tmp}"
    mv "${tmp}" "${file}"
  else
    # Fallback: append at end of file as an extra argument line
    echo "    - ${flag}=${value}" >> "${file}"
  fi
}

ensure_flag "--kubelet-client-certificate" "${CLIENT_CERT}" "${APISERVER_MANIFEST}"
ensure_flag "--kubelet-client-key" "${CLIENT_KEY}" "${APISERVER_MANIFEST}"

echo "Updated ${APISERVER_MANIFEST} with kubelet client certificate and key flags."
echo "NOTE: Editing a static pod manifest causes kubelet to restart the kube-apiserver pod."

# 4) Verification: confirm kube-apiserver is running with the expected flags
echo "Waiting for kube-apiserver process to pick up new flags..."
sleep 20

echo "Verification output (should show kubelet-client-certificate and kubelet-client-key):"
/bin/ps -ef | grep kube-apiserver | grep -v grep || {
  echo "WARNING: kube-apiserver process not found. Check kubelet and pod status." >&2
  exit 1
}

if /bin/ps -ef | grep kube-apiserver | grep -v grep | grep -q -- "--kubelet-client-certificate=${CLIENT_CERT}" \
   && /bin/ps -ef | grep kube-apiserver | grep -v grep | grep -q -- "--kubelet-client-key=${CLIENT_KEY}"; then
  echo "SUCCESS: kube-apiserver is running with the configured kubelet client certificate and key."
else
  echo "ERROR: kube-apiserver process does not show the expected kubelet client TLS flags." >&2
  echo "       Inspect ${APISERVER_MANIFEST} and kube-apiserver pod logs." >&2
  exit 1
fi

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​