Ensure Kubelet Certificate Authority Argument Is Appropriate

More Info:

Verify kubelets certificate before establishing connection.

Risk Level

Medium

Address

Security

Compliance Standards

• CIS Kubernetes

Triage and Remediation

Remediation

Remediation

Manual Steps

1. On every control plane node, identify the kube-apiserver static pod manifest and back it up:

sudo ls -l /etc/kubernetes/manifests/kube-apiserver.yaml
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml.bak.$(date +%F-%H%M%S)

2. Identify the correct CA file that signs kubelet serving certificates (commonly under /etc/kubernetes/pki/):

sudo ls -l /etc/kubernetes/pki/
sudo openssl x509 -in /etc/kubernetes/pki/ca.crt -text -noout | head -n 10

(Replace /etc/kubernetes/pki/ca.crt with the appropriate kubelet CA file if different in your environment.)

3. Edit the API server manifest to set --kubelet-certificate-authority to that CA file (this will restart the kube-apiserver because it is a static pod):

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Under spec.containers[0].command, either add or update the argument so that there is a line like:

    - --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt

4. Ensure the CA file is mounted into the kube-apiserver container if it is not already. In the same file, under spec.volumes, confirm or add:

  - name: k8s-certs
    hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate

And under spec.containers[0].volumeMounts, confirm or add:

    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true

5. Save the file and exit the editor. The kubelet will detect the manifest change and restart the kube-apiserver. Check that the static pod is running again:

sudo crictl ps | grep kube-apiserver || sudo docker ps | grep kube-apiserver

6. Verify that the kube-apiserver is now running with the correct --kubelet-certificate-authority argument:

/bin/ps -ef | grep kube-apiserver | grep -v grep | tr ' ' '\n' | grep -- --kubelet-certificate-authority

Using kubectl

kubectl cannot modify the kube-apiserver static pod manifest or its process flags, so this finding cannot be fixed via the Kubernetes API. To remediate it, you must edit /etc/kubernetes/manifests/kube-apiserver.yaml directly on every control plane node; see the Manual Steps section for the exact host-level procedure.

Automation

#!/usr/bin/env bash
#
# Automation: Ensure kube-apiserver is started with --kubelet-certificate-authority
# Applies to: every control plane node
# Run on: each control plane node over SSH
#
# This script:
#   - Backs up /etc/kubernetes/manifests/kube-apiserver.yaml
#   - Ensures a kubelet CA file exists at /etc/kubernetes/pki/kubelet-ca.crt
#   - Adds or updates --kubelet-certificate-authority in the kube-apiserver manifest
#   - Triggers kube-apiserver restart via kubelet static pod behavior
#   - Verifies the flag is present in the running process
#
# Assumptions:
#   - kube-apiserver is a static pod defined at /etc/kubernetes/manifests/kube-apiserver.yaml
#   - Desired CA path: /etc/kubernetes/pki/kubelet-ca.crt
#
# Adjust KUBELET_CA_PATH below if you use a different CA path.

set -euo pipefail

KUBE_APISERVER_MANIFEST="/etc/kubernetes/manifests/kube-apiserver.yaml"
BACKUP_SUFFIX=".bak-$(date +%Y%m%d%H%M%S)"
KUBELET_CA_PATH="/etc/kubernetes/pki/kubelet-ca.crt"

echo "==> Ensuring kubelet CA and kube-apiserver flag configuration"

# 1. Basic sanity checks
if [[ $EUID -ne 0 ]]; then
  echo "ERROR: This script must be run as root on each control plane node." >&2
  exit 1
fi

if [[ ! -f "$KUBE_APISERVER_MANIFEST" ]]; then
  echo "ERROR: kube-apiserver manifest not found at $KUBE_APISERVER_MANIFEST" >&2
  exit 1
fi

# 2. Ensure kubelet CA file exists (create placeholder if missing)
if [[ ! -f "$KUBELET_CA_PATH" ]]; then
  echo "WARNING: Kubelet CA not found at $KUBELET_CA_PATH."
  echo "Creating placeholder file. Replace this with the real CA certificate."
  mkdir -p "$(dirname "$KUBELET_CA_PATH")"
  touch "$KUBELET_CA_PATH"
  chmod 600 "$KUBELET_CA_PATH"
fi

# 3. Backup manifest
if ! grep -q "kubelet-certificate-authority" "$KUBE_APISERVER_MANIFEST"; then
  # Only create a backup once per run; always safe
  cp "$KUBE_APISERVER_MANIFEST" "${KUBE_APISERVER_MANIFEST}${BACKUP_SUFFIX}"
  echo "Backup created at ${KUBE_APISERVER_MANIFEST}${BACKUP_SUFFIX}"
fi

# 4. Ensure --kubelet-certificate-authority is set in the manifest
#    Idempotent update:
#      - If flag exists, replace its value with KUBELET_CA_PATH
#      - If flag does not exist, add a new - --kubelet-certificate-authority=... line

if grep -q -- "--kubelet-certificate-authority=" "$KUBE_APISERVER_MANIFEST"; then
  echo "Updating existing --kubelet-certificate-authority flag in manifest"
  # Replace entire argument value
  sed -i \
    "s#--kubelet-certificate-authority=.*#--kubelet-certificate-authority=${KUBELET_CA_PATH}#g" \
    "$KUBE_APISERVER_MANIFEST"
else
  echo "Adding --kubelet-certificate-authority flag to manifest"

  # Insert under the existing --kubelet-client-certificate or --kubelet-client-key if present,
  # otherwise append at the end of the args list.
  if grep -q -- "--kubelet-client-certificate" "$KUBE_APISERVER_MANIFEST"; then
    # Insert after kubelet-client-certificate line
    sed -i \
      "/--kubelet-client-certificate/a\    - --kubelet-certificate-authority=${KUBELET_CA_PATH}" \
      "$KUBE_APISERVER_MANIFEST"
  elif grep -q -- "--kubelet-client-key" "$KUBE_APISERVER_MANIFEST"; then
    # Insert after kubelet-client-key line
    sed -i \
      "/--kubelet-client-key/a\    - --kubelet-certificate-authority=${KUBELET_CA_PATH}" \
      "$KUBE_APISERVER_MANIFEST"
  else
    # Fallback: append at the end of args list
    # This is still idempotent because we only reach here if flag is absent.
    if grep -q "^- --" "$KUBE_APISERVER_MANIFEST"; then
      # append near the last argument line
      sed -i \
        '$a\    - --kubelet-certificate-authority='"${KUBELET_CA_PATH}" \
        "$KUBE_APISERVER_MANIFEST"
    else
      echo "WARNING: Could not reliably locate args list in $KUBE_APISERVER_MANIFEST."
      echo "Manually ensure the following argument is present under spec.containers.args:"
      echo "    - --kubelet-certificate-authority=${KUBELET_CA_PATH}"
    fi
  fi
fi

echo "Manifest updated. kube-apiserver static pod will be restarted automatically by kubelet."

# 5. Wait for kube-apiserver to restart and pick up new flags
echo "Waiting for kube-apiserver process to reflect updated flags..."
RETRY=30
SLEEP=5
SUCCESS=0

for ((i=1; i<=RETRY; i++)); do
  if /bin/ps -ef | grep kube-apiserver | grep -v grep | grep -q -- "--kubelet-certificate-authority=${KUBELET_CA_PATH}"; then
    SUCCESS=1
    break
  fi
  sleep "$SLEEP"
done

# 6. Verification (matches benchmark audit approach)
echo "==> Verification: checking running kube-apiserver flags"
/bin/ps -ef | grep kube-apiserver | grep -v grep || true

if [[ $SUCCESS -eq 1 ]]; then
  echo "PASSED: kube-apiserver is running with --kubelet-certificate-authority=${KUBELET_CA_PATH}"
  exit 0
else
  echo "FAILED: kube-apiserver process does not show --kubelet-certificate-authority=${KUBELET_CA_PATH}" >&2
  echo "Investigate kube-apiserver pod status and logs. You may need to fix YAML indentation or CA path."
  exit 1
fi

Additional Reading:

• https://kubernetes.io/docs/admin/kubelet-authentication-authorization/