API Server Should Set Etcd Client Certificate And Key
More Info:
Verifies that --etcd-certfile and --etcd-keyfile are set so the API server authenticates to etcd over mutual TLS, protecting the cluster datastore.
Risk Level
High
Address
Security
Compliance Standards
- CIS Kubernetes
Triage and Remediation
- Remediation
Remediation
Manual Steps
-
On every control plane node, confirm the API server static pod manifest path:
ls -l /etc/kubernetes/manifests/kube-apiserver.yamlIf the file is missing, locate it:
find /etc -maxdepth 4 -name 'kube-apiserver.yaml' -
On every control plane node, identify existing kubelet client cert/key files (or create them per your PKI process if they do not exist):
ls -l /etc/kubernetes/pki# Example (adjust to your environment):ls -l /etc/kubernetes/pki/kubelet-client.crt /etc/kubernetes/pki/kubelet-client.keyIf you must generate them, follow your organization’s CA process; ensure the certificate is trusted by kubelets.
-
On every control plane node, back up the API server manifest before editing:
cp /etc/kubernetes/manifests/kube-apiserver.yaml \/etc/kubernetes/manifests/kube-apiserver.yaml.bak.$(date +%Y%m%d%H%M%S) -
On every control plane node, edit the API server manifest to include the kubelet client certificate and key flags (this will restart the API server when the file changes):
sed -i '/- kube-apiserver/a\ - --kubelet-client-certificate=/etc/kubernetes/pki/kubelet-client.crt\n - --kubelet-client-key=/etc/kubernetes/pki/kubelet-client.key' \/etc/kubernetes/manifests/kube-apiserver.yamlAdjust the paths
/etc/kubernetes/pki/kubelet-client.crtand.keyto match your actual files. Ensure they appear as separate-arguments undercommand:in the manifest. -
On every control plane node, wait 30–60 seconds for the kubelet to detect the manifest change and restart the kube-apiserver container. Then verify the API server process includes the new flags:
/bin/ps -ef | grep kube-apiserver | grep -v grepConfirm you see both
--kubelet-client-certificate=/absolute/path/to/client-certificate-fileand--kubelet-client-key=/absolute/path/to/client-key-filein the command line.
Using kubectl
kubectl cannot modify the kube-apiserver static pod manifest or its process flags; this finding must be remediated by editing /etc/kubernetes/manifests/kube-apiserver.yaml directly on every control plane node. Refer to the Manual Steps section for the exact host-level changes and verification commands.
Automation
#!/usr/bin/env bash
#
# Fix CISKubernetes 1.2.4:
# Ensure --kubelet-client-certificate and --kubelet-client-key are set
# in /etc/kubernetes/manifests/kube-apiserver.yaml on every control plane node.
#
# Run on: every control plane node (as root).
#
# Assumptions:
# - kube-apiserver is a static pod defined at /etc/kubernetes/manifests/kube-apiserver.yaml
# - Client cert/key are already provisioned at:
# /etc/kubernetes/pki/apiserver-kubelet-client.crt
# /etc/kubernetes/pki/apiserver-kubelet-client.key
# (adjust CERT_PATH/KEY_PATH below if different)
#
# Operational impact:
# - Editing /etc/kubernetes/manifests/kube-apiserver.yaml will cause the
# kubelet to restart the kube-apiserver static pod automatically.
set -euo pipefail
CERT_PATH="/etc/kubernetes/pki/apiserver-kubelet-client.crt"
KEY_PATH="/etc/kubernetes/pki/apiserver-kubelet-client.key"
MANIFEST="/etc/kubernetes/manifests/kube-apiserver.yaml"
BACKUP_DIR="/etc/kubernetes/manifests/backup-cis-1.2.4"
TIMESTAMP="$(date +%Y%m%d-%H%M%S)"
echo ">>> [INFO] Verifying required files and paths"
if [[ ! -f "$MANIFEST" ]]; then
echo ">>> [ERROR] kube-apiserver manifest not found at $MANIFEST"
echo " This script expects a static pod manifest at that path."
exit 1
fi
if [[ ! -f "$CERT_PATH" ]]; then
echo ">>> [ERROR] kubelet client certificate not found at $CERT_PATH"
echo " Provision a valid client certificate, then re-run this script."
exit 1
fi
if [[ ! -f "$KEY_PATH" ]]; then
echo ">>> [ERROR] kubelet client key not found at $KEY_PATH"
echo " Provision a valid client key, then re-run this script."
exit 1
fi
echo ">>> [INFO] Creating backup directory $BACKUP_DIR (if needed)"
mkdir -p "$BACKUP_DIR"
BACKUP_FILE="${BACKUP_DIR}/kube-apiserver.yaml.${TIMESTAMP}"
cp "$MANIFEST" "$BACKUP_FILE"
echo ">>> [INFO] Backup created at $BACKUP_FILE"
echo ">>> [INFO] Ensuring --kubelet-client-certificate and --kubelet-client-key are set"
# Idempotently ensure arguments are present and correct in the manifest.
# We will:
# 1. Remove any existing occurrences of these flags.
# 2. Re-add them with the desired values under the kube-apiserver container args.
TMP_MANIFEST="$(mktemp)"
# Step 1: remove existing flags (if present)
# We match lines that contain the flags and trim them; this is safe to re-run.
sed -E \
-e 's@--kubelet-client-certificate=[^[:space:]]*@@g' \
-e 's@--kubelet-client-key=[^[:space:]]*@@g' \
"$MANIFEST" > "$TMP_MANIFEST"
# Step 2: normalize spacing so we don't leave empty args
# Remove any occurrences of double spaces that may have been introduced
sed -E -i 's/[[:space:]]+/ /g' "$TMP_MANIFEST"
# Step 3: ensure flags are present under the kube-apiserver container args
# We will append them as separate args lines right after the "- kube-apiserver" command.
# This is a simple, robust text insertion that is safe to re-run because
# we removed any existing flags above.
if ! grep -q -- "--kubelet-client-certificate=${CERT_PATH}" "$TMP_MANIFEST"; then
# insert after the first occurrence of "- kube-apiserver"
awk -v cert="--kubelet-client-certificate=${CERT_PATH}" \
-v key="--kubelet-client-key=${KEY_PATH}" '
/- kube-apiserver/ && inserted == 0 {
print $0
indent = gensub(/(^[[:space:]]*).*/, "\\1", "g", $0)
print indent " " cert
print indent " " key
inserted = 1
next
}
{ print $0 }
' "$TMP_MANIFEST" > "${TMP_MANIFEST}.new"
mv "${TMP_MANIFEST}.new" "$TMP_MANIFEST"
fi
# Move updated manifest into place atomically
mv "$TMP_MANIFEST" "$MANIFEST"
echo ">>> [INFO] Updated $MANIFEST"
echo ">>> [INFO] kubelet will restart the kube-apiserver static pod automatically."
# Optional: give kubelet some time to detect and restart
sleep 10
echo ">>> [INFO] Verification: checking kube-apiserver process flags"
/bin/ps -ef | grep kube-apiserver | grep -v grep || {
echo ">>> [WARN] kube-apiserver process not currently visible; it may still be restarting."
}
echo ">>> [INFO] Ensuring flags are present in running process (once it is up)"
if /bin/ps -ef | grep kube-apiserver | grep -v grep | grep -q -- "--kubelet-client-certificate=${CERT_PATH}"; then
echo ">>> [OK] --kubelet-client-certificate is set to ${CERT_PATH} in kube-apiserver process"
else
echo ">>> [FAIL] --kubelet-client-certificate is NOT visible in kube-apiserver process"
fi
if /bin/ps -ef | grep kube-apiserver | grep -v grep | grep -q -- "--kubelet-client-key=${KEY_PATH}"; then
echo ">>> [OK] --kubelet-client-key is set to ${KEY_PATH} in kube-apiserver process"
else
echo ">>> [FAIL] --kubelet-client-key is NOT visible in kube-apiserver process"
fi
echo ">>> [INFO] Final manifest snippet for kubelet client flags:"
grep -n -- "kubelet-client-" "$MANIFEST" || echo ">>> [WARN] Flags not found in manifest (check file formatting)."