Kubelet Client Certificate And Key Arguments Are Appropriate

More Info:

Enable certificate based kubelet authentication.

Risk Level

Medium

Address

Security

Compliance Standards

CIS Kubernetes

Triage and Remediation

Remediation

Remediation

Manual Steps

Create or obtain a valid client cert/key for the apiserver–kubelet connection (every control plane node)
- If you already have a designated client cert/key for kubelet auth, copy them to a secure location (e.g. /etc/kubernetes/pki/kubelet-apiserver-client.crt and /etc/kubernetes/pki/kubelet-apiserver-client.key) with permissions 600 and owned by root:root. Example:
```
sudo install -m 600 -o root -g root /path/to/client.crt /etc/kubernetes/pki/kubelet-apiserver-client.crt
sudo install -m 600 -o root -g root /path/to/client.key /etc/kubernetes/pki/kubelet-apiserver-client.key
```
- Ensure this certificate is trusted by the kubelets’ configured client CA and has appropriate client auth usage.

Back up the API server static pod manifest (every control plane node)

sudo cp -a /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml.bak.$(date +%F-%H%M%S)

Edit the kube-apiserver static pod manifest to add the flags (every control plane node)
- Open the file in a root editor:
```
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
```
- Under spec.containers[0].command, add or modify the following entries (one per line, preserving YAML indentation and existing args):
```
  - --kubelet-client-certificate=/etc/kubernetes/pki/kubelet-apiserver-client.crt
  - --kubelet-client-key=/etc/kubernetes/pki/kubelet-apiserver-client.key
```
- Save and exit.
  Operational impact: because this is a static pod manifest, the kubelet will automatically restart the kube-apiserver pod with the new configuration.
Ensure the cert/key paths are mounted into the apiserver container if needed (every control plane node)
- In the same /etc/kubernetes/manifests/kube-apiserver.yaml, confirm there is a volumeMounts entry pointing to the directory with your cert/key and a matching volumes entry. For example:
```
    volumeMounts:
      - mountPath: /etc/kubernetes/pki
        name: k8s-certs
        readOnly: true
...
  volumes:
    - name: k8s-certs
      hostPath:
        path: /etc/kubernetes/pki
        type: DirectoryOrCreate
```
- Adjust path/mountPath only if your cert/key directory differs. Save and exit; the apiserver will restart again if you changed the manifest.
Verify the API server is running with the correct flags (every control plane node)
After the apiserver pod has restarted and is stable, run:
```
/bin/ps -ef | grep kube-apiserver | grep -v grep
```
Confirm the printed command line includes both:
--kubelet-client-certificate=/etc/kubernetes/pki/kubelet-apiserver-client.crt
and
--kubelet-client-key=/etc/kubernetes/pki/kubelet-apiserver-client.key.

Using kubectl

kubectl cannot modify the kube-apiserver static pod manifest or its process flags, so this finding cannot be fixed via the Kubernetes API. The required changes must be made directly on every control plane node in /etc/kubernetes/manifests/kube-apiserver.yaml; see the Manual Steps section for the exact host-level remediation.

Automation

#!/usr/bin/env bash
#
# Purpose: Ensure kube-apiserver is configured with
#          --kubelet-client-certificate and --kubelet-client-key
# Scope:   Run on every control plane node (as root)
#
# NOTE: You MUST provide valid, existing certificate and key paths below.
#       These files must already exist on the node and be readable by the
#       kube-apiserver container.

set -euo pipefail

### USER CONFIGURATION - SET THESE PATHS CORRECTLY BEFORE RUNNING ###

KUBELET_CLIENT_CERT="/etc/kubernetes/pki/kubelet-apiserver-client.crt"
KUBELET_CLIENT_KEY="/etc/kubernetes/pki/kubelet-apiserver-client.key"
KUBE_APISERVER_MANIFEST="/etc/kubernetes/manifests/kube-apiserver.yaml"

#####################################################################

if [[ $EUID -ne 0 ]]; then
  echo "This script must be run as root on each control plane node." >&2
  exit 1
fi

# Basic sanity checks for configured paths
if [[ ! -f "$KUBE_APISERVER_MANIFEST" ]]; then
  echo "ERROR: kube-apiserver manifest not found at $KUBE_APISERVER_MANIFEST" >&2
  exit 1
fi

if [[ ! -f "$KUBELET_CLIENT_CERT" ]]; then
  echo "ERROR: kubelet client certificate not found at $KUBELET_CLIENT_CERT" >&2
  exit 1
fi

if [[ ! -f "$KUBELET_CLIENT_KEY" ]]; then
  echo "ERROR: kubelet client key not found at $KUBELET_CLIENT_KEY" >&2
  exit 1
fi

echo "Backing up $KUBE_APISERVER_MANIFEST ..."
cp -p "$KUBE_APISERVER_MANIFEST" "${KUBE_APISERVER_MANIFEST}.$(date +%Y%m%d%H%M%S).bak"

# Ensure flags exist and are set correctly (idempotent line-edit in manifest)
# We operate directly on the manifest flags lines under 'kube-apiserver' container args.

tmpfile="$(mktemp)"
trap 'rm -f "$tmpfile"' EXIT

# 1) Ensure --kubelet-client-certificate is present and set
if grep -q -- '--kubelet-client-certificate=' "$KUBE_APISERVER_MANIFEST"; then
  # Replace existing value
  sed -E "s#(--kubelet-client-certificate=)[^\" ]*#\1${KUBELET_CLIENT_CERT}#g" \
    "$KUBE_APISERVER_MANIFEST" > "$tmpfile"
  mv "$tmpfile" "$KUBE_APISERVER_MANIFEST"
  echo "Updated existing --kubelet-client-certificate to $KUBELET_CLIENT_CERT"
else
  # Insert new arg line under the kube-apiserver container args section
  # We insert after the --kubelet-certificate-authority or --client-ca-file if present,
  # otherwise after the first existing arg line.
  insert_done=false
  while IFS= read -r line; do
    echo "$line" >> "$tmpfile"
    if [[ "$insert_done" == "false" ]] && \
       [[ "$line" =~ --kubelet-certificate-authority= || "$line" =~ --client-ca-file= ]]; then
      echo "    - --kubelet-client-certificate=${KUBELET_CLIENT_CERT}" >> "$tmpfile"
      insert_done=true
    fi
  done < "$KUBE_APISERVER_MANIFEST"

  if [[ "$insert_done" == "false" ]]; then
    # Fallback: append near the end of args list
    mv "$tmpfile" "$tmpfile.1"
    insert_done=false
    while IFS= read -r line; do
      if [[ "$insert_done" == "false" && "$line" =~ ^[[:space:]]*args:[[:space:]]*$ ]]; then
        echo "$line" >> "$tmpfile"
        continue
      fi
      if [[ "$insert_done" == "false" && "$line" =~ ^[[:space:]]*-[[:space:]]*-- ]]; then
        echo "$line" >> "$tmpfile"
        # Lookahead: if next line is not an arg, insert here (simplistic but safe)
        read -r next || true
        if [[ "$next" =~ ^[[:space:]]*-[[:space:]]*-- ]]; then
          echo "$next" >> "$tmpfile"
        else
          if [[ -n "$next" ]]; then
            echo "$next" >> "$tmpfile"
          fi
          echo "    - --kubelet-client-certificate=${KUBELET_CLIENT_CERT}" >> "$tmpfile"
          insert_done=true
          continue
        fi
        continue
      fi
      echo "$line" >> "$tmpfile"
    done < "$tmpfile.1"
    rm -f "$tmpfile.1"
    if [[ "$insert_done" == "false" ]]; then
      echo "WARNING: could not auto-place --kubelet-client-certificate; manual review of $KUBE_APISERVER_MANIFEST required." >&2
    fi
  fi

  mv "$tmpfile" "$KUBE_APISERVER_MANIFEST"
  echo "Ensured --kubelet-client-certificate=${KUBELET_CLIENT_CERT} is present"
fi

# 2) Ensure --kubelet-client-key is present and set
tmpfile="$(mktemp)"
trap 'rm -f "$tmpfile"' EXIT

if grep -q -- '--kubelet-client-key=' "$KUBE_APISERVER_MANIFEST"; then
  sed -E "s#(--kubelet-client-key=)[^\" ]*#\1${KUBELET_CLIENT_KEY}#g" \
    "$KUBE_APISERVER_MANIFEST" > "$tmpfile"
  mv "$tmpfile" "$KUBE_APISERVER_MANIFEST"
  echo "Updated existing --kubelet-client-key to $KUBELET_CLIENT_KEY"
else
  insert_done=false
  while IFS= read -r line; do
    echo "$line" >> "$tmpfile"
    if [[ "$insert_done" == "false" && "$line" =~ --kubelet-client-certificate= ]]; then
      echo "    - --kubelet-client-key=${KUBELET_CLIENT_KEY}" >> "$tmpfile"
      insert_done=true
    fi
  done < "$KUBE_APISERVER_MANIFEST"

  if [[ "$insert_done" == "false" ]]; then
    # Fallback similar to above: append near args
    mv "$tmpfile" "$tmpfile.1"
    insert_done=false
    while IFS= read -r line; do
      if [[ "$insert_done" == "false" && "$line" =~ ^[[:space:]]*args:[[:space:]]*$ ]]; then
        echo "$line" >> "$tmpfile"
        continue
      fi
      if [[ "$insert_done" == "false" && "$line" =~ ^[[:space:]]*-[[:space:]]*-- ]]; then
        echo "$line" >> "$tmpfile"
        read -r next || true
        if [[ "$next" =~ ^[[:space:]]*-[[:space:]]*-- ]]; then
          echo "$next" >> "$tmpfile"
        else
          if [[ -n "$next" ]]; then
            echo "$next" >> "$tmpfile"
          fi
          echo "    - --kubelet-client-key=${KUBELET_CLIENT_KEY}" >> "$tmpfile"
          insert_done=true
          continue
        fi
        continue
      fi
      echo "$line" >> "$tmpfile"
    done < "$tmpfile.1"
    rm -f "$tmpfile.1"
    if [[ "$insert_done" == "false" ]]; then
      echo "WARNING: could not auto-place --kubelet-client-key; manual review of $KUBE_APISERVER_MANIFEST required." >&2
    fi
  fi

  mv "$tmpfile" "$KUBE_APISERVER_MANIFEST"
  echo "Ensured --kubelet-client-key=${KUBELET_CLIENT_KEY} is present"
fi

echo
echo "Manifest updated. Because this is a static pod under /etc/kubernetes/manifests,"
echo "kubelet will automatically restart the kube-apiserver with the new flags."
echo "Waiting up to 120 seconds for kube-apiserver to restart..."

# Simple wait loop for kube-apiserver process to be running with expected flags
end=$((SECONDS+120))
success=false

while (( SECONDS < end )); do
  if /bin/ps -ef | grep "[k]ube-apiserver" >/dev/null 2>&1; then
    if /bin/ps -ef | grep "[k]ube-apiserver" | grep -q -- "--kubelet-client-certificate=${KUBELET_CLIENT_CERT}" && \
       /bin/ps -ef | grep "[k]ube-apiserver" | grep -q -- "--kubelet-client-key=${KUBELET_CLIENT_KEY}"; then
      success=true
      break
    fi
  fi
  sleep 5
done

echo
if [[ "$success" == "true" ]]; then
  echo "Verification succeeded: kube-apiserver is running with the desired kubelet client certificate and key."
  echo "Current kube-apiserver command line:"
  /bin/ps -ef | grep "[k]ube-apiserver"
  exit 0
else
  echo "WARNING: kube-apiserver did not show the expected flags within the timeout." >&2
  echo "Please manually review the process output below and the manifest at $KUBE_APISERVER_MANIFEST" >&2
  /bin/ps -ef | grep kube-apiserver | grep -v grep || true
  exit 1
fi

Additional Reading:

https://kubernetes.io/docs/concepts/cluster-administration/master-node-communication/#apiserver---kubelet

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​

Additional Reading:​