> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Ensure Kubelet Client Certificate And Key Arguments Are Appropriate

### More Info:

Enable certificate based kubelet authentication.

### Risk Level

Medium

### Address

Security

### Compliance Standards

* CIS Kubernetes

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Manual Steps" defaultOpen="true">
        1. **Create or obtain a valid client cert/key for the apiserver–kubelet connection** (every control plane node)
           * If you already have a designated client cert/key for kubelet auth, copy them to a secure location (e.g. `/etc/kubernetes/pki/kubelet-apiserver-client.crt` and `/etc/kubernetes/pki/kubelet-apiserver-client.key`) with permissions `600` and owned by `root:root`. Example:
           ```sh theme={null}
           sudo install -m 600 -o root -g root /path/to/client.crt /etc/kubernetes/pki/kubelet-apiserver-client.crt
           sudo install -m 600 -o root -g root /path/to/client.key /etc/kubernetes/pki/kubelet-apiserver-client.key
           ```
           * Ensure this certificate is trusted by the kubelets’ configured client CA and has appropriate client auth usage.

        2. **Back up the API server static pod manifest** (every control plane node)
           ```sh theme={null}
           sudo cp -a /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml.bak.$(date +%F-%H%M%S)
           ```

        3. **Edit the kube-apiserver static pod manifest to add the flags** (every control plane node)
           * Open the file in a root editor:
           ```sh theme={null}
           sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
           ```
           * Under `spec.containers[0].command`, add or modify the following entries (one per line, preserving YAML indentation and existing args):
           ```yaml theme={null}
             - --kubelet-client-certificate=/etc/kubernetes/pki/kubelet-apiserver-client.crt
             - --kubelet-client-key=/etc/kubernetes/pki/kubelet-apiserver-client.key
           ```
           * Save and exit.\
             **Operational impact:** because this is a static pod manifest, the kubelet will automatically restart the `kube-apiserver` pod with the new configuration.

        4. **Ensure the cert/key paths are mounted into the apiserver container if needed** (every control plane node)
           * In the same `/etc/kubernetes/manifests/kube-apiserver.yaml`, confirm there is a `volumeMounts` entry pointing to the directory with your cert/key and a matching `volumes` entry. For example:
           ```yaml theme={null}
               volumeMounts:
                 - mountPath: /etc/kubernetes/pki
                   name: k8s-certs
                   readOnly: true
           ...
             volumes:
               - name: k8s-certs
                 hostPath:
                   path: /etc/kubernetes/pki
                   type: DirectoryOrCreate
           ```
           * Adjust `path`/`mountPath` only if your cert/key directory differs. Save and exit; the apiserver will restart again if you changed the manifest.

        5. **Verify the API server is running with the correct flags** (every control plane node)\
           After the apiserver pod has restarted and is stable, run:
           ```sh theme={null}
           /bin/ps -ef | grep kube-apiserver | grep -v grep
           ```
           Confirm the printed command line includes both:\
           `--kubelet-client-certificate=/etc/kubernetes/pki/kubelet-apiserver-client.crt`\
           and\
           `--kubelet-client-key=/etc/kubernetes/pki/kubelet-apiserver-client.key`.
      </Accordion>

      <Accordion title="Using kubectl">
        kubectl cannot modify the kube-apiserver static pod manifest or its process flags, so this finding cannot be fixed via the Kubernetes API. The required changes must be made directly on every control plane node in `/etc/kubernetes/manifests/kube-apiserver.yaml`; see the Manual Steps section for the exact host-level remediation.
      </Accordion>

      <Accordion title="Automation">
        ```bash theme={null}
        #!/usr/bin/env bash
        #
        # Purpose: Ensure kube-apiserver is configured with
        #          --kubelet-client-certificate and --kubelet-client-key
        # Scope:   Run on every control plane node (as root)
        #
        # NOTE: You MUST provide valid, existing certificate and key paths below.
        #       These files must already exist on the node and be readable by the
        #       kube-apiserver container.

        set -euo pipefail

        ### USER CONFIGURATION - SET THESE PATHS CORRECTLY BEFORE RUNNING ###

        KUBELET_CLIENT_CERT="/etc/kubernetes/pki/kubelet-apiserver-client.crt"
        KUBELET_CLIENT_KEY="/etc/kubernetes/pki/kubelet-apiserver-client.key"
        KUBE_APISERVER_MANIFEST="/etc/kubernetes/manifests/kube-apiserver.yaml"

        #####################################################################

        if [[ $EUID -ne 0 ]]; then
          echo "This script must be run as root on each control plane node." >&2
          exit 1
        fi

        # Basic sanity checks for configured paths
        if [[ ! -f "$KUBE_APISERVER_MANIFEST" ]]; then
          echo "ERROR: kube-apiserver manifest not found at $KUBE_APISERVER_MANIFEST" >&2
          exit 1
        fi

        if [[ ! -f "$KUBELET_CLIENT_CERT" ]]; then
          echo "ERROR: kubelet client certificate not found at $KUBELET_CLIENT_CERT" >&2
          exit 1
        fi

        if [[ ! -f "$KUBELET_CLIENT_KEY" ]]; then
          echo "ERROR: kubelet client key not found at $KUBELET_CLIENT_KEY" >&2
          exit 1
        fi

        echo "Backing up $KUBE_APISERVER_MANIFEST ..."
        cp -p "$KUBE_APISERVER_MANIFEST" "${KUBE_APISERVER_MANIFEST}.$(date +%Y%m%d%H%M%S).bak"

        # Ensure flags exist and are set correctly (idempotent line-edit in manifest)
        # We operate directly on the manifest flags lines under 'kube-apiserver' container args.

        tmpfile="$(mktemp)"
        trap 'rm -f "$tmpfile"' EXIT

        # 1) Ensure --kubelet-client-certificate is present and set
        if grep -q -- '--kubelet-client-certificate=' "$KUBE_APISERVER_MANIFEST"; then
          # Replace existing value
          sed -E "s#(--kubelet-client-certificate=)[^\" ]*#\1${KUBELET_CLIENT_CERT}#g" \
            "$KUBE_APISERVER_MANIFEST" > "$tmpfile"
          mv "$tmpfile" "$KUBE_APISERVER_MANIFEST"
          echo "Updated existing --kubelet-client-certificate to $KUBELET_CLIENT_CERT"
        else
          # Insert new arg line under the kube-apiserver container args section
          # We insert after the --kubelet-certificate-authority or --client-ca-file if present,
          # otherwise after the first existing arg line.
          insert_done=false
          while IFS= read -r line; do
            echo "$line" >> "$tmpfile"
            if [[ "$insert_done" == "false" ]] && \
               [[ "$line" =~ --kubelet-certificate-authority= || "$line" =~ --client-ca-file= ]]; then
              echo "    - --kubelet-client-certificate=${KUBELET_CLIENT_CERT}" >> "$tmpfile"
              insert_done=true
            fi
          done < "$KUBE_APISERVER_MANIFEST"

          if [[ "$insert_done" == "false" ]]; then
            # Fallback: append near the end of args list
            mv "$tmpfile" "$tmpfile.1"
            insert_done=false
            while IFS= read -r line; do
              if [[ "$insert_done" == "false" && "$line" =~ ^[[:space:]]*args:[[:space:]]*$ ]]; then
                echo "$line" >> "$tmpfile"
                continue
              fi
              if [[ "$insert_done" == "false" && "$line" =~ ^[[:space:]]*-[[:space:]]*-- ]]; then
                echo "$line" >> "$tmpfile"
                # Lookahead: if next line is not an arg, insert here (simplistic but safe)
                read -r next || true
                if [[ "$next" =~ ^[[:space:]]*-[[:space:]]*-- ]]; then
                  echo "$next" >> "$tmpfile"
                else
                  if [[ -n "$next" ]]; then
                    echo "$next" >> "$tmpfile"
                  fi
                  echo "    - --kubelet-client-certificate=${KUBELET_CLIENT_CERT}" >> "$tmpfile"
                  insert_done=true
                  continue
                fi
                continue
              fi
              echo "$line" >> "$tmpfile"
            done < "$tmpfile.1"
            rm -f "$tmpfile.1"
            if [[ "$insert_done" == "false" ]]; then
              echo "WARNING: could not auto-place --kubelet-client-certificate; manual review of $KUBE_APISERVER_MANIFEST required." >&2
            fi
          fi

          mv "$tmpfile" "$KUBE_APISERVER_MANIFEST"
          echo "Ensured --kubelet-client-certificate=${KUBELET_CLIENT_CERT} is present"
        fi

        # 2) Ensure --kubelet-client-key is present and set
        tmpfile="$(mktemp)"
        trap 'rm -f "$tmpfile"' EXIT

        if grep -q -- '--kubelet-client-key=' "$KUBE_APISERVER_MANIFEST"; then
          sed -E "s#(--kubelet-client-key=)[^\" ]*#\1${KUBELET_CLIENT_KEY}#g" \
            "$KUBE_APISERVER_MANIFEST" > "$tmpfile"
          mv "$tmpfile" "$KUBE_APISERVER_MANIFEST"
          echo "Updated existing --kubelet-client-key to $KUBELET_CLIENT_KEY"
        else
          insert_done=false
          while IFS= read -r line; do
            echo "$line" >> "$tmpfile"
            if [[ "$insert_done" == "false" && "$line" =~ --kubelet-client-certificate= ]]; then
              echo "    - --kubelet-client-key=${KUBELET_CLIENT_KEY}" >> "$tmpfile"
              insert_done=true
            fi
          done < "$KUBE_APISERVER_MANIFEST"

          if [[ "$insert_done" == "false" ]]; then
            # Fallback similar to above: append near args
            mv "$tmpfile" "$tmpfile.1"
            insert_done=false
            while IFS= read -r line; do
              if [[ "$insert_done" == "false" && "$line" =~ ^[[:space:]]*args:[[:space:]]*$ ]]; then
                echo "$line" >> "$tmpfile"
                continue
              fi
              if [[ "$insert_done" == "false" && "$line" =~ ^[[:space:]]*-[[:space:]]*-- ]]; then
                echo "$line" >> "$tmpfile"
                read -r next || true
                if [[ "$next" =~ ^[[:space:]]*-[[:space:]]*-- ]]; then
                  echo "$next" >> "$tmpfile"
                else
                  if [[ -n "$next" ]]; then
                    echo "$next" >> "$tmpfile"
                  fi
                  echo "    - --kubelet-client-key=${KUBELET_CLIENT_KEY}" >> "$tmpfile"
                  insert_done=true
                  continue
                fi
                continue
              fi
              echo "$line" >> "$tmpfile"
            done < "$tmpfile.1"
            rm -f "$tmpfile.1"
            if [[ "$insert_done" == "false" ]]; then
              echo "WARNING: could not auto-place --kubelet-client-key; manual review of $KUBE_APISERVER_MANIFEST required." >&2
            fi
          fi

          mv "$tmpfile" "$KUBE_APISERVER_MANIFEST"
          echo "Ensured --kubelet-client-key=${KUBELET_CLIENT_KEY} is present"
        fi

        echo
        echo "Manifest updated. Because this is a static pod under /etc/kubernetes/manifests,"
        echo "kubelet will automatically restart the kube-apiserver with the new flags."
        echo "Waiting up to 120 seconds for kube-apiserver to restart..."

        # Simple wait loop for kube-apiserver process to be running with expected flags
        end=$((SECONDS+120))
        success=false

        while (( SECONDS < end )); do
          if /bin/ps -ef | grep "[k]ube-apiserver" >/dev/null 2>&1; then
            if /bin/ps -ef | grep "[k]ube-apiserver" | grep -q -- "--kubelet-client-certificate=${KUBELET_CLIENT_CERT}" && \
               /bin/ps -ef | grep "[k]ube-apiserver" | grep -q -- "--kubelet-client-key=${KUBELET_CLIENT_KEY}"; then
              success=true
              break
            fi
          fi
          sleep 5
        done

        echo
        if [[ "$success" == "true" ]]; then
          echo "Verification succeeded: kube-apiserver is running with the desired kubelet client certificate and key."
          echo "Current kube-apiserver command line:"
          /bin/ps -ef | grep "[k]ube-apiserver"
          exit 0
        else
          echo "WARNING: kube-apiserver did not show the expected flags within the timeout." >&2
          echo "Please manually review the process output below and the manifest at $KUBE_APISERVER_MANIFEST" >&2
          /bin/ps -ef | grep kube-apiserver | grep -v grep || true
          exit 1
        fi
        ```
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>

### Additional Reading:

* [https://kubernetes.io/docs/concepts/cluster-administration/master-node-communication/#apiserver---kubelet](https://kubernetes.io/docs/concepts/cluster-administration/master-node-communication/#apiserver---kubelet)