The --rotate-certificates Argument Is Not Present Or Is Set

More Info:

Enabling certificate rotation lets the kubelet automatically renew its client certificates before they expire, reducing the risk of using long-lived or expired credentials.

Risk Level

High

Address

Security

Compliance Standards

• CIS EKS

Triage and Remediation

Remediation

Remediation

Manual Steps

1. On every worker node, back up the kubelet configuration and systemd drop-in files:

sudo cp -a /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak.$(date +%s)
sudo cp -a /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf /etc/systemd/system/kubelet-args.conf.bak.$(date +%s)

2. Edit /var/lib/kubelet/config.yaml to enable certificate rotation (YAML key name is case-sensitive to your deployment; if rotateCertificates exists, set it to true, otherwise add it under the top-level kubeletConfiguration block):

sudo vi /var/lib/kubelet/config.yaml

Ensure it contains:

rotateCertificates: true

3. Edit the kubelet systemd drop-in to ensure no flag disables rotation:

sudo vi /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf

In the KUBELET_ARGS (or Environment="KUBELET_ARGS=...") line:

• Remove any occurrence of --rotate-certificates=false.
• If you manage kubelet only via flags and not config file, ensure --rotate-certificates=true is present, for example:

Environment="KUBELET_ARGS=... --rotate-certificates=true"

4. Reload systemd and restart kubelet (this restarts kubelet and may briefly impact node status/workloads):

sudo systemctl daemon-reload
sudo systemctl restart kubelet

5. Verify kubelet is running with certificate rotation enabled:

/bin/ps -fC kubelet

Confirm the output either:

• Shows --rotate-certificates=true and does not contain --rotate-certificates=false, or
• Does not show any --rotate-certificates flag and you have rotateCertificates: true in /var/lib/kubelet/config.yaml.

Using kubectl

kubectl cannot configure kubelet process flags or its config file on worker nodes, so it cannot be used to set rotate-certificates or edit /var/lib/kubelet/config.yaml or the systemd drop-ins. Make the changes directly on each worker node’s host configuration as described in the Manual Steps section.

Automation

#!/usr/bin/env bash
#
# Enable kubelet certificate rotation on all worker nodes.
# Run on each worker node (e.g. via SSH/Ansible). Requires root.

set -euo pipefail

KUBELET_CFG_FILE="/var/lib/kubelet/config.yaml"
KUBELET_ARGS_DROPIN="/etc/systemd/system/kubelet.service.d/10-kubelet-args.conf"

backup_file() {
  local f="$1"
  if [ -f "$f" ] && [ ! -f "${f}.bak" ]; then
    cp -p "$f" "${f}.bak"
  fi
}

ensure_rotate_cert_in_config() {
  local f="$1"

  # Create minimal config if missing
  if [ ! -f "$f" ]; then
    mkdir -p "$(dirname "$f")"
    cat > "$f" <<'EOF'
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
rotateCertificates: true
EOF
    return
  fi

  # If key exists, set to true; else append it
  if grep -qE '^[[:space:]]*rotateCertificates:' "$f"; then
    # Replace any existing value with true (idempotent)
    sed -i -E 's/^[[:space:]]*rotateCertificates:[[:space:]]*(true|false|["'\'']?[^"'\''[:space:]]+["'\'']?).*/rotateCertificates: true/' "$f"
  else
    # Append under a reasonable place (end of file)
    printf '\nrotateCertificates: true\n' >> "$f"
  fi
}

ensure_rotate_cert_in_args() {
  local f="$1"

  [ -d "$(dirname "$f")" ] || mkdir -p "$(dirname "$f")"

  if [ ! -f "$f" ]; then
    cat > "$f" <<'EOF'
[Service]
Environment="KUBELET_ARGS=--rotate-certificates=true"
EOF
    return
  fi

  # Ensure no explicit false flag remains
  sed -i -E 's/--rotate-certificates=false//g' "$f"

  # If true already present, nothing more to do
  if grep -q -- '--rotate-certificates=true' "$f"; then
    return
  fi

  # If KUBELET_ARGS environment line exists, append the flag if not present
  if grep -q '^Environment="KUBELET_ARGS=' "$f"; then
    # Append flag safely inside the quoted string
    sed -i -E 's#^(Environment="KUBELET_ARGS=.*)"$#\1 --rotate-certificates=true"#' "$f"
  else
    # Add a new Environment line
    printf 'Environment="KUBELET_ARGS=--rotate-certificates=true"\n' >> "$f"
  fi
}

main() {
  if [ "$(id -u)" -ne 0 ]; then
    echo "This script must be run as root." >&2
    exit 1
  fi

  echo "Configuring kubelet certificate rotation on node: $(hostname)"

  # Backup files once
  backup_file "$KUBELET_CFG_FILE"
  backup_file "$KUBELET_ARGS_DROPIN"

  # Apply remediation method 1 (config file)
  ensure_rotate_cert_in_config "$KUBELET_CFG_FILE"

  # Ensure no conflicting false arg and/or explicitly set true if using args
  ensure_rotate_cert_in_args "$KUBELET_ARGS_DROPIN"

  # Reload systemd and restart kubelet to apply changes (causes kubelet restart)
  systemctl daemon-reload
  systemctl restart kubelet

  # Verification (adapted from audit command)
  echo "Verifying kubelet process flags on node: $(hostname)"
  /bin/ps -fC kubelet || {
    echo "kubelet process not found after restart." >&2
    exit 1
  }

  # Confirm that rotate-certificates flag is either absent or set to true
  if /bin/ps -fC kubelet | grep -q -- '--rotate-certificates=false'; then
    echo "Verification failed: kubelet still has --rotate-certificates=false" >&2
    exit 1
  fi

  echo "Verification passed: kubelet certificate rotation is enabled (flag absent or true)."
}

main "$@"