The RotateKubeletServerCertificate Argument Is Set To True

More Info:

Enabling the RotateKubeletServerCertificate feature gate lets the kubelet automatically request and rotate its serving certificate, avoiding the use of expired or long-lived server certificates.

Risk Level

High

Address

Security

Compliance Standards

CIS EKS

Triage and Remediation

Remediation

Remediation

Manual Steps

On every worker node, check how kubelet is configured:
```
ps -fC kubelet
```
Review the command line for any --config flag (config file) and any --rotate-kubelet-server-certificate flag (exec arg override).

If using a kubelet config file (recommended), edit it and enable the feature gate (example path from finding):

sudo sed -i 's/^featureGates:.*/featureGates:\n  RotateKubeletServerCertificate: true/' /var/lib/kubelet/config.yaml

If featureGates is not present, add this block under the top-level YAML:

sudo tee -a /var/lib/kubelet/config.yaml >/dev/null << 'EOF'

featureGates: RotateKubeletServerCertificate: true EOF

3. Ensure no systemd drop-in overrides the setting to false (if these files exist):  
```bash
sudo grep -R --color=always 'rotate-kubelet-server-certificate' /etc/systemd/system/kubelet.service.d || true

If you see --rotate-kubelet-server-certificate=false in /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf, edit it:

sudo sed -i 's/--rotate-kubelet-server-certificate=false/--rotate-kubelet-server-certificate=true/g' /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf

Or remove the flag entirely so the config file controls it.

If you are configuring kubelet only with executable arguments (no config file referenced in the ps output), append the flag in the systemd drop-in (create or edit as needed):
```
sudo mkdir -p /etc/systemd/system/kubelet.service.d
sudo tee /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf >/dev/null << 'EOF'
```

[Service] Environment="KUBELET_ARGS=--rotate-kubelet-server-certificate=true" EOF

5. Restart kubelet on the worker node to apply changes (operational impact: this briefly restarts the kubelet and may momentarily affect pod reporting):  
```bash
sudo systemctl daemon-reload
sudo systemctl restart kubelet.service
sudo systemctl status kubelet -l

Verify kubelet is running with the correct setting on the worker node:
```
ps -fC kubelet
```
Confirm either that --rotate-kubelet-server-certificate=true appears in the command line, or that no --rotate-kubelet-server-certificate=false flag is present and the config file has RotateKubeletServerCertificate: true.

Using kubectl

kubectl cannot be used to enable RotateKubeletServerCertificate because this setting is applied via kubelet host-level configuration (/var/lib/kubelet/config.yaml, /etc/kubernetes/kubelet/kubelet-config.json, and the kubelet systemd unit) on every worker node. To remediate this finding, make the changes directly on each node as described in the Manual Steps section.

Automation

#!/usr/bin/env bash
#
# Enable RotateKubeletServerCertificate on all worker nodes.
# Run on each worker node as root.
#
# Behavior:
# - Ensures featureGates.RotateKubeletServerCertificate=true in kubelet config file if present
# - Ensures no kubelet systemd args set it to false
# - Adds --rotate-kubelet-server-certificate=true to kubelet args if using flags
# - Restarts kubelet only if a change was made
# - Verifies setting via kubelet configz endpoint if possible, otherwise via process args

set -euo pipefail

CHANGED=0

echo "[INFO] Starting kubelet RotateKubeletServerCertificate remediation"

########################################
# Helper: backup a file once per run
########################################
backup_file() {
  local f="$1"
  if [[ -f "$f" ]]; then
    local backup="${f}.$(date +%Y%m%d%H%M%S).bak"
    cp -p "$f" "$backup"
    echo "[INFO] Backed up $f to $backup"
  fi
}

########################################
# Method 1: kubelet config file
########################################
# Primary path from finding
CONFIG_FILE="/var/lib/kubelet/config.yaml"
ALT_CONFIG_JSON="/etc/kubernetes/kubelet/kubelet-config.json"

update_yaml_feature_gate() {
  local file="$1"
  if ! command -v python3 >/dev/null 2>&1; then
    echo "[WARN] python3 not available; skipping structured edit of $file"
    return 1
  fi

  backup_file "$file"

  python3 - <<'PYEOF'
import sys, yaml, os, datetime

path = os.environ["KUBELET_CFG_PATH"]
with open(path) as f:
    data = yaml.safe_load(f) or {}

fg = data.get("featureGates")
if fg is None:
    fg = {}
    data["featureGates"] = fg

orig = fg.get("RotateKubeletServerCertificate")
if orig is True:
    # already correct
    sys.exit(10)

fg["RotateKubeletServerCertificate"] = True

bak = f"{path}.{datetime.datetime.now().strftime('%Y%m%d%H%M%S')}.pybak"
with open(bak, "w") as f:
    yaml.safe_dump(data, f, default_flow_style=False)

with open(path, "w") as f:
    yaml.safe_dump(data, f, default_flow_style=False)
PYEOF
  rc=$?
  if [[ $rc -eq 0 ]]; then
    echo "[INFO] Updated RotateKubeletServerCertificate in $file"
    return 0
  elif [[ $rc -eq 10 ]]; then
    echo "[INFO] RotateKubeletServerCertificate already true in $file"
    return 2
  else
    echo "[WARN] Failed to update $file via python/yaml"
    return 1
  fi
}

# Try YAML config first if it exists
if [[ -f "$CONFIG_FILE" ]]; then
  echo "[INFO] Detected kubelet config file at $CONFIG_FILE"
  KUBELET_CFG_PATH="$CONFIG_FILE" update_yaml_feature_gate "$CONFIG_FILE" || true
  case $? in
    0) CHANGED=1 ;;
    2) : ;; # no change
  esac
fi

# Try legacy JSON config if present (per remediation Method 1)
if [[ -f "$ALT_CONFIG_JSON" ]]; then
  echo "[INFO] Detected kubelet config file at $ALT_CONFIG_JSON"
  if command -v python3 >/dev/null 2>&1; then
    backup_file "$ALT_CONFIG_JSON"
    python3 - <<'PYEOF'
import sys, json, os, datetime

path = os.environ["KUBELET_CFG_JSON"]
with open(path) as f:
    data = json.load(f)

fg = data.get("featureGates")
if fg is None:
    fg = {}
    data["featureGates"] = fg

orig = fg.get("RotateKubeletServerCertificate")
if orig is True:
    sys.exit(10)

fg["RotateKubeletServerCertificate"] = True

bak = f"{path}.{datetime.datetime.now().strftime('%Y%m%d%H%M%S')}.pybak"
with open(bak, "w") as f:
    json.dump(data, f, indent=2, sort_keys=True)

with open(path, "w") as f:
    json.dump(data, f, indent=2, sort_keys=True)
PYEOF
    rc=$?
    if [[ $rc -eq 0 ]]; then
      echo "[INFO] Updated RotateKubeletServerCertificate in $ALT_CONFIG_JSON"
      CHANGED=1
    elif [[ $rc -eq 10 ]]; then
      echo "[INFO] RotateKubeletServerCertificate already true in $ALT_CONFIG_JSON"
    else
      echo "[WARN] Failed to update $ALT_CONFIG_JSON via python/json"
    fi
  else
    echo "[WARN] python3 not available; cannot safely edit $ALT_CONFIG_JSON"
  fi
fi

########################################
# Method 2: systemd kubelet args
########################################
KUBELET_DROPIN="/etc/systemd/system/kubelet.service.d/10-kubelet-args.conf"
if [[ -f "$KUBELET_DROPIN" ]]; then
  echo "[INFO] Processing kubelet systemd drop-in at $KUBELET_DROPIN"
  backup_file "$KUBELET_DROPIN"

  # Remove any explicit false override
  if grep -q -- '--rotate-kubelet-server-certificate=false' "$KUBELET_DROPIN"; then
    sed -i 's/--rotate-kubelet-server-certificate=false//g' "$KUBELET_DROPIN"
    echo "[INFO] Removed --rotate-kubelet-server-certificate=false from $KUBELET_DROPIN"
    CHANGED=1
  fi

  # Ensure true flag present in either Environment or KUBELET_ARGS
  if ! grep -q -- '--rotate-kubelet-server-certificate=true' "$KUBELET_DROPIN"; then
    if grep -q '^Environment="KUBELET_ARGS=' "$KUBELET_DROPIN"; then
      # Append to existing KUBELET_ARGS
      sed -i 's/^Environment="KUBELET_ARGS=\(.*\)"/Environment="KUBELET_ARGS=\1 --rotate-kubelet-server-certificate=true"/' "$KUBELET_DROPIN"
      echo "[INFO] Appended --rotate-kubelet-server-certificate=true to existing KUBELET_ARGS"
    else
      # Add a new Environment line
      cat <<'EOF' >> "$KUBELET_DROPIN"
Environment="KUBELET_ARGS=--rotate-kubelet-server-certificate=true"
EOF
      echo "[INFO] Added new KUBELET_ARGS with --rotate-kubelet-server-certificate=true"
    fi
    CHANGED=1
  else
    echo "[INFO] --rotate-kubelet-server-certificate=true already present in $KUBELET_DROPIN"
  fi
else
  echo "[INFO] No kubelet systemd drop-in at $KUBELET_DROPIN; assuming config file or other mechanism controls feature gates"
fi

########################################
# Restart kubelet if configuration changed
########################################
if [[ "$CHANGED" -eq 1 ]]; then
  echo "[INFO] Changes detected; restarting kubelet"
  systemctl daemon-reload
  systemctl restart kubelet.service
else
  echo "[INFO] No changes made; kubelet restart not required"
fi

echo "[INFO] Kubelet status:"
systemctl status kubelet -l --no-pager || true

########################################
# Verification (adapted from audit)
########################################
echo "[INFO] Verifying RotateKubeletServerCertificate setting"

# Preferred: configz endpoint (if running and accessible)
VERIFY_OK=0
if command -v curl >/dev/null 2>&1 && command -v jq >/dev/null 2>&1; then
  # Try common localhost configz endpoint
  if curl -fsS "http://127.0.0.1:10248/configz" >/dev/null 2>&1; then
    if curl -fsS "http://127.0.0.1:10248/configz" \
      | jq -e '.kubeletconfig.featureGates.RotateKubeletServerCertificate == true' >/dev/null 2>&1; then
      echo "[INFO] Verified via configz: RotateKubeletServerCertificate=true"
      VERIFY_OK=1
    else
      echo "[WARN] configz reachable but RotateKubeletServerCertificate is not true"
    fi
  fi
fi

# Fallback: check process arguments (original audit path)
if [[ "$VERIFY_OK" -ne 1 ]]; then
  echo "[INFO] Falling back to process-args verification"
  if /bin/ps -fC kubelet | grep -q -- '--rotate-kubelet-server-certificate=true'; then
    echo "[INFO] Verified: kubelet process includes --rotate-kubelet-server-certificate=true"
    VERIFY_OK=1
  else
    echo "[WARN] kubelet process does not show --rotate-kubelet-server-certificate=true"
  fi
fi

if [[ "$VERIFY_OK" -eq 1 ]]; then
  echo "[INFO] Remediation verification PASSED on this node"
  exit 0
else
  echo "[ERROR] Remediation verification FAILED on this node"
  exit 1
fi

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​