Controller Manager Should Set Service Account Private Key

More Info:

Verifies that --service-account-private-key-file is set so the controller manager can sign service account tokens with a dedicated private key.

Risk Level

High

Address

Security

Compliance Standards

CIS Kubernetes

Triage and Remediation

Remediation

Remediation

Manual Steps

On every control plane node, confirm the controller manager is running as a static pod and note the manifest path:

sudo crictl ps | grep kube-controller-manager || sudo docker ps | grep kube-controller-manager
ls -l /etc/kubernetes/manifests/kube-controller-manager.yaml

On every control plane node, ensure you have or create a dedicated private key file for service accounts (adjust path/permissions if you already have a key):

sudo mkdir -p /etc/kubernetes/pki
sudo openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
sudo chmod 600 /etc/kubernetes/pki/sa.key
sudo chown root:root /etc/kubernetes/pki/sa.key

On every control plane node, back up the existing static pod manifest:

sudo cp -a /etc/kubernetes/manifests/kube-controller-manager.yaml /etc/kubernetes/manifests/kube-controller-manager.yaml.bak

On every control plane node, edit the controller manager manifest to set the private key file flag. Open the file:
```
sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml
```
In the command: (or args:) list for the kube-controller-manager container, add (or update) this entry so it is present exactly once:
```
- --service-account-private-key-file=/etc/kubernetes/pki/sa.key
```
Save and exit. Editing a file in /etc/kubernetes/manifests causes the kubelet to restart the kube-controller-manager pod automatically.

On every control plane node, wait for the controller manager pod to restart and become ready:

sudo crictl ps | grep kube-controller-manager || sudo docker ps | grep kube-controller-manager

On every control plane node, verify that the controller manager process now includes the --service-account-private-key-file flag with the correct path:
```
/bin/ps -ef | grep kube-controller-manager | grep -v grep | grep -- '--service-account-private-key-file=/etc/kubernetes/pki/sa.key'
```

Using kubectl

kubectl cannot modify the kube-controller-manager static pod manifest or its process flags. This finding must be remediated directly on each control plane node by editing /etc/kubernetes/manifests/kube-controller-manager.yaml; see the Manual Steps section for exact instructions.

Automation

#!/usr/bin/env bash
# Remediation: Ensure kube-controller-manager sets --service-account-private-key-file
# Scope: run on every control plane node (with root privileges)
# Idempotent: safe to re-run

set -euo pipefail

KCM_MANIFEST="/etc/kubernetes/manifests/kube-controller-manager.yaml"
SA_KEY_DIR="/etc/kubernetes/pki"
SA_KEY_FILE="${SA_KEY_DIR}/sa.key"
SA_KEY_ARG="--service-account-private-key-file=${SA_KEY_FILE}"

echo "[INFO] Starting remediation for kube-controller-manager on host: $(hostname)"

# 1. Preconditions
if [[ $EUID -ne 0 ]]; then
  echo "[ERROR] This script must be run as root." >&2
  exit 1
fi

if [[ ! -f "${KCM_MANIFEST}" ]]; then
  echo "[ERROR] Controller manager manifest not found at ${KCM_MANIFEST}." >&2
  exit 1
fi

mkdir -p "${SA_KEY_DIR}"
chmod 700 "${SA_KEY_DIR}"

# 2. Ensure a private key exists (only create if missing)
if [[ ! -f "${SA_KEY_FILE}" ]]; then
  echo "[INFO] Service account private key not found, generating: ${SA_KEY_FILE}"
  openssl genrsa -out "${SA_KEY_FILE}" 2048
  chmod 600 "${SA_KEY_FILE}"
else
  echo "[INFO] Service account private key already exists at ${SA_KEY_FILE}, leaving unchanged."
fi

# 3. Backup manifest
BACKUP="${KCM_MANIFEST}.$(date +%Y%m%d%H%M%S).bak"
cp "${KCM_MANIFEST}" "${BACKUP}"
echo "[INFO] Backup created at ${BACKUP}"

# 4. Ensure volume and volumeMount for the key are present (idempotent)
#    - volumeMount name: sa-private-key
#    - mountPath: /etc/kubernetes/pki
#    - hostPath: /etc/kubernetes/pki

# Add volumeMount under containers[].volumeMounts if missing
if ! grep -q 'name: sa-private-key' "${KCM_MANIFEST}"; then
  echo "[INFO] Adding sa-private-key volumeMount and volume to manifest."

  # Insert volumeMount (assumes standard kubeadm-like structure)
  # Insert under the first occurrence of 'volumeMounts:' in the controller manager container
  python3 - "$KCM_MANIFEST" << 'PYEOF'
import sys, ruamel.yaml
from copy import deepcopy

path = sys.argv[1]
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True

with open(path) as f:
    data = yaml.load(f)

spec = data.get('spec', {})
containers = spec.get('containers', [])
if not containers:
    sys.exit(0)

changed = False
for c in containers:
    if c.get('name') != 'kube-controller-manager':
        continue
    vm = c.setdefault('volumeMounts', [])
    if not any(m.get('name') == 'sa-private-key' for m in vm):
        vm.append({
            'name': 'sa-private-key',
            'mountPath': '/etc/kubernetes/pki',
            'readOnly': True
        })
        changed = True

vols = spec.setdefault('volumes', [])
if not any(v.get('name') == 'sa-private-key' for v in vols):
    vols.append({
        'name': 'sa-private-key',
        'hostPath': {'path': '/etc/kubernetes/pki', 'type': 'DirectoryOrCreate'}
    })
    changed = True

if changed:
    with open(path, 'w') as f:
        yaml.dump(data, f)
PYEOF

else
  echo "[INFO] sa-private-key volumeMount/volume already present, leaving as is."
fi

# 5. Ensure --service-account-private-key-file argument is set (idempotent)
if grep -qE '^\s*-+\s*--service-account-private-key-file(=|\s)' "${KCM_MANIFEST}"; then
  echo "[INFO] Existing --service-account-private-key-file argument found; normalizing value."

  # Replace any existing value with the desired path
  sed -i \
    -E "s#(--service-account-private-key-file(=|\s))[^\"'\s]+#\1${SA_KEY_FILE}#g" \
    "${KCM_MANIFEST}"
else
  echo "[INFO] Adding --service-account-private-key-file argument to kube-controller-manager container."

  # Append argument under the args: list for kube-controller-manager
  python3 - "$KCM_MANIFEST" << 'PYEOF'
import sys, ruamel.yaml
path = sys.argv[1]
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True

SA_KEY_ARG = "--service-account-private-key-file=/etc/kubernetes/pki/sa.key"

with open(path) as f:
    data = yaml.load(f)

spec = data.get('spec', {})
containers = spec.get('containers', [])
for c in containers:
    if c.get('name') != 'kube-controller-manager':
        continue
    args = c.setdefault('args', [])
    if SA_KEY_ARG not in args:
        args.append(SA_KEY_ARG)

with open(path, 'w') as f:
    yaml.dump(data, f)
PYEOF

fi

echo "[INFO] Manifest updated. Kubelet will automatically restart the kube-controller-manager static pod."

# 6. Wait for kube-controller-manager to restart and pick up the new flag
echo "[INFO] Waiting for kube-controller-manager process to reflect new argument..."
RETRIES=30
SLEEP=5
ok=false

for i in $(seq 1 $RETRIES); do
  if /bin/ps -ef | grep kube-controller-manager | grep -v grep | grep -q -- "${SA_KEY_ARG}"; then
    ok=true
    break
  fi
  sleep "$SLEEP"
done

# 7. Verification (as per audit command, plus flag check)
/bin/ps -ef | grep kube-controller-manager | grep -v grep || true

if [[ "${ok}" == "true" ]]; then
  echo "[SUCCESS] kube-controller-manager is running with ${SA_KEY_ARG}"
  exit 0
else
  echo "[WARNING] kube-controller-manager did not show ${SA_KEY_ARG} within timeout." >&2
  echo "[WARNING] Check kubelet and static pod status manually. Audit output above." >&2
  exit 1
fi

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​