Skip to main content

Ensure Terminated Pod GC Threshold Argument Appropriate

More Info:​

Activate garbage collector on pod termination, as appropriate

Risk Level​

Low

Address​

Security

Compliance Standards​

  • CIS Kubernetes

Triage and Remediation​

Remediation​

Manual Steps

  1. On every control plane node, back up the current manifest so you can roll back if needed:

    sudo cp -a /etc/kubernetes/manifests/kube-controller-manager.yaml \
      /etc/kubernetes/manifests/kube-controller-manager.yaml.bak.$(date +%F-%H%M%S)

  2. On every control plane node, open the manifest for editing:

    sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml

  3. In the spec.containers[0].command or spec.containers[0].args list, add or modify the terminated pod GC flag to the threshold you decide is appropriate for your cluster (example: 10). For example, ensure there is a line similar to:

    - --terminated-pod-gc-threshold=10

    Place it alongside the other -- flags for kube-controller-manager. Save and exit.
    Note: Editing this static pod manifest will cause the kube-controller-manager pod to restart automatically.

  4. (Optional sanity check) Immediately after saving, confirm the static pod has been recreated and is running:

    sudo crictl ps | grep kube-controller-manager || sudo docker ps | grep kube-controller-manager

    (Use whichever container runtime CLI is available on your node.)

  5. On every control plane node, verify that the running process now includes the configured threshold value:

    /bin/ps -ef | grep kube-controller-manager | grep -v grep

    Confirm the output shows --terminated-pod-gc-threshold=10 (or your chosen value).

Using kubectl

kubectl cannot modify the kube-controller-manager static pod manifest or its process flags, so this setting cannot be fixed through the Kubernetes API. To remediate, you must edit /etc/kubernetes/manifests/kube-controller-manager.yaml directly on every control plane node; follow the instructions in the Manual Steps section.

Automation
#!/usr/bin/env bash
#
# Automation: Configure --terminated-pod-gc-threshold in kube-controller-manager manifest
# Scope: run on every control plane node
#
# Usage:
#   sudo bash ./fix-terminated-pod-gc-threshold.sh
#
# This script:
#   - Backs up /etc/kubernetes/manifests/kube-controller-manager.yaml
#   - Ensures --terminated-pod-gc-threshold=<VALUE> is present in the container args
#   - Is idempotent and safe to re-run
#   - Verifies by inspecting the running kube-controller-manager process
#
# NOTE: Editing a static pod manifest under /etc/kubernetes/manifests will restart
#       the kube-controller-manager on this control plane node.

set -euo pipefail

MANIFEST="/etc/kubernetes/manifests/kube-controller-manager.yaml"
BACKUP_DIR="/etc/kubernetes/manifests/backup-terminated-pod-gc-threshold"
# Set your desired threshold here (benchmark example uses 10)
DESIRED_THRESHOLD="10"

echo "[*] Configuring --terminated-pod-gc-threshold=${DESIRED_THRESHOLD} in ${MANIFEST}"

if [[ ! -f "${MANIFEST}" ]]; then
  echo "[!] Manifest not found at ${MANIFEST}. This script must run on a control plane node."
  exit 1
fi

mkdir -p "${BACKUP_DIR}"

TS="$(date -u +%Y%m%dT%H%M%SZ)"
cp -a "${MANIFEST}" "${BACKUP_DIR}/kube-controller-manager.yaml.${TS}"

echo "[*] Backed up current manifest to ${BACKUP_DIR}/kube-controller-manager.yaml.${TS}"

TMP_MANIFEST="$(mktemp)"
cp -a "${MANIFEST}" "${TMP_MANIFEST}"

# If an existing --terminated-pod-gc-threshold flag is present, normalize it to the desired value.
# This avoids duplicate flags and keeps the script idempotent.
if grep -q -- '--terminated-pod-gc-threshold=' "${TMP_MANIFEST}"; then
  echo "[*] Existing --terminated-pod-gc-threshold found; updating to ${DESIRED_THRESHOLD}"
  # Replace any existing value with the desired one
  sed -i "s/--terminated-pod-gc-threshold=[0-9]\+/--terminated-pod-gc-threshold=${DESIRED_THRESHOLD}/" "${TMP_MANIFEST}"
else
  echo "[*] No existing --terminated-pod-gc-threshold flag found; adding it to args"

  # We need to inject a new arg line under the kube-controller-manager container.
  # This uses a conservative awk transformation:
  # - finds the 'name: kube-controller-manager' container
  # - finds its 'args:' block
  # - adds a new '- --terminated-pod-gc-threshold=VALUE' entry if not present
  awk -v threshold="${DESIRED_THRESHOLD}" '
    BEGIN {
      in_kcm = 0
      in_args = 0
      inserted = 0
    }
    /name:[[:space:]]*kube-controller-manager/ {
      in_kcm = 1
    }
    in_kcm && /^[[:space:]]*args:[[:space:]]*$/ {
      in_args = 1
    }
    # Detect end of args: new top-level key at same or less indent than "args:"
    in_args && /^[[:space:]]*[A-Za-z0-9_-]+:/ && $1 != "args:" {
      if (!inserted) {
        # Insert the new arg just before leaving args block
        printf("          - --terminated-pod-gc-threshold=%s\n", threshold)
        inserted = 1
      }
      in_args = 0
    }
    { print }
    END {
      if (in_kcm && in_args && !inserted) {
        # If file ended while still in args block, append the arg
        printf("          - --terminated-pod-gc-threshold=%s\n", threshold)
      }
    }
  ' "${TMP_MANIFEST}" > "${TMP_MANIFEST}.new"

  mv "${TMP_MANIFEST}.new" "${TMP_MANIFEST}"
fi

# Replace the live manifest atomically
mv "${TMP_MANIFEST}" "${MANIFEST}"

echo "[*] Updated ${MANIFEST}. kube-controller-manager static pod will be restarted by kubelet."

echo "[*] Waiting for kube-controller-manager process to reflect new flag..."
# Simple wait loop to give kubelet time to restart the static pod
RETRY=30
SLEEP_SEC=5
OK=0

for i in $(seq 1 "${RETRY}"); do
  if /bin/ps -ef | grep kube-controller-manager | grep -v grep | grep -q -- "--terminated-pod-gc-threshold=${DESIRED_THRESHOLD}"; then
    OK=1
    break
  fi
  sleep "${SLEEP_SEC}"
done

echo
echo "[*] Verification (process flags):"
/bin/ps -ef | grep kube-controller-manager | grep -v grep || true

if [[ "${OK}" -eq 1 ]]; then
  echo "[*] SUCCESS: kube-controller-manager is running with --terminated-pod-gc-threshold=${DESIRED_THRESHOLD}"
  exit 0
else
  echo "[!] WARNING: kube-controller-manager process does not yet show --terminated-pod-gc-threshold=${DESIRED_THRESHOLD}"
  echo "    Check the pod status with: kubectl -n kube-system get pods -l component=kube-controller-manager"
  exit 1
fi

Additional Reading:​