> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Ensure Terminated Pod GC Threshold Argument Appropriate

### More Info:

Activate garbage collector on pod termination, as appropriate

### Risk Level

Low

### Address

Security

### Compliance Standards

* CIS Kubernetes

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Manual Steps" defaultOpen="true">
        1. On every control plane node, back up the current manifest so you can roll back if needed:
           ```bash theme={null}
           sudo cp -a /etc/kubernetes/manifests/kube-controller-manager.yaml \
             /etc/kubernetes/manifests/kube-controller-manager.yaml.bak.$(date +%F-%H%M%S)
           ```

        2. On every control plane node, open the manifest for editing:
           ```bash theme={null}
           sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml
           ```

        3. In the `spec.containers[0].command` or `spec.containers[0].args` list, add or modify the terminated pod GC flag to the threshold you decide is appropriate for your cluster (example: 10). For example, ensure there is a line similar to:
           ```yaml theme={null}
           - --terminated-pod-gc-threshold=10
           ```
           Place it alongside the other `--` flags for `kube-controller-manager`. Save and exit.\
           Note: Editing this static pod manifest will cause the kube-controller-manager pod to restart automatically.

        4. (Optional sanity check) Immediately after saving, confirm the static pod has been recreated and is running:
           ```bash theme={null}
           sudo crictl ps | grep kube-controller-manager || sudo docker ps | grep kube-controller-manager
           ```
           (Use whichever container runtime CLI is available on your node.)

        5. On every control plane node, verify that the running process now includes the configured threshold value:
           ```bash theme={null}
           /bin/ps -ef | grep kube-controller-manager | grep -v grep
           ```
           Confirm the output shows `--terminated-pod-gc-threshold=10` (or your chosen value).
      </Accordion>

      <Accordion title="Using kubectl">
        kubectl cannot modify the kube-controller-manager static pod manifest or its process flags, so this setting cannot be fixed through the Kubernetes API. To remediate, you must edit `/etc/kubernetes/manifests/kube-controller-manager.yaml` directly on every control plane node; follow the instructions in the Manual Steps section.
      </Accordion>

      <Accordion title="Automation">
        ```bash theme={null}
        #!/usr/bin/env bash
        #
        # Automation: Configure --terminated-pod-gc-threshold in kube-controller-manager manifest
        # Scope: run on every control plane node
        #
        # Usage:
        #   sudo bash ./fix-terminated-pod-gc-threshold.sh
        #
        # This script:
        #   - Backs up /etc/kubernetes/manifests/kube-controller-manager.yaml
        #   - Ensures --terminated-pod-gc-threshold=<VALUE> is present in the container args
        #   - Is idempotent and safe to re-run
        #   - Verifies by inspecting the running kube-controller-manager process
        #
        # NOTE: Editing a static pod manifest under /etc/kubernetes/manifests will restart
        #       the kube-controller-manager on this control plane node.

        set -euo pipefail

        MANIFEST="/etc/kubernetes/manifests/kube-controller-manager.yaml"
        BACKUP_DIR="/etc/kubernetes/manifests/backup-terminated-pod-gc-threshold"
        # Set your desired threshold here (benchmark example uses 10)
        DESIRED_THRESHOLD="10"

        echo "[*] Configuring --terminated-pod-gc-threshold=${DESIRED_THRESHOLD} in ${MANIFEST}"

        if [[ ! -f "${MANIFEST}" ]]; then
          echo "[!] Manifest not found at ${MANIFEST}. This script must run on a control plane node."
          exit 1
        fi

        mkdir -p "${BACKUP_DIR}"

        TS="$(date -u +%Y%m%dT%H%M%SZ)"
        cp -a "${MANIFEST}" "${BACKUP_DIR}/kube-controller-manager.yaml.${TS}"

        echo "[*] Backed up current manifest to ${BACKUP_DIR}/kube-controller-manager.yaml.${TS}"

        TMP_MANIFEST="$(mktemp)"
        cp -a "${MANIFEST}" "${TMP_MANIFEST}"

        # If an existing --terminated-pod-gc-threshold flag is present, normalize it to the desired value.
        # This avoids duplicate flags and keeps the script idempotent.
        if grep -q -- '--terminated-pod-gc-threshold=' "${TMP_MANIFEST}"; then
          echo "[*] Existing --terminated-pod-gc-threshold found; updating to ${DESIRED_THRESHOLD}"
          # Replace any existing value with the desired one
          sed -i "s/--terminated-pod-gc-threshold=[0-9]\+/--terminated-pod-gc-threshold=${DESIRED_THRESHOLD}/" "${TMP_MANIFEST}"
        else
          echo "[*] No existing --terminated-pod-gc-threshold flag found; adding it to args"

          # We need to inject a new arg line under the kube-controller-manager container.
          # This uses a conservative awk transformation:
          # - finds the 'name: kube-controller-manager' container
          # - finds its 'args:' block
          # - adds a new '- --terminated-pod-gc-threshold=VALUE' entry if not present
          awk -v threshold="${DESIRED_THRESHOLD}" '
            BEGIN {
              in_kcm = 0
              in_args = 0
              inserted = 0
            }
            /name:[[:space:]]*kube-controller-manager/ {
              in_kcm = 1
            }
            in_kcm && /^[[:space:]]*args:[[:space:]]*$/ {
              in_args = 1
            }
            # Detect end of args: new top-level key at same or less indent than "args:"
            in_args && /^[[:space:]]*[A-Za-z0-9_-]+:/ && $1 != "args:" {
              if (!inserted) {
                # Insert the new arg just before leaving args block
                printf("          - --terminated-pod-gc-threshold=%s\n", threshold)
                inserted = 1
              }
              in_args = 0
            }
            { print }
            END {
              if (in_kcm && in_args && !inserted) {
                # If file ended while still in args block, append the arg
                printf("          - --terminated-pod-gc-threshold=%s\n", threshold)
              }
            }
          ' "${TMP_MANIFEST}" > "${TMP_MANIFEST}.new"

          mv "${TMP_MANIFEST}.new" "${TMP_MANIFEST}"
        fi

        # Replace the live manifest atomically
        mv "${TMP_MANIFEST}" "${MANIFEST}"

        echo "[*] Updated ${MANIFEST}. kube-controller-manager static pod will be restarted by kubelet."

        echo "[*] Waiting for kube-controller-manager process to reflect new flag..."
        # Simple wait loop to give kubelet time to restart the static pod
        RETRY=30
        SLEEP_SEC=5
        OK=0

        for i in $(seq 1 "${RETRY}"); do
          if /bin/ps -ef | grep kube-controller-manager | grep -v grep | grep -q -- "--terminated-pod-gc-threshold=${DESIRED_THRESHOLD}"; then
            OK=1
            break
          fi
          sleep "${SLEEP_SEC}"
        done

        echo
        echo "[*] Verification (process flags):"
        /bin/ps -ef | grep kube-controller-manager | grep -v grep || true

        if [[ "${OK}" -eq 1 ]]; then
          echo "[*] SUCCESS: kube-controller-manager is running with --terminated-pod-gc-threshold=${DESIRED_THRESHOLD}"
          exit 0
        else
          echo "[!] WARNING: kube-controller-manager process does not yet show --terminated-pod-gc-threshold=${DESIRED_THRESHOLD}"
          echo "    Check the pod status with: kubectl -n kube-system get pods -l component=kube-controller-manager"
          exit 1
        fi
        ```
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>

### Additional Reading:

* [https://kubernetes.io/docs/admin/kube-controller-manager/](https://kubernetes.io/docs/admin/kube-controller-manager/)