API Server Request Timeout Should Be Set As Appropriate

More Info:

Verifies that the API server --request-timeout argument is set appropriately to limit how long requests may run and protect against slow or hung connections.

Risk Level

Medium

Address

Security

Compliance Standards

• CIS Kubernetes

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Gather current configuration (each control plane node)

   sudo grep -n -- '--request-timeout' /etc/kubernetes/manifests/kube-apiserver.yaml || echo "not set"
   sudo ps -ef | grep kube-apiserver | grep -v grep

   • Confirm whether --request-timeout is present and note its value (or absence).

2. Review cluster usage and risk tolerance (any machine with access to cluster context/info)

   • Identify workloads that legitimately run long API operations (e.g., large list/watch, bulk custom resources, aggregating APIs).
   • Discuss with app and platform owners what an acceptable upper bound is for API request duration vs. user-facing timeouts and SLAs.

3. Decide on an appropriate timeout value

   • Use a finite value; typical ranges: 60s–300s for general-purpose clusters.
   • Choose a higher value only if you have known, legitimate long-running API calls and can tolerate the associated resource usage and DoS risk.
   • Record the chosen value (for example 300s) in your ops/runbook documentation.

4. Update the kube-apiserver static pod manifest (each control plane node)
   Edit the manifest:

   sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

   In the command: (or args:) list for kube-apiserver, add or adjust one line such as:

     - --request-timeout=300s

   • Ensure there is exactly one --request-timeout entry.
   • Save the file; the kubelet will automatically restart the API server pod (control plane impact: brief API unavailability during restart).

5. Verify the new setting (each control plane node)
   After the API server pod is back in Running state:

   sudo grep -n -- '--request-timeout' /etc/kubernetes/manifests/kube-apiserver.yaml
   sudo ps -ef | grep kube-apiserver | grep -v grep | grep -- '--request-timeout'

   • Confirm the running process shows the intended --request-timeout value.

6. Monitor for side effects (any machine with kubectl access)

   • Watch for client-facing timeouts or failed long-running operations:

     kubectl get events -A --sort-by=.lastTimestamp | tail -n 50

   • If legitimate operations are being cut off, repeat steps 2–5 to adjust the timeout to a better-fitting value.

Using kubectl

kubectl cannot modify the API server’s --request-timeout flag because it is configured on the host in the static pod manifest /etc/kubernetes/manifests/kube-apiserver.yaml on every control plane node. To address this finding, follow the guidance in the Manual Steps section on each control plane node.

Automation

#!/usr/bin/env bash
#
# Report kube-apiserver --request-timeout settings on all control plane nodes.
# Requirements:
#   - Run on any machine with kubectl access and cluster-admin privileges.
#   - kubectl must be configured to talk to the target cluster.
#
# This is READ-ONLY: it does not change anything.

set -euo pipefail

echo "Collecting kube-apiserver --request-timeout settings from all nodes..."
echo

# 1) Find all nodes that are labeled as control plane / master
CONTROL_PLANE_NODES=$(
  kubectl get nodes \
    -l 'node-role.kubernetes.io/master=,node-role.kubernetes.io/control-plane=' \
    -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true
)

if [ -z "${CONTROL_PLANE_NODES}" ]; then
  echo "No nodes labeled with node-role.kubernetes.io/master or node-role.kubernetes.io/control-plane found."
  echo "If this is a self-managed control plane, you may need to adjust the label selector."
  exit 1
fi

# 2) For each control plane node, try to read the static pod manifest and inspect the flag
for NODE in ${CONTROL_PLANE_NODES}; do
  echo "=== Node: ${NODE} ==="

  # Path is from the finding; we assume standard kubeadm/static pod layout.
  MANIFEST_PATH="/etc/kubernetes/manifests/kube-apiserver.yaml"

  # Use kubectl debug (where available) or privileged pod to read the manifest.
  # This is implemented as: create a temporary pod on the node and cat the file.
  # It is read-only and safe, but does require permission to schedule pods.

  # Create a short-lived pod that mounts /etc/kubernetes as hostPath
  POD_NAME="apiserver-timeout-inspect-${NODE//./-}-$$"
  kubectl run "${POD_NAME}" \
    --restart=Never \
    --image=busybox:1.36 \
    --overrides="$(
      cat <<EOF
{
  "apiVersion": "v1",
  "kind": "Pod",
  "metadata": {
    "name": "${POD_NAME}"
  },
  "spec": {
    "nodeName": "${NODE}",
    "hostPID": true,
    "tolerations": [
      {
        "operator": "Exists"
      }
    ],
    "containers": [
      {
        "name": "inspect",
        "image": "busybox:1.36",
        "command": ["sleep", "300"],
        "volumeMounts": [
          {
            "name": "k8s-etc",
            "mountPath": "/host-etc-kubernetes",
            "readOnly": true
          }
        ]
      }
    ],
    "volumes": [
      {
        "name": "k8s-etc",
        "hostPath": {
          "path": "/etc/kubernetes",
          "type": "Directory"
        }
      }
    ]
  }
}
EOF
    " >/dev/null 2>&1

  # Wait briefly for the pod to be running or failed
  kubectl wait pod "${POD_NAME}" --for=condition=Ready --timeout=60s >/dev/null 2>&1 || true

  echo "- Manifest: ${MANIFEST_PATH}"

  # Try to read the manifest
  if kubectl exec "${POD_NAME}" -- sh -c "test -f '${MANIFEST_PATH}'" >/dev/null 2>&1; then
    # Extract the request-timeout flag from the manifest
    TIMEOUT_LINE=$(kubectl exec "${POD_NAME}" -- sh -c \
      "grep -E -- '--request-timeout=' '${MANIFEST_PATH}' || true")

    if [ -z "${TIMEOUT_LINE}" ]; then
      echo "  request-timeout: NOT SET (kube-apiserver will use its compiled default, often 1m0s)"
    else
      # Try to isolate the value
      TIMEOUT_VALUE=$(printf '%s\n' "${TIMEOUT_LINE}" | \
        sed -n 's/.*--request-timeout=\([^[:space:]]*\).*/\1/p' | head -n1)

      if [ -z "${TIMEOUT_VALUE}" ]; then
        echo "  request-timeout: PRESENT but could not parse line:"
        echo "    ${TIMEOUT_LINE}"
      else
        echo "  request-timeout: ${TIMEOUT_VALUE}"
      fi
    fi
  else
    echo "  ERROR: ${MANIFEST_PATH} not found or not readable on this node."
  fi

  # Clean up the temporary pod
  kubectl delete pod "${POD_NAME}" --ignore-not-found --grace-period=0 --force >/dev/null 2>&1

  echo
done

cat <<'EOF'
Interpretation guidance:
- Problematic / requires review when:
  * "request-timeout: NOT SET" -> timeout is relying on the default and may not match your risk appetite.
  * "request-timeout: <very large value>" (e.g., many minutes or '0') -> long-running or hung requests
    can tie up apiserver worker threads and expose you to slowloris-style attacks.
  * "ERROR: ... not found" -> kube-apiserver is not using the expected static pod manifest path and must be investigated.

- Acceptable (subject to your environment and SLAs) when:
  * "request-timeout: <reasonable value>" such as 300s as in the benchmark example, or another value
    that is explicitly chosen and documented for your environment.

This script does NOT change the configuration; it only reports what is currently set so you can decide
whether a manual adjustment to /etc/kubernetes/manifests/kube-apiserver.yaml is warranted on each
control plane node.
EOF