Ensure Request Timeout Argument Is Appropriate

More Info:

Set global request timeout for API server requests as appropriate.

Risk Level

Medium

Address

Security

Compliance Standards

• CIS Kubernetes

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Gather current configuration (control plane node)

   ps -ef | grep kube-apiserver | grep -v grep
   cat /etc/kubernetes/manifests/kube-apiserver.yaml

   In the manifest, look under spec.containers[].command or ...args for --request-timeout=. If it is missing, the apiserver is using the default (60s as of recent versions).

2. Assess workload and operational requirements (any machine with access to cluster context)
   Identify long-running API calls (e.g., big list/watch, backup/export, custom controllers):

   kubectl get --raw /metrics | grep -E 'apiserver_request_duration_seconds_bucket|apiserver_request_total' || true
   kubectl get apiservices -o wide
   kubectl get crd -A

   Work with application/platform owners to determine the maximum expected duration for legitimate API calls and whether any clients (operators, backups, CI/CD) regularly exceed 60s.

3. Determine an appropriate timeout value (off-cluster decision step)
   Based on step 2, agree on a value that:

   • Is longer than normal legitimate requests (e.g., 120–300s for clusters with heavy list operations).
   • Is short enough to avoid hung connections consuming resources indefinitely.
     Document the chosen value (e.g., 300s) and the rationale.

4. Update the kube-apiserver manifest (every control plane node)
   Open the manifest for editing:

   sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

   In the container command/args list, add or adjust the flag to the chosen value, for example:

   - --request-timeout=300s

   Save the file. Because this is a static pod manifest, the kubelet will automatically restart the API server pod with the new setting; expect a brief control-plane disruption during restart.

5. Verify the new setting (every control plane node)
   After the API server pod restarts, confirm the flag is applied:

   ps -ef | grep kube-apiserver | grep -- '--request-timeout' | grep -v grep

   Ensure the output shows --request-timeout=300s (or your chosen value).

6. Monitor for side effects (any machine with kubectl access)
   Watch for errors/timeouts from controllers and clients after the change:

   kubectl get events -A --sort-by=.lastTimestamp | tail -n 50

   If you observe legitimate operations failing due to timeouts, revisit steps 2–3 and adjust --request-timeout accordingly, repeating steps 4–5.

Using kubectl

kubectl cannot modify the kube-apiserver pod manifest or its process flags, so it cannot be used to set the --request-timeout argument. This setting must be changed directly in /etc/kubernetes/manifests/kube-apiserver.yaml on every control plane node; see the Manual Steps section for how to review and adjust it.

Automation

#!/usr/bin/env bash
# Report kube-apiserver --request-timeout for all control-plane nodes
# Run on: any machine with kubectl access and cluster-admin permissions

set -euo pipefail

echo "Discovering control-plane nodes..."
control_planes="$(kubectl get nodes -o jsonpath='{range .items[?(@.metadata.labels.node-role\.kubernetes\.io/control-plane=="")]}.metadata.name{"\n"}{end}')"
if [ -z "$control_planes" ]; then
  # Older clusters may use master label
  control_planes="$(kubectl get nodes -o jsonpath='{range .items[?(@.metadata.labels.node-role\.kubernetes\.io/master=="")]}.metadata.name{"\n"}{end}')"
fi

if [ -z "$control_planes" ]; then
  echo "No control-plane nodes found via standard labels." >&2
  exit 1
fi

echo "Control-plane nodes detected:"
echo "$control_planes"
echo

for node in $control_planes; do
  echo "===== Node: $node ====="

  # Find the kube-apiserver pod name on this node
  pod="$(kubectl get pods -n kube-system \
    --field-selector spec.nodeName="$node" \
    -l component=kube-apiserver,tier=control-plane \
    -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)"

  if [ -z "$pod" ]; then
    # Fallback: look for any pod with 'kube-apiserver' in the name
    pod="$(kubectl get pods -n kube-system \
      --field-selector spec.nodeName="$node" \
      -o jsonpath='{range .items[?contains(@.metadata.name,"kube-apiserver")]}.metadata.name{"\n"}{end}' \
      | head -n1 || true)"
  fi

  if [ -z "$pod" ]; then
    echo "  [WARN] No kube-apiserver pod found on this node via kubectl."
    echo
    continue
  fi

  echo "  kube-apiserver pod: $pod"

  # Pull the command and args from the pod spec
  echo "  Reported container args (including --request-timeout if present):"
  kubectl get pod "$pod" -n kube-system -o jsonpath='{.spec.containers[0].command}{" "}{.spec.containers[0].args}{"\n"}' \
    | sed 's/ -/\n  -/g'

  # Extract explicit --request-timeout value if present
  timeout_val="$(
    kubectl get pod "$pod" -n kube-system -o jsonpath='{.spec.containers[0].args}' \
      | tr ' ' '\n' \
      | awk -F= '/^--request-timeout=/ {print $2}'
  )"

  if [ -n "$timeout_val" ]; then
    echo "  Detected --request-timeout: $timeout_val"
  else
    echo "  [INFO] --request-timeout is not explicitly set in pod args."
    echo "         The API server will use its built-in default."
  fi

  echo
done

cat <<'EOF'
Interpretation guidance:
- Each control-plane node should show a kube-apiserver pod and its args.
- A potential problem is indicated when:
  * --request-timeout is missing (relying on an unknown/default value), or
  * --request-timeout is set to a value your organization deems too low
    (legitimate requests time out prematurely) or too high
    (hung/expensive requests tie up resources too long).
- Use this report to decide an appropriate, consistent value (for example
  --request-timeout=300s) and then update /etc/kubernetes/manifests/kube-apiserver.yaml
  on every control-plane node as per your policy.
EOF

Additional Reading:

• https://kubernetes.io/docs/admin/kube-apiserver/