Kubelet Streaming Connection Idle Timeout Not Set To 0

More Info:

A streaming connection idle timeout of 0 disables timeouts, leaving idle connections open and exposed to denial of service. Setting a non-zero value ensures idle streaming connections are closed.

Risk Level

High

Address

Security

Compliance Standards

CIS Kubernetes

Triage and Remediation

Remediation

Remediation

Manual Steps

On every worker node, back up the existing kubelet config file:

sudo cp -a /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak.$(date +%F-%H%M%S)

On every worker node, edit /var/lib/kubelet/config.yaml and set a non‑zero timeout (example: 5 minutes). If the key exists and is 0, change it; if it does not exist, add it under the top‑level config:
```
sudo sed -i 's/^[[:space:]]*streamingConnectionIdleTimeout: *0[[:space:]]*$/streamingConnectionIdleTimeout: 5m/' /var/lib/kubelet/config.yaml
```
If the line does not already exist, open the file with an editor and add, at top level (aligned with other keys like authentication / authorization):
```
streamingConnectionIdleTimeout: 5m
```
If your kubelet also uses command-line flags via systemd drop‑in, ensure it does not override this to 0. On every worker node, open the systemd drop‑in file:
```
sudo sed -n '1,160p' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
```
If you see --streaming-connection-idle-timeout=0 anywhere, edit the file with a text editor and change it to:
```
--streaming-connection-idle-timeout=5m
```
On every worker node, reload systemd and restart kubelet (this will temporarily disrupt kubelet on that node):
```
sudo systemctl daemon-reload
sudo systemctl restart kubelet.service
```
On every worker node, verify kubelet is running and the timeout is no longer 0:
```
/bin/ps -fC kubelet
```
Confirm that either the kubelet command line does not contain --streaming-connection-idle-timeout=0 (and if present, shows a non‑zero value like 5m), and that /var/lib/kubelet/config.yaml contains streamingConnectionIdleTimeout: 5m (or another non‑zero duration).

Using kubectl

kubectl cannot modify kubelet process flags or its config file at /var/lib/kubelet/config.yaml, so this setting cannot be fixed via the Kubernetes API. Apply the remediation directly on every worker node’s host configuration as described in the Manual Steps section.

Automation

#!/usr/bin/env bash
#
# Fix kubelet streamingConnectionIdleTimeout on all worker nodes.
#
# Usage:
#   1) Place this script on each worker node and run as root, OR
#   2) Run centrally with SSH access to each worker node:
#        WORKER_NODES="node1 node2" ./fix-kubelet-timeout.sh ssh
#
# This script is idempotent and safe to re-run.

set -euo pipefail

STREAM_TIMEOUT_VALUE="5m"
KUBELET_CONFIG="/var/lib/kubelet/config.yaml"
SYSTEMD_DROPIN="/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"

mode="${1:-local}"   # 'local' (default) or 'ssh'
workers="${WORKER_NODES:-}"

run_local() {
  echo "=== Running on worker node: $(hostname) ==="

  # 1. Ensure kubelet config file has streamingConnectionIdleTimeout != 0
  if [ -f "$KUBELET_CONFIG" ]; then
    echo "Updating $KUBELET_CONFIG ..."

    # If key exists, replace it; otherwise, add under top-level.
    if grep -qE '^[[:space:]]*streamingConnectionIdleTimeout:' "$KUBELET_CONFIG"; then
      # If already set to desired non-zero value, do nothing
      if grep -qE "^[[:space:]]*streamingConnectionIdleTimeout:[[:space:]]*${STREAM_TIMEOUT_VALUE}[[:space:]]*$" "$KUBELET_CONFIG"; then
        echo "  streamingConnectionIdleTimeout already set to ${STREAM_TIMEOUT_VALUE}"
      else
        # Replace any existing value (including 0 or other) with desired non-zero
        sed -i -E "s/^[[:space:]]*streamingConnectionIdleTimeout:.*/streamingConnectionIdleTimeout: ${STREAM_TIMEOUT_VALUE}/" "$KUBELET_CONFIG"
        echo "  Set streamingConnectionIdleTimeout: ${STREAM_TIMEOUT_VALUE}"
      fi
    else
      # Add key if missing
      printf "\nstreamingConnectionIdleTimeout: %s\n" "$STREAM_TIMEOUT_VALUE" >> "$KUBELET_CONFIG"
      echo "  Added streamingConnectionIdleTimeout: ${STREAM_TIMEOUT_VALUE}"
    fi
  else
    echo "WARNING: $KUBELET_CONFIG not found; skipping file-based config."
  fi

  # 2. Ensure systemd drop-in (command-line args) has non-zero timeout if used
  if [ -f "$SYSTEMD_DROPIN" ]; then
    echo "Checking $SYSTEMD_DROPIN for command-line timeout flag ..."

    if grep -q -- '--streaming-connection-idle-timeout' "$SYSTEMD_DROPIN"; then
      # If already desired value, do nothing
      if grep -q -- "--streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}" "$SYSTEMD_DROPIN"; then
        echo "  --streaming-connection-idle-timeout already set to ${STREAM_TIMEOUT_VALUE}"
      else
        # Replace any existing value (including 0 or other)
        sed -i -E "s/--streaming-connection-idle-timeout=[^[:space:]]*/--streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}/" "$SYSTEMD_DROPIN"
        echo "  Updated --streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}"
      fi
    else
      # Append flag to existing KUBELET_SYSTEM_PODS_ARGS or Environment line if present
      if grep -q 'KUBELET_SYSTEM_PODS_ARGS' "$SYSTEMD_DROPIN"; then
        sed -i -E "s/(KUBELET_SYSTEM_PODS_ARGS=\"[^\"]*)\"/\1 --streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}\"/" "$SYSTEMD_DROPIN"
        echo "  Appended flag to KUBELET_SYSTEM_PODS_ARGS"
      elif grep -q '^Environment=' "$SYSTEMD_DROPIN"; then
        sed -i -E "s/^(Environment=.*)\"$/\1 --streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}\"/" "$SYSTEMD_DROPIN"
        echo "  Appended flag to Environment line"
      else
        cat <<EOF >> "$SYSTEMD_DROPIN"

Environment="KUBELET_SYSTEM_PODS_ARGS=--streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}"
EOF
        echo "  Added new Environment line with timeout flag"
      fi
    fi
  else
    echo "INFO: $SYSTEMD_DROPIN not found; kubelet may be fully config-file driven."
  fi

  # 3. Reload and restart kubelet (operational impact: kubelet restart on this node)
  echo "Reloading systemd and restarting kubelet ..."
  systemctl daemon-reload
  systemctl restart kubelet.service

  # 4. Verification: inspect kubelet process args to ensure timeout is not 0
  echo "Verification: kubelet process arguments on $(hostname):"
  /bin/ps -fC kubelet || true

  # Fail if we still see an explicit '=0'
  if /bin/ps -fC kubelet 2>/dev/null | grep -q -- '--streaming-connection-idle-timeout=0'; then
    echo "ERROR: kubelet still running with --streaming-connection-idle-timeout=0 on $(hostname)" >&2
    exit 1
  fi

  echo "=== Completed on $(hostname) ==="
}

run_ssh() {
  if [ -z "$workers" ]; then
    echo "ERROR: In ssh mode, set WORKER_NODES=\"node1 node2\" environment variable." >&2
    exit 1
  fi

  for node in $workers; do
    echo "##### Processing worker node: $node #####"
    ssh -o BatchMode=yes -o StrictHostKeyChecking=no "$node" 'bash -s' <<'EOF'
set -euo pipefail

STREAM_TIMEOUT_VALUE="5m"
KUBELET_CONFIG="/var/lib/kubelet/config.yaml"
SYSTEMD_DROPIN="/etc/systemd/system/kubelet.service.d/10-kubeadm.conf"

echo "=== Running on worker node: $(hostname) ==="

if [ -f "$KUBELET_CONFIG" ]; then
  echo "Updating $KUBELET_CONFIG ..."
  if grep -qE '^[[:space:]]*streamingConnectionIdleTimeout:' "$KUBELET_CONFIG"; then
    if grep -qE "^[[:space:]]*streamingConnectionIdleTimeout:[[:space:]]*${STREAM_TIMEOUT_VALUE}[[:space:]]*$" "$KUBELET_CONFIG"; then
      echo "  streamingConnectionIdleTimeout already set to ${STREAM_TIMEOUT_VALUE}"
    else
      sed -i -E "s/^[[:space:]]*streamingConnectionIdleTimeout:.*/streamingConnectionIdleTimeout: ${STREAM_TIMEOUT_VALUE}/" "$KUBELET_CONFIG"
      echo "  Set streamingConnectionIdleTimeout: ${STREAM_TIMEOUT_VALUE}"
    fi
  else
    printf "\nstreamingConnectionIdleTimeout: %s\n" "$STREAM_TIMEOUT_VALUE" >> "$KUBELET_CONFIG"
    echo "  Added streamingConnectionIdleTimeout: ${STREAM_TIMEOUT_VALUE}"
  fi
else
  echo "WARNING: $KUBELET_CONFIG not found; skipping file-based config."
fi

if [ -f "$SYSTEMD_DROPIN" ]; then
  echo "Checking $SYSTEMD_DROPIN for command-line timeout flag ..."
  if grep -q -- '--streaming-connection-idle-timeout' "$SYSTEMD_DROPIN"; then
    if grep -q -- "--streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}" "$SYSTEMD_DROPIN"; then
      echo "  --streaming-connection-idle-timeout already set to ${STREAM_TIMEOUT_VALUE}"
    else
      sed -i -E "s/--streaming-connection-idle-timeout=[^[:space:]]*/--streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}/" "$SYSTEMD_DROPIN"
      echo "  Updated --streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}"
    fi
  else
    if grep -q 'KUBELET_SYSTEM_PODS_ARGS' "$SYSTEMD_DROPIN"; then
      sed -i -E "s/(KUBELET_SYSTEM_PODS_ARGS=\"[^\"]*)\"/\1 --streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}\"/" "$SYSTEMD_DROPIN"
      echo "  Appended flag to KUBELET_SYSTEM_PODS_ARGS"
    elif grep -q '^Environment=' "$SYSTEMD_DROPIN"; then
      sed -i -E "s/^(Environment=.*)\"$/\1 --streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}\"/" "$SYSTEMD_DROPIN"
      echo "  Appended flag to Environment line"
    else
      cat <<EOS >> "$SYSTEMD_DROPIN"

Environment="KUBELET_SYSTEM_PODS_ARGS=--streaming-connection-idle-timeout=${STREAM_TIMEOUT_VALUE}"
EOS
      echo "  Added new Environment line with timeout flag"
    fi
  fi
else
  echo "INFO: $SYSTEMD_DROPIN not found; kubelet may be fully config-file driven."
fi

echo "Reloading systemd and restarting kubelet ..."
systemctl daemon-reload
systemctl restart kubelet.service

echo "Verification: kubelet process arguments on $(hostname):"
/bin/ps -fC kubelet || true

if /bin/ps -fC kubelet 2>/dev/null | grep -q -- '--streaming-connection-idle-timeout=0'; then
  echo "ERROR: kubelet still running with --streaming-connection-idle-timeout=0 on $(hostname)" >&2
  exit 1
fi

echo "=== Completed on $(hostname) ==="
EOF
  done
}

if [ "$mode" = "local" ]; then
  run_local
elif [ "$mode" = "ssh" ]; then
  run_ssh
else
  echo "Usage: $0 [local|ssh]" >&2
  exit 1
fi

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​