Skip to main content

The --streaming-connection-idle-timeout Argument Is Not Set

More Info:

A streaming connection idle timeout of 0 disables the timeout, leaving idle exec/attach/port-forward connections open indefinitely and exposing the node to resource exhaustion. Set a non-zero value.

Risk Level

High

Address

Security

Compliance Standards

  • CIS EKS

Triage and Remediation

Remediation

Manual Steps
  1. On every worker node, check how kubelet is configured and whether the flag is present:

    ps -fC kubelet
    ls -R /etc/kubernetes /var/lib/kubelet /etc/systemd/system/kubelet.service.d || true
  2. If kubelet uses a config file (preferred) and /var/lib/kubelet/config.yaml exists, edit it and set a non-zero timeout (creates the key if missing):

    sudo sed -i '/^streamingConnectionIdleTimeout:/d' /var/lib/kubelet/config.yaml
    echo 'streamingConnectionIdleTimeout: "4h0m0s"' | sudo tee -a /var/lib/kubelet/config.yaml
  3. Ensure no conflicting --streaming-connection-idle-timeout flag exists in systemd drop-ins so the config file value is not overridden:

    sudo mkdir -p /etc/systemd/system/kubelet.service.d
    sudo sed -i 's/--streaming-connection-idle-timeout=[^"[:space:]]*//g' \
    /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf 2>/dev/null || true

    Manually open the file if it exists and confirm any residual double spaces or stray backslashes are cleaned up:

    sudo vi /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf
  4. If instead your kubelet is configured only with executable arguments (no config file), append a non-zero timeout flag to the kubelet args on each worker node:

    sudo mkdir -p /etc/systemd/system/kubelet.service.d
    sudo sed -i '/KUBELET_ARGS=/d' /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf 2>/dev/null || true
    echo 'KUBELET_ARGS="--streaming-connection-idle-timeout=4h0m0s"' | \
    sudo tee -a /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf

    Adjust the KUBELET_ARGS line manually if your distribution uses a different variable name or format.

  5. Restart kubelet on every worker node so the new settings take effect (this will restart the kubelet and may briefly impact pod scheduling on that node):

    sudo systemctl daemon-reload
    sudo systemctl restart kubelet.service
    sudo systemctl status kubelet -l
  6. Verify on every worker node that kubelet is running with a non-zero idle timeout (or no =0 setting present):

    /bin/ps -fC kubelet | grep -E -- '--streaming-connection-idle-timeout' || echo "Flag not set, using config file value"

    Optionally, if the config file is used, confirm the effective value:

    grep -i 'streamingConnectionIdleTimeout' /var/lib/kubelet/config.yaml
Using kubectl

kubectl cannot modify kubelet process flags or the host-level configuration files involved in this finding. To remediate, you must change the kubelet settings directly on every worker node (for example /etc/kubernetes/kubelet/kubelet-config.json, /etc/systemd/system/kubelet.service.d/10-kubelet-args.conf, or via the configz-based method) as described in the Manual Steps section.

Automation
#!/usr/bin/env bash
#
# Fix CISEKS 3.2.5: Ensure that the --streaming-connection-idle-timeout argument is not set to 0
#
# Run this script on EVERY WORKER NODE as root.
# It is safe to re-run; it preserves existing non-zero settings.

set -euo pipefail

KUBELET_CONFIG_FILE="/var/lib/kubelet/config.yaml"
KUBELET_DROPIN="/etc/systemd/system/kubelet.service.d/10-kubelet-args.conf"
DESIRED_TIMEOUT="4h0m0s"

echo "[INFO] Starting kubelet streamingConnectionIdleTimeout remediation"

#----------------------------
# Step 1: Ensure config file timeout is set and non-zero (Remediation Method 1)
#----------------------------
if [ -f "$KUBELET_CONFIG_FILE" ]; then
echo "[INFO] Updating $KUBELET_CONFIG_FILE"

if grep -Eq '^[[:space:]]*streamingConnectionIdleTimeout:' "$KUBELET_CONFIG_FILE"; then
current_val="$(grep -E '^[[:space:]]*streamingConnectionIdleTimeout:' "$KUBELET_CONFIG_FILE" \
| head -n1 \
| awk -F: '{gsub(/ /,"",$2); print $2}')"

if [ "$current_val" = "0" ] || [ "$current_val" = "0s" ] || [ "$current_val" = "\"0\"" ] || [ "$current_val" = "\"0s\"" ]; then
echo "[INFO] Found zero timeout in $KUBELET_CONFIG_FILE, updating to ${DESIRED_TIMEOUT}"
sed -i -E 's/^([[:space:]]*streamingConnectionIdleTimeout:).*/\1 '"${DESIRED_TIMEOUT}"'/' "$KUBELET_CONFIG_FILE"
else
echo "[INFO] Existing streamingConnectionIdleTimeout is non-zero ($current_val), leaving as-is"
fi
else
echo "[INFO] Adding streamingConnectionIdleTimeout: ${DESIRED_TIMEOUT} to $KUBELET_CONFIG_FILE"
printf "\nstreamingConnectionIdleTimeout: %s\n" "$DESIRED_TIMEOUT" >> "$KUBELET_CONFIG_FILE"
fi
else
echo "[WARN] $KUBELET_CONFIG_FILE not found; kubelet may be using flags/configz instead"
fi

#----------------------------
# Step 2: Ensure systemd drop-in does NOT override with 0 (Remediation Method 1 & 2)
# and does not set timeout to 0 via flags
#----------------------------
if [ -f "$KUBELET_DROPIN" ]; then
echo "[INFO] Checking $KUBELET_DROPIN for --streaming-connection-idle-timeout=0"

# Remove any explicit 0s settings
if grep -q -- '--streaming-connection-idle-timeout=0' "$KUBELET_DROPIN"; then
echo "[INFO] Removing --streaming-connection-idle-timeout=0 from $KUBELET_DROPIN"
sed -i 's/--streaming-connection-idle-timeout=0s\?//g' "$KUBELET_DROPIN"
sed -i 's/--streaming-connection-idle-timeout=0//g' "$KUBELET_DROPIN"
fi

# Normalize whitespace
sed -i 's/[[:space:]]\+/ /g' "$KUBELET_DROPIN"
else
echo "[INFO] $KUBELET_DROPIN not found; kubelet may be managed differently on this node"
fi

#----------------------------
# Step 3: Reload systemd and restart kubelet
#----------------------------
echo "[INFO] Reloading systemd and restarting kubelet (this will restart kubelet on this node)"
systemctl daemon-reload
systemctl restart kubelet.service

echo "[INFO] Checking kubelet status"
systemctl status kubelet -l --no-pager || {
echo "[ERROR] kubelet did not start correctly after changes"
exit 1
}

#----------------------------
# Step 4: Verification
#----------------------------
echo "[INFO] Verifying kubelet process flags and configz for streamingConnectionIdleTimeout"

echo "[INFO] kubelet process command line:"
/bin/ps -fC kubelet || true

# Verify via local config file
if [ -f "$KUBELET_CONFIG_FILE" ]; then
echo "[INFO] Value from $KUBELET_CONFIG_FILE:"
grep -E '^[[:space:]]*streamingConnectionIdleTimeout:' "$KUBELET_CONFIG_FILE" || echo " (not set)"
fi

# Best-effort configz check (requires kubectl proxy on a machine with cluster access)
NODE_NAME="$(hostname)"
echo "[INFO] If this node is part of a live cluster and kubectl proxy is running elsewhere,"
echo "[INFO] run the following from ANY MACHINE WITH KUBECTL ACCESS to confirm via configz:"
echo
echo " kubectl proxy --port=8001 &"
echo " export HOSTNAME_PORT=localhost:8001"
echo " export NODE_NAME=${NODE_NAME}"
echo " curl -sSL \"http://\${HOSTNAME_PORT}/api/v1/nodes/\${NODE_NAME}/proxy/configz\" | grep -i streamingConnectionIdleTimeout || echo 'field not present'"
echo
echo "[INFO] Remediation script completed on this worker node."