Etcd Peer-Cert-File And Peer-Key-File Arguments Are Set

More Info:

The --peer-cert-file and --peer-key-file arguments must be set so that etcd peer-to-peer traffic is served over TLS. Without them, replication traffic containing all cluster state and secrets travels unencrypted between etcd nodes.

Risk Level

Critical

Address

Security

Compliance Standards

• CIS Kubernetes

Triage and Remediation

Remediation

Remediation

Manual Steps

1. On every etcd (control plane) node, back up the current manifest and list existing etcd certs:

   sudo cp -a /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/manifests/etcd.yaml.bak.$(date +%F-%H%M%S)
   sudo ls -l /etc/kubernetes/pki/etcd

2. If you already have dedicated peer cert/key files, note their paths (for example /etc/kubernetes/pki/etcd/peer.crt and /etc/kubernetes/pki/etcd/peer.key). If not, generate them following your PKI process or, as a simple local CA example, run:

   cd /etc/kubernetes/pki/etcd
   sudo openssl genrsa -out peer.key 2048
   sudo openssl req -new -key peer.key -out peer.csr -subj "/CN=etcd-peer"
   # Sign peer.csr with your etcd CA; example using etcd-ca:
   # sudo openssl x509 -req -in peer.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out peer.crt -days 365 -sha256
   sudo chmod 600 peer.key
   sudo chown root:root peer.crt peer.key

3. Edit the etcd static pod manifest on every etcd node to add the peer TLS flags, using the correct cert paths from step 2:

   sudo vi /etc/kubernetes/manifests/etcd.yaml

   In the spec.containers[0].command list, ensure these entries exist (adjust paths if different in your environment):

   - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
   - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key

   Do not remove any existing TLS or cluster-related flags.

4. Still in /etc/kubernetes/manifests/etcd.yaml, if your etcd cluster uses peer CA verification, confirm a matching --peer-trusted-ca-file is set and that the file exists:

   - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt

   Save and exit the editor. The kubelet will automatically restart the etcd static pod when the manifest changes; this temporarily restarts etcd on that node.

5. If you have multiple etcd nodes, repeat steps 1–4 on each etcd node, using the correct peer certificate and key for that node and ensuring any cluster-related flags (e.g., --initial-cluster, --initial-advertise-peer-urls, --initial-cluster-state) remain consistent with your existing etcd cluster configuration.

6. On every etcd node, verify the process now includes the peer cert and key flags:

   /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep

   Confirm the output contains both --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt and --peer-key-file=/etc/kubernetes/pki/etcd/peer.key with the expected paths.

Using kubectl

kubectl cannot be used to configure etcd’s --peer-cert-file and --peer-key-file flags because they are set in the static Pod manifest on the host filesystem. Make the changes directly in /etc/kubernetes/manifests/etcd.yaml on every etcd node as described in the Manual Steps section.

Automation

#!/usr/bin/env bash
#
# Remediation: Ensure etcd --peer-cert-file and --peer-key-file are set
# Scope: Run on every control plane node that hosts /etc/kubernetes/manifests/etcd.yaml
#
# This script is idempotent and safe to re-run.
# It assumes that valid etcd peer TLS cert/key files already exist.

set -euo pipefail

ETCD_MANIFEST="/etc/kubernetes/manifests/etcd.yaml"

# CONFIGURATION: set these to the actual peer cert/key paths used in your environment
PEER_CERT_FILE="/etc/kubernetes/pki/etcd/peer.crt"
PEER_KEY_FILE="/etc/kubernetes/pki/etcd/peer.key"

# --------- sanity checks ---------
if [[ $EUID -ne 0 ]]; then
  echo "ERROR: This script must be run as root on each control plane node." >&2
  exit 1
fi

if [[ ! -f "$ETCD_MANIFEST" ]]; then
  echo "ERROR: $ETCD_MANIFEST not found on this node; nothing to do." >&2
  exit 1
fi

if [[ ! -f "$PEER_CERT_FILE" ]]; then
  echo "ERROR: Peer cert file $PEER_CERT_FILE does not exist. Configure etcd peer TLS first." >&2
  exit 1
fi

if [[ ! -f "$PEER_KEY_FILE" ]]; then
  echo "ERROR: Peer key file $PEER_KEY_FILE does not exist. Configure etcd peer TLS first." >&2
  exit 1
fi

# --------- backup manifest ---------
BACKUP="${ETCD_MANIFEST}.$(date +%Y%m%d%H%M%S).bak"
cp -p "$ETCD_MANIFEST" "$BACKUP"
echo "Backed up $ETCD_MANIFEST to $BACKUP"

# --------- ensure env vars for cert/key (optional but common pattern) ---------
# Add/patch ETCD_PEER_CERT_FILE and ETCD_PEER_KEY_FILE env entries if an env section exists.

if command -v python3 >/dev/null 2>&1; then
  python3 - "$ETCD_MANIFEST" "$PEER_CERT_FILE" "$PEER_KEY_FILE" << 'PYEOF'
import sys, ruamel.yaml
from pathlib import Path

manifest_path = Path(sys.argv[1])
peer_cert = sys.argv[2]
peer_key = sys.argv[3]

yaml = ruamel.yaml.YAML()
data = yaml.load(manifest_path.read_text())

containers = data.get('spec', {}).get('containers', [])
if not containers:
    sys.exit(0)

c = containers[0]
env = c.get('env', [])
# ensure list
if env is None:
    env = []
    c['env'] = env

def set_env(name, value):
    for item in env:
        if item.get('name') == name:
            item['value'] = value
            return
    env.append({'name': name, 'value': value})

set_env('ETCD_PEER_CERT_FILE', peer_cert)
set_env('ETCD_PEER_KEY_FILE', peer_key)

manifest_path.write_text("")
with manifest_path.open('w') as f:
    yaml.dump(data, f)
PYEOF
else
  echo "WARN: python3 not available; skipping env-var alignment (not required for this control)."
fi

# --------- ensure flags in container args ---------
# Use yq if available for structured edit, else fall back to sed injection.

ensure_flag_in_args() {
  local flag="$1"
  local value="$2"

  if command -v yq >/dev/null 2>&1; then
    # Structured: ensure arg exists or append
    if yq '(.spec.containers[0].command // []) | any(. == "'"$flag"'")' "$ETCD_MANIFEST" >/dev/null 2>&1; then
      # flag exists; patch its value (assumes next array element is value)
      idx=$(yq '(.spec.containers[0].command // []) | to_entries | map(select(.value == "'"$flag"'")) | .[0].key' "$ETCD_MANIFEST")
      yq -i ".spec.containers[0].command[$((idx+1))] = \"$value\"" "$ETCD_MANIFEST"
    else
      # append flag and value
      yq -i '.spec.containers[0].command += ["'"$flag"'", "'"$value"'"]' "$ETCD_MANIFEST"
    fi
  else
    # Fallback: simple sed to append if missing; assumes --peer-* not present yet
    if ! grep -q -- "$flag" "$ETCD_MANIFEST"; then
      # Inject into the first container command list
      # This assumes a common kubeadm-style etcd manifest format.
      sed -i "0,/command:/s//command:\n    - \"$flag\"\n    - \"$value\"/" "$ETCD_MANIFEST"
    fi
  fi
}

ensure_flag_in_args "--peer-cert-file" "$PEER_CERT_FILE"
ensure_flag_in_args "--peer-key-file" "$PEER_KEY_FILE"

echo "Updated $ETCD_MANIFEST with --peer-cert-file and --peer-key-file."

echo "NOTE: Because this is a static pod manifest under /etc/kubernetes/manifests,"
echo "      the kubelet will automatically restart the etcd pod to apply these changes."

# --------- wait for etcd pod restart ---------
echo "Waiting up to 120 seconds for etcd process to reflect new flags..."
end=$((SECONDS+120))
success=0
while (( SECONDS < end )); do
  if /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep | /bin/grep -q -- "--peer-cert-file=${PEER_CERT_FILE}"; then
    if /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep | /bin/grep -q -- "--peer-key-file=${PEER_KEY_FILE}"; then
      success=1
      break
    fi
  fi
  sleep 5
done

# --------- verification (audit-style) ---------
echo
echo "Verification output (/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep):"
echo "---------------------------------------------------------------------"
/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep || true
echo "---------------------------------------------------------------------"

if [[ $success -eq 1 ]]; then
  echo "Result: etcd is running with --peer-cert-file and --peer-key-file correctly set."
  exit 0
else
  echo "WARNING: etcd process does not yet show the expected peer TLS flags."
  echo "         Check kubelet status and the etcd pod logs for issues."
  exit 1
fi