Skip to main content

Ensure Cert File And Key File Arguments Are Appropriate

More Info:​

Configure TLS encryption for the etcd service.

Risk Level​

Medium

Address​

Security

Compliance Standards​

  • CIS Kubernetes

Triage and Remediation​

Remediation​

Manual Steps
  1. On every etcd (control plane) node, back up the current manifest and identify the existing TLS files:
sudo cp -p /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/manifests/etcd.yaml.backup.$(date +%F-%H%M%S)
sudo grep -E 'cert-file|key-file' -n /etc/kubernetes/manifests/etcd.yaml || true
ls -l /etc/kubernetes/pki/etcd || true
  1. If you do not already have an etcd server certificate and key, generate them (example using existing etcd CA under /etc/kubernetes/pki/etcd):
cd /etc/kubernetes/pki/etcd
sudo openssl req -newkey rsa:4096 -nodes -keyout server.key -out server.csr -subj "/CN=etcd"
sudo openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -extensions v3_req -extfile <(printf "subjectAltName=IP:127.0.0.1,IP:$(hostname -i)")
sudo chmod 600 server.key
sudo chmod 644 server.crt
  1. Edit the etcd static pod manifest on each etcd node to configure --cert-file and --key-file (this edit will automatically restart the etcd pod when saved):
sudo sed -i.bak 's#--cert-file=[^[:space:]]*#--cert-file=/etc/kubernetes/pki/etcd/server.crt#g' /etc/kubernetes/manifests/etcd.yaml
sudo sed -i.bak 's#--key-file=[^[:space:]]*#--key-file=/etc/kubernetes/pki/etcd/server.key#g' /etc/kubernetes/manifests/etcd.yaml

If those arguments are missing, open the file and add them under the etcd container command:

sudo vi /etc/kubernetes/manifests/etcd.yaml

Add (or ensure) lines like:

    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --key-file=/etc/kubernetes/pki/etcd/server.key
  1. Confirm the referenced files exist and are readable by the kubelet/container runtime:
sudo ls -l /etc/kubernetes/pki/etcd/server.crt /etc/kubernetes/pki/etcd/server.key
  1. Wait 30–60 seconds for the static pod to be recreated, then confirm the etcd pod is running (from any machine with kubectl access):
kubectl get pods -n kube-system -o wide | grep etcd
  1. Verification on every etcd node: ensure the etcd process is running with the desired --cert-file and --key-file arguments:
/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep

Confirm the output includes:

--cert-file=/etc/kubernetes/pki/etcd/server.crt
--key-file=/etc/kubernetes/pki/etcd/server.key
Using kubectl

kubectl cannot modify host-level static pod manifests or etcd process flags; this finding must be fixed by editing /etc/kubernetes/manifests/etcd.yaml directly on every etcd (control plane) node. See the Manual Steps section for how to update the manifest and verify the etcd --cert-file and --key-file arguments.

Automation
#!/usr/bin/env bash
#
# Remediation: Configure etcd --cert-file and --key-file in /etc/kubernetes/manifests/etcd.yaml
#
# Run on: every etcd node (control plane node)
# Usage:
#   ETCD_CERT_FILE="/etc/kubernetes/pki/etcd/server.crt" \
#   ETCD_KEY_FILE="/etc/kubernetes/pki/etcd/server.key" \
#   sudo -E bash ./fix-etcd-tls.sh
#
# Idempotent: safe to re-run; only updates if values differ.

set -euo pipefail

MANIFEST="/etc/kubernetes/manifests/etcd.yaml"

# ---- Configuration inputs (must be provided via environment) ----
CERT_FILE="${ETCD_CERT_FILE:-}"
KEY_FILE="${ETCD_KEY_FILE:-}"

if [[ -z "$CERT_FILE" || -z "$KEY_FILE" ]]; then
  echo "ERROR: ETCD_CERT_FILE and ETCD_KEY_FILE environment variables must be set."
  echo "Example:"
  echo "  ETCD_CERT_FILE=\"/etc/kubernetes/pki/etcd/server.crt\" \\"
  echo "  ETCD_KEY_FILE=\"/etc/kubernetes/pki/etcd/server.key\" \\"
  echo "  sudo -E bash $0"
  exit 1
fi

# ---- Basic checks ----

if [[ ! -f "$MANIFEST" ]]; then
  echo "ERROR: etcd manifest not found at $MANIFEST"
  exit 1
fi

if [[ ! -f "$CERT_FILE" ]]; then
  echo "ERROR: cert file not found: $CERT_FILE"
  exit 1
fi

if [[ ! -f "$KEY_FILE" ]]; then
  echo "ERROR: key file not found: $KEY_FILE"
  exit 1
fi

if [[ ! -w "$MANIFEST" ]]; then
  echo "ERROR: cannot write to $MANIFEST (need root?)."
  exit 1
fi

# ---- Backup manifest (once per day, idempotent enough) ----

BACKUP_DIR="/etc/kubernetes/manifests/backup-etcd-tls"
mkdir -p "$BACKUP_DIR"
TODAY="$(date +%F)"
BACKUP_FILE="$BACKUP_DIR/etcd.yaml.$TODAY"

if [[ ! -f "$BACKUP_FILE" ]]; then
  cp "$MANIFEST" "$BACKUP_FILE"
  echo "Backup created: $BACKUP_FILE"
else
  echo "Backup for today already exists: $BACKUP_FILE"
fi

# ---- Update or insert flags in manifest ----
# We operate on YAML text with sed in a conservative manner:
# - If --cert-file is present, replace its value.
# - Else, append it on the etcd container's command list.
# - Same for --key-file.
#
# Assumptions:
# - etcd is defined as a static pod with a 'command:' list in the manifest.
# - Lines for flags follow the typical kubeadm format:
#   - --cert-file=...
#   - --key-file=...

TMP_MANIFEST="$(mktemp)"
cp "$MANIFEST" "$TMP_MANIFEST"

# Ensure a trailing newline
printf '\n' >> "$TMP_MANIFEST"

update_flag() {
  local flag_name="$1"         # e.g. --cert-file
  local flag_value="$2"        # e.g. /etc/kubernetes/pki/etcd/server.crt
  local file="$3"

  if grep -qE "^[[:space:]]*-[[:space:]]*$flag_name=" "$file"; then
    # Replace existing flag value
    sed -i "s|^\([[:space:]]*-[[:space:]]*$flag_name=\).*|\1$flag_value|g" "$file"
  else
    # Append flag under etcd container command list.
    # Insert after the 'command:' list start for the etcd container.
    # This is heuristic but works with standard kubeadm manifests.
    awk -v FLAG="$flag_name" -v VALUE="$flag_value" '
      $0 ~ /name:[[:space:]]*etcd/ { in_etcd=1 }
      in_etcd && $0 ~ /^[[:space:]]*command:/ { in_command=1 }
      in_command && $0 ~ /^[[:space:]]*-[[:space:]]*"/ { in_command=0 } # defensive, not usually needed

      {
        print $0
        if (in_command && $0 ~ /^[[:space:]]*-[[:space:]]*etcd$/ && !added) {
          # will rely on following lines that already have flags; fallback append at first flag line
        }
      }

      END { }
    ' "$file" > "${file}.tmp"

    # If awk didn’t change file, or for simplicity, just append near existing etcd flags.
    # Fallback: append after an existing etcd flag line (e.g. --data-dir)
    if ! grep -q "$flag_name=$VALUE" "${file}.tmp" 2>/dev/null; then
      mv "${file}.tmp" "$file"
      # Append after first etcd flag line
      sed -i "/^[[:space:]]*-[[:space:]]*--data-dir=/a\ \ \ \ \ \ \ \ - $flag_name=$VALUE" "$file" || true
    else
      mv "${file}.tmp" "$file"
    fi
  fi
}

update_flag "--cert-file" "$CERT_FILE" "$TMP_MANIFEST"
update_flag "--key-file"  "$KEY_FILE"  "$TMP_MANIFEST"

# ---- If nothing changed, exit; else replace manifest ----

if cmp -s "$MANIFEST" "$TMP_MANIFEST"; then
  echo "No changes required in $MANIFEST (flags already set as desired)."
  rm -f "$TMP_MANIFEST"
else
  mv "$TMP_MANIFEST" "$MANIFEST"
  echo "Updated $MANIFEST with:"
  echo "  --cert-file=$CERT_FILE"
  echo "  --key-file=$KEY_FILE"
  echo
  echo "Note: because this is a static pod manifest, kubelet will restart the etcd pod automatically."
fi

# ---- Verification ----
# Wait briefly for etcd to restart and then confirm flags from process list.

echo "Waiting up to 60 seconds for etcd to restart with new flags..."
for i in $(seq 1 30); do
  if /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep >/dev/null 2>&1; then
    break
  fi
  sleep 2
done

echo "Verifying etcd process flags..."
/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep || {
  echo "ERROR: etcd process not found after manifest change."
  exit 1
}

if /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep | /bin/grep -q -- "--cert-file=$CERT_FILE" \
   && /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep | /bin/grep -q -- "--key-file=$KEY_FILE"; then
  echo "SUCCESS: etcd is running with --cert-file=$CERT_FILE and --key-file=$KEY_FILE"
  exit 0
else
  echo "WARNING: etcd is running but expected flags were not detected."
  echo "Current etcd command line:"
  /bin/ps -ef | /bin/grep etcd | /bin/grep -v grep
  exit 1
fi

Additional Reading:​