Multi-Replica Deployments Should Have A PodDisruptionBudget

More Info:

Advisory: define a PodDisruptionBudget for each multi-replica Deployment so node drains and rollouts keep a minimum number of pods available.

Risk Level

Informational

Address

Security

Compliance Standards

• Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

1. List multi-replica Deployments that currently lack a PodDisruptionBudget (run on any machine with kubectl access):

   { kubectl get poddisruptionbudgets --all-namespaces -o json \
     kubectl get deployments --all-namespaces -o json; } | jq -rs '
     .[0] as $pdbs | .[1] |
     [ .items[]
       | select((.spec.replicas // 1) > 1)
       | .metadata as $m
       | (.spec.template.metadata.labels // {}) as $podLabels
       | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
       | ([ $pdbs.items[]
            | select(.metadata.namespace == $m.namespace)
            | select((.spec.selector.matchLabels // {}) | length > 0)
            | select([ (.spec.selector.matchLabels | to_entries)[]
                       | $podLabels[.key] == .value ] | all)
          ] | length) as $count
       | select($count == 0)
       | {name: $m.name, namespace: $m.namespace, labels: $podLabels}
     ]'

2. For one non-compliant Deployment, inspect its spec to identify pod labels and replica count (replace NAMESPACE and DEPLOYMENT with real values from step 1; run on any machine with kubectl access):

   kubectl get deployment DEPLOYMENT -n NAMESPACE -o yaml

   Note the labels under spec.template.metadata.labels and confirm spec.replicas > 1.

3. Create a PodDisruptionBudget manifest for that Deployment, using the pod template labels as matchLabels and setting an appropriate availability policy (run on any machine with kubectl access):

   cat > pdb-DEPLOYMENT.yaml << 'EOF'
   apiVersion: policy/v1
   kind: PodDisruptionBudget
   metadata:
     name: DEPLOYMENT-pdb
     namespace: NAMESPACE
   spec:
     minAvailable: 1
     selector:
       matchLabels:
         app: REPLACE_WITH_APP_LABEL_VALUE
   EOF

   Replace:

   • NAMESPACE with the Deployment’s namespace
   • DEPLOYMENT with the Deployment name
   • app: REPLACE_WITH_APP_LABEL_VALUE (and/or add more keys) so that the label set exactly matches spec.template.metadata.labels of the Deployment’s pods. Only matchLabels is evaluated by the check; do not use matchExpressions if you want it counted as compliant.

4. Apply the PodDisruptionBudget to the cluster (run on any machine with kubectl access):

   kubectl apply -f pdb-DEPLOYMENT.yaml

5. Repeat steps 2–4 for each remaining non-compliant multi-replica Deployment, ensuring each has a corresponding PodDisruptionBudget whose spec.selector.matchLabels selects that Deployment’s pods.

6. Verify that all multi-replica Deployments now have at least one matching PodDisruptionBudget (run on any machine with kubectl access):

   { kubectl get poddisruptionbudgets --all-namespaces -o json \
     kubectl get deployments --all-namespaces -o json; } | jq -rs '
     .[0] as $pdbs | .[1] |
     [ .items[]
     | select((.spec.replicas // 1) > 1)
     | .metadata as $m
     | (.spec.template.metadata.labels // {}) as $podLabels
     | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
     | ([ $pdbs.items[]
          | select(.metadata.namespace == $m.namespace)
          | select((.spec.selector.matchLabels // {}) | length > 0)
          | select([ (.spec.selector.matchLabels | to_entries)[]
                     | $podLabels[.key] == .value ] | all)
        ] | length) as $count
     | "kind=Deployment ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=apps/v1"
       + " replicas=\(.spec.replicas) podDisruptionBudgets=\($count)"
       + " is_compliant=\(if $count > 0 then "true" else "false" end)"
     ] as $rows
     | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

   Confirm that every kind=Deployment line with replicas= greater than 1 now shows is_compliant=true.

Using kubectl

# 1) Inspect a non-compliant multi-replica Deployment to get its labels
# Run on: any machine with kubectl access
kubectl get deployments -A \
  -o jsonpath='{"NAMESPACE NAME LABELS\n"}{range .items[?(@.spec.replicas>1)]}{.metadata.namespace}{" "}{.metadata.name}{" "}{range $k,$v := .spec.template.metadata.labels}{$k}={"$v"}, {end}{"\n"}{end}'

# Example: assume deployment "my-app" in namespace "prod" with pod label app=my-app
# Adjust namespace, name, and labels to match your Deployment(s).

# 2) Create a PodDisruptionBudget manifest for that Deployment
cat > pdb-my-app.yaml << 'EOF'
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: prod
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-app
EOF

# 3) Apply the PodDisruptionBudget
kubectl apply -f pdb-my-app.yaml

# Repeat steps 2–3 for each multi-replica Deployment, ensuring:
# - metadata.namespace matches the Deployment namespace
# - spec.selector.matchLabels exactly matches a stable subset of the pod template labels
# - minAvailable or maxUnavailable reflect your availability/SLO requirements

# 4) Verification: rerun the benchmark-style check
{ kubectl get poddisruptionbudgets --all-namespaces -o json \
  ; kubectl get deployments --all-namespaces -o json; } | jq -rs '
  .[0] as $pdbs | .[1] |
  [ .items[]
  | select((.spec.replicas // 1) > 1)
  | .metadata as $m
  | (.spec.template.metadata.labels // {}) as $podLabels
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ $pdbs.items[]
       | select(.metadata.namespace == $m.namespace)
       | select((.spec.selector.matchLabels // {}) | length > 0)
       | select([ (.spec.selector.matchLabels | to_entries)[]
                  | $podLabels[.key] == .value ] | all)
     ] | length) as $count
  | "kind=Deployment ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=apps/v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $labels == "" then "" else " labels=\($labels)" end)
    + " replicas=\(.spec.replicas) podDisruptionBudgets=\($count)"
    + " is_compliant=\(if $count > 0 then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

Automation

#!/usr/bin/env bash
# Automation: Ensure every multi-replica Deployment has a matching PodDisruptionBudget
# Platform: AKS (or any cluster reachable via kubectl)
#
# Requirements:
# - Run on any machine with kubectl access and jq installed.
# - Uses kubectl current context.
#
# Behavior:
# - For each Deployment with replicas > 1:
#   - If no PDB exists whose selector.matchLabels is a subset of the pod template labels,
#     create a PDB named "<deployment-name>-pdb" with:
#       minAvailable: 1            if replicas == 2
#       minAvailable: 50% (string) if replicas >= 3
#   - PDB selector.matchLabels is set to the Deployment's pod template labels.
# - Safe to re-run: existing matching PDBs are left untouched; created PDBs are stable.

set -euo pipefail

# --- Configurable defaults -----------------------------------------------------

# Label key/value added to PDBs created by this script (optional but useful)
PDB_MANAGED_LABEL_KEY="pdb.k8s-benchmark/managed-by"
PDB_MANAGED_LABEL_VALUE="cbp-c5.1-automation"

# ------------------------------------------------------------------------------
command -v kubectl >/dev/null 2>&1 || { echo "kubectl is required in PATH" >&2; exit 1; }
command -v jq >/dev/null 2>&1 || { echo "jq is required in PATH" >&2; exit 1; }

echo "Discovering multi-replica Deployments (replicas > 1)..."
DEPLOY_JSON="$(kubectl get deployments --all-namespaces -o json)"
PDB_JSON="$(kubectl get poddisruptionbudgets --all-namespaces -o json || echo '{"items": []}')"

# Function: check if any PDB in the namespace covers the deployment's pod labels.
# Coverage rule: PDB.spec.selector.matchLabels is non-empty and is a subset of
# the Deployment's pod template labels.
deployment_needs_pdb() {
  local namespace="$1"
  local deploy_name="$2"

  # Extract pod template labels and replicas for this deployment
  local deploy_info
  deploy_info="$(jq -r \
    --arg ns "$namespace" \
    --arg name "$deploy_name" \
    '
    .items[]
    | select(.metadata.namespace == $ns and .metadata.name == $name)
    | {
        replicas: (.spec.replicas // 1),
        podLabels: (.spec.template.metadata.labels // {})
      }
    ' <<<"$DEPLOY_JSON")"

  if [[ -z "$deploy_info" ]]; then
    # Deployment disappeared between listing and processing
    return 1
  fi

  local pod_labels_json replicas
  pod_labels_json="$(jq -c '.podLabels' <<<"$deploy_info")"
  replicas="$(jq -r '.replicas' <<<"$deploy_info")"

  # Skip if not actually multi-replica
  if (( replicas <= 1 )); then
    return 1
  fi

  # Determine if any PDB matches
  local match_count
  match_count="$(jq -r \
    --arg ns "$namespace" \
    --argjson podLabels "$pod_labels_json" \
    '
    .items[]
    | select(.metadata.namespace == $ns)
    | select((.spec.selector.matchLabels // {}) | length > 0)
    | .spec.selector.matchLabels as $sel
    # check $sel is a subset of podLabels:
    | ( [ ($sel | to_entries)[]
          | ($podLabels[.key] == .value)
        ] | all ) as $isSubset
    | select($isSubset)
    | 1
    ' <<<"$PDB_JSON" | wc -l | tr -d ' ')"

  if [[ "$match_count" -gt 0 ]]; then
    return 1  # already covered
  fi

  return 0  # needs PDB
}

# Function: create PDB manifest for a deployment
create_pdb_for_deployment() {
  local namespace="$1"
  local deploy_name="$2"

  local deploy
  deploy="$(kubectl get deployment "$deploy_name" -n "$namespace" -o json 2>/dev/null || true)"
  if [[ -z "$deploy" ]]; then
    echo "  [WARN] Deployment $namespace/$deploy_name disappeared; skipping"
    return
  fi

  local replicas pod_labels_json pdb_name min_available_yaml

  replicas="$(jq -r '.spec.replicas // 1' <<<"$deploy")"
  pod_labels_json="$(jq -c '.spec.template.metadata.labels // {}' <<<"$deploy")"
  pdb_name="${deploy_name}-pdb"

  if (( replicas <= 1 )); then
    echo "  [INFO] $namespace/$deploy_name now has replicas <=1; skipping PDB creation"
    return
  fi

  # Set minAvailable based on replicas
  if (( replicas == 2 )); then
    min_available_yaml="minAvailable: 1"
  else
    min_available_yaml="minAvailable: \"50%\""
  fi

  # Build selector.matchLabels YAML from pod_labels_json
  local selector_yaml
  selector_yaml="$(jq -r '
    to_entries
    | map("      \(.key): \"\(.value)\"")
    | join("\n")
  ' <<<"$pod_labels_json")"

  if [[ -z "$selector_yaml" ]]; then
    echo "  [WARN] $namespace/$deploy_name has no pod template labels; cannot create selector-based PDB"
    return
  fi

  echo "  [ACTION] Creating PDB $namespace/$pdb_name for Deployment $deploy_name (replicas=$replicas)"

  cat <<EOF | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ${pdb_name}
  namespace: ${namespace}
  labels:
    ${PDB_MANAGED_LABEL_KEY}: "${PDB_MANAGED_LABEL_VALUE}"
spec:
  ${min_available_yaml}
  selector:
    matchLabels:
$(echo "${selector_yaml}")
EOF
}

# Iterate deployments and create PDBs where needed
echo "Processing Deployments..."
while IFS=$'\t' read -r ns name replicas; do
  # Filter to replicas > 1 to reduce work early
  if [[ "$replicas" -le 1 ]]; then
    continue
  fi

  echo "Checking Deployment $ns/$name (replicas=$replicas)..."
  if deployment_needs_pdb "$ns" "$name"; then
    create_pdb_for_deployment "$ns" "$name"
    # Refresh PDB_JSON after potential creation to keep checks consistent
    PDB_JSON="$(kubectl get poddisruptionbudgets --all-namespaces -o json || echo '{"items": []}')"
  else
    echo "  [OK] Covered by existing PodDisruptionBudget"
  fi
done < <(echo "$DEPLOY_JSON" | jq -r '
  .items[]
  | {ns: .metadata.namespace, name: .metadata.name, replicas: (.spec.replicas // 1)}
  | select(.replicas > 1)
  | "\(.ns)\t\(.name)\t\(.replicas)"
')

# --- Verification -------------------------------------------------------------
echo
echo "Verification: multi-replica Deployments and their PodDisruptionBudgets"

{
  kubectl get poddisruptionbudgets --all-namespaces -o json || echo '{"items": []}'
  kubectl get deployments --all-namespaces -o json
} | jq -rs '
  .[0] as $pdbs | .[1] |
  [ .items[]
  | select((.spec.replicas // 1) > 1)
  | .metadata as $m
  | (.spec.template.metadata.labels // {}) as $podLabels
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ $pdbs.items[]
       | select(.metadata.namespace == $m.namespace)
       | select((.spec.selector.matchLabels // {}) | length > 0)
       | select([ (.spec.selector.matchLabels | to_entries)[]
                  | $podLabels[.key] == .value ] | all)
     ] | length) as $count
  | "kind=Deployment ns=\($m.namespace) name=\($m.name) replicas=\(.spec.replicas) podDisruptionBudgets=\($count)"
    + " is_compliant=\(if $count > 0 then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end
'