Skip to main content

Multi-Replica Deployments Should Have A PodDisruptionBudget

More Info:​

Advisory: define a PodDisruptionBudget for each multi-replica Deployment so node drains and rollouts keep a minimum number of pods available.

Risk Level​

Informational

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. List all multi-replica Deployments and their selectors

    • Run on: any machine with kubectl access
    • Command: 
      kubectl get deploy -A -o jsonpath='{range .items[?(@.spec.replicas>1)]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.spec.replicas}{" "}{@.spec.selector.matchLabels}{"\n"}{end}'
    • Review: Identify which multi-replica Deployments are critical (user-facing, stateful backends, control-plane add-ons, etc.).

  2. List existing PodDisruptionBudgets and their selectors

    • Run on: any machine with kubectl access
    • Command: 
      kubectl get pdb -A -o yaml
    • Review: For each PDB, note .metadata.namespace, .metadata.name, .spec.selector.matchLabels, and either .spec.minAvailable or .spec.maxUnavailable.

  3. Map Deployments to PDBs and find gaps

    • For each multi-replica Deployment from step 1, check if there is a PDB in the same namespace whose spec.selector.matchLabels matches the Deployment’s spec.selector.matchLabels.
    • If no matching PDB exists, or the PDB selector is broader/narrower than intended, mark that Deployment as needing a new or corrected PDB.

  4. Design an appropriate availability policy per Deployment

    • For each marked Deployment, decide:
      • Whether to use minAvailable (e.g., "80%" or replicas-1) to guarantee a minimum number of pods, or
      • maxUnavailable (e.g., 1) to allow controlled disruptions.
    • Ensure the chosen value is compatible with the Deployment’s spec.replicas (e.g., do not set minAvailable equal to replicas if you still need to allow voluntary disruptions such as node drains).

  5. Create or adjust PodDisruptionBudgets

    • Run on: any machine with kubectl access
    • For a missing PDB, create a manifest like: 
      apiVersion: policy/v1
      kind: PodDisruptionBudget
      metadata:
        name: <deployment-name>-pdb
        namespace: <namespace>
      spec:
        minAvailable: 1
        selector:
          matchLabels:
            app: <label-from-deployment-selector>
    • Apply it: 
      kubectl apply -f <pdb-manifest>.yaml
    • For an existing but misconfigured PDB, edit and correct it: 
      kubectl -n <namespace> edit pdb <pdb-name>
      Adjust spec.selector.matchLabels to match the Deployment labels and tune minAvailable/maxUnavailable per step 4.

  6. Verify PDB coverage for all multi-replica Deployments

    • Run on: any machine with kubectl access
    • Command to re-check multi-replica Deployments: 
      kubectl get deploy -A -o jsonpath='{range .items[?(@.spec.replicas>1)]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.spec.selector.matchLabels}{"\n"}{end}'
    • For each listed Deployment, confirm there is at least one PDB in the same namespace whose spec.selector.matchLabels matches the Deployment’s selector: 
      kubectl get pdb -n <namespace> -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.selector.matchLabels}{"\n"}{end}'
    • Optionally, simulate a drain to ensure PDBs are enforced (non-disruptively on a test node/pool): 
      kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --dry-run=server
Using kubectl
# 1. List all Deployments and their replica counts
# Run on: any machine with kubectl access
kubectl get deploy -A -o custom-columns=NS:metadata.namespace,NAME:metadata.name,REPLICAS:spec.replicas --sort-by=.metadata.namespace
  • Focus review on Deployments where REPLICAS is 2 or more (single-replica workloads do not benefit from a PDB for disruption tolerance).
# 2. List all PodDisruptionBudgets and the selectors they use
kubectl get pdb -A -o wide

Key columns to review:

  • NAMESPACE, NAME: where the PDB lives.
  • MIN AVAILABLE / MAX UNAVAILABLE: ensures enough pods stay up.
  • SELECTOR: which pods the PDB matches.

A problem is indicated when:

  • A namespace has multi-replica Deployments but no PDBs at all, or
  • There are PDBs, but their SELECTOR labels do not correspond to any multi-replica Deployment’s pod labels.
# 3. For each namespace with multi-replica Deployments, compare Deployment labels vs PDB selectors

# Example: show Deployments with replicas >= 2 in a given namespace
NAMESPACE=prod
kubectl get deploy -n "$NAMESPACE" \
  -o jsonpath='{range .items[?(@.spec.replicas>=2)]}{.metadata.name}{" => "}{.spec.selector.matchLabels}{"\n"}{end}'

# Show PDB selectors in the same namespace
kubectl get pdb -n "$NAMESPACE" \
  -o jsonpath='{range .items}{.metadata.name}{" => "}{.spec.selector.matchLabels}{"\n"}{end}'

A problem is indicated when, for a multi-replica Deployment:

  • There is no PDB in the same namespace whose selector matchLabels would match that Deployment’s pod labels, or
  • The PDB exists but its MIN AVAILABLE / MAX UNAVAILABLE values are clearly inconsistent with the desired availability for that workload (this judgment must be made by a human).
# 4. Spot-check a specific multi-replica Deployment against existing PDBs

NAMESPACE=prod
DEPLOYMENT=my-app

# Show Deployment pod selector and replicas
kubectl get deploy "$DEPLOYMENT" -n "$NAMESPACE" -o yaml | \
  egrep "name: |replicas:|matchLabels:|  app:|  component:"

# Show PDBs that might match this Deployment (same namespace)
kubectl get pdb -n "$NAMESPACE" -o yaml | \
  egrep "name: |minAvailable:|maxUnavailable:|matchLabels:|  app:|  component:"

A problem is indicated when:

  • The Deployment’s spec.selector.matchLabels do not line up with any PDB spec.selector.matchLabels, meaning a node drain or rollout could evict all pods, or
  • No PDB exists at all for an application that you consider critical and that runs with multiple replicas.
# 5. Verification after you add or adjust PDBs (human judgment still required)

# Re-list Deployments and PDBs
kubectl get deploy -A -o custom-columns=NS:metadata.namespace,NAME:metadata.name,REPLICAS:spec.replicas --sort-by=.metadata.namespace
kubectl get pdb -A -o wide

# For each multi-replica Deployment you care about, confirm:
# - There is at least one PDB in the same namespace
# - The PDB selector matches the Deployment's pod labels
# - The minAvailable / maxUnavailable values reflect your availability requirements
Automation
#!/usr/bin/env bash
# Report multi-replica Deployments that lack an associated PodDisruptionBudget.
# Run on: any machine with kubectl access and current context set.

set -euo pipefail

echo "Collecting multi-replica Deployments (replicas >= 2)..."
# namespace/name  replicas  matchLabels(JSON)
kubectl get deploy -A -o json \
  | jq -r '
    .items[]
    | select((.spec.replicas // 1) >= 2)
    | [.metadata.namespace,
       .metadata.name,
       (.spec.replicas // 1),
       (.spec.selector.matchLabels // {})]
    | @tsv' \
  | while IFS=$'\t' read -r ns name replicas matchlabels_json; do
      # Build a label selector from matchLabels
      if [[ "$matchlabels_json" == "{}" ]]; then
        # No matchLabels: skip, PDB matching is ambiguous and must be reviewed manually
        echo -e "WARN\t${ns}\t${name}\t${replicas}\tNO matchLabels in selector (review manually)"
        continue
      fi

      # Convert JSON object of labels to a key=value,key2=value2 selector string
      selector=$(
        jq -r 'to_entries | map("\(.key)=\(.value)") | join(",")' <<<"$matchlabels_json"
      )

      # Look for at least one PDB in the same namespace whose selector matches these labels
      pdb_count=$(kubectl get pdb -n "$ns" -o json \
        | jq --arg sel "$selector" '
            .items[]
            | select(.spec.selector != null)
            | .spec.selector.matchLabels // {}
            | to_entries
            | map("\(.key)=\(.value)")
            | join(",")
            | select(. == $sel)
            ' \
        | wc -l | tr -d '[:space:]')

      if [[ "$pdb_count" -eq 0 ]]; then
        echo -e "MISSING_PDB\t${ns}\t${name}\t${replicas}\tselector=${selector}"
      else
        echo -e "OK\t${ns}\t${name}\t${replicas}\tselector=${selector}\tPDBs=${pdb_count}"
      fi
    done

Explanation of output (what indicates a problem):

  • Lines starting with MISSING_PDB:

    • Example: MISSING_PDB my-namespace web-frontend 3 selector=app=web,role=frontend
    • Interpretation: this Deployment has replicas >= 2 and no PodDisruptionBudget with a matching label selector was found in its namespace. This is a candidate that should have a PodDisruptionBudget defined and reviewed.

  • Lines starting with WARN:

    • Example: WARN my-namespace legacy-app 4 NO matchLabels in selector (review manually)
    • Interpretation: the Deployment’s .spec.selector.matchLabels is empty; automatic matching to a PDB is not reliable. Manually inspect this Deployment and any PDBs in the same namespace.

  • Lines starting with OK:

    • Example: OK my-namespace api-server 5 selector=app=api PDBs=1
    • Interpretation: at least one PodDisruptionBudget exists whose selector exactly matches the Deployment’s matchLabels. These look compliant but still deserve human review for correct minAvailable/maxUnavailable values.