Skip to main content

Multi-Replica Deployments Should Have A PodDisruptionBudget

More Info:​

Advisory: define a PodDisruptionBudget for each multi-replica Deployment so node drains and rollouts keep a minimum number of pods available.

Risk Level​

Informational

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. Identify multi-replica Deployments and their labels (run on any machine with kubectl access)

    kubectl get deployments --all-namespaces -o wide
    # For each multi-replica deployment, inspect pod template labels:
    kubectl get deployment <deployment-name> -n <namespace> -o jsonpath='{.spec.template.metadata.labels}{"\n"}'

  2. Decide the minimum available pods for each Deployment (planning step)
    For each multi-replica Deployment, choose either:

    • minAvailable: minimum number of pods that must stay available during disruptions, or
    • maxUnavailable: maximum number or percentage of pods that can be disrupted at once.
      Ensure the value makes sense for the workload’s availability/SLO requirements.

  3. Create a PodDisruptionBudget manifest that matches the pod labels (run on any machine with kubectl access)
    Create a file such as pdb-<deployment-name>.yaml with matchLabels exactly matching the pod template labels from step 1 (do not use matchExpressions, as the audit ignores them). Example structure:

    apiVersion: policy/v1
    kind: PodDisruptionBudget
    metadata:
      name: <deployment-name>-pdb
      namespace: <namespace>
    spec:
      minAvailable: 1         # or use maxUnavailable: 1 / "25%"
      selector:
        matchLabels:
          app: my-app         # replace with the exact key: value pairs from the pod template
          tier: backend       # add/remove keys to match the Deployment’s pod labels

  4. Apply the PodDisruptionBudget to the cluster (run on any machine with kubectl access)

    kubectl apply -f pdb-<deployment-name>.yaml

  5. Confirm that each PDB selector correctly matches pods from its Deployment (run on any machine with kubectl access)

    kubectl get poddisruptionbudgets -n <namespace>
    kubectl get poddisruptionbudget <deployment-name>-pdb -n <namespace> -o yaml | sed -n '1,80p'
    # Verify the selector.matchLabels exactly corresponds to the Deployment’s pod template labels:
    kubectl get deployment <deployment-name> -n <namespace> -o jsonpath='{.spec.template.metadata.labels}{"\n"}'

  6. Verify compliance using the benchmark audit logic (run on any machine with kubectl access)

    { kubectl get poddisruptionbudgets --all-namespaces -o json
      kubectl get deployments --all-namespaces -o json
    } | jq -rs '
      .[0] as $pdbs | .[1] |
      [ .items[]
      | select((.spec.replicas // 1) > 1)
      | .metadata as $m
      | (.spec.template.metadata.labels // {}) as $podLabels
      | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
      | ([ $pdbs.items[]
           | select(.metadata.namespace == $m.namespace)
           | select((.spec.selector.matchLabels // {}) | length > 0)
           | select([ (.spec.selector.matchLabels | to_entries)[]
                      | $podLabels[.key] == .value ] | all)
         ] | length) as $count
      | "kind=Deployment ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=apps/v1"
        + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
        + (if $labels == "" then "" else " labels=\($labels)" end)
        + " replicas=\(.spec.replicas) podDisruptionBudgets=\($count)"
        + " is_compliant=\(if $count > 0 then "true" else "false" end)"
      ] as $rows
      | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

    Ensure every multi-replica Deployment reports is_compliant=true.

Using kubectl

On any machine with kubectl access:

  1. Identify multi‑replica Deployments and their labels
kubectl get deployments --all-namespaces -o wide
kubectl get deployment -n YOUR_NAMESPACE YOUR_DEPLOYMENT -o jsonpath='{.spec.template.metadata.labels}{"\n"}'
  1. Create a PodDisruptionBudget manifest matching the Deployment’s pod labels

Example (edit namespace, name, and labels to match your Deployment):

# pdb-multi-replica-example.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: my-namespace
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-app
      tier: backend

Apply it:

kubectl apply -f pdb-multi-replica-example.yaml

Repeat as needed so each Deployment with .spec.replicas > 1 has at least one PDB whose spec.selector.matchLabels is a subset of the pod template labels for that Deployment.

  1. Verification

Re‑run the audit (any machine with kubectl access):

{ kubectl get poddisruptionbudgets --all-namespaces -o json
  kubectl get deployments --all-namespaces -o json
} | jq -rs '
  .[0] as $pdbs | .[1] |
  [ .items[]
  | select((.spec.replicas // 1) > 1)
  | .metadata as $m
  | (.spec.template.metadata.labels // {}) as $podLabels
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ $pdbs.items[]
       | select(.metadata.namespace == $m.namespace)
       | select((.spec.selector.matchLabels // {}) | length > 0)
       | select([ (.spec.selector.matchLabels | to_entries)[]
                  | $podLabels[.key] == .value ] | all)
     ] | length) as $count
  | "kind=Deployment ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=apps/v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $labels == "" then "" else " labels=\($labels)" end)
    + " replicas=\(.spec.replicas) podDisruptionBudgets=\($count)"
    + " is_compliant=\(if $count > 0 then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Automation
#!/usr/bin/env bash
#
# Ensure every multi-replica Deployment has a matching PodDisruptionBudget
# that uses only matchLabels in its selector (per CBP C5.1).
#
# Run on: any machine with kubectl access and jq installed.
# Idempotent: safe to re-run; only creates PDBs that are missing.

set -euo pipefail

# --- prerequisites check ---
if ! command -v kubectl >/dev/null 2>&1; then
  echo "ERROR: kubectl not found in PATH" >&2
  exit 1
fi

if ! command -v jq >/dev/null 2>&1; then
  echo "ERROR: jq not found in PATH" >&2
  exit 1
fi

KCTX="$(kubectl config current-context 2>/dev/null || true)"
if [ -z "$KCTX" ]; then
  echo "ERROR: kubectl has no current-context configured" >&2
  exit 1
fi

echo "Using kubectl context: $KCTX"
echo

# --- discovery: find non-compliant Deployments (replicas > 1 without covering PDB) ---
echo "Discovering multi-replica Deployments without a matching PodDisruptionBudget..."

NONCOMPLIANT_JSON="$(
  {
    kubectl get poddisruptionbudgets --all-namespaces -o json 2>/dev/null || echo '{"items":[]}'
    kubectl get deployments --all-namespaces -o json
  } | jq -rs '
    .[0] as $pdbs | .[1] |
    [ .items[]
      | select((.spec.replicas // 1) > 1)
      | .metadata as $m
      | (.spec.template.metadata.labels // {}) as $podLabels
      | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
      | ([ $pdbs.items[]
           | select(.metadata.namespace == $m.namespace)
           | select((.spec.selector.matchLabels // {}) | length > 0)
           | select([ (.spec.selector.matchLabels | to_entries)[]
                      | $podLabels[.key] == .value ] | all)
         ] | length) as $count
      | select($count == 0)
      | {
          namespace: $m.namespace,
          name: $m.name,
          replicas: (.spec.replicas // 1),
          podLabels: $podLabels
        }
    ]'
)"

if [ "$(echo "$NONCOMPLIANT_JSON" | jq 'length')" -eq 0 ]; then
  echo "All multi-replica Deployments already have a matching PodDisruptionBudget."
  echo
  echo "Verification output (benchmark-style):"
  {
    kubectl get poddisruptionbudgets --all-namespaces -o json 2>/dev/null || echo '{"items":[]}'
    kubectl get deployments --all-namespaces -o json
  } | jq -rs '
    .[0] as $pdbs | .[1] |
    [ .items[]
    | select((.spec.replicas // 1) > 1)
    | .metadata as $m
    | (.spec.template.metadata.labels // {}) as $podLabels
    | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
    | ([ $pdbs.items[]
         | select(.metadata.namespace == $m.namespace)
         | select((.spec.selector.matchLabels // {}) | length > 0)
         | select([ (.spec.selector.matchLabels | to_entries)[]
                    | $podLabels[.key] == .value ] | all)
       ] | length) as $count
    | "kind=Deployment ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=apps/v1"
      + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
      + (if $labels == "" then "" else " labels=\($labels)" end)
      + " replicas=\(.spec.replicas) podDisruptionBudgets=\($count)"
      + " is_compliant=\(if $count > 0 then "true" else "false" end)"
    ] as $rows
    | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
  exit 0
fi

echo "$NONCOMPLIANT_JSON" | jq -r '.[] | "- Namespace: \(.namespace), Deployment: \(.name), replicas: \(.replicas)"'
echo

# --- create PDBs for each non-compliant Deployment ---
# Strategy: for each Deployment, create a PDB:
# - name: "<deployment-name>-pdb"
# - selector.matchLabels = pod template labels
# - use a conservative minAvailable of 1 (can be tuned per-Deployment later)
#
# If a PDB with that name already exists in the namespace, it is left as-is
# (to avoid overwriting manual settings), and we still verify compliance.

TMPDIR="$(mktemp -d)"
trap 'rm -rf "$TMPDIR"' EXIT

echo "Creating PodDisruptionBudgets where missing..."

echo "$NONCOMPLIANT_JSON" | jq -c '.[]' | while read -r item; do
  ns="$(echo "$item" | jq -r '.namespace')"
  dep="$(echo "$item" | jq -r '.name')"
  replicas="$(echo "$item" | jq -r '.replicas')"
  pdb_name="${dep}-pdb"

  # Build matchLabels yaml fragment from podLabels
  labels_yaml="$(
    echo "$item" | jq -r '.podLabels | to_entries[]? | "      \(.key): \"\(.value)\""' || true
  )"

  if [ -z "$labels_yaml" ]; then
    echo "WARNING: Deployment ${ns}/${dep} has no pod template labels; skipping PDB creation." >&2
    continue
  fi

  # Skip if a PDB with this name already exists
  if kubectl get poddisruptionbudget "$pdb_name" -n "$ns" >/dev/null 2>&1; then
    echo "PDB ${ns}/${pdb_name} already exists; not modifying it."
    continue
  fi

  cat > "${TMPDIR}/${ns}-${dep}-pdb.yaml" <<EOF
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ${pdb_name}
  namespace: ${ns}
spec:
  minAvailable: 1
  selector:
    matchLabels:
$(echo "$labels_yaml")
EOF

  echo "Applying PDB for ${ns}/${dep} (replicas=${replicas})..."
  kubectl apply -f "${TMPDIR}/${ns}-${dep}-pdb.yaml"
done

echo
echo "PDB creation complete."
echo

# --- verification (benchmark-style) ---
echo "Verification output (benchmark-style):"
{
  kubectl get poddisruptionbudgets --all-namespaces -o json 2>/dev/null || echo '{"items":[]}'
  kubectl get deployments --all-namespaces -o json
} | jq -rs '
  .[0] as $pdbs | .[1] |
  [ .items[]
  | select((.spec.replicas // 1) > 1)
  | .metadata as $m
  | (.spec.template.metadata.labels // {}) as $podLabels
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ $pdbs.items[]
       | select(.metadata.namespace == $m.namespace)
       | select((.spec.selector.matchLabels // {}) | length > 0)
       | select([ (.spec.selector.matchLabels | to_entries)[]
                  | $podLabels[.key] == .value ] | all)
     ] | length) as $count
  | "kind=Deployment ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=apps/v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $labels == "" then "" else " labels=\($labels)" end)
    + " replicas=\(.spec.replicas) podDisruptionBudgets=\($count)"
    + " is_compliant=\(if $count > 0 then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'