Multi-Replica Deployments Should Have A PodDisruptionBudget

More Info:

Advisory: define a PodDisruptionBudget for each multi-replica Deployment so node drains and rollouts keep a minimum number of pods available.

Risk Level

Informational

Address

Security

Compliance Standards

Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

Identify multi-replica Deployments that lack a PodDisruptionBudget
Run on: any machine with kubectl access

{ kubectl get poddisruptionbudgets --all-namespaces -o json \
  kubectl get deployments --all-namespaces -o json; } | jq -rs '
  .[0] as $pdbs | .[1] |
  [ .items[]
  | select((.spec.replicas // 1) > 1)
  | .metadata as $m
  | (.spec.template.metadata.labels // {}) as $podLabels
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ $pdbs.items[]
       | select(.metadata.namespace == $m.namespace)
       | select((.spec.selector.matchLabels // {}) | length > 0)
       | select([ (.spec.selector.matchLabels | to_entries)[]
                  | $podLabels[.key] == .value ] | all)
     ] | length) as $count
  | select($count == 0)
  | "ns=\($m.namespace) name=\($m.name) replicas=\(.spec.replicas)"
  ][]'

For one non-compliant Deployment, capture its labels to use in the PodDisruptionBudget selector
Run on: any machine with kubectl access
Replace NAMESPACE and DEPLOYMENT_NAME as needed:
```
kubectl get deployment DEPLOYMENT_NAME -n NAMESPACE -o jsonpath='{.spec.template.metadata.labels}' | jq
```
Note the key/value pairs; these must be copied exactly into the PodDisruptionBudget’s spec.selector.matchLabels.
Decide the availability requirement for this Deployment and choose the PodDisruptionBudget spec
Run on: any machine with kubectl access
Typical choices:
- To always keep at least one pod running:
```
minAvailable: 1
```
- Or to allow only 1 pod at a time to be disrupted in a 3+ replica Deployment:
```
maxUnavailable: 1
```
Ensure the chosen value makes sense given .spec.replicas for this Deployment.

Create a PodDisruptionBudget manifest for the Deployment using its pod labels
Run on: any machine with kubectl access
Example template; replace NAMESPACE, PDB_NAME, and the labels under matchLabels with the exact values from step 2:

cat > pdb-deployment-DEPLOYMENT_NAME.yaml << 'EOF'
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: PDB_NAME
  namespace: NAMESPACE
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-app            # replace with real label key/value
      tier: backend          # remove or adjust keys to match exactly
EOF

Ensure matchLabels matches a subset of the pod template labels of the target Deployment so the PDB actually selects its pods.

Apply the PodDisruptionBudget to the cluster
Run on: any machine with kubectl access
```
kubectl apply -f pdb-deployment-DEPLOYMENT_NAME.yaml
```

Verification: confirm all multi-replica Deployments now have at least one matching PodDisruptionBudget
Run on: any machine with kubectl access

{ kubectl get poddisruptionbudgets --all-namespaces -o json \
  kubectl get deployments --all-namespaces -o json; } | jq -rs '
  .[0] as $pdbs | .[1] |
  [ .items[]
  | select((.spec.replicas // 1) > 1)
  | .metadata as $m
  | (.spec.template.metadata.labels // {}) as $podLabels
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ $pdbs.items[]
       | select(.metadata.namespace == $m.namespace)
       | select((.spec.selector.matchLabels // {}) | length > 0)
       | select([ (.spec.selector.matchLabels | to_entries)[]
                  | $podLabels[.key] == .value ] | all)
     ] | length) as $count
  | "kind=Deployment ns=\($m.namespace) name=\($m.name) replicas=\(.spec.replicas) podDisruptionBudgets=\($count) is_compliant=\(if $count > 0 then "true" else "false" end)"
  ] as $rows
  | if ($rows | map(select(. | test("is_compliant=false"))) | length) == 0
    then "is_compliant=true"
    else $rows[] end'

Ensure all listed Deployments show is_compliant=true; if any show is_compliant=false, repeat steps 2–5 for those Deployments.

Using kubectl

On any machine with kubectl access:

Identify a multi‑replica Deployment needing a PDB (example: namespace production, deployment web-app):

kubectl -n production get deploy web-app -o yaml | sed -n '1,80p'

Note the pod template labels under spec.template.metadata.labels. Example:

spec:
  template:
    metadata:
      labels:
        app: web-app
        tier: frontend

Create a PodDisruptionBudget manifest that selects those labels and enforces a minimum number of available pods. Example (adjust labels, name, and minAvailable/maxUnavailable per your SLOs):

cat <<'EOF' > pdb-web-app.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-app-pdb
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web-app
      tier: frontend
EOF

Apply the PDB:

kubectl apply -f pdb-web-app.yaml

Repeat steps 1–3 for each Deployment with spec.replicas > 1, ensuring each PDB’s spec.selector.matchLabels exactly matches the Deployment’s pod template labels.
Verification (cluster‑wide, same logic as the audit):

{ kubectl get poddisruptionbudgets --all-namespaces -o json
  kubectl get deployments --all-namespaces -o json
} | jq -rs '
  .[0] as $pdbs | .[1] |
  [ .items[]
  | select((.spec.replicas // 1) > 1)
  | .metadata as $m
  | (.spec.template.metadata.labels // {}) as $podLabels
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ $pdbs.items[]
       | select(.metadata.namespace == $m.namespace)
       | select((.spec.selector.matchLabels // {}) | length > 0)
       | select([ (.spec.selector.matchLabels | to_entries)[]
                  | $podLabels[.key] == .value ] | all)
     ] | length) as $count
  | "kind=Deployment ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=apps/v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $labels == "" then "" else " labels=\($labels)" end)
    + " replicas=\(.spec.replicas) podDisruptionBudgets=\($count)"
    + " is_compliant=\(if $count > 0 then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

Automation

#!/usr/bin/env bash
#
# Ensure every multi-replica Deployment has a matching PodDisruptionBudget
# that uses matchLabels equal to the Deployment's pod-template labels.
#
# Requirements:
#   - Run on any machine with kubectl access to the GKE cluster
#   - kubectl, jq, and bash installed
#   - Current kube-context set to the target cluster
#
# Behaviour:
#   - Skips Deployments with replicas <= 1
#   - Creates/updates one PDB per Deployment:
#       name: <deployment-name>-pdb
#       selector.matchLabels: deployment's pod template labels
#       spec.minAvailable: 1 (if replicas == 2) else "50%" (if replicas > 2)
#   - Only manages PDBs it creates (matching the naming pattern)
#   - Safe to re-run (idempotent)
set -euo pipefail

echo "=== Discovering multi-replica Deployments and existing PDBs ==="

# Collect PDBs and Deployments as JSON arrays
PDB_JSON="$(kubectl get poddisruptionbudgets --all-namespaces -o json)"
DEPLOY_JSON="$(kubectl get deployments --all-namespaces -o json)"

# Function: determine if a Deployment already has a covering PDB (matchLabels only)
has_covering_pdb() {
  local ns="$1" name="$2"
  jq -er --arg ns "$ns" --arg name "$name" \
    --argjson pdbs "$PDB_JSON" --argjson deps "$DEPLOY_JSON" '
      ($deps.items[]
       | select(.metadata.namespace == $ns and .metadata.name == $name)) as $dep
      | ($dep.spec.template.metadata.labels // {}) as $podLabels
      | [ $pdbs.items[]
          | select(.metadata.namespace == $ns)
          | select((.spec.selector.matchLabels // {}) | length > 0)
          | select(
              # matchLabels must be a subset of pod labels and not empty
              (.spec.selector.matchLabels // {}) as $sel
              | ($sel | length) > 0
              and ([ $sel | to_entries[]
                     | ($podLabels[.key] == .value) ] | all)
            )
        ] | length
      | . > 0
    ' >/dev/null 2>&1
}

# Iterate all Deployments and reconcile required PDBs
echo "=== Reconciling PodDisruptionBudgets for multi-replica Deployments ==="

echo "$DEPLOY_JSON" | jq -r '
  .items[]
  | select((.spec.replicas // 1) > 1)
  | [.metadata.namespace, .metadata.name, (.spec.replicas // 1)] | @tsv
' | while IFS=$'\t' read -r NS NAME REPLICAS; do
  # Extract pod template labels for this Deployment
  POD_LABELS_JSON="$(echo "$DEPLOY_JSON" | jq -r --arg ns "$NS" --arg name "$NAME" '
    .items[]
    | select(.metadata.namespace == $ns and .metadata.name == $name)
    | (.spec.template.metadata.labels // {})
  ')"

  # Skip if there are no pod template labels (PDB selector would be empty)
  if [[ "$(echo "$POD_LABELS_JSON" | jq 'length')" -eq 0 ]]; then
    echo "Skipping ${NS}/${NAME}: no pod template labels to select on"
    continue
  fi

  if has_covering_pdb "$NS" "$NAME"; then
    echo "OK: ${NS}/${NAME} already has a covering PDB"
    continue
  fi

  # Decide minAvailable:
  # - 1 if exactly 2 replicas (to maximize availability)
  # - "50%" if more than 2 replicas (balanced disruption)
  MIN_AVAILABLE=""
  if [[ "$REPLICAS" -eq 2 ]]; then
    MIN_AVAILABLE="1"
  else
    MIN_AVAILABLE="50%"
  fi

  PDB_NAME="${NAME}-pdb"

  echo "Reconciling PDB for ${NS}/${NAME} -> ${NS}/${PDB_NAME} (minAvailable=${MIN_AVAILABLE})"

  # Build a minimal PDB manifest using the deployment's pod-template labels as matchLabels
  # Note: We include only matchLabels, as the audit check ignores matchExpressions.
  PDB_MANIFEST="$(jq -n \
    --arg apiVersion "policy/v1" \
    --arg kind "PodDisruptionBudget" \
    --arg ns "$NS" \
    --arg name "$PDB_NAME" \
    --argjson labels "$POD_LABELS_JSON" \
    --arg minAvail "$MIN_AVAILABLE" '
    {
      apiVersion: $apiVersion,
      kind: $kind,
      metadata: {
        name: $name,
        namespace: $ns,
        labels: {
          "managed-by": "pdb-automation-script"
        }
      },
      spec: {
        minAvailable: (try ( $minAvail|tonumber ) catch $minAvail),
        selector: {
          matchLabels: $labels
        }
      }
    }
  ')"

  # Apply the PDB manifest (create or update)
  echo "$PDB_MANIFEST" | kubectl apply -f -
done

echo "=== Verification: re-running compliance-style check ==="

{
  kubectl get poddisruptionbudgets --all-namespaces -o json
  kubectl get deployments --all-namespaces -o json
} | jq -rs '
  .[0] as $pdbs | .[1] |
  [ .items[]
  | select((.spec.replicas // 1) > 1)
  | .metadata as $m
  | (.spec.template.metadata.labels // {}) as $podLabels
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ $pdbs.items[]
       | select(.metadata.namespace == $m.namespace)
       | select((.spec.selector.matchLabels // {}) | length > 0)
       | select([ (.spec.selector.matchLabels | to_entries)[]
                  | $podLabels[.key] == .value ] | all)
     ] | length) as $count
  | "kind=Deployment ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=apps/v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $labels == "" then "" else " labels=\($labels)" end)
    + " replicas=\(.spec.replicas) podDisruptionBudgets=\($count)"
    + " is_compliant=\(if $count > 0 then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end
'

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​