Containers Should Define Liveness And Readiness Probes

More Info:

Advisory: long-running containers should define livenessProbe and readinessProbe so Kubernetes can restart hung pods and keep traffic off pods that are not ready.

Risk Level

Informational

Address

Security

Compliance Standards

Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

On any machine with kubectl access, list non-compliant pods and note their namespaces, names, and owning controllers (Deployment/StatefulSet/Job/etc.):

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | (.livenessProbe != null) as $live
  | (.readinessProbe != null) as $ready
  | select(( $live and $ready ) | not)
  | "ns=\($m.namespace) pod=\($m.name) container=\(.name)"
    + (if $own == null then "" else " ownerKind=\($own.kind) ownerName=\($own.name)" end)
  ][]'

For pods managed by a controller (recommended), edit the controller manifest and add probes. Example for a Deployment my-app in namespace prod:
```
kubectl -n prod edit deployment my-app
```
In the spec.template.spec.containers[] entry for each long-running container, add both probes, adapting paths/ports and thresholds to the app:
```
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
```
Save and exit; the Deployment will roll out new pods automatically.
For pods defined directly by a Pod manifest (no controller), edit the manifest and re-apply it. First, export the manifest:
```
kubectl -n my-namespace get pod my-pod -o yaml > /tmp/my-pod.yaml
```
Edit /tmp/my-pod.yaml, add appropriate livenessProbe and readinessProbe under the container, then delete and recreate the pod from the file:
```
kubectl -n my-namespace delete pod my-pod
kubectl -n my-namespace apply -f /tmp/my-pod.yaml
```
If your GKE workloads are managed by GitOps or another IaC system, locate the corresponding manifest in the repository and add the same livenessProbe and readinessProbe fields under each long-running container, then commit and let your pipeline apply the changes instead of using kubectl edit.

After the updated workloads have rolled out and pods are running, verify compliance from any machine with kubectl access:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.containers // [])[]
  | (.livenessProbe != null) as $live
  | (.readinessProbe != null) as $ready
  | select(( $live and $ready ) | not)
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else "is_compliant=false" end'

The output should be is_compliant=true.

Using kubectl

On any machine with kubectl access:

Identify a non-compliant pod and export its owning workload

Use the audit output’s owner= field (e.g., Deployment/default/web-app/...). Then export the current manifest for that owner resource:

# Example for a Deployment
kubectl get deployment web-app -n default -o yaml > web-app-deploy.yaml

(For other controllers, substitute deployment with statefulset, daemonset, etc.)

Edit the manifest to add probes

Open the file and, for each long-running container, add both livenessProbe and readinessProbe under spec.template.spec.containers[]. Example HTTP-based probes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  namespace: default
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: web-app
          image: gcr.io/PROJECT_ID/web-app:1.0.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /healthz/live
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 20
            failureThreshold: 3

Adjust paths, ports, and timings to match the application’s health endpoints and startup characteristics.

Apply the updated manifest

kubectl apply -f web-app-deploy.yaml

This will roll out new pods with the probes defined.

For bare Pods (no controller)

If the audit output shows owner= is empty (a standalone Pod), fetch, modify, and re-create it (Pods cannot be updated in-place for some fields; recreate is safer):

kubectl get pod standalone-app -n default -o yaml > standalone-app-pod.yaml

Edit standalone-app-pod.yaml to add livenessProbe and readinessProbe under spec.containers[] as above, then delete and recreate:

kubectl delete pod standalone-app -n default
kubectl apply -f standalone-app-pod.yaml

Verification

Run the same style of audit to confirm the probes are now present:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.containers // [])[]
  | (.livenessProbe != null) as $live
  | (.readinessProbe != null) as $ready
  | select(($live and $ready) | not)
  ] | if length == 0 then "is_compliant=true" else . end'

Automation

#!/usr/bin/env bash
set -euo pipefail

# Automation to ensure all long-running containers define livenessProbe and readinessProbe.
# Platform: GKE
# Runs on: any machine with kubectl, jq, and yq installed and authenticated to the cluster.
#
# IMPORTANT:
# - This script cannot safely guess a correct probe for your application.
# - It adds a simple HTTP 200 check on "/" at containerPort 80 as a generic default.
# - You MUST review and customize the probes per workload after running.
#
# Requirements:
#   kubectl (configured to target the cluster)
#   jq
#   yq (https://mikefarah.gitbook.io/yq/) – v4 syntax

set -o pipefail

# Namespace filter: exclude system namespaces per audit command
EXCLUDED_NS_REGEX='^(kube-system|kube-public|kube-node-lease)$'

# Default probe (HTTP GET on / port 80). Change as needed before running.
DEFAULT_PROBE_YAML='
livenessProbe:
  httpGet:
    path: /
    port: 80
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /
    port: 80
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3
'

# Ensure required tools
for bin in kubectl jq yq; do
  if ! command -v "$bin" >/dev/null 2>&1; then
    echo "ERROR: $bin is required but not found in PATH" >&2
    exit 1
  fi
done

echo "Discovering non-compliant pods (missing livenessProbe or readinessProbe)..."

# Get non-compliant pods as JSON
NON_COMPLIANT_JSON="$(kubectl get pods --all-namespaces -o json | jq '
  .items[]
  | select(.metadata.namespace | test("'"$EXCLUDED_NS_REGEX"'" ) | not)
  | . as $pod
  | .spec.containers[]
  | select((.livenessProbe == null) or (.readinessProbe == null))
  | {
      namespace: $pod.metadata.namespace,
      podName: $pod.metadata.name,
      containerName: .name
    }
' 2>/dev/null || true)"

if [[ -z "$NON_COMPLIANT_JSON" ]]; then
  echo "No non-compliant containers found. Cluster already compliant."
  exit 0
fi

# Build a unique list of (namespace, ownerKind, ownerName) for pod owners
echo "Identifying owning controllers for non-compliant pods..."

OWNER_LIST="$(kubectl get pods --all-namespaces -o json | jq -r '
  .items[]
  | select(.metadata.namespace | test("'"$EXCLUDED_NS_REGEX"'") | not)
  | . as $pod
  | .spec.containers[]
  | select((.livenessProbe == null) or (.readinessProbe == null))
  | $pod
  | (.metadata.ownerReferences // [])[]
  | select(.controller == true)
  | [ $pod.metadata.namespace, .kind, .name ]
  | @tsv
' | sort -u)"

if [[ -z "$OWNER_LIST" ]]; then
  echo "Non-compliant pods have no controlling owner (e.g., bare Pods)."
  echo "Updating individual Pods in-place is not recommended for long-running workloads."
  echo "Please add probes manually to the Pod specs or their higher-level controllers if any."
else
  echo "Controllers to patch (namespace kind name):"
  echo "$OWNER_LIST"
fi

# Function: patch a controller manifest with default probes where missing
patch_controller() {
  local namespace="$1"
  local kind="$2"
  local name="$3"

  echo "Processing ${kind}/${namespace}/${name} ..."

  # Fetch full manifest
  if ! kubectl get "$kind" "$name" -n "$namespace" -o yaml >"/tmp/probe_fix_${kind}_${namespace}_${name}.yaml" 2>/dev/null; then
    echo "  WARN: Could not fetch ${kind}/${namespace}/${name}, skipping."
    return
  fi

  local manifest="/tmp/probe_fix_${kind}_${namespace}_${name}.yaml"
  local patched="/tmp/probe_fix_${kind}_${namespace}_${name}.patched.yaml"

  # Use yq to ensure probes exist on each container
  # This is idempotent: if probe already exists, it is left unchanged.
  yq eval '
    .spec.template.spec.containers |=
    ( map(
        . as $c
        | ( .livenessProbe // "'"$(printf '%s\n' "$DEFAULT_PROBE_YAML" | sed -n '2,/^readinessProbe:/p' | sed '$d')"'" | from_yaml.livenessProbe // .livenessProbe ) as $lp
        | ( .readinessProbe // "'"$(printf '%s\n' "$DEFAULT_PROBE_YAML" | sed -n '/^readinessProbe:/,$p')"'" | from_yaml.readinessProbe // .readinessProbe ) as $rp
        | .livenessProbe = $lp
        | .readinessProbe = $rp
      ))
  ' "$manifest" > "$patched"

  # Detect if there is any change; if not, skip apply
  if diff -q "$manifest" "$patched" >/dev/null 2>&1; then
    echo "  No changes needed; all containers already define probes."
    rm -f "$manifest" "$patched"
    return
  fi

  echo "  Applying patched manifest..."
  kubectl apply -f "$patched"

  rm -f "$manifest" "$patched"
}

# Patch each owner
if [[ -n "$OWNER_LIST" ]]; then
  while IFS=$'\t' read -r ns kind name; do
    [[ -z "$ns" || -z "$kind" || -z "$name" ]] && continue
    patch_controller "$ns" "$kind" "$name"
  done <<< "$OWNER_LIST"
fi

echo "Waiting for updated pods to roll out..."
sleep 10

# Verification: rerun the audit command
echo "Re-running compliance audit..."
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.containers // [])[]
  | (.livenessProbe != null) as $live
  | (.readinessProbe != null) as $ready
  | "ns=\($m.namespace) pod=\($m.name) container=\(.name) is_compliant=\(if ($live and $ready) then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end
'

echo "NOTE: For each workload, review and customize the livenessProbe and readinessProbe to accurately reflect application health."

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​