Skip to main content

Containers Should Define Liveness And Readiness Probes

More Info:​

Advisory: long-running containers should define livenessProbe and readinessProbe so Kubernetes can restart hung pods and keep traffic off pods that are not ready.

Risk Level​

Informational

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. Identify noncompliant pods and their owning controllers (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | (.spec.containers // [])[]
      | (.livenessProbe != null) as $live
      | (.readinessProbe != null) as $ready
      | "kind=Pod ns=\($m.namespace) name=\($m.name) labels=\($labels)"
        + (if $own == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)" end)
        + " container=\(.name) image=\(.image) livenessProbe=\($live) readinessProbe=\($ready)"
        + " is_compliant=\(if ($live and $ready) then "true" else "false" end)"
      ][] | select(. | test("is_compliant=false"))'

  2. For a pod managed by a controller (e.g., Deployment), edit the controller manifest to add probes (run on any machine with kubectl access). Example for a Deployment my-deploy in namespace my-namespace:

    kubectl -n my-namespace edit deployment my-deploy

    In the spec.template.spec.containers[] entry for each long-running container, add something like:

    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5

    Adjust paths, ports, and timings to match the application.

  3. For a bare Pod without an owning controller (used only for testing or debugging), either:

    • Add probes directly by editing: 
      kubectl -n my-namespace edit pod my-pod
      then update the container spec with livenessProbe and readinessProbe as above, understanding this will not persist if the pod is recreated; or
    • Preferably, re-create it from a proper manifest: 
      kubectl -n my-namespace get pod my-pod -o yaml > /tmp/my-pod.yaml
      Edit /tmp/my-pod.yaml to:
      • Remove metadata.uid, metadata.resourceVersion, metadata.creationTimestamp, status, and any ownerReferences.
      • Add livenessProbe and readinessProbe under each long-running container. Then apply:
      kubectl -n my-namespace delete pod my-pod
      kubectl apply -f /tmp/my-pod.yaml

  4. If your workloads are managed through GitOps or IaC (e.g., manifests in OCI DevOps or a Git repo), make the same livenessProbe and readinessProbe additions in the source manifests for each long-running container, then let your normal deployment process apply them. This ensures changes are not overwritten.

  5. After edits, wait for the new pods to become Ready (run on any machine with kubectl access):

    kubectl get pods --all-namespaces

    Confirm that pods from updated controllers are in Running state and READY columns show all containers ready (e.g., 1/1, 2/2).

  6. Verify compliance (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.containers // [])[]
      | (.livenessProbe != null) as $live
      | (.readinessProbe != null) as $ready
      | "ns=\($m.namespace) pod=\($m.name) container=\(.name) is_compliant=\(if ($live and $ready) then "true" else "false" end)"
      ] as $rows
      | if ($rows | map(select(. | test("is_compliant=false"))) | length) == 0
        then "is_compliant=true"
        else $rows[] end'

    Confirm output is is_compliant=true or that no lines show is_compliant=false for long-running containers you expect to be covered.

Using kubectl

On any machine with kubectl access:

  1. Identify non-compliant pods and their owners
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | (.livenessProbe != null) as $live
  | (.readinessProbe != null) as $ready
  | select((($live and $ready) | not))
  | "ns=\($m.namespace) pod=\($m.name) container=\(.name) image=\(.image)"
    + (if $own == null then "" else " ownerKind=\($own.kind) ownerName=\($own.name)" end)
  ][]'
  1. Edit the owning workload manifests and add probes

For workloads managed via kubectl manifests, edit the appropriate object (e.g. Deployment, StatefulSet, DaemonSet, Job, CronJob) and add both probes to each long‑running container.

Example patch for a Deployment container (adapt paths/ports to your app):

kubectl -n my-namespace patch deployment my-app-deployment --type merge -p '{
  "spec": {
    "template": {
      "spec": {
        "containers": [
          {
            "name": "my-app-container",
            "livenessProbe": {
              "httpGet": {
                "path": "/healthz",
                "port": 8080
              },
              "initialDelaySeconds": 30,
              "periodSeconds": 10,
              "timeoutSeconds": 5,
              "failureThreshold": 3,
              "successThreshold": 1
            },
            "readinessProbe": {
              "httpGet": {
                "path": "/readyz",
                "port": 8080
              },
              "initialDelaySeconds": 10,
              "periodSeconds": 5,
              "timeoutSeconds": 3,
              "failureThreshold": 3,
              "successThreshold": 1
            }
          }
        ]
      }
    }
  }
}'

If your container should use TCP or exec probes instead, adjust accordingly, for example:

kubectl -n my-namespace patch deployment my-app-deployment --type merge -p '{
  "spec": {
    "template": {
      "spec": {
        "containers": [
          {
            "name": "my-app-container",
            "livenessProbe": {
              "tcpSocket": { "port": 5432 },
              "initialDelaySeconds": 30,
              "periodSeconds": 10
            },
            "readinessProbe": {
              "exec": { "command": ["sh", "-c", "pg_isready -q"] },
              "initialDelaySeconds": 10,
              "periodSeconds": 5
            }
          }
        ]
      }
    }
  }
}'

For objects first defined in YAML, you can instead edit and re-apply:

kubectl -n my-namespace get deployment my-app-deployment -o yaml > my-app-deployment.yaml
# Edit my-app-deployment.yaml to add livenessProbe and readinessProbe under spec.template.spec.containers[]
kubectl apply -f my-app-deployment.yaml
  1. For bare Pods created directly (not recommended for long‑running apps)

Export, edit, and recreate with probes:

kubectl -n my-namespace get pod my-pod -o yaml > my-pod.yaml

Edit my-pod.yaml:

  • Remove fields under metadata such as creationTimestamp, resourceVersion, uid, selfLink, managedFields.
  • Remove status: completely.
  • Under each long‑running spec.containers[], add livenessProbe and readinessProbe.

Then recreate:

kubectl -n my-namespace delete pod my-pod
kubectl apply -f my-pod.yaml
  1. Verification

Run the original audit command and confirm all listed containers show is_compliant=true:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | (.livenessProbe != null) as $live
  | (.readinessProbe != null) as $ready
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\(.image) livenessProbe=\($live) readinessProbe=\($ready)"
    + " is_compliant=\(if ($live and $ready) then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Automation
#!/usr/bin/env bash
#
# automation-fix-probes.sh
#
# Idempotently ensure long-running containers have livenessProbe and readinessProbe
# by adding a noop tcpSocket probe on the container port or a simple HTTP probe on /
# for Deployment, StatefulSet, and DaemonSet workloads in an OKE cluster.
#
# REQUIREMENTS:
#   - Run on any machine with kubectl access and jq installed.
#   - Assumes you will review and adjust generated probe settings for your apps.
#
# NOTE:
#   - This script cannot safely infer ideal probe settings for every workload.
#   - It adds minimal, generic probes only where BOTH livenessProbe and
#     readinessProbe are currently missing for a container.
#   - You should test in a non-production environment first.

set -euo pipefail

# Namespace filter: exclude system namespaces as in the audit command
EXCLUDED_NS_REGEX='^(kube-system|kube-public|kube-node-lease)$'

timestamp() { date -u +"%Y-%m-%dT%H:%M:%SZ"; }

echo "[$(timestamp)] Starting probe remediation for long-running workloads..."

# Ensure jq exists
if ! command -v jq >/dev/null 2>&1; then
  echo "jq is required but not installed. Please install jq and re-run." >&2
  exit 1
fi

# Find all workloads with at least one container missing both probes
echo "[$(timestamp)] Discovering Deployments, StatefulSets, and DaemonSets with missing probes..."

kubectl get deploy,sts,ds --all-namespaces -o json \
  | jq -r '
    .items[]
    | select(.metadata.namespace | test("'"$EXCLUDED_NS_REGEX"'") | not)
    | {
        kind: .kind,
        apiVersion: .apiVersion,
        namespace: .metadata.namespace,
        name: .metadata.name,
        spec: .spec
      }
    ' \
  > /tmp/_probe_workloads_raw.json

if ! [ -s /tmp/_probe_workloads_raw.json ]; then
  echo "[$(timestamp)] No relevant workloads found."
else
  # Filter to only those with containers missing BOTH probes
  jq -c '
    select(
      (
        ( .spec.template.spec.containers // [] )
        | map(
            ( .livenessProbe // null ) == null and ( .readinessProbe // null ) == null
          )
        | any
      )
    )
  ' /tmp/_probe_workloads_raw.json > /tmp/_probe_workloads_target.json || true
fi

if ! [ -s /tmp/_probe_workloads_target.json ]; then
  echo "[$(timestamp)] No workloads have containers missing both probes. Nothing to change."
else
  echo "[$(timestamp)] Found workloads needing default probes:"
  jq -r '.namespace + " " + .kind + " " + .name' /tmp/_probe_workloads_target.json | sort | uniq

  echo "[$(timestamp)] Generating patched manifests with generic probes..."

  PATCH_DIR="/tmp/probe-patches-$(date +%s)"
  mkdir -p "$PATCH_DIR"

  # Build per-workload patched manifests
  while read -r item; do
    ns=$(echo "$item" | jq -r '.namespace')
    kind=$(echo "$item" | jq -r '.kind')
    name=$(echo "$item" | jq -r '.name')

    echo "[$(timestamp)] Processing $kind $ns/$name"

    # Fetch live object with full spec
    kubectl get "$kind" "$name" -n "$ns" -o json > "$PATCH_DIR/${ns}--${kind}--${name}.orig.json"

    # Build a patched version:
    # For each container:
    #   - If both livenessProbe and readinessProbe are missing, add generic probes.
    #   - If either probe exists, leave as-is (idempotent).
    jq '
      .spec.template.spec.containers |=
        map(
          if ((.livenessProbe // null) == null and (.readinessProbe // null) == null) then
            # Heuristic: if container has a single containerPort, use tcpSocket on that.
            # Otherwise, use basic HTTP GET on / over the first declared port if present,
            # or fall back to 8080.
            . as $c
            | (
                if ($c.ports // [] | length) > 0 then
                  ($c.ports[0].containerPort // 8080)
                else
                  8080
                end
              ) as $port
            | .livenessProbe = {
                initialDelaySeconds: 30,
                periodSeconds: 10,
                timeoutSeconds: 1,
                failureThreshold: 3,
                successThreshold: 1,
                tcpSocket: { port: $port }
              }
            | .readinessProbe = {
                initialDelaySeconds: 5,
                periodSeconds: 10,
                timeoutSeconds: 1,
                failureThreshold: 3,
                successThreshold: 1,
                tcpSocket: { port: $port }
              }
          else
            .
          end
        )
    ' "$PATCH_DIR/${ns}--${kind}--${name}.orig.json" > "$PATCH_DIR/${ns}--${kind}--${name}.patched.json"

    # Apply patched manifest (idempotent: same patch re-applies cleanly)
    echo "[$(timestamp)] Applying patched manifest to $kind $ns/$name"
    kubectl apply -f "$PATCH_DIR/${ns}--${kind}--${name}.patched.json"

  done < <(jq -c '.' /tmp/_probe_workloads_target.json)

  echo "[$(timestamp)] Waiting for updated pods to roll out (this may take time)..."
  # Perform rollout status for changed workloads
  while read -r item; do
    ns=$(echo "$item" | jq -r '.namespace')
    kind=$(echo "$item" | jq -r '.kind')
    name=$(echo "$item" | jq -r '.name')

    case "$kind" in
      Deployment)
        kubectl rollout status deploy/"$name" -n "$ns" --timeout=5m || true
        ;;
      StatefulSet)
        kubectl rollout status sts/"$name" -n "$ns" --timeout=5m || true
        ;;
      DaemonSet)
        kubectl rollout status ds/"$name" -n "$ns" --timeout=5m || true
        ;;
    esac
  done < <(jq -c '.' /tmp/_probe_workloads_target.json)

fi

echo "[$(timestamp)] Verifying compliance with liveness and readiness probes..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.containers // [])[]
  | (.livenessProbe != null) as $live
  | (.readinessProbe != null) as $ready
  | "ns=\($m.namespace) pod=\($m.name) container=\(.name) livenessProbe=\($live) readinessProbe=\($ready)"
    + " is_compliant=\(if ($live and $ready) then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end
'

echo "[$(timestamp)] Probe remediation script completed."

Usage (run on any machine with kubectl access):

bash automation-fix-probes.sh