Containers Should Define Liveness And Readiness Probes

More Info:

Advisory: long-running containers should define livenessProbe and readinessProbe so Kubernetes can restart hung pods and keep traffic off pods that are not ready.

Risk Level

Informational

Address

Security

Compliance Standards

Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

Identify long-running pods and candidate containers
- On any machine with kubectl access:
```
kubectl get pods --all-namespaces -o wide
```
- Focus on:
  - Workload pods (Deployments, StatefulSets, DaemonSets, Jobs that run indefinitely).
  - Exclude obviously short-lived Jobs/CronJobs and completed pods.
Inspect existing probes on target pods
- For each candidate pod, inspect its spec:
```
kubectl get pod <pod-name> -n <namespace> -o yaml
```
- Under .spec.containers[].livenessProbe and .spec.containers[].readinessProbe, verify that:
  - Both probes exist.
  - The command or HTTP/TCP/GRPC check is appropriate for determining health/readiness.
Locate and edit the owning workload manifest
- Determine the owner (e.g., Deployment, StatefulSet) for pods missing probes:
```
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.metadata.ownerReferences[*].kind}{" "}{.metadata.ownerReferences[*].name}{"\n"}'
```
- Fetch the current manifest for editing:
```
kubectl get <KIND> <NAME> -n <namespace> -o yaml > /tmp/<kind>-<name>.yaml
```
- In /tmp/<kind>-<name>.yaml, under each long-running container in spec.template.spec.containers[], add or refine livenessProbe and readinessProbe per the application’s behavior (e.g., HTTP GET on a health endpoint, exec check, or TCP socket).
Apply the updated workload manifests
- On any machine with kubectl access, apply the edited manifest:
```
kubectl apply -f /tmp/<kind>-<name>.yaml
```
- Be aware: updating a Pod template will trigger a rollout; pods will be recreated. Confirm this is acceptable or coordinate a maintenance window.

Verify probes are present and functioning

After rollout completes, confirm probes on the new pods:

kubectl get pods -n <namespace> -l <app-label>=<value>
kubectl get pod <new-pod-name> -n <namespace> -o yaml | \
  grep -A5 -E 'livenessProbe|readinessProbe'

Check probe status and events for failures:

kubectl describe pod <new-pod-name> -n <namespace> | grep -i "Liveness\|Readiness" -A3

Document exceptions for workloads without probes
- For any container where probes are intentionally omitted (e.g., very short-lived Jobs), record:
  - Workload name, namespace, and rationale.
  - Evidence of behavior:
    kubectl get <kind> <name> -n <namespace> -o yaml > /tmp/<kind>-<name>-documented-exception.yaml
- Store this documentation in your configuration/IaC repo or security exception register.

Using kubectl

# 1) List all pods and their namespaces
# Run on: any machine with kubectl access
kubectl get pods --all-namespaces -o wide

Look for long-running workloads (services, APIs, workers, cron controllers, etc.) as candidates to review. Short-lived batch jobs may reasonably not use probes.

# 2) Show container specs (including probes) for a specific pod
# Replace <namespace> and <pod-name>
kubectl get pod <pod-name> -n <namespace> -o yaml

In the spec.containers[] section of the output, inspect each container:

Problem indication: livenessProbe is missing or empty.
Problem indication: readinessProbe is missing or empty.
These fields must be under each container, not just in initContainers.

# 3) Quickly list pods that appear to have no liveness/readiness probes
# (heuristic: shows pods where *no* container has a given probe)
kubectl get pods --all-namespaces -o json \
  | jq -r '
    .items[]
    | {
        ns: .metadata.namespace,
        pod: .metadata.name,
        containers: [.spec.containers[].name],
        liveness_missing: (
          [ .spec.containers[]
            | has("livenessProbe") and (.livenessProbe != null)
          ] | any | not
        ),
        readiness_missing: (
          [ .spec.containers[]
            | has("readinessProbe") and (.readinessProbe != null)
          ] | any | not
        )
      }
    | select(.liveness_missing or .readiness_missing)
    | "\(.ns) \(.pod) containers=\(.containers|join(",")) liveness_missing=\(.liveness_missing) readiness_missing=\(.readiness_missing)"
  '

Interpretation:

Each line is a pod with at least one issue.
liveness_missing=true: none of the containers in this pod define a livenessProbe.
readiness_missing=true: none of the containers in this pod define a readinessProbe.
These pods need human review to decide whether probes are appropriate for each container.

# 4) Inspect a higher-level controller (recommended: adjust at the controller, not the pod)
# Example: Deployment
kubectl get deploy <deployment-name> -n <namespace> -o yaml

Again, inspect spec.template.spec.containers[]:

Problem: long-running containers under this controller lack livenessProbe and/or readinessProbe.
Fix decisions should be made here (Deployment/StatefulSet/DaemonSet spec), then applied via your normal GitOps/manifest process.

Automation

#!/usr/bin/env bash
# Report pods whose containers lack livenessProbe and/or readinessProbe
# Run on: any machine with kubectl access
# Requires: kubectl, jq

set -euo pipefail

# Namespace selector: "" = all namespaces
NAMESPACE_SELECTOR="--all-namespaces"

echo "Scanning pods for missing liveness/readiness probes..."
echo

kubectl get pods ${NAMESPACE_SELECTOR} -o json | jq -r '
  .items[]
  | {
      ns: .metadata.namespace,
      pod: .metadata.name,
      owner: (
        (.metadata.ownerReferences[0].kind + "/" + .metadata.ownerReferences[0].name)
        // "POD_DIRECT"
      ),
      containers: [
        .spec.containers[]
        | {
            name,
            hasLiveness: (has("livenessProbe")),
            hasReadiness: (has("readinessProbe"))
          }
      ]
    }
  | . as $pod
  | $pod.containers[]
  | select((.hasLiveness == false) or (.hasReadiness == false))
  | @tsv "\($pod.ns)\t\($pod.pod)\t\($pod.owner)\t\(.name)\t\(.hasLiveness)\t\(.hasReadiness)"
' | awk '
BEGIN {
  OFS="\t";
  print "NAMESPACE","POD","OWNER","CONTAINER","HAS_LIVENESS","HAS_READINESS"
} { print }'

echo
echo "Legend:"
echo "  OWNER:"
echo "    * KIND/NAME  = higher-level controller (Deployment, DaemonSet, etc.)"
echo "    * POD_DIRECT = pod created directly (no ownerReference)"
echo
echo "  HAS_LIVENESS / HAS_READINESS:"
echo "    true  = probe present on this container"
echo "    false = probe missing on this container"

How to interpret the output:

Any line where HAS_LIVENESS is false indicates a container without a livenessProbe.
Any line where HAS_READINESS is false indicates a container without a readinessProbe.
Focus review on long-running applications; short-lived batch/Job pods may legitimately omit probes.
Use the OWNER column to locate and update the underlying controller manifest (e.g., Deployment) rather than editing individual pods.

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​