> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Containers Should Define Liveness And Readiness Probes

### More Info:

Advisory: long-running containers should define livenessProbe and readinessProbe so Kubernetes can restart hung pods and keep traffic off pods that are not ready.

### Risk Level

Informational

### Address

Security

### Compliance Standards

* Cloudanix Best Practice

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Manual Steps" defaultOpen="true">
        1. **Identify long-running pods and candidate containers**
           * On any machine with kubectl access:
             ```bash theme={null}
             kubectl get pods --all-namespaces -o wide
             ```
           * Focus on:
             * Workload pods (Deployments, StatefulSets, DaemonSets, Jobs that run indefinitely).
             * Exclude obviously short-lived Jobs/CronJobs and completed pods.

        2. **Inspect existing probes on target pods**
           * For each candidate pod, inspect its spec:
             ```bash theme={null}
             kubectl get pod <pod-name> -n <namespace> -o yaml
             ```
           * Under `.spec.containers[].livenessProbe` and `.spec.containers[].readinessProbe`, verify that:
             * Both probes exist.
             * The command or HTTP/TCP/GRPC check is appropriate for determining health/readiness.

        3. **Locate and edit the owning workload manifest**
           * Determine the owner (e.g., Deployment, StatefulSet) for pods missing probes:
             ```bash theme={null}
             kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.metadata.ownerReferences[*].kind}{" "}{.metadata.ownerReferences[*].name}{"\n"}'
             ```
           * Fetch the current manifest for editing:
             ```bash theme={null}
             kubectl get <KIND> <NAME> -n <namespace> -o yaml > /tmp/<kind>-<name>.yaml
             ```
           * In `/tmp/<kind>-<name>.yaml`, under each long-running container in `spec.template.spec.containers[]`, add or refine `livenessProbe` and `readinessProbe` per the application’s behavior (e.g., HTTP GET on a health endpoint, exec check, or TCP socket).

        4. **Apply the updated workload manifests**
           * On any machine with kubectl access, apply the edited manifest:
             ```bash theme={null}
             kubectl apply -f /tmp/<kind>-<name>.yaml
             ```
           * Be aware: updating a Pod template will trigger a rollout; pods will be recreated. Confirm this is acceptable or coordinate a maintenance window.

        5. **Verify probes are present and functioning**
           * After rollout completes, confirm probes on the new pods:
             ```bash theme={null}
             kubectl get pods -n <namespace> -l <app-label>=<value>
             kubectl get pod <new-pod-name> -n <namespace> -o yaml | \
               grep -A5 -E 'livenessProbe|readinessProbe'
             ```
           * Check probe status and events for failures:
             ```bash theme={null}
             kubectl describe pod <new-pod-name> -n <namespace> | grep -i "Liveness\|Readiness" -A3
             ```

        6. **Document exceptions for workloads without probes**
           * For any container where probes are intentionally omitted (e.g., very short-lived Jobs), record:
             * Workload name, namespace, and rationale.
             * Evidence of behavior:
               ```bash theme={null}
               kubectl get <kind> <name> -n <namespace> -o yaml > /tmp/<kind>-<name>-documented-exception.yaml
               ```
           * Store this documentation in your configuration/IaC repo or security exception register.
      </Accordion>

      <Accordion title="Using kubectl">
        ```bash theme={null}
        # 1) List all pods and their namespaces
        # Run on: any machine with kubectl access
        kubectl get pods --all-namespaces -o wide
        ```

        Look for long-running workloads (services, APIs, workers, cron controllers, etc.) as candidates to review. Short-lived batch jobs may reasonably not use probes.

        ```bash theme={null}
        # 2) Show container specs (including probes) for a specific pod
        # Replace <namespace> and <pod-name>
        kubectl get pod <pod-name> -n <namespace> -o yaml
        ```

        In the `spec.containers[]` section of the output, inspect each container:

        * Problem indication: `livenessProbe` is missing or empty.
        * Problem indication: `readinessProbe` is missing or empty.
        * These fields must be under each container, not just in initContainers.

        ```bash theme={null}
        # 3) Quickly list pods that appear to have no liveness/readiness probes
        # (heuristic: shows pods where *no* container has a given probe)
        kubectl get pods --all-namespaces -o json \
          | jq -r '
            .items[]
            | {
                ns: .metadata.namespace,
                pod: .metadata.name,
                containers: [.spec.containers[].name],
                liveness_missing: (
                  [ .spec.containers[]
                    | has("livenessProbe") and (.livenessProbe != null)
                  ] | any | not
                ),
                readiness_missing: (
                  [ .spec.containers[]
                    | has("readinessProbe") and (.readinessProbe != null)
                  ] | any | not
                )
              }
            | select(.liveness_missing or .readiness_missing)
            | "\(.ns) \(.pod) containers=\(.containers|join(",")) liveness_missing=\(.liveness_missing) readiness_missing=\(.readiness_missing)"
          '
        ```

        Interpretation:

        * Each line is a pod with at least one issue.
        * `liveness_missing=true`: none of the containers in this pod define a livenessProbe.
        * `readiness_missing=true`: none of the containers in this pod define a readinessProbe.
        * These pods need human review to decide whether probes are appropriate for each container.

        ```bash theme={null}
        # 4) Inspect a higher-level controller (recommended: adjust at the controller, not the pod)
        # Example: Deployment
        kubectl get deploy <deployment-name> -n <namespace> -o yaml
        ```

        Again, inspect `spec.template.spec.containers[]`:

        * Problem: long-running containers under this controller lack `livenessProbe` and/or `readinessProbe`.
        * Fix decisions should be made here (Deployment/StatefulSet/DaemonSet spec), then applied via your normal GitOps/manifest process.
      </Accordion>

      <Accordion title="Automation">
        ```bash theme={null}
        #!/usr/bin/env bash
        # Report pods whose containers lack livenessProbe and/or readinessProbe
        # Run on: any machine with kubectl access
        # Requires: kubectl, jq

        set -euo pipefail

        # Namespace selector: "" = all namespaces
        NAMESPACE_SELECTOR="--all-namespaces"

        echo "Scanning pods for missing liveness/readiness probes..."
        echo

        kubectl get pods ${NAMESPACE_SELECTOR} -o json | jq -r '
          .items[]
          | {
              ns: .metadata.namespace,
              pod: .metadata.name,
              owner: (
                (.metadata.ownerReferences[0].kind + "/" + .metadata.ownerReferences[0].name)
                // "POD_DIRECT"
              ),
              containers: [
                .spec.containers[]
                | {
                    name,
                    hasLiveness: (has("livenessProbe")),
                    hasReadiness: (has("readinessProbe"))
                  }
              ]
            }
          | . as $pod
          | $pod.containers[]
          | select((.hasLiveness == false) or (.hasReadiness == false))
          | @tsv "\($pod.ns)\t\($pod.pod)\t\($pod.owner)\t\(.name)\t\(.hasLiveness)\t\(.hasReadiness)"
        ' | awk '
        BEGIN {
          OFS="\t";
          print "NAMESPACE","POD","OWNER","CONTAINER","HAS_LIVENESS","HAS_READINESS"
        } { print }'

        echo
        echo "Legend:"
        echo "  OWNER:"
        echo "    * KIND/NAME  = higher-level controller (Deployment, DaemonSet, etc.)"
        echo "    * POD_DIRECT = pod created directly (no ownerReference)"
        echo
        echo "  HAS_LIVENESS / HAS_READINESS:"
        echo "    true  = probe present on this container"
        echo "    false = probe missing on this container"
        ```

        How to interpret the output:

        * Any line where `HAS_LIVENESS` is `false` indicates a container without a `livenessProbe`.
        * Any line where `HAS_READINESS` is `false` indicates a container without a `readinessProbe`.
        * Focus review on long-running applications; short-lived batch/Job pods may legitimately omit probes.
        * Use the `OWNER` column to locate and update the underlying controller manifest (e.g., Deployment) rather than editing individual pods.
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>