> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Containers Should Define Liveness And Readiness Probes

### More Info:

Advisory: long-running containers should define livenessProbe and readinessProbe so Kubernetes can restart hung pods and keep traffic off pods that are not ready.

### Risk Level

Informational

### Address

Security

### Compliance Standards

* Cloudanix Best Practice

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Manual Steps" defaultOpen="true">
        1. Identify noncompliant pods and their owning controllers (run on any machine with kubectl access):
           ```bash theme={null}
           kubectl get pods --all-namespaces -o json | jq -r '
             [ .items[]
             | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
             | .metadata as $m
             | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
             | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
             | (.spec.containers // [])[]
             | (.livenessProbe != null) as $live
             | (.readinessProbe != null) as $ready
             | "kind=Pod ns=\($m.namespace) name=\($m.name) labels=\($labels)"
               + (if $own == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)" end)
               + " container=\(.name) image=\(.image) livenessProbe=\($live) readinessProbe=\($ready)"
               + " is_compliant=\(if ($live and $ready) then "true" else "false" end)"
             ][] | select(. | test("is_compliant=false"))'
           ```

        2. For a pod managed by a controller (e.g., Deployment), edit the controller manifest to add probes (run on any machine with kubectl access). Example for a Deployment `my-deploy` in namespace `my-namespace`:
           ```bash theme={null}
           kubectl -n my-namespace edit deployment my-deploy
           ```
           In the `spec.template.spec.containers[]` entry for each long-running container, add something like:
           ```yaml theme={null}
           livenessProbe:
             httpGet:
               path: /healthz
               port: 8080
             initialDelaySeconds: 30
             periodSeconds: 10
           readinessProbe:
             httpGet:
               path: /ready
               port: 8080
             initialDelaySeconds: 5
             periodSeconds: 5
           ```
           Adjust paths, ports, and timings to match the application.

        3. For a bare Pod without an owning controller (used only for testing or debugging), either:
           * Add probes directly by editing:
             ```bash theme={null}
             kubectl -n my-namespace edit pod my-pod
             ```
             then update the container spec with `livenessProbe` and `readinessProbe` as above, understanding this will not persist if the pod is recreated; or
           * Preferably, re-create it from a proper manifest:
             ```bash theme={null}
             kubectl -n my-namespace get pod my-pod -o yaml > /tmp/my-pod.yaml
             ```
             Edit `/tmp/my-pod.yaml` to:
             * Remove `metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp`, `status`, and any `ownerReferences`.
             * Add `livenessProbe` and `readinessProbe` under each long-running container.
               Then apply:
             ```bash theme={null}
             kubectl -n my-namespace delete pod my-pod
             kubectl apply -f /tmp/my-pod.yaml
             ```

        4. If your workloads are managed through GitOps or IaC (e.g., manifests in OCI DevOps or a Git repo), make the same `livenessProbe` and `readinessProbe` additions in the source manifests for each long-running container, then let your normal deployment process apply them. This ensures changes are not overwritten.

        5. After edits, wait for the new pods to become Ready (run on any machine with kubectl access):
           ```bash theme={null}
           kubectl get pods --all-namespaces
           ```
           Confirm that pods from updated controllers are in `Running` state and `READY` columns show all containers ready (e.g., `1/1`, `2/2`).

        6. Verify compliance (run on any machine with kubectl access):
           ```bash theme={null}
           kubectl get pods --all-namespaces -o json | jq -r '
             [ .items[]
             | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
             | .metadata as $m
             | (.spec.containers // [])[]
             | (.livenessProbe != null) as $live
             | (.readinessProbe != null) as $ready
             | "ns=\($m.namespace) pod=\($m.name) container=\(.name) is_compliant=\(if ($live and $ready) then "true" else "false" end)"
             ] as $rows
             | if ($rows | map(select(. | test("is_compliant=false"))) | length) == 0
               then "is_compliant=true"
               else $rows[] end'
           ```
           Confirm output is `is_compliant=true` or that no lines show `is_compliant=false` for long-running containers you expect to be covered.
      </Accordion>

      <Accordion title="Using kubectl">
        On any machine with kubectl access:

        1. Identify non-compliant pods and their owners

        ```bash theme={null}
        kubectl get pods --all-namespaces -o json | jq -r '
          [ .items[]
          | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
          | .metadata as $m
          | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
          | (.spec.containers // [])[]
          | (.livenessProbe != null) as $live
          | (.readinessProbe != null) as $ready
          | select((($live and $ready) | not))
          | "ns=\($m.namespace) pod=\($m.name) container=\(.name) image=\(.image)"
            + (if $own == null then "" else " ownerKind=\($own.kind) ownerName=\($own.name)" end)
          ][]'
        ```

        2. Edit the owning workload manifests and add probes

        For workloads managed via kubectl manifests, edit the appropriate object (e.g. Deployment, StatefulSet, DaemonSet, Job, CronJob) and add both probes to each long‑running container.

        Example patch for a Deployment container (adapt paths/ports to your app):

        ```bash theme={null}
        kubectl -n my-namespace patch deployment my-app-deployment --type merge -p '{
          "spec": {
            "template": {
              "spec": {
                "containers": [
                  {
                    "name": "my-app-container",
                    "livenessProbe": {
                      "httpGet": {
                        "path": "/healthz",
                        "port": 8080
                      },
                      "initialDelaySeconds": 30,
                      "periodSeconds": 10,
                      "timeoutSeconds": 5,
                      "failureThreshold": 3,
                      "successThreshold": 1
                    },
                    "readinessProbe": {
                      "httpGet": {
                        "path": "/readyz",
                        "port": 8080
                      },
                      "initialDelaySeconds": 10,
                      "periodSeconds": 5,
                      "timeoutSeconds": 3,
                      "failureThreshold": 3,
                      "successThreshold": 1
                    }
                  }
                ]
              }
            }
          }
        }'
        ```

        If your container should use TCP or exec probes instead, adjust accordingly, for example:

        ```bash theme={null}
        kubectl -n my-namespace patch deployment my-app-deployment --type merge -p '{
          "spec": {
            "template": {
              "spec": {
                "containers": [
                  {
                    "name": "my-app-container",
                    "livenessProbe": {
                      "tcpSocket": { "port": 5432 },
                      "initialDelaySeconds": 30,
                      "periodSeconds": 10
                    },
                    "readinessProbe": {
                      "exec": { "command": ["sh", "-c", "pg_isready -q"] },
                      "initialDelaySeconds": 10,
                      "periodSeconds": 5
                    }
                  }
                ]
              }
            }
          }
        }'
        ```

        For objects first defined in YAML, you can instead edit and re-apply:

        ```bash theme={null}
        kubectl -n my-namespace get deployment my-app-deployment -o yaml > my-app-deployment.yaml
        # Edit my-app-deployment.yaml to add livenessProbe and readinessProbe under spec.template.spec.containers[]
        kubectl apply -f my-app-deployment.yaml
        ```

        3. For bare Pods created directly (not recommended for long‑running apps)

        Export, edit, and recreate with probes:

        ```bash theme={null}
        kubectl -n my-namespace get pod my-pod -o yaml > my-pod.yaml
        ```

        Edit `my-pod.yaml`:

        * Remove fields under `metadata` such as `creationTimestamp`, `resourceVersion`, `uid`, `selfLink`, `managedFields`.
        * Remove `status:` completely.
        * Under each long‑running `spec.containers[]`, add `livenessProbe` and `readinessProbe`.

        Then recreate:

        ```bash theme={null}
        kubectl -n my-namespace delete pod my-pod
        kubectl apply -f my-pod.yaml
        ```

        4. Verification

        Run the original audit command and confirm all listed containers show `is_compliant=true`:

        ```bash theme={null}
        kubectl get pods --all-namespaces -o json | jq -r '
          [ .items[]
          | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
          | .metadata as $m
          | (.spec.nodeName // "") as $node
          | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
          | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
          | (.spec.containers // [])[]
          | (.livenessProbe != null) as $live
          | (.readinessProbe != null) as $ready
          | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
            + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
            + (if $node   == ""   then "" else " node=\($node)" end)
            + (if $labels == ""   then "" else " labels=\($labels)" end)
            + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
            + " container=\(.name) image=\(.image) livenessProbe=\($live) readinessProbe=\($ready)"
            + " is_compliant=\(if ($live and $ready) then "true" else "false" end)"
          ] as $rows
          | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
        ```
      </Accordion>

      <Accordion title="Automation">
        ```bash theme={null}
        #!/usr/bin/env bash
        #
        # automation-fix-probes.sh
        #
        # Idempotently ensure long-running containers have livenessProbe and readinessProbe
        # by adding a noop tcpSocket probe on the container port or a simple HTTP probe on /
        # for Deployment, StatefulSet, and DaemonSet workloads in an OKE cluster.
        #
        # REQUIREMENTS:
        #   - Run on any machine with kubectl access and jq installed.
        #   - Assumes you will review and adjust generated probe settings for your apps.
        #
        # NOTE:
        #   - This script cannot safely infer ideal probe settings for every workload.
        #   - It adds minimal, generic probes only where BOTH livenessProbe and
        #     readinessProbe are currently missing for a container.
        #   - You should test in a non-production environment first.

        set -euo pipefail

        # Namespace filter: exclude system namespaces as in the audit command
        EXCLUDED_NS_REGEX='^(kube-system|kube-public|kube-node-lease)$'

        timestamp() { date -u +"%Y-%m-%dT%H:%M:%SZ"; }

        echo "[$(timestamp)] Starting probe remediation for long-running workloads..."

        # Ensure jq exists
        if ! command -v jq >/dev/null 2>&1; then
          echo "jq is required but not installed. Please install jq and re-run." >&2
          exit 1
        fi

        # Find all workloads with at least one container missing both probes
        echo "[$(timestamp)] Discovering Deployments, StatefulSets, and DaemonSets with missing probes..."

        kubectl get deploy,sts,ds --all-namespaces -o json \
          | jq -r '
            .items[]
            | select(.metadata.namespace | test("'"$EXCLUDED_NS_REGEX"'") | not)
            | {
                kind: .kind,
                apiVersion: .apiVersion,
                namespace: .metadata.namespace,
                name: .metadata.name,
                spec: .spec
              }
            ' \
          > /tmp/_probe_workloads_raw.json

        if ! [ -s /tmp/_probe_workloads_raw.json ]; then
          echo "[$(timestamp)] No relevant workloads found."
        else
          # Filter to only those with containers missing BOTH probes
          jq -c '
            select(
              (
                ( .spec.template.spec.containers // [] )
                | map(
                    ( .livenessProbe // null ) == null and ( .readinessProbe // null ) == null
                  )
                | any
              )
            )
          ' /tmp/_probe_workloads_raw.json > /tmp/_probe_workloads_target.json || true
        fi

        if ! [ -s /tmp/_probe_workloads_target.json ]; then
          echo "[$(timestamp)] No workloads have containers missing both probes. Nothing to change."
        else
          echo "[$(timestamp)] Found workloads needing default probes:"
          jq -r '.namespace + " " + .kind + " " + .name' /tmp/_probe_workloads_target.json | sort | uniq

          echo "[$(timestamp)] Generating patched manifests with generic probes..."

          PATCH_DIR="/tmp/probe-patches-$(date +%s)"
          mkdir -p "$PATCH_DIR"

          # Build per-workload patched manifests
          while read -r item; do
            ns=$(echo "$item" | jq -r '.namespace')
            kind=$(echo "$item" | jq -r '.kind')
            name=$(echo "$item" | jq -r '.name')

            echo "[$(timestamp)] Processing $kind $ns/$name"

            # Fetch live object with full spec
            kubectl get "$kind" "$name" -n "$ns" -o json > "$PATCH_DIR/${ns}--${kind}--${name}.orig.json"

            # Build a patched version:
            # For each container:
            #   - If both livenessProbe and readinessProbe are missing, add generic probes.
            #   - If either probe exists, leave as-is (idempotent).
            jq '
              .spec.template.spec.containers |=
                map(
                  if ((.livenessProbe // null) == null and (.readinessProbe // null) == null) then
                    # Heuristic: if container has a single containerPort, use tcpSocket on that.
                    # Otherwise, use basic HTTP GET on / over the first declared port if present,
                    # or fall back to 8080.
                    . as $c
                    | (
                        if ($c.ports // [] | length) > 0 then
                          ($c.ports[0].containerPort // 8080)
                        else
                          8080
                        end
                      ) as $port
                    | .livenessProbe = {
                        initialDelaySeconds: 30,
                        periodSeconds: 10,
                        timeoutSeconds: 1,
                        failureThreshold: 3,
                        successThreshold: 1,
                        tcpSocket: { port: $port }
                      }
                    | .readinessProbe = {
                        initialDelaySeconds: 5,
                        periodSeconds: 10,
                        timeoutSeconds: 1,
                        failureThreshold: 3,
                        successThreshold: 1,
                        tcpSocket: { port: $port }
                      }
                  else
                    .
                  end
                )
            ' "$PATCH_DIR/${ns}--${kind}--${name}.orig.json" > "$PATCH_DIR/${ns}--${kind}--${name}.patched.json"

            # Apply patched manifest (idempotent: same patch re-applies cleanly)
            echo "[$(timestamp)] Applying patched manifest to $kind $ns/$name"
            kubectl apply -f "$PATCH_DIR/${ns}--${kind}--${name}.patched.json"

          done < <(jq -c '.' /tmp/_probe_workloads_target.json)

          echo "[$(timestamp)] Waiting for updated pods to roll out (this may take time)..."
          # Perform rollout status for changed workloads
          while read -r item; do
            ns=$(echo "$item" | jq -r '.namespace')
            kind=$(echo "$item" | jq -r '.kind')
            name=$(echo "$item" | jq -r '.name')

            case "$kind" in
              Deployment)
                kubectl rollout status deploy/"$name" -n "$ns" --timeout=5m || true
                ;;
              StatefulSet)
                kubectl rollout status sts/"$name" -n "$ns" --timeout=5m || true
                ;;
              DaemonSet)
                kubectl rollout status ds/"$name" -n "$ns" --timeout=5m || true
                ;;
            esac
          done < <(jq -c '.' /tmp/_probe_workloads_target.json)

        fi

        echo "[$(timestamp)] Verifying compliance with liveness and readiness probes..."

        kubectl get pods --all-namespaces -o json | jq -r '
          [ .items[]
          | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
          | .metadata as $m
          | (.spec.containers // [])[]
          | (.livenessProbe != null) as $live
          | (.readinessProbe != null) as $ready
          | "ns=\($m.namespace) pod=\($m.name) container=\(.name) livenessProbe=\($live) readinessProbe=\($ready)"
            + " is_compliant=\(if ($live and $ready) then "true" else "false" end)"
          ] as $rows
          | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end
        '

        echo "[$(timestamp)] Probe remediation script completed."
        ```

        **Usage (run on any machine with kubectl access):**

        ```bash theme={null}
        bash automation-fix-probes.sh
        ```
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>