> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Containers Should Define Liveness And Readiness Probes

### More Info:

Advisory: long-running containers should define livenessProbe and readinessProbe so Kubernetes can restart hung pods and keep traffic off pods that are not ready.

### Risk Level

Informational

### Address

Security

### Compliance Standards

* Cloudanix Best Practice

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Manual Steps" defaultOpen="true">
        1. On any machine with kubectl access, list non-compliant pods and note their namespaces, names, and owning controllers (Deployment/StatefulSet/Job/etc.):
           ```bash theme={null}
           kubectl get pods --all-namespaces -o json | jq -r '
             [ .items[]
             | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
             | .metadata as $m
             | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
             | (.spec.containers // [])[]
             | (.livenessProbe != null) as $live
             | (.readinessProbe != null) as $ready
             | select(( $live and $ready ) | not)
             | "ns=\($m.namespace) pod=\($m.name) container=\(.name)"
               + (if $own == null then "" else " ownerKind=\($own.kind) ownerName=\($own.name)" end)
             ][]'
           ```

        2. For pods managed by a controller (recommended), edit the controller manifest and add probes. Example for a Deployment `my-app` in namespace `prod`:
           ```bash theme={null}
           kubectl -n prod edit deployment my-app
           ```
           In the `spec.template.spec.containers[]` entry for each long-running container, add both probes, adapting paths/ports and thresholds to the app:
           ```yaml theme={null}
           livenessProbe:
             httpGet:
               path: /healthz
               port: 8080
             initialDelaySeconds: 30
             periodSeconds: 10
           readinessProbe:
             httpGet:
               path: /ready
               port: 8080
             initialDelaySeconds: 5
             periodSeconds: 5
           ```
           Save and exit; the Deployment will roll out new pods automatically.

        3. For pods defined directly by a Pod manifest (no controller), edit the manifest and re-apply it. First, export the manifest:
           ```bash theme={null}
           kubectl -n my-namespace get pod my-pod -o yaml > /tmp/my-pod.yaml
           ```
           Edit `/tmp/my-pod.yaml`, add appropriate `livenessProbe` and `readinessProbe` under the container, then delete and recreate the pod from the file:
           ```bash theme={null}
           kubectl -n my-namespace delete pod my-pod
           kubectl -n my-namespace apply -f /tmp/my-pod.yaml
           ```

        4. If your GKE workloads are managed by GitOps or another IaC system, locate the corresponding manifest in the repository and add the same `livenessProbe` and `readinessProbe` fields under each long-running container, then commit and let your pipeline apply the changes instead of using `kubectl edit`.

        5. After the updated workloads have rolled out and pods are running, verify compliance from any machine with kubectl access:
           ```bash theme={null}
           kubectl get pods --all-namespaces -o json | jq -r '
             [ .items[]
             | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
             | .metadata as $m
             | (.spec.containers // [])[]
             | (.livenessProbe != null) as $live
             | (.readinessProbe != null) as $ready
             | select(( $live and $ready ) | not)
             ] as $rows
             | if ($rows | length) == 0 then "is_compliant=true" else "is_compliant=false" end'
           ```
           The output should be `is_compliant=true`.
      </Accordion>

      <Accordion title="Using kubectl">
        On any machine with kubectl access:

        1. Identify a non-compliant pod and export its owning workload

        Use the audit output’s `owner=` field (e.g., `Deployment/default/web-app/...`). Then export the current manifest for that owner resource:

        ```bash theme={null}
        # Example for a Deployment
        kubectl get deployment web-app -n default -o yaml > web-app-deploy.yaml
        ```

        (For other controllers, substitute `deployment` with `statefulset`, `daemonset`, etc.)

        2. Edit the manifest to add probes

        Open the file and, for each long-running container, add both `livenessProbe` and `readinessProbe` under `spec.template.spec.containers[]`. Example HTTP-based probes:

        ```yaml theme={null}
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: web-app
          namespace: default
        spec:
          replicas: 3
          template:
            spec:
              containers:
                - name: web-app
                  image: gcr.io/PROJECT_ID/web-app:1.0.0
                  ports:
                    - containerPort: 8080
                  readinessProbe:
                    httpGet:
                      path: /healthz/ready
                      port: 8080
                    initialDelaySeconds: 5
                    periodSeconds: 10
                    failureThreshold: 3
                  livenessProbe:
                    httpGet:
                      path: /healthz/live
                      port: 8080
                    initialDelaySeconds: 15
                    periodSeconds: 20
                    failureThreshold: 3
        ```

        Adjust paths, ports, and timings to match the application’s health endpoints and startup characteristics.

        3. Apply the updated manifest

        ```bash theme={null}
        kubectl apply -f web-app-deploy.yaml
        ```

        This will roll out new pods with the probes defined.

        4. For bare Pods (no controller)

        If the audit output shows `owner=` is empty (a standalone Pod), fetch, modify, and re-create it (Pods cannot be updated in-place for some fields; recreate is safer):

        ```bash theme={null}
        kubectl get pod standalone-app -n default -o yaml > standalone-app-pod.yaml
        ```

        Edit `standalone-app-pod.yaml` to add `livenessProbe` and `readinessProbe` under `spec.containers[]` as above, then delete and recreate:

        ```bash theme={null}
        kubectl delete pod standalone-app -n default
        kubectl apply -f standalone-app-pod.yaml
        ```

        5. Verification

        Run the same style of audit to confirm the probes are now present:

        ```bash theme={null}
        kubectl get pods --all-namespaces -o json | jq -r '
          [ .items[]
          | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
          | .metadata as $m
          | (.spec.containers // [])[]
          | (.livenessProbe != null) as $live
          | (.readinessProbe != null) as $ready
          | select(($live and $ready) | not)
          ] | if length == 0 then "is_compliant=true" else . end'
        ```
      </Accordion>

      <Accordion title="Automation">
        ```bash theme={null}
        #!/usr/bin/env bash
        set -euo pipefail

        # Automation to ensure all long-running containers define livenessProbe and readinessProbe.
        # Platform: GKE
        # Runs on: any machine with kubectl, jq, and yq installed and authenticated to the cluster.
        #
        # IMPORTANT:
        # - This script cannot safely guess a correct probe for your application.
        # - It adds a simple HTTP 200 check on "/" at containerPort 80 as a generic default.
        # - You MUST review and customize the probes per workload after running.
        #
        # Requirements:
        #   kubectl (configured to target the cluster)
        #   jq
        #   yq (https://mikefarah.gitbook.io/yq/) – v4 syntax

        set -o pipefail

        # Namespace filter: exclude system namespaces per audit command
        EXCLUDED_NS_REGEX='^(kube-system|kube-public|kube-node-lease)$'

        # Default probe (HTTP GET on / port 80). Change as needed before running.
        DEFAULT_PROBE_YAML='
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 10
          failureThreshold: 3
        '

        # Ensure required tools
        for bin in kubectl jq yq; do
          if ! command -v "$bin" >/dev/null 2>&1; then
            echo "ERROR: $bin is required but not found in PATH" >&2
            exit 1
          fi
        done

        echo "Discovering non-compliant pods (missing livenessProbe or readinessProbe)..."

        # Get non-compliant pods as JSON
        NON_COMPLIANT_JSON="$(kubectl get pods --all-namespaces -o json | jq '
          .items[]
          | select(.metadata.namespace | test("'"$EXCLUDED_NS_REGEX"'" ) | not)
          | . as $pod
          | .spec.containers[]
          | select((.livenessProbe == null) or (.readinessProbe == null))
          | {
              namespace: $pod.metadata.namespace,
              podName: $pod.metadata.name,
              containerName: .name
            }
        ' 2>/dev/null || true)"

        if [[ -z "$NON_COMPLIANT_JSON" ]]; then
          echo "No non-compliant containers found. Cluster already compliant."
          exit 0
        fi

        # Build a unique list of (namespace, ownerKind, ownerName) for pod owners
        echo "Identifying owning controllers for non-compliant pods..."

        OWNER_LIST="$(kubectl get pods --all-namespaces -o json | jq -r '
          .items[]
          | select(.metadata.namespace | test("'"$EXCLUDED_NS_REGEX"'") | not)
          | . as $pod
          | .spec.containers[]
          | select((.livenessProbe == null) or (.readinessProbe == null))
          | $pod
          | (.metadata.ownerReferences // [])[]
          | select(.controller == true)
          | [ $pod.metadata.namespace, .kind, .name ]
          | @tsv
        ' | sort -u)"

        if [[ -z "$OWNER_LIST" ]]; then
          echo "Non-compliant pods have no controlling owner (e.g., bare Pods)."
          echo "Updating individual Pods in-place is not recommended for long-running workloads."
          echo "Please add probes manually to the Pod specs or their higher-level controllers if any."
        else
          echo "Controllers to patch (namespace kind name):"
          echo "$OWNER_LIST"
        fi

        # Function: patch a controller manifest with default probes where missing
        patch_controller() {
          local namespace="$1"
          local kind="$2"
          local name="$3"

          echo "Processing ${kind}/${namespace}/${name} ..."

          # Fetch full manifest
          if ! kubectl get "$kind" "$name" -n "$namespace" -o yaml >"/tmp/probe_fix_${kind}_${namespace}_${name}.yaml" 2>/dev/null; then
            echo "  WARN: Could not fetch ${kind}/${namespace}/${name}, skipping."
            return
          fi

          local manifest="/tmp/probe_fix_${kind}_${namespace}_${name}.yaml"
          local patched="/tmp/probe_fix_${kind}_${namespace}_${name}.patched.yaml"

          # Use yq to ensure probes exist on each container
          # This is idempotent: if probe already exists, it is left unchanged.
          yq eval '
            .spec.template.spec.containers |=
            ( map(
                . as $c
                | ( .livenessProbe // "'"$(printf '%s\n' "$DEFAULT_PROBE_YAML" | sed -n '2,/^readinessProbe:/p' | sed '$d')"'" | from_yaml.livenessProbe // .livenessProbe ) as $lp
                | ( .readinessProbe // "'"$(printf '%s\n' "$DEFAULT_PROBE_YAML" | sed -n '/^readinessProbe:/,$p')"'" | from_yaml.readinessProbe // .readinessProbe ) as $rp
                | .livenessProbe = $lp
                | .readinessProbe = $rp
              ))
          ' "$manifest" > "$patched"

          # Detect if there is any change; if not, skip apply
          if diff -q "$manifest" "$patched" >/dev/null 2>&1; then
            echo "  No changes needed; all containers already define probes."
            rm -f "$manifest" "$patched"
            return
          fi

          echo "  Applying patched manifest..."
          kubectl apply -f "$patched"

          rm -f "$manifest" "$patched"
        }

        # Patch each owner
        if [[ -n "$OWNER_LIST" ]]; then
          while IFS=$'\t' read -r ns kind name; do
            [[ -z "$ns" || -z "$kind" || -z "$name" ]] && continue
            patch_controller "$ns" "$kind" "$name"
          done <<< "$OWNER_LIST"
        fi

        echo "Waiting for updated pods to roll out..."
        sleep 10

        # Verification: rerun the audit command
        echo "Re-running compliance audit..."
        kubectl get pods --all-namespaces -o json | jq -r '
          [ .items[]
          | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
          | .metadata as $m
          | (.spec.containers // [])[]
          | (.livenessProbe != null) as $live
          | (.readinessProbe != null) as $ready
          | "ns=\($m.namespace) pod=\($m.name) container=\(.name) is_compliant=\(if ($live and $ready) then "true" else "false" end)"
          ] as $rows
          | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end
        '

        echo "NOTE: For each workload, review and customize the livenessProbe and readinessProbe to accurately reflect application health."
        ```
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>