> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Containers Should Define Liveness And Readiness Probes

### More Info:

Advisory: long-running containers should define livenessProbe and readinessProbe so Kubernetes can restart hung pods and keep traffic off pods that are not ready.

### Risk Level

Informational

### Address

Security

### Compliance Standards

* Cloudanix Best Practice

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Manual Steps" defaultOpen="true">
        1. Identify offending pods and their owners (run on any machine with kubectl access):
           ```bash theme={null}
           kubectl get pods --all-namespaces -o json | jq -r '
             [ .items[]
             | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
             | .metadata as $m
             | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
             | (.spec.containers // [])[]
             | (.livenessProbe != null) as $live
             | (.readinessProbe != null) as $ready
             | select(( $live and $ready ) | not)
             | "ns=\($m.namespace) pod=\($m.name) container=\(.name) image=\(.image)"
               + (if $own == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)" end)
             ][]'
           ```

        2. For each non-compliant workload that is controlled by a higher-level object (e.g., Deployment, StatefulSet, DaemonSet), export its manifest for editing (run on any machine with kubectl access; replace values with those from step 1):
           ```bash theme={null}
           kubectl get deployment <deployment-name> -n <namespace> -o yaml > /tmp/deployment-<deployment-name>.yaml
           ```
           (Similarly use `statefulset` or `daemonset` in place of `deployment` as needed.)

        3. Edit the manifest to add `livenessProbe` and `readinessProbe` for each long-running container (run on the same machine where you saved the file):
           ```bash theme={null}
           vi /tmp/deployment-<deployment-name>.yaml
           ```
           Under `spec.template.spec.containers[]` for each long-running container, add probes appropriate to the application, for example:
           ```yaml theme={null}
           livenessProbe:
             httpGet:
               path: /healthz
               port: 8080
             initialDelaySeconds: 30
             periodSeconds: 10
           readinessProbe:
             httpGet:
               path: /ready
               port: 8080
             initialDelaySeconds: 5
             periodSeconds: 5
           ```
           Adjust paths, ports, and timings to match the container’s behavior.

        4. Apply the updated manifest so the controller recreates pods with probes (run on any machine with kubectl access):
           ```bash theme={null}
           kubectl apply -f /tmp/deployment-<deployment-name>.yaml
           ```
           Repeat steps 2–4 for each affected Deployment/StatefulSet/DaemonSet.

        5. For any standalone Pod objects that you intentionally manage directly (no controller owner), edit and re-apply the Pod manifest (run on any machine with kubectl access):
           ```bash theme={null}
           kubectl get pod <pod-name> -n <namespace> -o yaml > /tmp/pod-<namespace>-<pod-name>.yaml
           vi /tmp/pod-<namespace>-<pod-name>.yaml
           ```
           Add `livenessProbe` and `readinessProbe` under `spec.containers[]` as in step 3, then:
           ```bash theme={null}
           kubectl delete pod <pod-name> -n <namespace>
           kubectl apply -f /tmp/pod-<namespace>-<pod-name>.yaml
           ```

        6. Verify that all long-running containers now define both probes (run on any machine with kubectl access):
           ```bash theme={null}
           kubectl get pods --all-namespaces -o json | jq -r '
             [ .items[]
             | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
             | .metadata as $m
             | (.spec.containers // [])[]
             | (.livenessProbe != null) as $live
             | (.readinessProbe != null) as $ready
             | "ns=\($m.namespace) pod=\($m.name) container=\(.name) livenessProbe=\($live) readinessProbe=\($ready)"
             ] as $rows
             | if ([ .items[]
                     | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
                     | .spec.containers[] 
                     | (.livenessProbe != null) as $live
                     | (.readinessProbe != null) as $ready
                     | select(( $live and $ready ) | not)
                   ] | length) == 0
               then "is_compliant=true"
               else $rows[] end'
           ```
      </Accordion>

      <Accordion title="Using kubectl">
        On any machine with kubectl access:

        1. Identify non-compliant pods and their controllers (Deployments, etc.):

        ```bash theme={null}
        kubectl get pods --all-namespaces -o json | jq -r '
          [ .items[]
          | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
          | .metadata as $m
          | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
          | (.spec.containers // [])[]
          | (.livenessProbe != null) as $live
          | (.readinessProbe != null) as $ready
          | select(( $live and $ready ) | not)
          | "\($own.kind) \($m.namespace) \($own.name)"
          ] | unique[]' | sort -u
        ```

        2. For each owning controller (example: Deployment `my-app` in namespace `prod`), export its manifest:

        ```bash theme={null}
        kubectl -n prod get deploy my-app -o yaml > /tmp/deploy-my-app.yaml
        ```

        3. Edit `/tmp/deploy-my-app.yaml` and add `livenessProbe` and `readinessProbe` to each long-running container. Example HTTP-based probes:

        ```yaml theme={null}
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: my-app
          namespace: prod
        spec:
          template:
            spec:
              containers:
              - name: app
                image: my-registry.example.com/app:1.0.0
                ports:
                - containerPort: 8080
                livenessProbe:
                  httpGet:
                    path: /healthz
                    port: 8080
                  initialDelaySeconds: 30
                  periodSeconds: 10
                  timeoutSeconds: 5
                  failureThreshold: 3
                readinessProbe:
                  httpGet:
                    path: /readyz
                    port: 8080
                  initialDelaySeconds: 10
                  periodSeconds: 5
                  timeoutSeconds: 3
                  failureThreshold: 3
        ```

        Apply the updated manifest:

        ```bash theme={null}
        kubectl apply -f /tmp/deploy-my-app.yaml
        ```

        Repeat for all affected controllers (Deployments, StatefulSets, DaemonSets, Jobs/CronJobs if they are long-running).

        4. If any pod is standalone (kind: Pod) and managed directly (not recommended in AKS), edit in place:

        ```bash theme={null}
        kubectl -n my-namespace edit pod my-standalone-pod
        ```

        Add `livenessProbe` and `readinessProbe` under the container spec as shown above, then save and exit; the pod will be recreated if the spec changes in a controller, or updated in place for a bare pod.

        5. Verification (any machine with kubectl access):

        ```bash theme={null}
        kubectl get pods --all-namespaces -o json | jq -r '
          [ .items[]
          | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
          | .metadata as $m
          | (.spec.containers // [])[]
          | (.livenessProbe != null) as $live
          | (.readinessProbe != null) as $ready
          | select(( $live and $ready ) | not)
          ] | if (length == 0) then "is_compliant=true" else . end'
        ```
      </Accordion>

      <Accordion title="Automation">
        ```bash theme={null}
        #!/usr/bin/env bash
        # Purpose:
        #   Report long-running containers (non-job pods outside AKS system namespaces)
        #   that are missing livenessProbe and/or readinessProbe.
        #
        # NOTE:
        #   Probe configuration is application-specific; this script CANNOT safely
        #   auto-add probes without knowing container ports, paths, or health semantics.
        #   Use this as an automation aid to identify and review/fix manifests.
        #
        # Requirements:
        #   - Run on any machine with kubectl access to the AKS cluster
        #   - Tools: kubectl, jq, awk
        #
        # Usage:
        #   ./check_probes.sh
        set -euo pipefail

        echo "Scanning for containers missing liveness/readiness probes..."

        # Reuse the authoritative audit logic to gather non-compliant containers
        NON_COMPLIANT_JSON="$(kubectl get pods --all-namespaces -o json \
          | jq -c '
            [ .items[]
            | select(.metadata.namespace as $n
                     | ["kube-system","kube-public","kube-node-lease","gatekeeper-system","azure-arc"]  # extend ignore list as needed
                     | index($n)
                     | not)
            | .metadata as $m
            | (.spec.containers // [])[]
            | select((.livenessProbe == null) or (.readinessProbe == null))
            | {
                namespace: $m.namespace,
                pod: $m.name,
                container: .name,
                liveness_defined: (.livenessProbe != null),
                readiness_defined: (.readinessProbe != null),
                owner_kind: ( ($m.ownerReferences // []) | map(select(.controller)) | first | .kind // "Pod"),
                owner_name: ( ($m.ownerReferences // []) | map(select(.controller)) | first | .name // $m.name)
              }
            ]')"

        if [[ "${NON_COMPLIANT_JSON}" == "[]" ]]; then
          echo "All scanned containers already define both livenessProbe and readinessProbe."
          exit 0
        fi

        echo
        echo "The following containers are missing one or both probes:"
        echo

        echo "${NON_COMPLIANT_JSON}" | jq -r '
          .[]
          | "namespace=\(.namespace)\tpod=\(.pod)\tcontainer=\(.container)\towner=\(.owner_kind)/\(.owner_name)\tliveness=\(.liveness_defined)\treadiness=\(.readiness_defined)"
        ' | column -t -s $'\t'

        cat <<'EOF'

        NEXT STEPS (manual, per workload):

        1. Identify the owning object and manifest source:
           - For native Kubernetes objects:
               kubectl -n <namespace> get deployment/<name> -o yaml
               kubectl -n <namespace> get statefulset/<name> -o yaml
               kubectl -n <namespace> get daemonset/<name> -o yaml
           - For Helm-managed workloads:
               helm list -A | grep <release>
               helm get values -n <namespace> <release> > values.yaml

        2. For EACH listed container, edit its manifest/values to add both probes, for example:
           # Example ONLY – adjust path/port/thresholds for your app
           livenessProbe:
             httpGet:
               path: /healthz
               port: 8080
             initialDelaySeconds: 30
             periodSeconds: 10
             timeoutSeconds: 5
             failureThreshold: 3
           readinessProbe:
             httpGet:
               path: /ready
               port: 8080
             initialDelaySeconds: 5
             periodSeconds: 5
             timeoutSeconds: 3
             failureThreshold: 3

        3. Apply the change using the same mechanism used to deploy:
           - kubectl apply -f <manifest>.yaml
           - or: helm upgrade -n <namespace> <release> <chart> -f values.yaml
           - or: update CI/CD/IaC (e.g., GitOps) and let it reconcile.

        4. Verify the fix (idempotent verification):

           kubectl get pods --all-namespaces -o json | jq -r '
             [ .items[]
             | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease","gatekeeper-system","azure-arc"] | index($n) | not)
             | .metadata as $m
             | (.spec.containers // [])[]
             | (.livenessProbe != null) as $live
             | (.readinessProbe != null) as $ready
             | "ns=\($m.namespace) pod=\($m.name) container=\(.name) livenessProbe=\($live) readinessProbe=\($ready)"
               + " is_compliant=\(if ($live and $ready) then "true" else "false" end)"
             ] as $rows
             | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end
           ' | grep -v 'is_compliant=true' || echo "All remaining scanned containers are compliant."
        EOF
        ```
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>