Containers Should Set CPU And Memory Requests

More Info:

Verifies every container sets resources.requests so the scheduler can place the pod correctly and QoS is not BestEffort.

Risk Level

Low

Address

Security

Compliance Standards

• Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

1. List all non-exempt pods and identify non-compliant containers

   • Run on: any machine with kubectl access

   kubectl get pods --all-namespaces -o json | jq -r '
     [ .items[]
     | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
     | .metadata as $m
     | (.spec.nodeName // "") as $node
     | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
     | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
     | (.spec.containers // [])[]
     | ((.resources.requests.cpu != null) and (.resources.requests.memory != null)) as $ok
     | select($ok | not)
     | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
       + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
       + (if $node   == ""   then "" else " node=\($node)" end)
       + (if $labels == ""   then "" else " labels=\($labels)" end)
       + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
       + " container=\(.name) image=\(.image)"
       + " requestsCpu=\(.resources.requests.cpu // "unset") requestsMemory=\(.resources.requests.memory // "unset")"
       + " is_compliant=false"
     ][]'

2. For pods owned by higher-level controllers, patch the controller to add requests

   • Run on: any machine with kubectl access
   • For each non-compliant line with owner=<Kind>/<namespace>/<name>/..., edit that owner, not the pod. Example for a Deployment:

   kubectl -n NAMESPACE patch deployment DEPLOYMENT_NAME --type merge -p '
   {
     "spec": {
       "template": {
         "spec": {
           "containers": [
             {
               "name": "CONTAINER_NAME",
               "resources": {
                 "requests": {
                   "cpu": "100m",
                   "memory": "128Mi"
                 }
               }
             }
           ]
         }
       }
     }
   }'

   • Adjust NAMESPACE, DEPLOYMENT_NAME, CONTAINER_NAME, and choose CPU/memory values appropriate for the workload.

3. For standalone Pods (no owner), export, edit, and re-apply with requests

   • Run on: any machine with kubectl access

   kubectl -n NAMESPACE get pod POD_NAME -o yaml > /tmp/pod-POD_NAME.yaml

   • Edit /tmp/pod-POD_NAME.yaml and under each affected container add, or update, for example:

   resources:
     requests:
       cpu: "100m"
       memory: "128Mi"

   • Delete and recreate the pod from the edited manifest (standalone pods only):

   kubectl -n NAMESPACE delete pod POD_NAME
   kubectl apply -f /tmp/pod-POD_NAME.yaml

4. For controllers managed via GitOps or IaC, update the source manifests

   • Run in your Git/IaC workflow; apply from any machine with kubectl access.
   • In each manifest (e.g., Deployment/StatefulSet/DaemonSet), ensure every container under spec.template.spec.containers defines resources.requests.cpu and resources.requests.memory as in step 3, then apply:

   kubectl apply -f PATH/TO/MANIFEST.yaml

5. Wait for rollouts to complete

   • Run on: any machine with kubectl access

   kubectl -n NAMESPACE rollout status deployment/DEPLOYMENT_NAME
   kubectl -n NAMESPACE rollout status statefulset/STATEFULSET_NAME
   kubectl -n NAMESPACE rollout status daemonset/DAEMONSET_NAME

6. Verification (re-run the audit and confirm no is_compliant=false)

   • Run on: any machine with kubectl access

   kubectl get pods --all-namespaces -o json | jq -r '
     [ .items[]
     | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
     | .metadata as $m
     | (.spec.nodeName // "") as $node
     | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
     | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
     | (.spec.containers // [])[]
     | ((.resources.requests.cpu != null) and (.resources.requests.memory != null)) as $ok
     | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
       + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
       + (if $node   == ""   then "" else " node=\($node)" end)
       + (if $labels == ""   then "" else " labels=\($labels)" end)
       + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
       + " container=\(.name) image=\(.image)"
       + " requestsCpu=\(.resources.requests.cpu // "unset") requestsMemory=\(.resources.requests.memory // "unset")"
       + " is_compliant=\(if $ok then "true" else "false" end)"
     ] as $rows
     | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

   • Confirm there are no lines with is_compliant=false.

Using kubectl

On any machine with kubectl access:

1. Identify non-compliant Pods and their controllers

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | ((.resources.requests.cpu != null) and (.resources.requests.memory != null)) as $ok
  | select($ok | not)
  | "ns=\($m.namespace) pod=\($m.name) container=\(.name)"
    + (if $own == null then "" else " ownerKind=\($own.kind) ownerName=\($own.name)" end)
  ][]'

For each line, note:

• ownerKind/ownerName if present (Deployment, StatefulSet, DaemonSet, Job, CronJob, etc.).
• If no ownerKind, you will edit the Pod template directly where it is defined (usually a standalone Pod manifest).

2. Patch a controller-managed workload (example: Deployment)

Replace NAMESPACE, DEPLOYMENT_NAME, CONTAINER_NAME, and the CPU/memory values with your desired requests. This modifies the Pod template so all new Pods are compliant.

kubectl patch deployment DEPLOYMENT_NAME \
  -n NAMESPACE \
  --type='json' \
  -p='[
    {
      "op": "add",
      "path": "/spec/template/spec/containers/0/resources",
      "value": {
        "requests": {
          "cpu": "100m",
          "memory": "128Mi"
        }
      }
    }
  ]'

If the target container is not index 0, get its index first:

kubectl get deployment DEPLOYMENT_NAME -n NAMESPACE -o json | jq '.spec.template.spec.containers | to_entries[] | "\(.key):\(.value.name)"'

Then use that index instead of 0 in the JSON patch path.

For other controllers, use the corresponding kind:

# StatefulSet
kubectl patch statefulset STATEFULSET_NAME -n NAMESPACE --type='json' -p='[ ... ]'

# DaemonSet
kubectl patch daemonset DAEMONSET_NAME -n NAMESPACE --type="json" -p='[ ... ]'

# Job
kubectl patch job JOB_NAME -n NAMESPACE --type="json" -p='[ ... ]'

# CronJob (note the jobTemplate)
kubectl patch cronjob CRONJOB_NAME -n NAMESPACE --type="json" -p='[
  {
    "op": "add",
    "path": "/spec/jobTemplate/spec/template/spec/containers/0/resources",
    "value": {
      "requests": {
        "cpu": "100m",
        "memory": "128Mi"
      }
    }
  }
]'

3. Patch a standalone Pod (not recommended long term but possible)

For Pods with no ownerKind/ownerName, you must edit the source manifest and re-apply it. If you only have the live object, you can patch it (changes are lost when the Pod is recreated):

kubectl patch pod POD_NAME \
  -n NAMESPACE \
  --type='json' \
  -p='[
    {
      "op": "add",
      "path": "/spec/containers/0/resources",
      "value": {
        "requests": {
          "cpu": "100m",
          "memory": "128Mi"
        }
      }
    }
  ]'

Again, adjust the container index as needed using:

kubectl get pod POD_NAME -n NAMESPACE -o json | jq '.spec.containers | to_entries[] | "\(.key):\(.value.name)"'

4. Prefer declarative manifests for ongoing management

Edit your YAML manifests (Deployment, StatefulSet, etc.) so each container has:

resources:
  requests:
    cpu: "100m"
    memory: "128Mi"

Then apply:

kubectl apply -f path/to/manifest.yaml

5. Verification

Run the original audit command to confirm all containers have CPU and memory requests set:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | ((.resources.requests.cpu != null) and (.resources.requests.memory != null)) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\(.image)"
    + " requestsCpu=\(.resources.requests.cpu // "unset") requestsMemory=\(.resources.requests.memory // "unset")"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

Automation

#!/usr/bin/env bash
# Purpose: Ensure every non-exempt container has CPU and memory requests set,
#          by patching owning workload manifests via kubectl.
# Scope:   Run on any machine with kubectl access and current context set to the target GKE cluster.
# Notes:
#   - This script is idempotent: it only sets requests where they are currently unset.
#   - It skips kube-system, kube-public, kube-node-lease as per the audit.
#   - Default values are intentionally conservative; adjust if needed before running.

set -euo pipefail

# --------- CONFIGURABLE DEFAULTS (EDIT AS NEEDED) ----------
DEFAULT_CPU_REQUEST="50m"
DEFAULT_MEM_REQUEST="64Mi"
# Optional: label to mark already-processed workloads (prevents reprocessing if you later change defaults)
PROCESSED_LABEL_KEY="cbp-c1-9-patched"
PROCESSED_LABEL_VALUE="true"
# -----------------------------------------------------------

require_cmds() {
  for c in kubectl jq; do
    if ! command -v "$c" >/dev/null 2>&1; then
      echo "ERROR: Required command '$c' not found in PATH" >&2
      exit 1
    fi
  done
}

# Get all non-exempt pods and containers that are non-compliant
find_non_compliant() {
  kubectl get pods --all-namespaces -o json | jq -r '
    .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | [ ($m.ownerReferences // [])[] | select(.controller) ] | first as $own
    | .spec.containers[]
    | ((.resources.requests.cpu != null) and (.resources.requests.memory != null)) as $ok
    | select($ok | not)
    | {
        podNamespace: $m.namespace,
        podName: $m.name,
        containerName: .name,
        ownerKind: ($own.kind // "Pod"),
        ownerName: ($own.name // $m.name),
        ownerUid:  ($own.uid  // $m.uid)
      }
    | @base64
  '
}

decode_b64json() {
  echo "$1" | base64 --decode
}

# Patch a specific workload (Deployment/StatefulSet/DaemonSet/Job/CronJob/Pod) in-place
patch_workload() {
  local ns="$1"
  local kind="$2"
  local name="$3"
  local container="$4"

  echo "Patching $kind $ns/$name container '$container'..."

  # Fetch current spec
  local tmpfile
  tmpfile="$(mktemp)"
  kubectl get "$kind" "$name" -n "$ns" -o json > "$tmpfile"

  # Skip if already labeled as processed (idempotence helper)
  if jq -e --arg k "$PROCESSED_LABEL_KEY" --arg v "$PROCESSED_LABEL_VALUE" \
      '.metadata.labels[$k] == $v' "$tmpfile" >/dev/null 2>&1; then
    echo "  Skipping: workload already labeled $PROCESSED_LABEL_KEY=$PROCESSED_LABEL_VALUE"
    rm -f "$tmpfile"
    return
  fi

  # Add requests only where missing
  local patched
  patched="$(jq \
    --arg cname "$container" \
    --arg cpu "$DEFAULT_CPU_REQUEST" \
    --arg mem "$DEFAULT_MEM_REQUEST" \
    --arg k "$PROCESSED_LABEL_KEY" \
    --arg v "$PROCESSED_LABEL_VALUE" '
    # Ensure metadata.labels exists
    (.metadata.labels //= {}) |
    .metadata.labels[$k] = $v |
    # Handle top-level pod template (covers Deployments, StatefulSets, DaemonSets, Jobs)
    (if .spec.template.spec.containers then
      .spec.template.spec.containers |=
      (map(
         if .name == $cname then
           .resources.requests.cpu   //= $cpu |
           .resources.requests.memory//= $mem
         else .
         end
      ))
     else . end) |
    # Handle CronJobs (spec.jobTemplate)
    (if .kind == "CronJob" and .spec.jobTemplate.spec.template.spec.containers then
      .spec.jobTemplate.spec.template.spec.containers |=
      (map(
         if .name == $cname then
           .resources.requests.cpu   //= $cpu |
           .resources.requests.memory//= $mem
         else .
         end
      ))
     else . end) |
    # Handle plain Pods (ownerKind Pod or direct pod)
    (if .kind == "Pod" and .spec.containers then
      .spec.containers |=
      (map(
         if .name == $cname then
           .resources.requests.cpu   //= $cpu |
           .resources.requests.memory//= $mem
         else .
         end
      ))
     else . end)
  ' "$tmpfile")"

  # Apply patched manifest
  printf '%s\n' "$patched" | kubectl apply -n "$ns" -f -

  rm -f "$tmpfile"
}

main() {
  require_cmds

  echo "Discovering non-compliant containers..."
  mapfile -t items < <(find_non_compliant)

  if [ "${#items[@]}" -eq 0 ]; then
    echo "No non-compliant containers found. Cluster is already compliant."
  else
    for item in "${items[@]}"; do
      j="$(decode_b64json "$item")"
      ns="$(echo "$j" | jq -r '.podNamespace')"
      ownerKind="$(echo "$j" | jq -r '.ownerKind')"
      ownerName="$(echo "$j" | jq -r '.ownerName')"
      cname="$(echo "$j" | jq -r '.containerName')"

      # Normalize owner kind to API resource
      case "$ownerKind" in
        Deployment|StatefulSet|DaemonSet|Job|CronJob|Pod)
          patch_workload "$ns" "$ownerKind" "$ownerName" "$cname"
          ;;
        ReplicaSet)
          # Try to escalate to owning Deployment when possible
          dname="$(kubectl get rs "$ownerName" -n "$ns" -o jsonpath='{.metadata.ownerReferences[?(@.controller==true)].name}' 2>/dev/null || true)"
          if [ -n "$dname" ]; then
            patch_workload "$ns" "Deployment" "$dname" "$cname"
          else
            echo "Owner ReplicaSet $ns/$ownerName has no Deployment owner; patching ReplicaSet directly."
            patch_workload "$ns" "ReplicaSet" "$ownerName" "$cname"
          fi
          ;;
        *)
          echo "Unknown/unsupported owner kind '$ownerKind' for $ns/$ownerName; attempting direct patch."
          patch_workload "$ns" "$ownerKind" "$ownerName" "$cname" || true
          ;;
      esac
    done
  fi

  echo
  echo "Verifying compliance..."
  kubectl get pods --all-namespaces -o json | jq -r '
    [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.containers // [])[]
    | ((.resources.requests.cpu != null) and (.resources.requests.memory != null)) as $ok
    | select($ok | not)
    ] as $rows
    | if ($rows | length) == 0 then
        "is_compliant=true"
      else
        "is_compliant=false (there are still containers without requests set)"
      end
  '
}

main "$@"