Skip to main content

Containers Should Set CPU And Memory Requests

More Info:​

Verifies every container sets resources.requests so the scheduler can place the pod correctly and QoS is not BestEffort.

Risk Level​

Low

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. List all non-exempt pods and identify offenders (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.containers // [])[]
      | ((.resources.requests.cpu != null) and (.resources.requests.memory != null)) as $ok
      | select($ok | not)
      | "ns=\($m.namespace) pod=\($m.name) container=\(.name)"
    ] | unique[]'

  2. For a pod managed by a higher-level controller (Deployment/ReplicaSet/StatefulSet/DaemonSet/Job/CronJob), get and edit the owning object’s manifest (example for a Deployment; run on any machine with kubectl access):

    kubectl -n NAMESPACE get deployment DEPLOYMENT_NAME -o yaml > /tmp/deploy.yaml
    vi /tmp/deploy.yaml

    Under each container in spec.template.spec.containers, add or update:

    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"

    (Choose values appropriate for the application.)

  3. Apply the updated controller manifest (run on any machine with kubectl access):

    kubectl apply -f /tmp/deploy.yaml

    Repeat steps 2–3 for other controller types (kubectl get statefulset, kubectl get daemonset, kubectl get job, kubectl get cronjob) that own non-compliant pods.

  4. For stand-alone Pods (no controller ownerReference), edit the Pod spec directly (run on any machine with kubectl access):

    kubectl -n NAMESPACE edit pod POD_NAME

    In each spec.containers[], ensure:

    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"

    Note: editing certain fields of a running Pod may be restricted; if so, recreate the Pod with a corrected manifest:

    kubectl -n NAMESPACE get pod POD_NAME -o yaml > /tmp/pod.yaml
    # edit /tmp/pod.yaml: add resources.requests and remove fields under metadata.status, status, and cluster-assigned fields like metadata.uid, resourceVersion
    vi /tmp/pod.yaml
    kubectl -n NAMESPACE delete pod POD_NAME
    kubectl -n NAMESPACE apply -f /tmp/pod.yaml

  5. For workloads managed by AKS add-ons or third-party operators, adjust their Helm chart values or operator configuration so that the generated Pod templates include resources.requests.cpu and resources.requests.memory for all containers, then redeploy using the add-on’s/Helm’s normal process.

  6. Verify compliance for all pods (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.containers // [])[]
      | ((.resources.requests.cpu != null) and (.resources.requests.memory != null)) as $ok
      | select($ok | not)
    ] as $rows
    | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

    The output should be is_compliant=true.

Using kubectl

On any machine with kubectl access:

  1. Identify non-compliant pods and their controllers
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | ((.resources.requests.cpu != null) and (.resources.requests.memory != null)) as $ok
  | select($ok | not)
  | "ns=\($m.namespace) pod=\($m.name) ownerKind=\($own.kind // "Pod") ownerName=\($own.name // "")"
  ][]'

Focus on the owning workload (Deployment, StatefulSet, Job, etc.), not the pod itself.

  1. Export the owning workload manifest (example for a Deployment)
kubectl -n NAMESPACE get deploy DEPLOYMENT_NAME -o yaml > /tmp/deploy-with-requests.yaml
  1. Edit containers to add resources.requests (CPU and memory) in the manifest

In /tmp/deploy-with-requests.yaml, under each container in spec.template.spec.containers, ensure a block like:

        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"

Adjust values to match application requirements. Do this for every container that currently lacks requests or has only one of CPU/memory.

  1. Apply the updated manifest
kubectl apply -f /tmp/deploy-with-requests.yaml

Repeat steps 2–4 for each affected controller type (e.g., kubectl get statefulset, kubectl get job, etc.).

  1. For standalone Pods managed directly (no ownerReferences)

Export, edit, and re-create (pods are immutable):

kubectl -n NAMESPACE get pod POD_NAME -o yaml > /tmp/pod-with-requests.yaml

Edit each container to add:

    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"

Remove runtime-only fields before re-creating:

yq 'del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .metadata.managedFields, .status)' \
  /tmp/pod-with-requests.yaml > /tmp/pod-with-requests-clean.yaml

kubectl -n NAMESPACE delete pod POD_NAME
kubectl apply -f /tmp/pod-with-requests-clean.yaml
  1. Verification

Run the original audit and confirm all listed rows have is_compliant=true (or that it prints just is_compliant=true):

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | ((.resources.requests.cpu != null) and (.resources.requests.memory != null)) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\(.image)"
    + " requestsCpu=\(.resources.requests.cpu // "unset") requestsMemory=\(.resources.requests.memory // "unset")"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Automation
#!/usr/bin/env bash
#
# Fix CBP C1.9: Ensure every container sets CPU and memory requests
# Target: any machine with kubectl access to the AKS cluster
#
# This script:
#   - Scans all non-system namespaces for Pods whose containers lack CPU and/or memory requests
#   - For workload-backed Pods (Deployment/ReplicaSet/StatefulSet/DaemonSet/Job/CronJob),
#     patches the owning workload to add default requests where missing
#   - Skips bare Pods (no controller ownerReferences) — these must be fixed manually
#   - Re-runs the audit to verify
#
# NOTE:
#   - You MUST review and customize the default request values below for your cluster.
#   - Changing workload specs will create new Pods; expect rolling restarts.
#

set -euo pipefail

#-----------------------------
# Configurable defaults
#-----------------------------
# Default requests to apply where missing. Adjust per your standards.
DEFAULT_CPU_REQUEST="100m"
DEFAULT_MEM_REQUEST="128Mi"

# Namespaces to exclude from modification (system namespaces)
EXCLUDED_NAMESPACES_REGEX='^(kube-system|kube-public|kube-node-lease)$'

#-----------------------------
# Preconditions
#-----------------------------
command -v kubectl >/dev/null 2>&1 || {
  echo "ERROR: kubectl not found in PATH" >&2
  exit 1
}

command -v jq >/dev/null 2>&1 || {
  echo "ERROR: jq not found in PATH" >&2
  exit 1
}

# Ensure we can talk to the cluster
if ! kubectl version --request-timeout=10s >/dev/null 2>&1; then
  echo "ERROR: kubectl cannot reach the cluster. Check context and network." >&2
  exit 1
fi

#-----------------------------
# Helper functions
#-----------------------------

# Patch a workload's pod template to ensure all containers have requests.
# Arguments:
#   1: namespace
#   2: workload kind (Deployment/StatefulSet/DaemonSet/Job/CronJob/ReplicaSet)
#   3: workload name
patch_workload() {
  local ns="$1"
  local kind="$2"
  local name="$3"

  echo "Processing ${kind}/${ns}/${name} ..."

  # Get current spec template
  local tmpl
  if ! tmpl="$(kubectl -n "${ns}" get "${kind}" "${name}" -o json)"; then
    echo "  WARN: Could not fetch ${kind}/${ns}/${name}, skipping" >&2
    return
  fi

  # Build a patched pod template with ensured requests for all containers
  local patched
  patched="$(echo "${tmpl}" | jq --arg cpu "${DEFAULT_CPU_REQUEST}" --arg mem "${DEFAULT_MEM_REQUEST}" '
    .spec.template.spec as $spec
    | .spec.template.spec.containers |= (map(
        .resources.requests |= (
          if . == null then
            {cpu: $cpu, memory: $mem}
          else
            .cpu    |= (if . == null then $cpu else . end)
            | .memory |= (if . == null then $mem else . end)
          end
        )
      ))
  ')"

  # Apply patch via strategic merge (idempotent)
  # Extract only the template piece to avoid unintended changes
  local patch
  patch="$(echo "${patched}" | jq '{spec: {template: .spec.template}}')"

  echo "${patch}" | kubectl -n "${ns}" patch "${kind}" "${name}" --type merge -p "$(cat)" >/dev/null

  echo "  Patched ${kind}/${ns}/${name}"
}

# Determine patchable owner kind (top-level) from ownerReferences
# Takes JSON of ownerReferences[]; prints KIND NAMESPACE NAME on success, nothing otherwise
get_owner_ref() {
  local pod_json="$1"

  # Owner kind we will patch (only controller owners)
  echo "${pod_json}" | jq -r '
    .metadata as $m
    | ([($m.ownerReferences // [])[] | select(.controller == true)] | first) as $own
    | if $own == null then empty else
        "\($own.kind) \($m.namespace) \($own.name)"
      end
  '
}

#-----------------------------
# MAIN
#-----------------------------

echo "Scanning for non-compliant containers (missing CPU and/or memory requests)..."

# Get all Pods in non-system namespaces
pods_json="$(kubectl get pods --all-namespaces -o json)"

# Collect non-compliant Pods
non_compliant_pods="$(echo "${pods_json}" | jq -c '
  .items[]
  | select(.metadata.namespace | test("'"${EXCLUDED_NAMESPACES_REGEX}"'") | not)
  | . as $pod
  | (.spec.containers // [])[] as $c
  | ((.resources.requests.cpu != null) and (.resources.requests.memory != null)) as $ok
  | select($ok | not)
  | $pod
' | jq -s '.')"

if [ "$(echo "${non_compliant_pods}" | jq 'length')" -eq 0 ]; then
  echo "No non-compliant pods found; nothing to do."
else
  echo "Found non-compliant pods; identifying owning workloads..."

  # Map of owner (kind/ns/name) to any (dummy) value, to deduplicate
  # We'll feed this through jq to extract unique owners
  owners="$(echo "${non_compliant_pods}" | jq -c '
    .[]
    | . as $p
    | $p.metadata.namespace as $ns
    | ($p.metadata.ownerReferences // []) as $owners
    | [ $owners[] | select(.controller == true) ] | first as $own
    | if $own == null then
        empty
      else
        {
          kind: $own.kind,
          namespace: $ns,
          name: $own.name
        }
      end
  ' | jq -s 'unique_by(.kind, .namespace, .name)')"

  # Log bare Pods (no controller)
  bare_pods="$(echo "${non_compliant_pods}" | jq -r '
    .[]
    | .metadata as $m
    | [($m.ownerReferences // [])[] | select(.controller == true)] | first as $own
    | select($own == null)
    | "Bare Pod: ns=\($m.namespace) name=\($m.name)"
  ')"

  if [ -n "${bare_pods}" ]; then
    echo "NOTICE: The following non-compliant Pods are not controlled by a workload and must be fixed manually:"
    echo "${bare_pods}"
    echo
  fi

  # Patch each unique owner
  echo "${owners}" | jq -c '.[]' | while read -r o; do
    kind="$(echo "${o}" | jq -r '.kind')"
    ns="$(echo "${o}" | jq -r '.namespace')"
    name="$(echo "${o}" | jq -r '.name')"

    case "${kind}" in
      Deployment|StatefulSet|DaemonSet|Job|ReplicaSet)
        patch_workload "${ns}" "${kind}" "${name}"
        ;;
      CronJob)
        # For CronJob, the Pod template is at spec.jobTemplate.spec.template
        echo "Processing CronJob/${ns}/${name} ..."
        cj_json="$(kubectl -n "${ns}" get CronJob "${name}" -o json)" || {
          echo "  WARN: Could not fetch CronJob/${ns}/${name}, skipping" >&2
          continue
        }

        patched_cj="$(echo "${cj_json}" | jq --arg cpu "${DEFAULT_CPU_REQUEST}" --arg mem "${DEFAULT_MEM_REQUEST}" '
          .spec.jobTemplate.spec.template.spec.containers |= (map(
            .resources.requests |= (
              if . == null then
                {cpu: $cpu, memory: $mem}
              else
                .cpu    |= (if . == null then $cpu else . end)
                | .memory |= (if . == null then $mem else . end)
              end
            )
          ))
        ')"

        patch="$(echo "${patched_cj}" | jq '{spec: {jobTemplate: {spec: {template: .spec.jobTemplate.spec.template}}}}')"
        echo "${patch}" | kubectl -n "${ns}" patch CronJob "${name}" --type merge -p "$(cat)" >/dev/null
        echo "  Patched CronJob/${ns}/${name}"
        ;;
      *)
        echo "WARN: Owner kind ${kind}/${ns}/${name} is not handled automatically; fix its manifest manually." >&2
        ;;
    esac
  done
fi

echo
echo "Waiting 10 seconds for workloads to roll out new Pods..."
sleep 10

echo
echo "Re-running compliance audit..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | ((.resources.requests.cpu != null) and (.resources.requests.memory != null)) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\(.image)"
    + " requestsCpu=\(.resources.requests.cpu // "unset") requestsMemory=\(.resources.requests.memory // "unset")"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | map(select(. | test("is_compliant=false"))) | length) == 0
    then "is_compliant=true"
    else $rows[]
    end
'