Skip to main content

Containers Should Set CPU And Memory Limits

More Info:​

Verifies every container sets resources.limits.cpu and resources.limits.memory so a single workload cannot exhaust a node.

Risk Level​

Medium

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. Identify noncompliant pods and their controllers (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | (.spec.containers // [])[]
      | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
      | select($ok | not)
      | "ns=\($m.namespace) pod=\($m.name) container=\(.name)"
        + (if $own == null then "" else " ownerKind=\($own.kind) ownerName=\($own.name)" end)
    ][]'

  2. For a pod managed by a controller (e.g., Deployment), edit the controller manifest to add limits (run on any machine with kubectl access):

    kubectl -n <namespace> edit deployment <deployment-name>

    In each container under spec.template.spec.containers, ensure:

    resources:
      limits:
        cpu: "500m"
        memory: "256Mi"

    Adjust values to match your capacity and SLOs, then save and exit to let Kubernetes roll out updated pods.

  3. For other controllers (StatefulSet, DaemonSet, Job, CronJob), edit similarly (run on any machine with kubectl access):

    kubectl -n <namespace> edit statefulset <name>
    kubectl -n <namespace> edit daemonset <name>
    kubectl -n <namespace> edit job <name>
    kubectl -n <namespace> edit cronjob <name>

    Add or update resources.limits.cpu and resources.limits.memory for every container in spec.template.spec.containers (or spec.jobTemplate.spec.template.spec.containers for CronJob).

  4. For standalone pods without controllers that must be kept, export, modify, and recreate (run on any machine with kubectl access):

    kubectl -n <namespace> get pod <pod-name> -o yaml > /tmp/pod-with-limits.yaml

    Edit /tmp/pod-with-limits.yaml:

    • Remove fields status, metadata.resourceVersion, metadata.uid, metadata.creationTimestamp, and metadata.ownerReferences.
    • Under every container in spec.containers, add: 
      resources:
        limits:
          cpu: "500m"
          memory: "256Mi"

    Then recreate:

    kubectl -n <namespace> delete pod <pod-name>
    kubectl -n <namespace> apply -f /tmp/pod-with-limits.yaml

  5. Optionally enforce future compliance with a LimitRange in each namespace (run on any machine with kubectl access):

    cat << 'EOF' | kubectl apply -f -
    apiVersion: v1
    kind: LimitRange
    metadata:
      name: default-container-limits
      namespace: <namespace>
    spec:
      limits:
      - type: Container
        default:
          cpu: "500m"
          memory: "256Mi"
        defaultRequest:
          cpu: "250m"
          memory: "128Mi"
    EOF

  6. Verify all non-system pods now have CPU and memory limits (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.containers // [])[]
      | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
      | select($ok | not)
    ] as $rows
    | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Using kubectl

On any machine with kubectl access:

  1. Identify non‑compliant pods and their owning controllers
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | select($ok | not)
  | "\($own.kind)//\($m.namespace)//\($own.name)"
  ] | unique[]' | sed 's#//# #g'

This outputs lines like: Deployment default my-app

  1. Patch a controller to add limits (example for a Deployment)

Export the manifest, edit, and re-apply:

kubectl get deployment my-app -n default -o yaml > /tmp/my-app-deploy.yaml

Edit /tmp/my-app-deploy.yaml and, for each container under spec.template.spec.containers, ensure a resources block like:

        resources:
          limits:
            cpu: "500m"
            memory: "256Mi"
          requests:
            cpu: "250m"
            memory: "128Mi"

Apply the updated manifest:

kubectl apply -f /tmp/my-app-deploy.yaml

Repeat this export/edit/apply pattern for each non‑compliant controller kind (e.g., StatefulSet, DaemonSet, Job, CronJob, ReplicaSet if managed directly), always updating the containers in spec.template.spec.containers.

  1. For bare Pods (no ownerReferences)

Export, edit, and re‑create:

kubectl get pod my-pod -n default -o yaml > /tmp/my-pod.yaml

In /tmp/my-pod.yaml:

  • Remove the entire status: section.
  • Under spec.containers[], add resources.limits.cpu and resources.limits.memory as above.

Delete and recreate:

kubectl delete pod my-pod -n default
kubectl apply -f /tmp/my-pod.yaml
  1. Verification

On any machine with kubectl access:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | select($ok | not)
  ] | if (length == 0) then "is_compliant=true" else . end'
Automation
#!/usr/bin/env bash
#
# Enforce CPU and memory limits on all non-excluded Pods by patching their controllers.
# - Skips kube-system, kube-public, kube-node-lease
# - For each container with missing limits, sets:
#     limits.cpu: "500m"
#     limits.memory: "512Mi"
# - Patches Deployments, StatefulSets, DaemonSets, ReplicaSets, Jobs, CronJobs.
# - Standalone Pods are reported but NOT modified (edit their manifests/IaC).
#
# Run on: any machine with kubectl and jq access to the cluster.
# Idempotent: safe to re-run; existing limits are preserved.

set -euo pipefail

DEFAULT_CPU_LIMIT="500m"
DEFAULT_MEM_LIMIT="512Mi"

# Ensure kubectl and jq are available
command -v kubectl >/dev/null 2>&1 || { echo "kubectl not found in PATH" >&2; exit 1; }
command -v jq       >/dev/null 2>&1 || { echo "jq not found in PATH" >&2; exit 1; }

echo "Scanning pods for containers missing CPU or memory limits..."

# Get all relevant pods as JSON once
PODS_JSON="$(kubectl get pods --all-namespaces -o json)"

# Build a list of affected containers with their owner (if any)
# Output format (tab-separated):
# ns  podName  ownerKind  ownerName  ownerUID  containerName  hasCpu  hasMem
AFFECTED_CONTAINERS="$(
  echo "${PODS_JSON}" | jq -r '
    .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | (.spec.containers // [])[]
    | ((.resources.limits.cpu != null)   ) as $hasCpu
    | ((.resources.limits.memory != null)) as $hasMem
    | select( ( $hasCpu and $hasMem ) | not )
    | [
        $m.namespace,
        $m.name,
        (if $own == null then "" else $own.kind end),
        (if $own == null then "" else $own.name end),
        (if $own == null then "" else $own.uid end),
        .name,
        (if $hasCpu then "true" else "false" end),
        (if $hasMem then "true" else "false" end)
      ]
      | @tsv
  '
)"

if [[ -z "${AFFECTED_CONTAINERS}" ]]; then
  echo "All scanned containers already have CPU and memory limits set."
  echo "Re-running verification command for confirmation..."
  kubectl get pods --all-namespaces -o json | jq -r '
    [ .items[]
    | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | .metadata as $m
    | (.spec.nodeName // "") as $node
    | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
    | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
    | (.spec.containers // [])[]
    | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
    | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
      + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
      + (if $node   == ""   then "" else " node=\($node)" end)
      + (if $labels == ""   then "" else " labels=\($labels)" end)
      + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
      + " container=\(.name) image=\(.image)"
      + " limitsCpu=\(.resources.limits.cpu // "unset") limitsMemory=\(.resources.limits.memory // "unset")"
      + " is_compliant=\(if $ok then "true" else "false" end)"
    ] as $rows
    | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
  exit 0
fi

echo "Found containers missing limits. Grouping by owning controller..."

# Build a unique list of owner controllers to patch
# Output: ns<TAB>ownerKind<TAB>ownerName
OWNERS_TO_PATCH="$(
  printf "%s\n" "${AFFECTED_CONTAINERS}" | awk -F'\t' '
    $3 != "" && $4 != "" {
      key = $1 "\t" $3 "\t" $4
      if (!(key in seen)) {
        seen[key]=1
        print key
      }
    }
  '
)"

# Track standalone pods (no owner) so users can fix their manifests manually
STANDALONE_PODS_REPORTED=()

if [[ -z "${OWNERS_TO_PATCH}" ]]; then
  echo "No controller-backed workloads to patch; only standalone Pods are non-compliant."
else
  echo "Patching the following controllers with default CPU=${DEFAULT_CPU_LIMIT}, Memory=${DEFAULT_MEM_LIMIT}:"
  printf "%s\n" "${OWNERS_TO_PATCH}" | sed $'s/\t/ /g'

  # For each controller, construct and apply a strategic merge patch
  while IFS=$'\t' read -r NS OWNER_KIND OWNER_NAME; do
    [[ -z "${NS}" || -z "${OWNER_KIND}" || -z "${OWNER_NAME}" ]] && continue

    # Map kind -> kubectl resource type
    case "${OWNER_KIND}" in
      Deployment)   RES_TYPE="deployment" ;;
      StatefulSet)  RES_TYPE="statefulset" ;;
      DaemonSet)    RES_TYPE="daemonset" ;;
      ReplicaSet)   RES_TYPE="replicaset" ;;
      Job)          RES_TYPE="job" ;;
      CronJob)      RES_TYPE="cronjob" ;;
      *)
        echo "Skipping unsupported controller kind ${OWNER_KIND} (${NS}/${OWNER_NAME})" >&2
        continue
        ;;
    esac

    echo "Processing ${OWNER_KIND} ${NS}/${OWNER_NAME}..."

    # Get the current pod template spec for this controller
    CTRL_JSON="$(kubectl -n "${NS}" get "${RES_TYPE}" "${OWNER_NAME}" -o json)"

    # Build a patched pod template with limits added where missing and existing limits preserved
    PATCH_TEMPLATE="$(
      echo "${CTRL_JSON}" | jq --arg cpu "${DEFAULT_CPU_LIMIT}" --arg mem "${DEFAULT_MEM_LIMIT}" '
        .spec.template.spec
        |= (
          .containers |= map(
            .resources.limits.cpu    |= (if . == null then $cpu else . end)
            | .resources.limits.memory |= (if . == null then $mem else . end)
          )
        )
      '
    )"

    # Extract just the spec.template part for a strategic merge patch
    PATCH="$(
      echo "${PATCH_TEMPLATE}" | jq '{ spec: { template: . } }'
    )"

    echo "${PATCH}" | kubectl -n "${NS}" patch "${RES_TYPE}" "${OWNER_NAME}" --type merge -p "$(cat)"
    echo "Patched ${OWNER_KIND} ${NS}/${OWNER_NAME}"
  done <<< "${OWNERS_TO_PATCH}"
fi

# Report standalone pods (no owning controller) requiring manual manifest/IaC updates
echo
echo "Checking for standalone Pods (no owning controller) with missing limits..."
while IFS=$'\t' read -r NS POD OWNER_KIND OWNER_NAME OWNER_UID CONTAINER HASCPU HASMEM; do
  if [[ -z "${OWNER_KIND}" || -z "${OWNER_NAME}" ]]; then
    ID="${NS}/${POD}"
    if [[ " ${STANDALONE_PODS_REPORTED[*]-} " != *" ${ID} "* ]]; then
      STANDALONE_PODS_REPORTED+=("${ID}")
      echo "Standalone Pod requires manual fix: ${NS}/${POD} (container ${CONTAINER})"
    fi
  fi
done <<< "${AFFECTED_CONTAINERS}"

if ((${#STANDALONE_PODS_REPORTED[@]} > 0)); then
  echo
  echo "For each standalone Pod above, edit its manifest/IaC to set resources.limits.cpu and resources.limits.memory,"
  echo "then recreate the Pod from the updated definition."
fi

echo
echo "Re-running verification command to confirm compliance..."
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\(.image)"
    + " limitsCpu=\(.resources.limits.cpu // "unset") limitsMemory=\(.resources.limits.memory // "unset")"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'