Skip to main content

Containers Should Set CPU And Memory Limits

More Info:​

Verifies every container sets resources.limits.cpu and resources.limits.memory so a single workload cannot exhaust a node.

Risk Level​

Medium

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. On any machine with kubectl access, list all non-system pods and identify containers missing limits:

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.containers // [])[]
      | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
      | select($ok | not)
      | "ns=\($m.namespace) pod=\($m.name) container=\(.name) image=\(.image)"
      ][]'

  2. For a pod managed by a higher-level controller (e.g., Deployment), find the owning resource and export its manifest:

    # Show owner for a specific pod
    kubectl get pod POD_NAME -n NAMESPACE -o jsonpath='{.metadata.ownerReferences}' | jq

    # Example: export the owning Deployment
    kubectl get deploy DEPLOYMENT_NAME -n NAMESPACE -o yaml > /tmp/deployment-NAMESPACE-DEPLOYMENT_NAME.yaml

  3. Edit the manifest and add resources.limits.cpu and resources.limits.memory for every container in spec.template.spec.containers (replace values with appropriate limits):

    vi /tmp/deployment-NAMESPACE-DEPLOYMENT_NAME.yaml

    Under each container:

    resources:
      limits:
        cpu: "500m"
        memory: "512Mi"
    # keep existing requests if present; otherwise you may add them if desired

  4. Apply the updated manifest so Kubernetes recreates pods with limits:

    kubectl apply -f /tmp/deployment-NAMESPACE-DEPLOYMENT_NAME.yaml

  5. For standalone Pods (no controller owner), export, edit, and recreate them with limits (this will delete and recreate the pod):

    kubectl get pod POD_NAME -n NAMESPACE -o yaml > /tmp/pod-NAMESPACE-POD_NAME.yaml

    # Remove fields that prevent re-creation
    sed -i '/^  resourceVersion:/d;/^  uid:/d;/^  creationTimestamp:/d;/^  managedFields:/d' \
      /tmp/pod-NAMESPACE-POD_NAME.yaml

    vi /tmp/pod-NAMESPACE-POD_NAME.yaml
    # Add resources.limits.cpu and resources.limits.memory to each container as in step 3

    kubectl delete pod POD_NAME -n NAMESPACE
    kubectl apply -f /tmp/pod-NAMESPACE-POD_NAME.yaml

  6. Verify all non-system containers now have both CPU and memory limits using the benchmark audit command on any machine with kubectl access:

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.nodeName // "") as $node
      | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | (.spec.containers // [])[]
      | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
      | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
        + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
        + (if $node   == ""   then "" else " node=\($node)" end)
        + (if $labels == ""   then "" else " labels=\($labels)" end)
        + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
        + " container=\(.name) image=\(.image)"
        + " limitsCpu=\(.resources.limits.cpu // "unset") limitsMemory=\(.resources.limits.memory // "unset")"
        + " is_compliant=\(if $ok then "true" else "false" end)"
      ] as $rows
      | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Using kubectl

On any machine with kubectl access:

  1. Identify non-compliant pods and their controllers
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | select($ok | not)
  | "ns=\($m.namespace) pod=\($m.name) container=\(.name)"
    + (if $own == null then "" else " ownerKind=\($own.kind) ownerName=\($own.name)" end)
  ][]'

Focus on the owning controllers (e.g. Deployment, StatefulSet, DaemonSet, Job, CronJob) rather than editing bare Pods where possible.

  1. Patch a Deployment to add limits (example)

Replace namespace, name, and limit values as appropriate.

kubectl -n default patch deployment my-deployment --type merge -p '
spec:
  template:
    spec:
      containers:
      - name: my-container
        resources:
          limits:
            cpu: "500m"
            memory: "256Mi"
'
  1. Patch a StatefulSet (example)
kubectl -n default patch statefulset my-statefulset --type merge -p '
spec:
  template:
    spec:
      containers:
      - name: my-container
        resources:
          limits:
            cpu: "1"
            memory: "1Gi"
'
  1. Patch a DaemonSet (example)
kubectl -n default patch daemonset my-daemonset --type merge -p '
spec:
  template:
    spec:
      containers:
      - name: my-container
        resources:
          limits:
            cpu: "200m"
            memory: "128Mi"
'
  1. Patch a Job or CronJob (examples)

Job:

kubectl -n default patch job my-job --type merge -p '
spec:
  template:
    spec:
      containers:
      - name: my-container
        resources:
          limits:
            cpu: "250m"
            memory: "256Mi"
'

CronJob:

kubectl -n default patch cronjob my-cronjob --type merge -p '
spec:
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: my-container
            resources:
              limits:
                cpu: "250m"
                memory: "256Mi"
'
  1. For workloads managed by manifests (GitOps/IaC)

Edit the manifest used to create the controller, for each container:

spec:
  template:
    spec:
      containers:
      - name: my-container
        image: gcr.io/project/image:tag
        resources:
          limits:
            cpu: "500m"
            memory: "256Mi"

Apply:

kubectl apply -f path/to/manifest.yaml
  1. Verification

Run the audit command again and confirm either full compliance or only excluded namespaces appear:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\(.image)"
    + " limitsCpu=\(.resources.limits.cpu // "unset") limitsMemory=\(.resources.limits.memory // "unset")"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Automation
#!/usr/bin/env bash
#
# Enforce CPU and memory limits on all non-exempt containers in the cluster.
# Scope: any machine with kubectl access to the GKE cluster.
#
# Behavior:
# - Skips kube-system, kube-public, kube-node-lease.
# - Skips pods/owners that already have both limits set.
# - Patches pod templates on owning workload APIs (Deployment/StatefulSet/DaemonSet/Job/CronJob/ReplicaSet/ReplicationController).
# - For naked Pods (no controller), warns and prints a kubectl patch template.
#
# WARNING: This script chooses default limits if none exist:
#   CPU limit:   500m
#   Memory limit: 512Mi
# Adjust DEFAULT_CPU_LIMIT and DEFAULT_MEM_LIMIT to match your policy
# BEFORE running.
set -euo pipefail

DEFAULT_CPU_LIMIT="500m"
DEFAULT_MEM_LIMIT="512Mi"

# Namespaces to exclude from enforcement
EXCLUDED_NAMESPACES_REGEX='^(kube-system|kube-public|kube-node-lease)$'

# Ensure prerequisites
command -v kubectl >/dev/null 2>&1 || {
  echo "kubectl not found in PATH" >&2
  exit 1
}
command -v jq >/dev/null 2>&1 || {
  echo "jq not found in PATH" >&2
  exit 1
}

echo "Scanning cluster for containers without CPU and/or memory limits..."

# Get all pods (excluding system namespaces) as JSON
PODS_JSON="$(kubectl get pods --all-namespaces -o json)"

# Build list of non-compliant containers with owner reference info
NON_COMPLIANT_JSON="$(echo "${PODS_JSON}" | jq -c '
  .items[]
  | select(.metadata.namespace | test("'"${EXCLUDED_NAMESPACES_REGEX}"'") | not)
  | . as $pod
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | select($ok | not)
  | {
      podNamespace: $pod.metadata.namespace,
      podName: $pod.metadata.name,
      containerName: .name,
      hasCpuLimit: (.resources.limits.cpu != null),
      hasMemLimit: (.resources.limits.memory != null),
      owner: (
        [ ($pod.metadata.ownerReferences // [])[] | select(.controller) ] | first // null
      )
    }
')"

if [[ -z "${NON_COMPLIANT_JSON}" ]]; then
  echo "No non-compliant containers found (excluding system namespaces)."
else
  echo "Found non-compliant containers. Updating owners where possible..."
fi

# Helper: patch a workload template to ensure all its containers have limits
patch_workload() {
  local kind namespace name
  kind="$1"      # e.g. Deployment
  namespace="$2" # e.g. default
  name="$3"      # resource name

  echo "Processing ${kind}/${namespace}/${name}"

  # Get the existing object
  local obj jsonpath
  obj="$(kubectl -n "${namespace}" get "${kind,,}.${kind}.apps" "${name}" -o json 2>/dev/null || \
        kubectl -n "${namespace}" get "${kind}" "${name}" -o json)"

  # Determine where the pod template spec lives
  local template_path
  case "${kind}" in
    Deployment|StatefulSet|DaemonSet|ReplicaSet)
      template_path=".spec.template.spec"
      ;;
    Job)
      template_path=".spec.template.spec"
      ;;
    CronJob)
      # v1 CronJob: spec.jobTemplate.spec.template.spec
      template_path=".spec.jobTemplate.spec.template.spec"
      ;;
    ReplicationController)
      template_path=".spec.template.spec"
      ;;
    *)
      echo "Unsupported owner kind ${kind}; skipping." >&2
      return 0
      ;;
  esac

  # Build a patched containers array with ensured limits
  local patched
  patched="$(echo "${obj}" | jq --arg cpu "${DEFAULT_CPU_LIMIT}" --arg mem "${DEFAULT_MEM_LIMIT}" --arg tp "${template_path}" '
    . as $root
    | ($tp | split(".") | reduce .[] as $part ( $root; .[$part]) ) as $spec
    | $spec.containers as $cs
    | ($cs | map(
        . as $c
        | .resources |= (
            (. // {}) as $r
            | $r.limits |= (
                (. // {}) as $l
                | if ($l.cpu == null) then .cpu = $cpu else . end
                | if ($l.memory == null) then .memory = $mem else . end
              )
          )
      )) as $newcs
    | (
        $tp | ".[\"" + (split(".") | join("\"][\"")) + "\"].containers"
      ) as $ptr
    | ( $ptr | "setpath(" + (.[1:-1] | @json) + "; $newcs)" ) as $prog
    | ( . | ( $prog | fromjson ) )
  ')"

  # If jq failed, skip
  if [[ -z "${patched}" ]]; then
    echo "Failed to compute patch for ${kind}/${namespace}/${name}; skipping." >&2
    return 1
  fi

  # Apply patch using strategic-merge
  echo "${patched}" | kubectl -n "${namespace}" apply -f - >/dev/null
}

# Track processed owners to keep idempotency and avoid repeated patches
declare -A PROCESSED_OWNERS

# Iterate over non-compliant containers and patch their controllers
while IFS= read -r line; do
  [[ -z "${line}" ]] && continue
  pod_ns="$(echo "${line}"        | jq -r '.podNamespace')"
  pod_name="$(echo "${line}"      | jq -r '.podName')"
  container_name="$(echo "${line}"| jq -r '.containerName')"
  owner_kind="$(echo "${line}"    | jq -r '.owner.kind // ""')"
  owner_name="$(echo "${line}"    | jq -r '.owner.name // ""')"

  if [[ -z "${owner_kind}" || -z "${owner_name}" ]]; then
    echo "Naked Pod ${pod_ns}/${pod_name} (container: ${container_name}) has no owning controller."
    echo "Manually add limits to its spec, for example:"
    echo "  kubectl -n ${pod_ns} patch pod ${pod_name} --type=json -p='[
      {\"op\":\"add\",\"path\":\"/spec/containers/0/resources\",\"value\":{\"limits\":{\"cpu\":\"${DEFAULT_CPU_LIMIT}\",\"memory\":\"${DEFAULT_MEM_LIMIT}\"}}}
    ]'"
    continue
  fi

  owner_key="${owner_kind}/${pod_ns}/${owner_name}"
  if [[ -n "${PROCESSED_OWNERS[${owner_key}]+x}" ]]; then
    continue
  fi
  PROCESSED_OWNERS["${owner_key}"]=1

  patch_workload "${owner_kind}" "${pod_ns}" "${owner_name}" || true
done <<< "${NON_COMPLIANT_JSON}"

echo "Waiting for updated workloads to roll out..."
kubectl wait --all-namespaces --for=condition=Ready pod --all --timeout=300s >/dev/null 2>&1 || true

echo "Re-running compliance check:"
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | select($ok | not)
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'