Containers Should Set CPU And Memory Limits

More Info:

Verifies every container sets resources.limits.cpu and resources.limits.memory so a single workload cannot exhaust a node.

Risk Level

Medium

Address

Security

Compliance Standards

Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

List non-compliant pods (run on any machine with kubectl access):

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | select($ok | not)
  | "ns=\($m.namespace) pod=\($m.name)"
  ] | unique[]'

Choose one non-compliant pod and check if it is controlled by a higher-level object (run on any machine with kubectl access):
```
kubectl get pod POD_NAME -n NAMESPACE -o jsonpath='{.metadata.ownerReferences}'
```
- If empty: fix the Pod manifest directly (step 3).
- If it has an owner (e.g., Deployment, StatefulSet, Job, CronJob, ReplicaSet, DaemonSet): fix that owner resource (step 4); do not edit the pod directly.

For a standalone Pod: export its manifest, edit, and re-apply (run on any machine with kubectl access):

kubectl get pod POD_NAME -n NAMESPACE -o yaml > /tmp/pod-POD_NAME.yaml

Edit /tmp/pod-POD_NAME.yaml and under each .spec.containers[].resources ensure:

resources:
  limits:
    cpu: "500m"        # example value, choose per your policy
    memory: "512Mi"    # example value, choose per your policy

Then delete the running pod and recreate it from the edited manifest:

kubectl delete pod POD_NAME -n NAMESPACE
kubectl apply -f /tmp/pod-POD_NAME.yaml

For controller-managed pods: edit the controller so all its containers set limits (run on any machine with kubectl access). Example for a Deployment:
```
kubectl edit deployment DEPLOYMENT_NAME -n NAMESPACE
```
In the editor, for each container under spec.template.spec.containers[], ensure:
```
resources:
  limits:
    cpu: "500m"
    memory: "512Mi"
```
Save and exit. Kubernetes will roll out updated pods automatically.
Use the equivalent kubectl edit statefulset, kubectl edit daemonset, kubectl edit job, or kubectl edit cronjob for other controllers.

(Optional but recommended) Enforce limits via a LimitRange in each namespace so new pods must set them (run on any machine with kubectl access):

cat << 'EOF' > /tmp/limits-namespace.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: default-container-limits
  namespace: TARGET_NAMESPACE
spec:
  limits:
  - type: Container
    max:
      cpu: "2"
      memory: "2Gi"
    default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "250m"
      memory: "256Mi"
EOF

kubectl apply -f /tmp/limits-namespace.yaml

Verify all non-exempt pods and containers now have CPU and memory limits (run on any machine with kubectl access):

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | select($ok | not)
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

The output should be is_compliant=true.

Using kubectl

On any machine with kubectl access:

Identify pods/containers missing limits (example, focused view):

kubectl get pods --all-namespaces -o json | jq -r '
  .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | .spec.containers[]
  | select((.resources.limits.cpu == null) or (.resources.limits.memory == null))
  | "ns=\($m.namespace) pod=\($m.name) container=\(.name)"
'

For each affected pod, edit the owning workload manifest to add resources.limits.cpu and resources.limits.memory for every container.

Example patch for a Deployment (namespace and name adjusted as needed):

kubectl -n default patch deployment my-app \
  --type merge \
  -p '{
    "spec": {
      "template": {
        "spec": {
          "containers": [
            {
              "name": "my-container",
              "resources": {
                "limits": {
                  "cpu": "500m",
                  "memory": "256Mi"
                }
              }
            }
          ]
        }
      }
    }
  }'

If multiple containers exist, include each container as a separate entry in the containers array with its own name and resources.limits.

For objects managed via manifests (GitOps, IaC), update the YAML instead and apply:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
spec:
  template:
    spec:
      containers:
        - name: my-container
          image: my-image:tag
          resources:
            limits:
              cpu: "500m"
              memory: "256Mi"

Apply the manifest:

kubectl apply -f /absolute/path/to/my-app-deployment.yaml

Repeat similar edits for other controllers (StatefulSet, DaemonSet, Job, CronJob) so all their containers define both CPU and memory limits.

Verify compliance:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | select($ok | not)
  ] | if (length == 0) then "is_compliant=true" else .[] end'

Automation

#!/usr/bin/env bash
#
# Enforce CPU and memory limits on all containers in AKS pods
# that currently have either limit missing.
#
# REQUIREMENTS:
# - Run on any machine with kubectl access and jq installed.
# - You must have permission to update the relevant API resources.

set -euo pipefail

# -------- Configuration (adjust as needed) --------
# Default limits for pods whose limits are missing.
# Choose values appropriate for your AKS cluster.
DEFAULT_CPU_LIMIT="500m"
DEFAULT_MEM_LIMIT="512Mi"

# Namespaces to exclude (system namespaces already excluded by the audit query).
EXCLUDED_NAMESPACES_REGEX='^(kube-system|kube-public|kube-node-lease)$'

# -------- Helper functions --------
log() {
  printf '[%s] %s\n' "$(date -u +'%Y-%m-%dT%H:%M:%SZ')" "$*" >&2
}

# -------- Discover non-compliant pods --------
log "Discovering pods with missing CPU or memory limits..."

NON_COMPLIANT_JSON=$(kubectl get pods --all-namespaces -o json | jq --arg re "$EXCLUDED_NAMESPACES_REGEX" '
  .items[]
  | select(.metadata.namespace | test($re) | not)
  | . as $pod
  | (.spec.containers // []) as $containers
  | [ range(0; $containers | length) ] as $idxs
  | select(
      any($idxs[]; ($containers[.] | (.resources.limits.cpu == null or .resources.limits.memory == null)))
    )
  | {
      kind: "Pod",
      apiVersion: "v1",
      namespace: .metadata.namespace,
      name: .metadata.name
    }' | jq -s '.')

COUNT=$(printf '%s\n' "${NON_COMPLIANT_JSON}" | jq 'length')
if [ "${COUNT}" -eq 0 ]; then
  log "All pods are already compliant; nothing to do."
else
  log "Found ${COUNT} non-compliant pods to process."
fi

# -------- Patch function --------
patch_pod_limits() {
  local ns name
  ns="$1"
  name="$2"

  # Build a strategic merge patch that fills in missing limits for each container.
  # This does NOT modify existing limits.
  local patch
  patch=$(kubectl get pod "${name}" -n "${ns}" -o json | jq --arg cpu "${DEFAULT_CPU_LIMIT}" --arg mem "${DEFAULT_MEM_LIMIT}" '
    {
      spec: {
        containers: (
          (.spec.containers // []) | map(
            .resources |= (
              if . == null then {} else . end
            )
            | .resources.limits |= (
              if . == null then {} else . end
            )
            | if (.resources.limits.cpu == null) then
                .resources.limits.cpu = $cpu
              else .
              end
            | if (.resources.limits.memory == null) then
                .resources.limits.memory = $mem
              else .
              end
          )
        )
      }
    }')

  # If patch is empty (no changes), skip.
  if [ "$(printf '%s' "${patch}" | jq -c '.spec.containers')" = "[]" ]; then
    log "No containers to patch in ${ns}/${name}; skipping."
    return 0
  fi

  log "Patching pod ${ns}/${name} with default limits cpu=${DEFAULT_CPU_LIMIT}, mem=${DEFAULT_MEM_LIMIT}..."
  kubectl patch pod "${name}" -n "${ns}" --type merge -p "${patch}" >/dev/null
}

# -------- Apply patches --------
if [ "${COUNT}" -gt 0 ]; then
  echo "${NON_COMPLIANT_JSON}" | jq -r '.[] | [.namespace, .name] | @tsv' | \
  while IFS=$'\t' read -r ns name; do
    # Pods owned by controllers (Deployments, StatefulSets, etc.) will be
    # recreated from their controllers without these limits. For automation
    # safety, this script only patches the current pods.
    # For a durable fix, update the owning workload manifests separately.
    patch_pod_limits "${ns}" "${name}" || log "WARNING: Failed to patch pod ${ns}/${name}"
  done
fi

# -------- Verification (re-run audit) --------
log "Verifying that all pods now have CPU and memory limits..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\(.image)"
    + " limitsCpu=\(.resources.limits.cpu // "unset") limitsMemory=\(.resources.limits.memory // "unset")"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | map(select(test("is_compliant=false"))) | length) == 0
    then "is_compliant=true"
    else ($rows[] | select(test("is_compliant=false")))
    end' | tee /dev/stderr

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​