Containers Should Set CPU And Memory Limits

More Info:

Verifies every container sets resources.limits.cpu and resources.limits.memory so a single workload cannot exhaust a node.

Risk Level

Medium

Address

Security

Compliance Standards

• Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

1. Identify noncompliant pods (run on any machine with kubectl access):

   kubectl get pods --all-namespaces -o json | jq -r '
     [ .items[]
     | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
     | .metadata as $m
     | (.spec.containers // [])[]
     | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
     | select($ok | not)
     | "ns=\($m.namespace) pod=\($m.name) container=\(.name)"
     ][]'

2. Choose a noncompliant workload and fetch its manifest (any machine with kubectl access). Example for a pod in namespace default named myapp-pod:

   kubectl get pod myapp-pod -n default -o yaml > /tmp/myapp-pod.yaml

3. Edit the manifest to add CPU and memory limits for each container (any machine with kubectl access). Open the file:

   vi /tmp/myapp-pod.yaml

   Under each .spec.containers[].resources section, ensure something like:

   resources:
     limits:
       cpu: "500m"
       memory: "512Mi"

   Adjust values per your capacity and application requirements.

4. Recreate the pod with the updated manifest (any machine with kubectl access). Pods created by higher-level controllers (Deployments, StatefulSets, etc.) should be fixed at the controller level instead; for a standalone pod:

   kubectl delete pod myapp-pod -n default
   kubectl apply -f /tmp/myapp-pod.yaml

5. For pods managed by controllers (recommended in EKS), patch the controller instead of individual pods (any machine with kubectl access). Example for a Deployment myapp-deploy in default namespace:

   kubectl edit deployment myapp-deploy -n default

   In the editor, under spec.template.spec.containers[], add:

   resources:
     limits:
       cpu: "500m"
       memory: "512Mi"

   Save and exit so the Deployment rolls out updated pods.

6. Verify all non-system pods now have CPU and memory limits (any machine with kubectl access):

   kubectl get pods --all-namespaces -o json | jq -r '
     [ .items[]
     | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
     | .metadata as $m
     | (.spec.containers // [])[]
     | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
     | select($ok | not)
     ] as $rows
     | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

   Ensure the output is is_compliant=true.

Using kubectl

On any machine with kubectl access:

1. Identify non‑compliant Pods and their controllers

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | select($ok | not)
  | "ns=\($m.namespace) pod=\($m.name) ownerKind=\($own.kind // "Pod") ownerName=\($own.name // $m.name)"
  ] | unique[]'

For each line, note ownerKind and ownerName. Always patch the controller (Deployment/StatefulSet/DaemonSet/Job/CronJob), not the Pod, so the setting persists.

2. Example: patch a Deployment’s containers to add limits

Edit the Deployment manifest:

kubectl -n <namespace> get deploy <deployment-name> -o yaml > /tmp/deploy.yaml

In /tmp/deploy.yaml, under each .spec.template.spec.containers[].resources, add:

resources:
  limits:
    cpu: "500m"
    memory: "256Mi"

(adjust values to your policy; repeat for every container.)

Apply the updated manifest:

kubectl apply -f /tmp/deploy.yaml

3. Example: patch a single container in place (quick fix)

kubectl -n <namespace> patch deploy <deployment-name> \
  --type='json' \
  -p='[
    {
      "op": "add",
      "path": "/spec/template/spec/containers/0/resources",
      "value": {
        "limits": {
          "cpu": "500m",
          "memory": "256Mi"
        }
      }
    }
  ]'

Adjust the container index in the path and the limit values as needed.

Use the same approach (kubectl get ... -o yaml → edit → kubectl apply) for StatefulSets, DaemonSets, Jobs, and CronJobs.

4. For standalone Pods (no ownerReference)

Export, edit, and re‑create with a controller (recommended) or as a Pod:

kubectl -n <namespace> get pod <pod-name> -o yaml > /tmp/pod.yaml

Edit /tmp/pod.yaml: remove fields under status:, remove metadata.uid, metadata.resourceVersion, metadata.creationTimestamp, and set resources.limits as above for each container. Then:

kubectl delete -n <namespace> pod <pod-name>
kubectl apply -f /tmp/pod.yaml

5. Verification

Re‑run the benchmark audit command from any machine with kubectl access:

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " container=\(.name) image=\(.image)"
    + " limitsCpu=\(.resources.limits.cpu // "unset") limitsMemory=\(.resources.limits.memory // "unset")"
    + " is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

All reported containers should now show non‑unset limitsCpu and limitsMemory and is_compliant=true.

Automation

#!/usr/bin/env bash
# Remediate: ensure all containers in non-system namespaces have CPU and memory limits.
# Scope: any machine with kubectl access to the EKS cluster.
# Requirements: kubectl, jq, and yq (https://github.com/mikefarah/yq) installed and in PATH.

set -euo pipefail

# --- Configuration: default limits if missing (adjust as appropriate for your cluster) ---
DEFAULT_CPU_LIMIT="500m"
DEFAULT_MEMORY_LIMIT="512Mi"

# --- Pre-flight checks ---
for cmd in kubectl jq yq; do
  if ! command -v "$cmd" >/dev/null 2>&1; then
    echo "ERROR: $cmd is required but not installed or not in PATH." >&2
    exit 1
  fi
done

echo "Discovering non-system pods without CPU or memory limits..."

# Get all non-system pods in YAML form
PODS_YAML="$(kubectl get pods --all-namespaces \
  --field-selector=status.phase!=Succeeded,status.phase!=Failed \
  -o yaml \
  | yq 'del(.items[].metadata.namespace | select(. == "kube-system" or . == "kube-public" or . == "kube-node-lease"))'
)"

# If there are no non-system pods, exit early
if [ "$(printf '%s\n' "$PODS_YAML" | yq '.items | length')" -eq 0 ]; then
  echo "No non-system pods found. Nothing to remediate."
  exit 0
fi

# Build a list of owning controllers (kind/namespace/name) from the pods
# We skip pods with no controller ownerReference because patching them directly often suffices.
echo "Identifying owning workloads to patch (Deployments, StatefulSets, DaemonSets, Jobs, CronJobs, ReplicaSets, ReplicationControllers)..."

OWNERS_JSON="$(printf '%s\n' "$PODS_YAML" \
  | yq -o=json '.items[]
      | . as $pod
      | ($pod.metadata.ownerReferences // [])[]
      | select(.controller == true)
      | {
          kind: .kind,
          apiVersion: $pod.apiVersion,
          namespace: $pod.metadata.namespace,
          name: .name
        }' 2>/dev/null || true
)"

if [ -z "$OWNERS_JSON" ]; then
  echo "No owning controllers detected; only standalone pods will be processed (if any)."
  OWNERS_UNIQUE=""
else
  OWNERS_UNIQUE="$(printf '%s\n' "$OWNERS_JSON" \
    | jq -s 'unique_by(.kind,.namespace,.name)[]')"
fi

# Function: patch all templates in a given controller object
patch_controller_limits() {
  local kind="$1"
  local namespace="$2"
  local name="$3"

  echo "Patching ${kind}/${namespace}/${name} ..."

  # Fetch current object in YAML
  local obj_yaml
  if ! obj_yaml="$(kubectl -n "$namespace" get "$kind" "$name" -o yaml 2>/dev/null)"; then
    echo "  WARN: Unable to get ${kind}/${namespace}/${name}, skipping." >&2
    return 0
  fi

  # Patch all containers in pod template: set limits if missing, leave existing ones unchanged
  local patched_yaml
  patched_yaml="$(printf '%s\n' "$obj_yaml" | yq "
    (.. | select(has(\"spec\") and has(\"template\")) | .spec.template.spec.containers[])
      |= (
        .resources.limits.cpu   //= \"${DEFAULT_CPU_LIMIT}\" |
        .resources.limits.memory //= \"${DEFAULT_MEMORY_LIMIT}\"
      )
  ")"

  # Apply patch (idempotent; re-running maintains same limits)
  printf '%s\n' "$patched_yaml" | kubectl apply -f - >/dev/null

  echo "  Patched ${kind}/${namespace}/${name}"
}

# Patch each unique owner
if [ -n "$OWNERS_UNIQUE" ]; then
  echo "$OWNERS_UNIQUE" | jq -c '.' | while read -r owner; do
    kind="$(echo "$owner" | jq -r '.kind')"
    namespace="$(echo "$owner" | jq -r '.namespace')"
    name="$(echo "$owner" | jq -r '.name')"

    # Map owner kinds to the actual resource kinds we can kubectl get/apply
    case "$kind" in
      Deployment|StatefulSet|DaemonSet|Job|CronJob|ReplicaSet|ReplicationController)
        patch_controller_limits "$kind" "$namespace" "$name"
        ;;
      *)
        echo "  INFO: Skipping unsupported owner kind $kind for ${namespace}/${name}" >&2
        ;;
    esac
  done
fi

# Handle standalone pods (no controller ownerReferences) directly.
echo "Patching standalone pods (no controller owner) if needed..."

STANDALONE_PODS="$(printf '%s\n' "$PODS_YAML" \
  | yq -o=json '.items[]
      | select((.metadata.ownerReferences // []) | length == 0)
      | {
          namespace: .metadata.namespace,
          name: .metadata.name
        }' 2>/dev/null || true
)"

if [ -z "$STANDALONE_PODS" ]; then
  echo "No standalone pods detected."
else
  echo "$STANDALONE_PODS" | jq -c '.' | while read -r pod; do
    ns="$(echo "$pod" | jq -r '.namespace')"
    name="$(echo "$pod" | jq -r '.name')"

    echo "  Checking standalone pod ${ns}/${name} ..."

    pod_yaml="$(kubectl -n "$ns" get pod "$name" -o yaml 2>/dev/null || true)"
    [ -z "$pod_yaml" ] && { echo "    WARN: Pod ${ns}/${name} no longer exists, skipping."; continue; }

    # Only patch if any container in this pod is missing cpu or memory limits
    needs_patch="$(printf '%s\n' "$pod_yaml" | yq '
      .spec.containers[]
      | ((.resources.limits.cpu == null) or (.resources.limits.memory == null))
      ' 2>/dev/null || true
    )"

    if [ -z "$needs_patch" ]; then
      echo "    Already compliant; no changes."
      continue
    fi

    patched_pod_yaml="$(printf '%s\n' "$pod_yaml" | yq "
      .spec.containers[]
        |= (
          .resources.limits.cpu   //= \"${DEFAULT_CPU_LIMIT}\" |
          .resources.limits.memory //= \"${DEFAULT_MEMORY_LIMIT}\"
        )
    ")"

    printf '%s\n' "$patched_pod_yaml" | kubectl apply -f - >/dev/null
    echo "    Patched pod ${ns}/${name}"
  done
fi

# --- Verification ---
echo
echo "Verifying that all non-system pod containers have CPU and memory limits..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.containers // [])[]
  | ((.resources.limits.cpu != null) and (.resources.limits.memory != null)) as $ok
  | "ns=\($m.namespace) pod=\($m.name) container=\(.name) limitsCpu=\(.resources.limits.cpu // "unset") limitsMemory=\(.resources.limits.memory // "unset") is_compliant=\(if $ok then "true" else "false" end)"
  ] as $rows
  | if ([.[].|select(contains("is_compliant=false"))] | length) == 0
    then "is_compliant=true"
    else .[]
    end
'

echo
echo "NOTE: If any line above shows is_compliant=false, inspect and update the owning workload manifest manually."