Skip to main content

Tenant Namespaces Should Have A ResourceQuota

More Info:​

Advisory: create a ResourceQuota per tenant namespace to bound aggregate CPU, memory and object counts, preventing one tenant from starving others.

Risk Level​

Low

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. List all tenant namespaces (excluding system namespaces). Run on any machine with kubectl access:

    kubectl get namespaces \
      --no-headers \
      | awk '!/kube-system/ && !/kube-public/ && !/kube-node-lease/ {print $1}'

  2. For each tenant namespace (replace TENANT_NAMESPACE with the actual name), create a baseline ResourceQuota manifest file locally, for example rq-tenant-TENANT_NAMESPACE.yaml:

    apiVersion: v1
    kind: ResourceQuota
    metadata:
      name: tenant-quota
      namespace: TENANT_NAMESPACE
    spec:
      hard:
        requests.cpu: "4"
        requests.memory: "8Gi"
        limits.cpu: "8"
        limits.memory: "16Gi"
        pods: "50"
        services: "10"
        configmaps: "20"
        persistentvolumeclaims: "10"
        secrets: "50"
        replicationcontrollers: "20"
        resourcequotas: "1"

    Adjust the values to match your tenant sizing and capacity planning.

  3. Apply the ResourceQuota for each tenant namespace. Run on any machine with kubectl access:

    kubectl apply -f rq-tenant-TENANT_NAMESPACE.yaml

  4. (Optional) Review the applied ResourceQuota and confirm it matches expectations. Run on any machine with kubectl access:

    kubectl get resourcequota -n TENANT_NAMESPACE tenant-quota -o yaml

  5. If you already use labels to identify tenant namespaces (for example tenant=true), you can target only those namespaces. Run on any machine with kubectl access:

    kubectl get ns -l tenant=true -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' \
    | while read ns; do
        sed "s/TENANT_NAMESPACE/$ns/g" rq-tenant-template.yaml | kubectl apply -f -
      done

    Where rq-tenant-template.yaml is the manifest from step 2 with TENANT_NAMESPACE as a placeholder.

  6. Verify that every tenant namespace now has at least one ResourceQuota. Run on any machine with kubectl access:

    { kubectl get resourcequotas --all-namespaces -o json
      kubectl get namespaces -o json
    } | jq -rs '
      .[0] as $quotas | .[1] |
      [ .items[]
      | select(.metadata.name as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | ([ $quotas.items[] | select(.metadata.namespace == $m.name) ] | length) as $count
      | "name=\($m.name) resourceQuotas=\($count) is_compliant=\(if $count > 0 then "true" else "false" end)"
      ][]'

    Confirm that is_compliant=true for all tenant namespaces.

Using kubectl
# 1) List tenant namespaces that lack a ResourceQuota
# Run on: any machine with kubectl access
{ kubectl get resourcequotas --all-namespaces -o json
  kubectl get namespaces -o json
} | jq -rs '
  .[0] as $quotas | .[1] |
  [ .items[]
  | select(.metadata.name as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ $quotas.items[] | select(.metadata.namespace == $m.name) ] | length) as $count
  | select($count == 0)
  | .name
  ][]'

# Suppose the output contains tenant namespaces "team-a", "team-b".
# 2) Create a ResourceQuota manifest per tenant namespace (edit names/limits as needed)
# Run on: any machine with kubectl access

cat > team-a-resourcequota.yaml << 'EOF'
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "4"
    requests.memory: "8Gi"
    limits.cpu: "8"
    limits.memory: "16Gi"
    pods: "50"
    services: "20"
    configmaps: "50"
    secrets: "100"
    persistentvolumeclaims: "20"
EOF

cat > team-b-resourcequota.yaml << 'EOF'
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
  namespace: team-b
spec:
  hard:
    requests.cpu: "4"
    requests.memory: "8Gi"
    limits.cpu: "8"
    limits.memory: "16Gi"
    pods: "50"
    services: "20"
    configmaps: "50"
    secrets: "100"
    persistentvolumeclaims: "20"
EOF

# 3) Apply the ResourceQuota objects
kubectl apply -f team-a-resourcequota.yaml
kubectl apply -f team-b-resourcequota.yaml

# 4) Verification: rerun the audit to confirm each tenant namespace has at least one ResourceQuota
{ kubectl get resourcequotas --all-namespaces -o json
  kubectl get namespaces -o json
} | jq -rs '
  .[0] as $quotas | .[1] |
  [ .items[]
  | select(.metadata.name as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ $quotas.items[] | select(.metadata.namespace == $m.name) ] | length) as $count
  | "kind=Namespace name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $labels == "" then "" else " labels=\($labels)" end)
    + " resourceQuotas=\($count)"
    + " is_compliant=\(if $count > 0 then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Automation
#!/usr/bin/env bash
# Purpose: Ensure every tenant namespace in an OKE cluster has at least one ResourceQuota.
# Safe to re-run. Requires: kubectl, jq in PATH, current context set.

set -euo pipefail

#-----------------------------
# Configuration
#-----------------------------
# System namespaces to exclude from tenant treatment
EXCLUDED_NAMESPACES=("kube-system" "kube-public" "kube-node-lease")

# Default ResourceQuota name to create when none exists
RQ_NAME="tenant-default-quota"

# Default quota values – adjust to fit your cluster’s sizing and SLOs
# These are example bounds only.
CPU_REQUESTS="4"
CPU_LIMITS="8"
MEMORY_REQUESTS="8Gi"
MEMORY_LIMITS="16Gi"
PODS="200"
SERVICES="50"
CONFIGMAPS="100"
SECRETS="100"
PERSISTENTVOLUMECLAIMS="50"

#-----------------------------
# Helper functions
#-----------------------------
is_excluded_ns() {
  local ns="$1"
  for e in "${EXCLUDED_NAMESPACES[@]}"; do
    if [[ "$ns" == "$e" ]]; then
      return 0
    fi
  done
  return 1
}

#-----------------------------
# Main logic
#-----------------------------
echo "Discovering namespaces..."
# Any machine with kubectl access
mapfile -t ALL_NAMESPACES < <(kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')

if [[ "${#ALL_NAMESPACES[@]}" -eq 0 ]]; then
  echo "No namespaces found, nothing to do."
  exit 0
fi

for ns in "${ALL_NAMESPACES[@]}"; do
  if is_excluded_ns "$ns"; then
    echo "Skipping excluded namespace: $ns"
    continue
  fi

  echo "Processing tenant namespace: $ns"

  # Check if any ResourceQuota exists in this namespace
  if kubectl get resourcequota -n "$ns" >/dev/null 2>&1; then
    existing_count=$(kubectl get resourcequota -n "$ns" --no-headers 2>/dev/null | wc -l || true)
  else
    existing_count=0
  fi

  if [[ "$existing_count" -gt 0 ]]; then
    echo "  Namespace $ns already has $existing_count ResourceQuota object(s); leaving as-is."
    continue
  fi

  echo "  No ResourceQuota found in $ns; creating $RQ_NAME..."

  cat <<EOF | kubectl apply -n "$ns" -f -
apiVersion: v1
kind: ResourceQuota
metadata:
  name: ${RQ_NAME}
spec:
  hard:
    requests.cpu: "${CPU_REQUESTS}"
    limits.cpu: "${CPU_LIMITS}"
    requests.memory: "${MEMORY_REQUESTS}"
    limits.memory: "${MEMORY_LIMITS}"
    pods: "${PODS}"
    services: "${SERVICES}"
    configmaps: "${CONFIGMAPS}"
    secrets: "${SECRETS}"
    persistentvolumeclaims: "${PERSISTENTVOLUMECLAIMS}"
EOF

done

#-----------------------------
# Verification
#-----------------------------
echo
echo "Verifying that each tenant namespace has at least one ResourceQuota..."

{ kubectl get resourcequotas --all-namespaces -o json
  kubectl get namespaces -o json
} | jq -rs '
  .[0] as $quotas | .[1] |
  [ .items[]
  | select(.metadata.name as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ $quotas.items[] | select(.metadata.namespace == $m.name) ] | length) as $count
  | "kind=Namespace name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $labels == "" then "" else " labels=\($labels)" end)
    + " resourceQuotas=\($count)"
    + " is_compliant=\(if $count > 0 then "true" else "false" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

echo
echo "Automation complete."