Skip to main content

Hostile Multi Tenant Workloads

More Info:​

Restricting Untrusted Workloads Can Be Achieved By Using Aci Along With Aks. What Is Aci? Aci Lets You Quickly Deploy Container Instances Without Additional Infrastructure Overhead. When You Connect With Aks, Aci Becomes A Secured, Logical Extension Of Your Aks Cluster. The Virtual Nodes Component, Which Is Based On Virtual Kubelet, Is Installed In Your Aks Cluster That Presents Aci As A Virtual Kubernetes Node. Kubernetes Can Then Schedule Pods That Run As Aci Instances Through Virtual Nodes, Not As Pods On Vm Nodes Directly In Your Aks Cluster.

Risk Level​

Low

Address​

Security

Compliance Standards​

  • CIS AKS

Triage and Remediation​

Remediation​

Manual Steps

  1. Identify hostile / untrusted / multi‑tenant workloads and namespaces

    • On any machine with az and kubectl access: 
      # List all namespaces
      kubectl get ns -o wide

      # List pods with their service accounts and nodes
      kubectl get pods -A -o wide

      # (Optional) Label namespaces you consider untrusted / multi‑tenant
      kubectl label ns <namespace> tenant=untrusted
    • Document which namespaces / apps are considered hostile or untrusted (e.g., external customer workloads, third‑party images, CI preview environments).

  2. Check whether virtual nodes / ACI integration are already in use

    • On any machine with az access: 
      # Show AKS cluster configuration and add‑ons
      az aks show \
        --resource-group <AKS_RESOURCE_GROUP> \
        --name <AKS_CLUSTER_NAME> \
        --output table
    • In the Azure Portal, open the AKS cluster → “Node pools” and check if any node pool of type “Virtual nodes” exists and which subnet it uses.

  3. Inspect how untrusted pods are currently scheduled

    • On any machine with kubectl access: 
      # See which nodes run untrusted workloads
      kubectl get pods -n <untrusted-namespace> -o wide

      # Get node types and labels
      kubectl get nodes --show-labels -o wide
    • If untrusted pods are running directly on regular VM‑backed nodes, note the namespaces and deployments for potential isolation via ACI.

  4. Decide tenant‑isolation policy using ACI / virtual nodes

    • Using your security and platform teams, decide:
      • Which namespaces / workloads must never run on VM nodes and should run only on ACI (virtual nodes).
      • Whether you need a dedicated subnet / resource group for ACI to isolate network and billing.
    • Record the policy in cluster documentation (e.g., “Namespaces labeled tenant=untrusted MUST schedule only on virtual node(s) backed by ACI”).

  5. Configure or adjust ACI / virtual nodes and scheduling constraints

    • If ACI/virtual nodes are not enabled, enable them (Portal or CLI) for the cluster in an appropriate subnet, following Azure AKS documentation.
    • On any machine with kubectl access, inspect the virtual node’s labels to drive scheduling decisions: 
      # After enabling virtual nodes, check the virtual node labels
      kubectl get nodes -o wide --show-labels | grep virtual-kubelet || true
    • For each untrusted namespace or deployment, configure nodeSelector, affinity, or taints/tolerations (via your existing IaC / manifests) so that untrusted pods only schedule to the virtual node(s) and are excluded from regular VM nodes. (Use your existing deployment tooling; this cannot be done purely from the managed control‑plane side.)

  6. Verify isolation is in effect

    • On any machine with kubectl access: 
      # Confirm untrusted workloads now run only on virtual nodes
      kubectl get pods -n <untrusted-namespace> -o wide

      # Confirm no untrusted pods are on VM nodes
      kubectl get pods -A -o wide | grep -v virtual-kubelet
    • Ensure the observed scheduling matches your documented policy (untrusted pods only on virtual nodes/ACI, trusted workloads on VM nodes as intended).
Using kubectl

kubectl cannot change this setting because it is configured at the AKS/ACI integration and cluster configuration level in Azure, not via Kubernetes API objects. To address this finding, make the required changes in the Azure portal/CLI/Terraform for AKS and ACI as described in the Manual Steps section.

Automation
#!/usr/bin/env bash
#
# Purpose:
#   Help review hostile multi-tenant workload risk in AKS by:
#   - Detecting use of virtual nodes / ACI (aci-connector)
#   - Listing pods scheduled to virtual nodes
#   - Highlighting pods without basic isolation controls
#
# Run on:
#   Any machine with kubectl access and context set to the target AKS cluster.
#
# Requirements:
#   - kubectl
#   - jq

set -euo pipefail

echo "=== 1) Detecting AKS virtual nodes / ACI usage ==================================="

echo "[INFO] Searching for virtual-node / aci-connector DaemonSets in kube-system..."
kubectl -n kube-system get ds \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.template.spec.containers[*].image \
  | grep -E 'virtual-node|aci-connector' || echo "[INFO] No virtual-node / aci-connector DaemonSet found."

echo
echo "[INFO] Listing nodes that look like virtual nodes / ACI backed..."
kubectl get nodes -o wide \
  | grep -Ei 'virtual-node|aci|virtual-kubelet' || echo "[INFO] No obvious virtual-node / ACI nodes found."

echo
echo "=== 2) Listing pods running on potential virtual / ACI nodes ====================="

# Capture nodes that look like virtual / ACI nodes by name pattern.
VIRTUAL_NODES=$(kubectl get nodes -o json \
  | jq -r '.items[]
    | select(.metadata.name | test("virtual-node|aci|virtual-kubelet"; "i"))
    | .metadata.name')

if [ -z "${VIRTUAL_NODES}" ]; then
  echo "[INFO] No virtual / ACI nodes detected by naming convention."
else
  echo "[INFO] Detected potential virtual / ACI nodes:"
  echo "${VIRTUAL_NODES}" | sed 's/^/  - /'
  echo

  echo "[INFO] Pods scheduled on these nodes (all namespaces):"
  for NODE in ${VIRTUAL_NODES}; do
    echo
    echo "---- Node: ${NODE} ----"
    kubectl get pods --all-namespaces --field-selector spec.nodeName="${NODE}" -o wide
  done
fi

echo
echo "=== 3) Basic isolation posture for non-system namespaces ========================="
echo "[INFO] Reviewing pods (excluding kube-system) for weak isolation settings."
echo "[INFO] This does NOT decide multi-tenancy safety; it highlights risky defaults."

kubectl get pods --all-namespaces -o json \
  | jq -r '
    .items[]
    | select(.metadata.namespace != "kube-system")
    | . as $pod
    | ($pod.spec.nodeName // "UNSCHEDULED") as $node
    | ($pod.metadata.namespace) as $ns
    | ($pod.metadata.name) as $name
    | ($pod.spec.containers[]? // {} ) as $c
    | ($c.name // "unknown") as $cname
    | (
        ( $pod.spec.hostNetwork == true ) or
        ( $c.securityContext.privileged == true ) or
        ( $c.securityContext.allowPrivilegeEscalation == true ) or
        ( ($c.securityContext.runAsUser // 0) == 0 ) or
        ( $pod.spec.securityContext.runAsUser == 0 ) or
        ( $c.securityContext.capabilities.add != null and ($c.securityContext.capabilities.add|length) > 0 )
      ) as $risky
    | select($risky == true)
    | @tsv
      "\($ns)\t\($name)\t\($cname)\t\($node)\t" +
      "hostNetwork=\($pod.spec.hostNetwork // false)" +
      ",privileged=\($c.securityContext.privileged // false)" +
      ",allowPrivEsc=\($c.securityContext.allowPrivilegeEscalation // false)" +
      ",runAsUserPod=\($pod.spec.securityContext.runAsUser // "null")" +
      ",runAsUserCtr=\($c.securityContext.runAsUser // "null")" +
      ",capsAdd=\($c.securityContext.capabilities.add // [])"
  ' \
  | awk 'BEGIN { print "NAMESPACE\tPOD\tCONTAINER\tNODE\tRISK_FLAGS" }1'

echo
echo "=== 4) Namespaces that may represent distinct tenants ============================"
echo "[INFO] Namespaces with label/annotation suggesting tenant separation."
echo "[INFO] Use this to cross-check which tenants are using ACI / virtual nodes."

echo
echo "[INFO] Namespaces with labels containing \"tenant\" or \"team\":"
kubectl get ns -o json \
  | jq -r '
      .items[]
      | select(
          (
            (.metadata.labels // {} | tostring | test("tenant|team"; "i")) or
            (.metadata.annotations // {} | tostring | test("tenant|team"; "i"))
          )
        )
      | @tsv "\(.metadata.name)\t\(.metadata.labels // {} | tostring)"
    ' \
  | awk 'BEGIN { print "NAMESPACE\tLABELS" }1' || echo "[INFO] None detected."

echo
echo "=== Interpretation ==============================================================="
echo "Potential problem indicators:"
echo "1) You see virtual-node / aci-connector present AND many different tenant/team"
echo "   namespaces scheduling to those virtual nodes without clear isolation policy."
echo "2) Pods on virtual / ACI nodes show risky flags above (privileged, hostNetwork,"
echo "   allowPrivilegeEscalation, runAs root, extra capabilities)."
echo
echo "This script only surfaces data for review; decisions on whether workloads are"
echo "hostile / multi-tenant and whether ACI/virtual nodes are appropriate remain manual."

What output indicates a potential problem

  • Section 1–2:

    • Virtual / ACI nodes or virtual-node / aci-connector DaemonSets are present.
    • Many pods from different business/tenant namespaces are scheduled on those nodes.

  • Section 3:

    • Pods, especially those on virtual / ACI nodes, appear with:
      • hostNetwork=true
      • privileged=true
      • allowPrivilegeEscalation=true
      • runAsUser = 0 (root)
      • Added Linux capabilities These increase risk when running hostile or untrusted multi-tenant workloads.

  • Section 4:

    • Namespaces labeled/annotated by tenant/team that are also shown in Section 2 as using virtual / ACI nodes, indicating cross-tenant sharing of that infrastructure and requiring manual risk assessment.

Additional Reading:​