Skip to main content

Pods Should Be Managed By A Controller

More Info:​

Verifies pods are owned by a controller (Deployment, StatefulSet, DaemonSet, Job). A naked pod is not rescheduled if its node dies.

Risk Level​

Low

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. List all non-system “naked” Pods (no controller)

    • Run on: any machine with kubectl access
    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | select($own == null)
      | "ns=\($m.namespace) name=\($m.name)"
      ][]'

  2. For each naked Pod, export its manifest

    • Replace <namespace> and <pod-name> for each entry from step 1.
    • Run on: any machine with kubectl access
    kubectl get pod <pod-name> -n <namespace> -o yaml > /tmp/<namespace>-<pod-name>.pod.yaml

  3. Convert the Pod manifest into a controller manifest (example: Deployment)

    • Edit the exported file on your workstation and save as a new file, e.g. /tmp/<namespace>-<pod-name>.deploy.yaml.
    • Minimal transformation pattern (keep metadata.labels and spec, remove fields set by the cluster like status, resourceVersion, uid, etc.):
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: <pod-name>
      namespace: <namespace>
      labels:
        app: <pod-name>
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: <pod-name>
      template:
        metadata:
          labels:
            app: <pod-name>
        spec:
          # Paste the original pod.spec here (containers, volumes, securityContext, etc.),
          # but remove fields such as nodeName, hostIP, status, etc.
    • Choose StatefulSet/DaemonSet/Job instead of Deployment if that better matches the workload semantics, following the same pattern (controller spec.template contains the old Pod spec).

  4. Delete the original naked Pod and create the controller

    • This will cause a brief disruption for that workload.
    • Run on: any machine with kubectl access
    kubectl delete pod <pod-name> -n <namespace> --wait=true
    kubectl apply -f /tmp/<namespace>-<pod-name>.deploy.yaml
    • Wait for the new Pod to become Ready:
    kubectl get pods -n <namespace> -l app=<pod-name>

  5. Repeat for all naked Pods

    • Perform steps 2–4 for each Pod listed in step 1, selecting the appropriate controller type (Deployment, StatefulSet, DaemonSet, or Job) based on how the workload should behave on EKS.

  6. Verify no remaining naked Pods

    • Run on: any machine with kubectl access
    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.nodeName // "") as $node
      | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
        + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
        + (if $node   == ""   then "" else " node=\($node)" end)
        + (if $labels == ""   then "" else " labels=\($labels)" end)
        + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
        + " is_compliant=\(if $own == null then "false" else "true" end)"
      ] as $rows
      | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
    • Confirm that all entries show is_compliant=true and no remaining Pods have owner= missing or is_compliant=false.
Using kubectl

On any machine with kubectl access:

  1. Identify naked pods (excluding system namespaces):
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | select($own == null)
  | "\($m.namespace) \($m.name)"
  ][]'
  1. For each non-system naked pod you want to keep, export its spec (example for namespace default, pod my-pod):
kubectl get pod my-pod -n default -o yaml > /tmp/my-pod-raw.yaml
  1. Create a controller manifest from the pod spec, for example a Deployment (most common). Edit a file like /tmp/my-pod-deploy.yaml to look like:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-pod
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-pod
  template:
    metadata:
      labels:
        app: my-pod
    spec:
      containers:
        - name: my-container
          image: registry.example.com/my-image:tag
          # copy over other container fields (ports, env, resources, etc.)
      # copy over other pod-level fields you need (volumes, serviceAccountName, nodeSelector, etc.)

Populate containers and other fields from /tmp/my-pod-raw.yaml’s spec: section.

  1. Apply the controller:
kubectl apply -f /tmp/my-pod-deploy.yaml
  1. Wait for the new pod(s) to be ready:
kubectl rollout status deployment/my-pod -n default
  1. Migrate traffic or data if needed (e.g., ensure Services point via labels app: my-pod), then delete the original naked pod:
kubectl delete pod my-pod -n default
  1. Repeat steps 2–6 for each naked pod, choosing the appropriate controller type:
  • Long-running stateless: Deployment
  • Long-running stateful with stable identity/storage: StatefulSet
  • One per node: DaemonSet
  • Finite/one-off work: Job (or CronJob for scheduled jobs)
  1. Verification (same command used by the check):
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | (.spec.nodeName // "") as $node
  | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
    + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
    + (if $node   == ""   then "" else " node=\($node)" end)
    + (if $labels == ""   then "" else " labels=\($labels)" end)
    + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
    + " is_compliant=\(if $own == null then "false" else "true" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Automation
#!/usr/bin/env bash
#
# automate_naked_pod_migration.sh
#
# Idempotently detect non-system naked Pods and generate/apply equivalent
# Deployments so Pods are managed by a controller.
#
# Requirements:
# - Run on any machine with kubectl access and jq installed.
# - Current kube-context must point to the target EKS cluster.
#
# What it does:
# 1. Detect Pods without a controller in non-system namespaces.
# 2. For each naked Pod:
#    - Create an apps/v1 Deployment manifest with pod spec copied.
#    - Use a deterministic, idempotent Deployment name.
#    - Label the Deployment to mark it as auto-migrated.
# 3. Apply the Deployment.
# 4. Optionally delete the original naked Pod (prompted).
# 5. Re-run audit command to verify compliance.
#
# Limitations / operator review:
# - Does NOT touch Pods where deletion might be dangerous:
#   * Pods with ownerReferences (already managed).
#   * Pods in kube-system, kube-public, kube-node-lease.
#   * Static Pods or mirror Pods (no namespace / node-only).
# - Does NOT handle stateful workloads (PVCs, stable identities).
#   Review and convert such Pods manually into a StatefulSet instead.
# - Assumes replicas=1 is appropriate; adjust post-migration if needed.

set -euo pipefail

if ! command -v kubectl >/dev/null 2>&1; then
  echo "kubectl not found in PATH; install it and ensure it can reach the EKS cluster." >&2
  exit 1
fi

if ! command -v jq >/dev/null 2>&1; then
  echo "jq not found in PATH; install jq to continue." >&2
  exit 1
fi

# Confirm cluster access
kubectl version --short >/dev/null

WORKDIR="$(pwd)/naked-pod-migration-$(date +%Y%m%d-%H%M%S)"
mkdir -p "${WORKDIR}"
echo "Working directory: ${WORKDIR}"

echo "Detecting naked Pods (no controller) in non-system namespaces..."

NAKED_JSON="${WORKDIR}/naked_pods.json"

kubectl get pods --all-namespaces -o json | jq '
  .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | select((.metadata.ownerReferences // []) | length == 0)
' > "${NAKED_JSON}"

if [[ ! -s "${NAKED_JSON}" ]]; then
  echo "No naked Pods found in non-system namespaces. Nothing to do."
  exit 0
fi

echo "The following naked Pods were found:"
jq -r '. | "ns=\(.metadata.namespace) name=\(.metadata.name)"' "${NAKED_JSON}" | sort

echo
read -r -p "Proceed to create Deployments for these Pods? [y/N]: " PROCEED
PROCEED=${PROCEED:-N}
if [[ "${PROCEED}" != "y" && "${PROCEED}" != "Y" ]]; then
  echo "Aborting by user choice."
  exit 0
fi

# Function: generate Deployment manifest YAML for a given Pod JSON
generate_deployment_manifest() {
  local pod_json="$1"
  local ns name deploy_name
  ns=$(echo "${pod_json}" | jq -r '.metadata.namespace')
  name=$(echo "${pod_json}" | jq -r '.metadata.name')

  # Deterministic Deployment name: original-name + "-deployment"
  deploy_name="${name}-deployment"

  # Remove fields from pod metadata that are not valid in pod template
  # Also strip nodeName and hostNetwork-specific assignments to let scheduler work.
  echo "${pod_json}" | jq -r --arg deploy_name "${deploy_name}" '
    {
      apiVersion: "apps/v1",
      kind: "Deployment",
      metadata: {
        name: $deploy_name,
        namespace: .metadata.namespace,
        labels: (.metadata.labels // {}) + { "naked-pod-migrated": "true" },
        annotations: (.metadata.annotations // {})
      },
      spec: {
        replicas: 1,
        selector: {
          matchLabels: (.metadata.labels // { "app": .metadata.name })
        },
        template: {
          metadata: {
            labels: (.metadata.labels // { "app": .metadata.name }) + { "naked-pod-migrated": "true" },
            annotations: (.metadata.annotations // {})
          },
          spec: (
            .spec
            | del(
                .nodeName,
                .hostname,
                .subdomain,
                .restartPolicy,
                .serviceAccount,
                .serviceAccountName,
                .priority,
                .priorityClassName,
                .schedulerName,
                .tolerations[]
                | select(.key == "CriticalAddonsOnly")
              )
          )
        }
      }
    }
  '
}

MANIFEST_DIR="${WORKDIR}/manifests"
mkdir -p "${MANIFEST_DIR}"

echo
echo "Generating Deployment manifests..."

index=0
while read -r pod; do
  index=$((index + 1))
  ns=$(echo "${pod}" | jq -r '.metadata.namespace')
  name=$(echo "${pod}" | jq -r '.metadata.name')
  manifest="${MANIFEST_DIR}/${ns}-${name}-deployment.yaml"

  generate_deployment_manifest "${pod}" > "${manifest}"

  echo "  Created: ${manifest}"
done < <(jq -c '. as $p | $p' "${NAKED_JSON}")

echo
echo "Applying Deployment manifests..."
kubectl apply -f "${MANIFEST_DIR}"

echo
read -r -p "Delete the original naked Pods after Deployments are created? [y/N]: " DELETE_PODS
DELETE_PODS=${DELETE_PODS:-N}

if [[ "${DELETE_PODS}" == "y" || "${DELETE_PODS}" == "Y" ]]; then
  echo "Deleting original naked Pods..."
  while read -r pod; do
    ns=$(echo "${pod}" | jq -r '.metadata.namespace')
    name=$(echo "${pod}" | jq -r '.metadata.name')
    echo "  Deleting pod ${ns}/${name}"
    kubectl delete pod "${name}" -n "${ns}" --ignore-not-found=true
  done < <(jq -c '. as $p | $p' "${NAKED_JSON}")
else
  echo "Original naked Pods were NOT deleted. Verify new Deployments and manually delete old Pods when safe."
fi

echo
echo "Waiting for new Deployment Pods to become Ready (best-effort)..."
kubectl get deploy --all-namespaces -l naked-pod-migrated=true -o name | while read -r d; do
  ns=$(echo "${d}" | cut -d'/' -f1)
  name=$(echo "${d}" | cut -d'/' -f2)
  echo "  Waiting for ${ns}/${name}..."
  kubectl -n "${ns}" rollout status "deployment/${name}" --timeout=120s || \
    echo "  WARNING: rollout for ${ns}/${name} did not complete within 120s. Check manually."
done

echo
echo "Re-running compliance audit to verify that Pods are now managed by a controller..."
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | "kind=Pod ns=\($m.namespace) name=\($m.name) is_compliant=\(if $own == null then "false" else "true" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end
'

echo
echo "Automation complete. Review any remaining 'is_compliant=false' Pods above and migrate them manually as needed."