Skip to main content

Pods Should Be Managed By A Controller

More Info:​

Verifies pods are owned by a controller (Deployment, StatefulSet, DaemonSet, Job). A naked pod is not rescheduled if its node dies.

Risk Level​

Low

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. Identify all naked pods (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.nodeName // "") as $node
      | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
        + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
        + (if $node   == ""   then "" else " node=\($node)" end)
        + (if $labels == ""   then "" else " labels=\($labels)" end)
        + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
        + " is_compliant=\(if $own == null then "false" else "true" end)"
      ] as $rows
      | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end' \
    | grep 'is_compliant=false'

  2. For each naked pod, export its manifest so you can base a controller on it (replace NAMESPACE and PODNAME; run on any machine with kubectl access):

    kubectl get pod PODNAME -n NAMESPACE -o yaml > PODNAME-pod.yaml

  3. Create an appropriate controller manifest from the pod spec (run on any machine with kubectl access; edit locally with your editor):

    • For stateless workloads, convert to a Deployment: 
      cat PODNAME-pod.yaml | sed '1,/^spec:/d' > PODNAME-pod-spec.yaml
      Then create PODNAME-deploy.yaml with content like: 
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: PODNAME
        namespace: NAMESPACE
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: PODNAME
        template:
          metadata:
            labels:
              app: PODNAME
          spec:
      Now paste the contents of PODNAME-pod-spec.yaml under the spec: line above (indented two spaces) and adjust labels, selectors, and any fields that must be unique (for example, remove nodeName if you don’t want pinning).
    • For workloads that must run on every node, build a DaemonSet instead by changing kind: Deployment to kind: DaemonSet and removing the replicas: field.
    • For stateful or single-instance workloads that need stable identities, use kind: StatefulSet and add a serviceName and volumeClaimTemplates as appropriate.

  4. Apply the new controller to the cluster (run on any machine with kubectl access):

    kubectl apply -f PODNAME-deploy.yaml

    Wait for the new managed pod(s) to be ready:

    kubectl get pods -n NAMESPACE -l app=PODNAME

  5. Once the controller-managed pod is running and serving traffic, delete the original naked pod (run on any machine with kubectl access):

    kubectl delete pod PODNAME -n NAMESPACE

    If the pod was fronted by a Service, confirm the Service’s selector matches the labels used by the controller (for example, app=PODNAME):

    kubectl get service -n NAMESPACE
    kubectl get service SERVICENAME -n NAMESPACE -o yaml | grep -A3 'selector:'

  6. Verify no remaining naked pods exist (run on any machine with kubectl access):

    kubectl get pods --all-namespaces -o json | jq -r '
      [ .items[]
      | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
      | .metadata as $m
      | (.spec.nodeName // "") as $node
      | (($m.labels // {}) | to_entries | map("\(.key):\(.value)") | join(",")) as $labels
      | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
      | "kind=Pod ns=\($m.namespace) name=\($m.name) uid=\($m.uid) apiVersion=v1"
        + (if ($m.creationTimestamp // "") == "" then "" else " created=\($m.creationTimestamp)" end)
        + (if $node   == ""   then "" else " node=\($node)" end)
        + (if $labels == ""   then "" else " labels=\($labels)" end)
        + (if $own    == null then "" else " owner=\($own.kind)/\($m.namespace)/\($own.name)/\($own.uid)" end)
        + " is_compliant=\(if $own == null then "false" else "true" end)"
      ] as $rows
      | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'

    Confirm that either the output is is_compliant=true or that no lines contain is_compliant=false.

Using kubectl

On any machine with kubectl access:

  1. Identify “naked” pods (no controller ownerReference) outside system namespaces:
kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | select($own == null)
  | "\($m.namespace) \($m.name)"
  ][]'
  1. For each listed pod, export its spec to base a controller on (example for namespace app-namespace, pod my-app-pod):
kubectl get pod my-app-pod -n app-namespace -o yaml > /tmp/my-app-pod.yaml
  1. Create a matching controller manifest. For a typical stateless app, prefer a Deployment. Strip pod fields that must not be templated (status, metadata.uid, resourceVersion, etc.) and wrap spec under template. Example deployment-my-app.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: app-namespace
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: your-registry.example.com/my-app:1.0.0
          ports:
            - containerPort: 8080
          env:
            - name: ENVIRONMENT
              value: "prod"
          # copy over other needed fields from the original pod spec
          # (resources, probes, volumeMounts, etc.)
      # copy over volumes, node selectors, tolerations, etc. as required

Apply it:

kubectl apply -f deployment-my-app.yaml
  1. If the naked pod is a singleton system-style agent that should run on all nodes, instead create a DaemonSet. Example daemonset-my-agent.yaml:
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: my-agent
  namespace: app-namespace
spec:
  selector:
    matchLabels:
      app: my-agent
  template:
    metadata:
      labels:
        app: my-agent
    spec:
      containers:
        - name: my-agent
          image: your-registry.example.com/my-agent:1.0.0
          # copy other required fields from the original pod spec

Apply it:

kubectl apply -f daemonset-my-agent.yaml
  1. After the new controller-created pods are running and traffic is confirmed, delete the original naked pod(s), one at a time, by name:
kubectl delete pod my-app-pod -n app-namespace

  1. Repeat steps 2–5 for each naked pod, choosing Deployment/StatefulSet/DaemonSet/Job as appropriate to the workload semantics (stateful storage, one-shot jobs, per-node agents).

  2. Verification (same audit as the check, run after all conversions):

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | "kind=Pod ns=\($m.namespace) name=\($m.name) is_compliant=\(if $own == null then "false" else "true" end)"
  ] as $rows
  | if ($rows | length) == 0 then "is_compliant=true" else $rows[] end'
Automation
#!/usr/bin/env bash
#
# Remediation: Ensure pods are managed by a controller (C5.2) on OKE
#
# Usage:
#   1) Ensure kubectl is configured (on any machine with kubectl access).
#   2) Run: bash fix-naked-pods.sh
#
# What this does:
#   - Detects "naked" pods (no controlling ownerReference) outside
#     kube-system, kube-public, kube-node-lease.
#   - Generates equivalent controller manifests (Deployment by default)
#     into ./generated-controllers/.
#   - Applies those manifests.
#   - Deletes the naked pods (so the controllers take over).
#   - Re-runs the audit at the end to verify.
#
# IMPORTANT:
#   - This script cannot infer the correct controller type (Deployment vs
#     StatefulSet vs DaemonSet vs Job) from a single pod. It uses
#     Deployment by default.
#   - Review the generated manifests before applying in production.
#   - For stateful or singleton workloads, you may want StatefulSet or
#     Job instead; adjust the generated YAML accordingly before apply.
#   - This is best-effort automation and may not be appropriate for all
#     naked pods. You are responsible for reviewing and approving changes.
set -euo pipefail

# -------- configuration --------

# Default controller kind used to manage previously naked pods.
# Options you might set after review: Deployment / StatefulSet / Job.
DEFAULT_CONTROLLER_KIND="Deployment"

# Directory to store generated manifests
OUT_DIR="./generated-controllers"
mkdir -p "${OUT_DIR}"

# -------- helper functions --------

error() { printf 'ERROR: %s\n' "$*" >&2; }
info()  { printf 'INFO: %s\n' "$*"; }

require_bin() {
  if ! command -v "$1" >/dev/null 2>&1; then
    error "Required binary '$1' not found in PATH"
    exit 1
  fi
}

# -------- prerequisites --------

require_bin kubectl
require_bin jq

# Simple cluster access check
if ! kubectl version --request-timeout=10s >/dev/null 2>&1; then
  error "kubectl cannot reach the cluster or is not authenticated"
  exit 1
fi

info "Detecting naked pods (no controller) in all non-system namespaces..."

# -------- discover naked pods --------

# JSON list of naked pods with minimal fields needed to reconstruct a spec
NAKED_PODS_JSON=$(
  kubectl get pods --all-namespaces -o json \
  | jq '
    .items[]
    | select(.metadata.namespace as $n
             | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
    | . as $pod
    | ((.metadata.ownerReferences // []) | map(select(.controller)) | length) as $ctlCount
    | select($ctlCount == 0)
    | {
        namespace: .metadata.namespace,
        name: .metadata.name,
        labels: (.metadata.labels // {}),
        annotations: (.metadata.annotations // {}),
        containers: .spec.containers,
        initContainers: (.spec.initContainers // []),
        nodeSelector: (.spec.nodeSelector // {}),
        tolerations: (.spec.tolerations // []),
        affinity: (.spec.affinity // {}),
        volumes: (.spec.volumes // []),
        serviceAccountName: (.spec.serviceAccountName // ""),
        restartPolicy: (.spec.restartPolicy // "Always")
      }'
)

if [[ -z "${NAKED_PODS_JSON}" ]] || [[ "${NAKED_PODS_JSON}" == "null" ]]; then
  info "No naked pods found. Cluster is already compliant."
  exit 0
fi

# Count naked pods
NAKED_COUNT=$(printf '%s\n' "${NAKED_PODS_JSON}" | jq -s 'length')
if [[ "${NAKED_COUNT}" -eq 0 ]]; then
  info "No naked pods found. Cluster is already compliant."
  exit 0
fi

info "Found ${NAKED_COUNT} naked pod(s). Generating ${DEFAULT_CONTROLLER_KIND} manifests in ${OUT_DIR}..."

# -------- generate controller manifests --------

# Note: Safe to re-run; files are overwritten.
printf '%s\n' "${NAKED_PODS_JSON}" | jq -c '.' | while read -r POD; do
  NS=$(printf '%s\n' "${POD}" | jq -r '.namespace')
  NAME=$(printf '%s\n' "${POD}" | jq -r '.name')

  # Controller name (avoid clash with existing controller)
  CTRL_NAME="${NAME}-managed"

  FILE="${OUT_DIR}/${NS}-${NAME}-${DEFAULT_CONTROLLER_KIND,,}.yaml"

  info "Generating ${DEFAULT_CONTROLLER_KIND} for Pod ${NS}/${NAME} -> ${CTRL_NAME} (${FILE})"

  # Convert pod labels into a selector; if none, fabricate one.
  LABELS_JSON=$(printf '%s\n' "${POD}" | jq '.labels')
  HAVE_LABELS=$(printf '%s\n' "${LABELS_JSON}" | jq 'length')
  if [[ "${HAVE_LABELS}" -eq 0 ]]; then
    LABELS_JSON='{ "app": "'"${CTRL_NAME}"'" }'
  fi

  # Restart policy only relevant for some controllers; Deployments must use Always.
  RESTART_POLICY=$(printf '%s\n' "${POD}" | jq -r '.restartPolicy')
  if [[ "${DEFAULT_CONTROLLER_KIND}" == "Deployment" ]]; then
    RESTART_POLICY="Always"
  fi

  # Build YAML
  cat > "${FILE}" <<EOF
apiVersion: apps/v1
kind: ${DEFAULT_CONTROLLER_KIND}
metadata:
  name: ${CTRL_NAME}
  namespace: ${NS}
  labels: $(printf '%s\n' "${LABELS_JSON}" | jq -c '.' )
spec:
  replicas: 1
  selector:
    matchLabels: $(printf '%s\n' "${LABELS_JSON}" | jq -c '.' )
  template:
    metadata:
      labels: $(printf '%s\n' "${LABELS_JSON}" | jq -c '.' )
      annotations: $(printf '%s\n' "${POD}" | jq -c '.annotations' )
    spec:
      serviceAccountName: $(printf '%s\n' "${POD}" | jq -r '.serviceAccountName')
      restartPolicy: ${RESTART_POLICY}
      containers: $(printf '%s\n' "${POD}" | jq -c '.containers')
      initContainers: $(printf '%s\n' "${POD}" | jq -c '.initContainers')
      nodeSelector: $(printf '%s\n' "${POD}" | jq -c '.nodeSelector')
      tolerations: $(printf '%s\n' "${POD}" | jq -c '.tolerations')
      affinity: $(printf '%s\n' "${POD}" | jq -c '.affinity')
      volumes: $(printf '%s\n' "${POD}" | jq -c '.volumes')
EOF

done

info "Generated manifests. Review them in ${OUT_DIR} before proceeding."
info "Applying generated controller manifests..."

kubectl apply -f "${OUT_DIR}"

info "Deleting original naked pods so controllers can recreate them..."

# Delete only if pods still exist (idempotent)
printf '%s\n' "${NAKED_PODS_JSON}" | jq -r '.namespace + " " + .name' | while read -r NS NAME; do
  if kubectl get pod "${NAME}" -n "${NS}" >/dev/null 2>&1; then
    info "Deleting naked pod ${NS}/${NAME}"
    kubectl delete pod "${NAME}" -n "${NS}" --wait=false
  else
    info "Pod ${NS}/${NAME} already gone; skipping delete"
  fi
done

info "Waiting briefly for controllers to create replacement pods..."
sleep 15

# -------- verification (re-run audit) --------

info "Re-running compliance check to verify pods are managed by controllers..."

kubectl get pods --all-namespaces -o json | jq -r '
  [ .items[]
  | select(.metadata.namespace as $n | ["kube-system","kube-public","kube-node-lease"] | index($n) | not)
  | .metadata as $m
  | ([ ($m.ownerReferences // [])[] | select(.controller) ] | first) as $own
  | (if $own == null then "non_compliant" else empty end)
  ] as $rows
  | if ($rows | length) == 0 then "All non-system pods are now managed by controllers (is_compliant=true)" else "Some naked pods remain (is_compliant=false)" end
'