Skip to main content

OCI OKE Kubelet Configuration File Ownership Should Be

More Info:​

The kubelet configuration file (typically /etc/kubernetes/kubelet.conf) should be owned by root:root. Incorrect ownership lets unprivileged users read or modify kubelet credentials and bootstrap state.

Risk Level​

High

Address​

Compliance, Security

Compliance Standards​

  • CIS OKE

Triage and Remediation​

Remediation​

Using Console

In OKE you can’t directly “click a button” in the console to change file ownership inside worker nodes; you have to use the console to push a startup script (cloud‑init) via the node pool configuration so each node fixes it on boot.

Below are step‑by‑step instructions to remediate this using only the OCI Console (no manual SSH per node).

1. Identify the kubelet config file path on OKE nodes​

On standard OKE managed node images, the kubelet config is typically at:

  • /var/lib/kubelet/config.yaml
    (and sometimes a secondary config at /etc/kubernetes/kubelet.conf)

You should fix ownership for both, to be safe.

2. Prepare a cloud‑init script (user data)​

You’ll use a cloud‑init script that runs on each node at startup and enforces ownership and permissions.

Example script:

#!/bin/bash
# Fix kubelet config file ownership/permissions

for f in /var/lib/kubelet/config.yaml /etc/kubernetes/kubelet.conf; do
  if [ -f "$f" ]; then
    chown root:root "$f"
    chmod 600 "$f"
  fi
done

# Restart kubelet so it picks up correct permissions if needed
systemctl restart kubelet || true

You’ll paste this into the node pool’s “User data” section in the console.

3. Update an existing Node Pool to apply the script​

  1. In OCI Console, go to:
    • Developer Services → Kubernetes Clusters (OKE).
  2. Click your cluster.
  3. In the cluster details page, go to Node Pools tab.
  4. Click the node pool you want to fix.
  5. Click Edit (or Update Node Pool Configuration, depending on UI).
  6. Find the Node Metadata / Cloud-Init / User Data (or “Boot volume, Metadata” or similar) section.
  7. In User data, select Plain text and paste the script from step 2.
  8. Save the changes.

This ensures any new/replaced nodes in this pool will automatically set the kubelet config file to root:root and 600 on boot.

Note: Existing running nodes won’t automatically re‑run cloud‑init just because you updated user data. You need to cycle them.

4. Recycle nodes so the script runs​

To ensure all current worker nodes pick up the fix:

  1. In the same node pool page, note the Number of nodes.
  2. Option A – Rolling replacement by scaling:
    • Temporarily increase the node count (e.g., from 3 to 4).
    • Wait for the new node(s) to be Active and Ready in the cluster.
    • Then decrease the node count back (e.g., from 4 to 3), which terminates old nodes.
  3. Option B – Delete and let the pool replace nodes (if “Autoscale” or node draining is configured appropriately):
    • From Compute → Instances, filter by the node pool’s compartment and name.
    • Terminate nodes one by one; the node pool will create new ones using the updated user data.

New nodes created after the update will run the script at first boot and fix file ownership.

5. (Optional) Verify via Kubernetes / SSH​

If you can SSH (or via bastion) to a node:

stat -c '%U:%G %a %n' /var/lib/kubelet/config.yaml /etc/kubernetes/kubelet.conf

Expected:

  • Owner: root
  • Group: root
  • Mode: 600 (or at least not group/world writable or readable)

Example output:

root:root 600 /var/lib/kubelet/config.yaml
root:root 600 /etc/kubernetes/kubelet.conf

6. Apply this pattern to all Node Pools​

Repeat steps 3–4 for every node pool in the cluster (and across clusters) so every worker node is remediated consistently.

Summary via OCI Console:

  • Use Node Pool → Edit → User data to add a small boot script (chown root:root + chmod 600 + systemctl restart kubelet).
  • Recycle/replace nodes through node pool scaling so the new config is enforced cluster‑wide.
Using CLI

You fix this on the worker node OS; the OCI CLI is used to target the instances and run a remote command that corrects file ownership.

Below is a minimal, end‑to‑end way using Instance Agent Commands via OCI CLI.

1. Identify the OKE node pool and nodes​

If you don’t already have the node pool OCID:

oci ce node-pool list \
  --compartment-id <COMPARTMENT_OCID> \
  --cluster-id <OKE_CLUSTER_OCID> \
  --all

Then get the node pool details to see the worker nodes:

oci ce node-pool get \
  --node-pool-id <NODEPOOL_OCID> \
  --query "data.nodes[].id" \
  --raw-output

Each node has an associated Compute instance. Get them:

oci ce node get \
  --node-id <NODE_OCID> \
  --query "data.instance-id" \
  --raw-output

Repeat or use --all and a jmespath query to list all instance OCIDs.

2. Ensure Instance Agent is enabled​

On each instance OCID, the Oracle Cloud Agent must be enabled and running (this is the default for OKE images):

oci compute instance get \
  --instance-id <INSTANCE_OCID> \
  --query "data.\"agent-config\""

"is-management-disabled": false and "is-monitoring-disabled": false is what you want.

3. Run the remediation command via OCI CLI​

On OKE worker nodes, kubelet config is typically at:

  • /var/lib/kubelet/config.yaml (most OKE images)
  • or /etc/kubernetes/kubelet.conf (older/custom images)

You can set ownership to root:root for both safely.

For each instance:

oci compute instance-agent command create \
  --instance-id <INSTANCE_OCID> \
  --display-name "Fix kubelet config ownership" \
  --content "$(cat << 'EOF'
{
  "contentType": "TEXT",
  "scriptContent": "#!/bin/bash\n\
set -e\n\
if [ -f /var/lib/kubelet/config.yaml ]; then\n\
  chown root:root /var/lib/kubelet/config.yaml\n\
fi\n\
if [ -f /etc/kubernetes/kubelet.conf ]; then\n\
  chown root:root /etc/kubernetes/kubelet.conf\n\
fi\n"
}
EOF
)"

Notes:

  • The script is run as root by the agent.

  • You can add permissions hardening if desired, e.g.:

    chmod 600 /var/lib/kubelet/config.yaml /etc/kubernetes/kubelet.conf 2>/dev/null || true

4. Verify ownership on each node​

Option A – via another instance-agent command:

oci compute instance-agent command create \
  --instance-id <INSTANCE_OCID> \
  --display-name "Check kubelet ownership" \
  --content "$(cat << 'EOF'
{
  "contentType": "TEXT",
  "scriptContent": "#!/bin/bash\n\
ls -l /var/lib/kubelet/config.yaml 2>/dev/null || true\n\
ls -l /etc/kubernetes/kubelet.conf 2>/dev/null || true\n"
}
EOF
)"

Retrieve output:

oci compute instance-agent command get \
  --instance-agent-command-id <COMMAND_OCID> \
  --query "data.results[].output" \
  --raw-output

You should see root root as owner and group.

Option B – SSH into a node and:

ls -l /var/lib/kubelet/config.yaml /etc/kubernetes/kubelet.conf 2>/dev/null

5. Make it persistent for new nodes​

For future OKE nodes (new node pools or scale‑out), ensure this is applied automatically, e.g.:

  • Use a custom node image with correct ownership baked in, or
  • Add an OKE node pool cloud‑init script that fixes ownership on boot:
#cloud-config
runcmd:
  - chown root:root /var/lib/kubelet/config.yaml 2>/dev/null || true
  - chown root:root /etc/kubernetes/kubelet.conf 2>/dev/null || true

Then attach this via --node-config-details / --node-metadata (cloud-init user-data) when creating the node pool.

Using Python

In OKE you can’t SSH into every node manually at scale, so the standard pattern is:

  1. Run a privileged DaemonSet that:
    • Mounts the host’s /var/lib/kubelet/config.yaml (or the kubelet config directory).
    • Runs chown root:root and chmod 600 (or whatever you require) on the host file.
  2. Create/apply that DaemonSet using Python (either via kubectl or the Kubernetes Python client).

Below is a minimal, end‑to‑end example using Python and the Kubernetes Python client.

1. Identify kubelet config path on OKE nodes​

On OKE node pools today it is typically:

/var/lib/kubelet/config.yaml

Confirm on one node (via Cloud Shell + ssh into the node, or via an existing debug DaemonSet):

sudo ls -l /var/lib/kubelet/config.yaml

Use that path in the script below.

2. Python script to deploy a DaemonSet that fixes ownership​

This script:

  • Connects to your OKE cluster using local kubeconfig.
  • Creates a DaemonSet in kube-system that:
    • Runs privileged.
    • Mounts the host’s /var/lib/kubelet/config.yaml.
    • Executes chown root:root /host/kubelet/config.yaml && chmod 600 /host/kubelet/config.yaml.

After it runs on all nodes once, you can optionally delete the DaemonSet.

from kubernetes import client, config

NAMESPACE = "kube-system"
DAEMONSET_NAME = "fix-kubelet-config-perms"
KUBELET_CONFIG_HOST_PATH = "/var/lib/kubelet/config.yaml"
CONTAINER_IMAGE = "busybox:1.36"   # any tiny image with chown/chmod/sh

def main():
    # Load kubeconfig (e.g., ~/.kube/config configured for your OKE cluster)
    config.load_kube_config()

    apps_v1 = client.AppsV1Api()

    # Define Pod spec
    container = client.V1Container(
        name="fix-kubelet-config",
        image=CONTAINER_IMAGE,
        security_context=client.V1SecurityContext(
            privileged=True,
        ),
        command=[
            "sh",
            "-c",
            (
                "if [ -f /host/kubelet/config.yaml ]; then "
                "  chown root:root /host/kubelet/config.yaml && "
                "  chmod 600 /host/kubelet/config.yaml; "
                "else "
                "  echo 'config.yaml not found'; "
                "fi; "
                "sleep 3600"
            ),
        ],
        volume_mounts=[
            client.V1VolumeMount(
                name="kubelet-config",
                mount_path="/host/kubelet/config.yaml",
                read_only=False,
            )
        ],
    )

    # HostPath volume pointing to kubelet config on host
    volume = client.V1Volume(
        name="kubelet-config",
        host_path=client.V1HostPathVolumeSource(
            path=KUBELET_CONFIG_HOST_PATH,
            type="File"
        ),
    )

    pod_spec = client.V1PodSpec(
        containers=[container],
        restart_policy="Always",
        host_network=True,  # not strictly required but common for node utilities
        tolerations=[
            client.V1Toleration(
                operator="Exists"
            )
        ],
        volumes=[volume],
    )

    # Pod template
    template = client.V1PodTemplateSpec(
        metadata=client.V1ObjectMeta(labels={"app": DAEMONSET_NAME}),
        spec=pod_spec,
    )

    # DaemonSet spec
    ds_spec = client.V1DaemonSetSpec(
        selector=client.V1LabelSelector(
            match_labels={"app": DAEMONSET_NAME}
        ),
        template=template,
    )

    daemonset = client.V1DaemonSet(
        api_version="apps/v1",
        kind="DaemonSet",
        metadata=client.V1ObjectMeta(
            name=DAEMONSET_NAME,
            namespace=NAMESPACE,
        ),
        spec=ds_spec,
    )

    # Create or replace DaemonSet
    try:
        existing = apps_v1.read_namespaced_daemon_set(
            name=DAEMONSET_NAME,
            namespace=NAMESPACE,
        )
        print(f"DaemonSet {DAEMONSET_NAME} exists, replacing...")
        daemonset.metadata.resource_version = existing.metadata.resource_version
        apps_v1.replace_namespaced_daemon_set(
            name=DAEMONSET_NAME,
            namespace=NAMESPACE,
            body=daemonset,
        )
    except client.exceptions.ApiException as e:
        if e.status == 404:
            print(f"Creating DaemonSet {DAEMONSET_NAME}...")
            apps_v1.create_namespaced_daemon_set(
                namespace=NAMESPACE,
                body=daemonset,
            )
        else:
            raise

    print("DaemonSet applied. Wait until all pods are Running, then verify and optionally delete it.")

if __name__ == "__main__":
    main()

3. Verify the remediation​

After the DaemonSet pods are Running on all nodes:

# On each node (or via another debug DaemonSet):
sudo ls -l /var/lib/kubelet/config.yaml
# Should show: -rw------- 1 root root ...

When you’re satisfied:

kubectl -n kube-system delete daemonset fix-kubelet-config-perms

Notes​

  • Adjust KUBELET_CONFIG_HOST_PATH if your OKE version uses a different location.
  • If your policy requires different permissions (e.g., 640), change the chmod value in the command.
  • If you prefer, replace the sleep 3600 with sleep infinity or keep the DaemonSet only long enough to fix the files, then delete it.
Using Terraform
# There is no Terraform argument on oci_containerengine_node_pool (or any other
# OCI Container Engine resource) that controls ownership of /etc/kubernetes/kubelet.conf.
# This setting is inside the node OS and cannot be remediated at the node pool API level.

# You must enforce root:root ownership using:
# - A custom node image where kubelet.conf permissions are pre-hardened, or
# - Cloud-init / bootstrap scripts baked into that image or into your instance config.

# Example (conceptual) node pool definition; note there is no way to set kubelet.conf ownership here.

resource "oci_containerengine_node_pool" "OKE_NODEPOOL" {
  compartment_id = "OCID_OF_COMPARTMENT"
  cluster_id     = "OCID_OF_CLUSTER"
  name           = "NODEPOOL_NAME"
  kubernetes_version = "K8S_VERSION"

  node_shape = "VM.Standard3.Flex"

  node_source_details {
    source_type = "IMAGE"
    image_id    = "OCID_OF_HARDENED_CUSTOM_IMAGE" # Image where kubelet.conf is already owned by root:root
  }

  node_config_details {
    size = 3
    placement_configs {
      availability_domain = "AVAILABILITY_DOMAIN"
      subnet_id           = "OCID_OF_SUBNET"
    }
  }
}

# This finding cannot be fixed directly by Terraform on the oci_containerengine_node_pool resource.
# Use the Console or custom image build pipeline to ensure /etc/kubernetes/kubelet.conf is owned by root:root
# in the node OS. Then recreate/roll the node pool to pick up the hardened image.

# Verification: `terraform plan` will show no changes specifically about kubelet.conf ownership,
# because that configuration is outside Terraform’s OCI provider surface.