OCI OKE kubelet-config.json File Permissions Should Be 644

More Info:

The kubelet-config.json file should have permissions of 644 or more restrictive. Loose permissions allow unprivileged users to read or modify kubelet configuration on the node.

Risk Level

High

Address

Compliance, Security

Compliance Standards

• CIS OKE

Triage and Remediation

Remediation

Remediation

Using Console

In OKE you can’t click a “fix permissions” button in the OCI Console, but you can remediate this cluster‑wide using the Console to deploy a DaemonSet that corrects the file permissions on every worker node.

Below are step‑by‑step instructions using only the OCI Console (plus the built‑in editor).

---

1. Open your OKE cluster in the OCI Console

1. Sign in to OCI Console.
2. In the left menu: Developer Services → Kubernetes Clusters (OKE).
3. Select the Compartment that contains your cluster.
4. Click on your cluster name to open its details page.

---

2. Use the Console Workloads UI to create a DaemonSet

1. In the cluster details page, go to the Workloads tab (or Workloads → DaemonSets depending on your UI version).
2. Click Create → choose DaemonSet (or Create resource and select DaemonSet).
3. Switch to the YAML editor view.
4. Paste the following DaemonSet manifest, adjusting the namespace if desired (default is fine):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fix-kubelet-config-perms
  namespace: default
spec:
  selector:
    matchLabels:
      app: fix-kubelet-config-perms
  template:
    metadata:
      labels:
        app: fix-kubelet-config-perms
    spec:
      hostPID: true
      hostNetwork: true
      tolerations:
      - operator: "Exists"
      containers:
      - name: chmod-kubelet-config
        image: alpine:3.18
        securityContext:
          privileged: true
        command:
        - /bin/sh
        - -c
        - |
          set -e
          # Adjust path if needed for your OKE image:
          if [ -f /host/etc/oci-oke/kubelet-config.json ]; then
            chmod 644 /host/etc/oci-oke/kubelet-config.json
          elif [ -f /host/etc/kubernetes/kubelet-config.json ]; then
            chmod 644 /host/etc/kubernetes/kubelet-config.json
          fi
          # Sleep long enough to ensure the change is applied, then exit so the pod can be removed later
          sleep 60
        volumeMounts:
        - name: host-etc
          mountPath: /host/etc
      restartPolicy: Always
      volumes:
      - name: host-etc
        hostPath:
          path: /etc
          type: Directory

5. Click Create.

This will schedule one privileged pod on each node. Each pod will:

• Mount the node’s /etc directory.
• Run chmod 644 on kubelet-config.json (common OKE paths checked).
• Exit after a short delay.

---

3. Confirm the permissions

1. In the Console, go to the Workloads → Pods view (or DaemonSets → Pods) and wait until all fix-kubelet-config-perms-* pods show as Running and then Completed or Terminated.

2. Use Cloud Shell or your own kubectl (with the cluster’s kubeconfig) to open a debug pod and check one node:

   kubectl get nodes -o wide
   # Pick a node name, then:
   kubectl debug node/<NODE_NAME> -it --image=alpine:3.18 -- chroot /host sh
   # Now inside the node filesystem:
   ls -l /etc/oci-oke/kubelet-config.json || ls -l /etc/kubernetes/kubelet-config.json
   # Expect: -rw-r--r-- (0644)
   exit

---

4. Clean up the DaemonSet

Once you’ve verified the permissions:

1. In the Console, go back to Workloads → DaemonSets.
2. Click the fix-kubelet-config-perms DaemonSet.
3. Click Delete to remove it (the permissions it set on the nodes will remain).

---

5. Make it persistent for future nodes (optional)

To ensure new worker nodes also get 0644:

• Update your node pool to use a custom cloud‑init / user‑data script that runs:

  chmod 644 /etc/oci-oke/kubelet-config.json 2>/dev/null || \
  chmod 644 /etc/kubernetes/kubelet-config.json 2>/dev/null

You configure that in the node pool’s Node configuration → Advanced options / Custom bootstrap script section in the OCI Console.

Using CLI

To remediate this using OCI CLI, you’ll typically do it non-interactively via the Compute Instance Agent “run-command” feature on each worker node in the OKE node pool.

Below are the minimal steps.

---

1. Identify the OKE node pool and worker nodes

If you already know the node pool OCID, skip to step 2.

# List OKE clusters (to find the cluster OCID)
oci ce cluster list --compartment-id <COMPARTMENT_OCID>

# List nodepools in the cluster
oci ce node-pool list \
  --compartment-id <COMPARTMENT_OCID> \
  --cluster-id <CLUSTER_OCID>

# Get details of the nodepool (to see instance/instance-pool IDs)
oci ce node-pool get --node-pool-id <NODEPOOL_OCID>

From the node pool details, note the instancePoolId or instance IDs depending on node pool type.

If it’s a Compute Instance Pool:

oci compute-management instance-pool list-instances \
  --compartment-id <COMPARTMENT_OCID> \
  --instance-pool-id <INSTANCE_POOL_OCID> \
  --all

This returns the worker instance OCIDs.

---

2. Prepare the chmod command payload

Assuming the file is at /var/lib/kubelet/kubelet-config.json (adjust path if needed):

{
  "commands": [
    "sudo chown root:root /var/lib/kubelet/kubelet-config.json || true",
    "sudo chmod 644 /var/lib/kubelet/kubelet-config.json"
  ]
}

Save this as kubelet-perms.json.

---

3. Run the command on each worker node via OCI CLI

For each worker instance OCID:

oci compute instance-agent command execute \
  --instance-id <INSTANCE_OCID> \
  --endpoint-id <INSTANCE_AGENT_ENDPOINT_OCID-or-omit-if-default> \
  --content file://kubelet-perms.json \
  --execution-mode PARALLEL \
  --wait-for-state SUCCEEDED

If you don’t specify --endpoint-id, the default instance agent endpoint is used. Ensure:

• Compute Instance Agent is enabled on the worker image.
• IAM policy allows INSTANCE_AGENT_COMMAND_EXECUTION for your principal.

Example IAM policy (for reference):

Allow group <GROUP_NAME> to use instance-family in compartment <COMPARTMENT_NAME>
Allow group <GROUP_NAME> to use instance-agent-command-execution-family in compartment <COMPARTMENT_NAME>

---

4. Verify permissions on each node

You can either:

1. Use another instance-agent command:

   # verification payload: kubelet-perms-check.json
   {
     "commands": [
       "ls -l /var/lib/kubelet/kubelet-config.json"
     ]
   }

   Then:

   oci compute instance-agent command execute \
     --instance-id <INSTANCE_OCID> \
     --content file://kubelet-perms-check.json \
     --execution-mode PARALLEL \
     --wait-for-state SUCCEEDED

   And inspect the output via:

   oci compute instance-agent command get \
     --instance-agent-command-id <COMMAND_OCID_FROM_PREVIOUS_OUTPUT>

2. Or SSH into a worker and run:

   ls -l /var/lib/kubelet/kubelet-config.json

Expected:

-rw-r--r-- 1 root root ... /var/lib/kubelet/kubelet-config.json

---

5. Make it persistent for future nodes (optional but recommended)

If you use custom images or custom node configuration scripts, ensure the image or bootstrap script sets:

sudo chown root:root /var/lib/kubelet/kubelet-config.json || true
sudo chmod 644 /var/lib/kubelet/kubelet-config.json || true

For example, in a cloud-init or node boot script referenced by the node pool.

---

If you provide your node pool type (managed/unmanaged, image type), I can give you the exact oci commands tailored to your configuration.

Using Python

Below is one practical way to remediate this in OKE using Python: connect (via SSH) to each worker node and correct the file’s permissions.

Assumptions (adjust as needed):

• File path: /etc/oci/kubelet/kubelet-config.json
• You have SSH access to worker nodes (via a bastion or public IP).
• You have OCI API credentials (config file ~/.oci/config) and know the OCID of the OKE cluster or node pool.

---

1. High-level remediation steps

1. Identify the worker nodes of your OKE cluster / node pool.
2. Get their IP addresses (public or private through a bastion).
3. SSH to each node.
4. Change the file permission to 644:

   sudo chmod 644 /etc/oci/kubelet/kubelet-config.json

5. (Optional but recommended) Enforce this automatically using:
   • A DaemonSet, or
   • A custom image / cloud-init script for future nodes.

---

2. Python example using OCI SDK + SSH (paramiko)

Install requirements:

pip install oci paramiko

2.1. Script to:

• List instances for a given node pool
• SSH to each node
• Run chmod 644 on kubelet-config.json

import oci
import paramiko

# ---------- CONFIGURE THESE ----------
OCI_PROFILE = "DEFAULT"                   # profile in ~/.oci/config
NODE_POOL_OCID = "ocid1.nodepool.oc1..."  # your node pool OCID
SSH_USERNAME = "opc"                      # or ubuntu / other image user
SSH_KEY_PATH = "/path/to/private_key"     # your SSH private key
FILE_PATH = "/etc/oci/kubelet/kubelet-config.json"
# -------------------------------------


def get_nodepool_instances():
    # Load OCI config
    config = oci.config.from_file("~/.oci/config", OCI_PROFILE)
    container_engine_client = oci.container_engine.ContainerEngineClient(config)
    compute_client = oci.core.ComputeClient(config)
    network_client = oci.core.VirtualNetworkClient(config)

    # 1) Get node pool details to obtain instance IDs
    nodepool = container_engine_client.get_node_pool(NODE_POOL_OCID).data

    instance_ids = []
    for node in nodepool.nodes:
        if node.instance_id:
            instance_ids.append(node.instance_id)

    # 2) Resolve instance -> IP
    instances_info = []
    for instance_id in instance_ids:
        inst = compute_client.get_instance(instance_id).data
        vnic_attachments = oci.pagination.list_call_get_all_results(
            compute_client.list_vnic_attachments,
            compartment_id=inst.compartment_id,
            instance_id=instance_id
        ).data

        for vnic_attachment in vnic_attachments:
            vnic = network_client.get_vnic(vnic_attachment.vnic_id).data

            # Prefer public IP if present; otherwise, use private IP (through bastion/VPN)
            ip_address = vnic.public_ip or vnic.private_ip
            if ip_address:
                instances_info.append({
                    "instance_id": instance_id,
                    "ip": ip_address
                })
                break

    return instances_info


def chmod_file_on_node(ip, username, key_path, file_path, mode="644"):
    """SSH to node and run chmod."""
    print(f"Connecting to {ip} ...")
    key = paramiko.RSAKey.from_private_key_file(key_path)

    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect(ip, username=username, pkey=key, timeout=15)

    cmd = f"sudo chmod {mode} {file_path}"
    stdin, stdout, stderr = ssh.exec_command(cmd)
    exit_status = stdout.channel.recv_exit_status()

    if exit_status == 0:
        print(f"[OK] {file_path} on {ip} set to {mode}")
    else:
        print(f"[FAIL] {file_path} on {ip}. Exit code: {exit_status}")
        print("stderr:", stderr.read().decode())

    ssh.close()


def main():
    nodes = get_nodepool_instances()
    if not nodes:
        print("No instances found for the node pool.")
        return

    for node in nodes:
        chmod_file_on_node(
            ip=node["ip"],
            username=SSH_USERNAME,
            key_path=SSH_KEY_PATH,
            file_path=FILE_PATH,
            mode="644"
        )


if __name__ == "__main__":
    main()

---

3. Optional: Make the fix persistent

For long-term compliance:

• Cloud-init / custom image:
  Bake into your node image or cloud-init:

  echo "sudo chmod 644 /etc/oci/kubelet/kubelet-config.json" | sudo tee /usr/local/bin/fix-kubelet-perms.sh
  sudo chmod +x /usr/local/bin/fix-kubelet-perms.sh
  echo "@reboot root /usr/local/bin/fix-kubelet-perms.sh" | sudo tee -a /etc/crontab

• DaemonSet:
  Create a privileged DaemonSet that runs a small container to chmod 644 the hostPath-mounted file on every node at startup.

If you tell me your exact file path (from the scanner output) and your connectivity pattern (public IP vs bastion), I can tailor the script further.

Using Terraform

resource "oci_containerengine_node_pool" "OKE_NODEPOOL" {
  # SUBSTITUTE: Your existing node pool settings
  compartment_id     = "OCID_OF_COMPARTMENT"
  cluster_id         = "OCID_OF_CLUSTER"
  kubernetes_version = "VERSION"
  name               = "OKE_NODEPOOL_NAME"
  subnet_ids         = ["OCID_OF_SUBNET"]

  node_shape = "VM.SHAPE"

  node_source_details {
    source_type             = "IMAGE"
    image_id                = "OCID_OF_IMAGE"
    boot_volume_size_in_gbs = 50
  }

  node_config_details {
    size = 3

    placement_configs {
      availability_domain = "AVAILABILITY_DOMAIN_NAME"
      subnet_id           = "OCID_OF_SUBNET"
    }
  }

  # Ensure kubelet-config.json permissions are 0644 via cloud-init
  # NOTE: Changing node_metadata typically triggers a rolling replacement
  # of the nodes in this pool (potential service disruption).
  node_metadata = {
    user_data = base64encode(<<-EOT
      #cloud-config
      write_files:
        - path: /usr/local/bin/fix-kubelet-config-perms.sh
          permissions: '0755'
          owner: root:root
          content: |
            #!/bin/bash
            set -e
            FILE="/etc/kubernetes/kubelet/kubelet-config.json"
            if [ -f "$FILE" ]; then
              chmod 0644 "$FILE"
            fi
      runcmd:
        - [ /usr/local/bin/fix-kubelet-config-perms.sh ]
      EOT
    )
  }

  # ... any other existing arguments you already use ...
}

This uses node_metadata.user_data cloud-init to enforce 0644 on kubelet-config.json on every node in the pool; updating this usually causes the existing worker VMs to be recreated, so plan for a rolling disruption.

To verify, terraform plan should show an in-place update to oci_containerengine_node_pool.OKE_NODEPOOL with a change in the node_metadata.user_data field (and no other unrelated changes).