Skip to main content

Audit Logging Should Be Enabled And Shipped Off-Cluster

More Info:​

Advisory: Kubernetes API audit logging should be enabled and forwarded to an external, tamper-resistant store so control-plane activity is retained independently of the cluster.

Risk Level​

Medium

Address​

Security

Compliance Standards​

  • Cloudanix Best Practice

Triage and Remediation​

Remediation​

Manual Steps

  1. Identify your environment and provider-specific audit feature

    • Determine whether this is EKS, AKS, GKE, OKE, or another managed service, and locate that provider’s Kubernetes API/audit logging feature in its docs or console (e.g., “control plane logging”, “master API audit logs”, “Kubernetes audit logs”).
    • If you use IaC (Terraform/CloudFormation/ARM/Deployment Manager), locate the module or stack that defines the cluster.

  2. Check if API/audit logging is enabled (console and IaC)

    • In the cloud console, open the cluster’s control-plane/monitoring/logging settings and verify whether Kubernetes API/audit logs are enabled and which log types are selected (API server, audit, controller manager, scheduler, etc.).
    • In your IaC code, verify the logging blocks match the console and ensure audit/API logs are enabled there as the source of truth (for example, look for fields like enable_kubernetes_audit_logs, cluster_logging, control_plane_logging, or similar).
    • If audit/API logging is disabled or only partially enabled, plan to enable full API/audit logging via the same mechanism you normally use (preferably IaC, then reconcile manually in the console).

  3. Verify logs are being produced by the control plane

    • From any machine with access to your logging backend (or via the provider log viewer), confirm that new Kubernetes API/audit events appear when you perform actions in the cluster (e.g., list pods, create/delete a test resource).
    • Example test: on any machine with kubectl access, run: 
      kubectl get pods -A >/dev/null
      Then in the logging backend or log viewer for the control-plane/audit logs, search for recent entries involving the get verb on pods from your user or IP.

  4. Assess whether logs are shipped to an external, tamper-resistant store

    • Determine the current storage location: confirm whether logs are only in an in-console log viewer or short-lived provider storage, or if they are exported to an external system (e.g., SIEM, separate log account/project, object storage with write-once / immutability / restricted access).
    • In the provider or logging service UI/CLI, review any configured log sinks/exports (e.g., “log sinks”, “subscriptions”, “streams”, “export rules”) to see where Kubernetes audit/API logs are being sent.

  5. Configure or strengthen off-cluster forwarding and retention

    • If Kubernetes API/audit logs are not forwarded off-cluster, create or update a log export/sink/stream to send them to an external, centralized, and access-controlled destination (e.g., dedicated logging account/project, immutable bucket, or SIEM).
    • In that destination, configure:
      • Retention to meet your policy.
      • Restricted write-only/append-only access for the logging pipeline.
      • Read access limited to appropriate security/operations roles.
    • Ensure this configuration is captured in your IaC (e.g., log sink resources, IAM/policy definitions, storage lifecycle rules).

  6. Re‑verify end-to-end and document

    • Trigger a new API action (e.g., create/delete a test namespace with kubectl) and confirm that a corresponding audit/API log record appears in the external store, not just the provider console.
    • Record in your runbook: how audit logging is enabled, where logs are exported, retention settings, and who has access; schedule periodic review to ensure the configuration and exports remain in place.
Using kubectl

kubectl cannot enable or configure Kubernetes API audit logging, nor can it direct logs to an external store; these settings are only available in your cloud provider’s managed control-plane configuration (console, CLI, or IaC). Refer to the Manual Steps section for provider-specific guidance on enabling audit logging and shipping it off-cluster.

Automation
#!/usr/bin/env bash
#
# audit-api-audit-logging.sh
#
# Purpose:
#   Gather evidence about Kubernetes API audit logging and whether
#   audit logs are being shipped off-cluster, for manual review.
#
# Run on:
#   Any machine with:
#     - kubectl configured for the cluster
#     - access to cloud CLI (optional but recommended: aws / az / gcloud / oci)
#
# Usage:
#   chmod +x audit-api-audit-logging.sh
#   ./audit-api-audit-logging.sh > audit-api-audit-logging-$(date +%F).txt

set -euo pipefail

echo "=== Audit: Kubernetes API Audit Logging & Off-Cluster Shipping ==="
echo "Timestamp: $(date -Iseconds)"
echo

echo "Cluster info (for context)"
echo "------------------------------------------------------------"
kubectl version --short 2>/dev/null || echo "kubectl version: unable to query (check kubeconfig)"
echo
kubectl cluster-info 2>/dev/null || echo "kubectl cluster-info: unable to query (check permissions)"
echo

###############################################################################
# 1. Managed-control-plane detection (best-effort, for operator context)
###############################################################################

echo "Control plane type detection (best effort)"
echo "------------------------------------------------------------"
# Try common hints but do NOT assume result is authoritative.
PROVIDER_HINT="unknown"

if kubectl get ns kube-system >/dev/null 2>&1; then
  if kubectl get configmap -n kube-system aws-auth >/dev/null 2>&1; then
    PROVIDER_HINT="EKS"
  elif kubectl get nodes -o jsonpath='{.items[0].metadata.labels.eks\.amazonaws\.com/nodegroup}' >/dev/null 2>&1; then
    PROVIDER_HINT="EKS"
  elif kubectl get nodes -o jsonpath='{.items[0].metadata.labels.cloud\.google\.com/gke-nodepool}' >/dev/null 2>&1; then
    PROVIDER_HINT="GKE"
  elif kubectl get nodes -o jsonpath='{.items[0].metadata.labels.agentpool}' >/dev/null 2>&1; then
    PROVIDER_HINT="AKS"
  fi
fi

echo "Provider hint (heuristic, verify manually): ${PROVIDER_HINT}"
echo

###############################################################################
# 2. In-cluster evidence of API audit logging usage
###############################################################################

echo "In-cluster evidence related to API audit logging"
echo "------------------------------------------------------------"

echo
echo "2.1. Check for namespaces and components commonly used for log shipping"
echo "-----------------------------------------------------------------"
kubectl get ns 2>/dev/null | sed 's/^/  /' || echo "  Unable to list namespaces"

echo
echo "2.2. Check for logging / audit shipping DaemonSets (cluster-wide)"
echo "-----------------------------------------------------------------"
kubectl get daemonset -A 2>/dev/null | sed 's/^/  /' || echo "  Unable to list DaemonSets"

echo
echo "2.3. Check for logging / audit shipping Deployments (cluster-wide)"
echo "-----------------------------------------------------------------"
kubectl get deploy -A 2>/dev/null | sed 's/^/  /' || echo "  Unable to list Deployments"

echo
echo "2.4. Look for components suggesting API server audit ingestion"
echo "-----------------------------------------------------------------"
echo "  (grep for 'audit' in Deployments/DaemonSets names)"
{
  kubectl get deploy -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name --no-headers 2>/dev/null
  kubectl get daemonset -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name --no-headers 2>/dev/null
} | grep -i "audit" || echo "  No Deployments/DaemonSets with 'audit' in the name found (this may be OK)."

###############################################################################
# 3. Provider-specific hints (manual review required)
###############################################################################
# NOTE:
#   For managed control planes, *enabling API audit logging and off-cluster
#   shipping* is configured in the cloud provider console / API / IaC.
#   The following commands only help you discover current logging-related
#   config; they do not guarantee that audit logging is properly enabled.
###############################################################################

echo
echo "Provider-specific evidence (you must review & interpret manually)"
echo "------------------------------------------------------------"

case "${PROVIDER_HINT}" in
  "EKS")
    echo "Detected EKS (heuristic)."
    echo
    echo "3.1. List EKS clusters and logging configuration via AWS CLI"
    echo "     (Run these on a machine with 'aws' configured and proper IAM perms)"
    cat <<'EOF'
# List clusters
aws eks list-clusters

# For each cluster, show enabled control-plane logs, including audit
aws eks describe-cluster \
  --name <CLUSTER_NAME> \
  --query 'cluster.logging.clusterLogging[*].{types:types, enabled:enabled}'
EOF
    echo
    echo "Interpretation (problem indicators):"
    echo "  - If 'audit' is NOT present in 'types' with 'enabled: true',"
    echo "    then API audit logging is not enabled for that EKS cluster."
    echo "  - Even when enabled, confirm that CloudWatch Logs or another"
    echo "    external, tamper-resistant destination retains logs according"
    echo "    to your policy (verify in AWS console/IaC)."
    ;;
  "GKE")
    echo "Detected GKE (heuristic)."
    echo
    echo "3.1. List GKE clusters and logging config via gcloud"
    echo "     (Run these on a machine with 'gcloud' configured and proper perms)"
    cat <<'EOF'
# List clusters with logging details
gcloud container clusters list \
  --format="table(name,location,loggingService,loggingConfig)"

# For a specific cluster, get full logging config (includes audit logs)
gcloud container clusters describe <CLUSTER_NAME> \
  --region <REGION> \
  --format="yaml(loggingConfig,loggingService)"
EOF
    echo
    echo "Interpretation (problem indicators):"
    echo "  - If 'loggingService' is 'none', control-plane logs (including audit)"
    echo "    are not being exported."
    echo "  - If 'loggingConfig.componentConfig.enableComponents' does NOT include"
    echo "    'APISERVER', then API server logs (including audits) may not be"
    echo "    collected."
    echo "  - Confirm logs are sent to Cloud Logging with retention and access"
    echo "    controls that make them tamper-resistant."
    ;;
  "AKS")
    echo "Detected AKS (heuristic)."
    echo
    echo "3.1. List AKS clusters and diagnostic settings via Azure CLI"
    echo "     (Run these on a machine with 'az' configured and proper perms)"
    cat <<'EOF'
# List AKS clusters
az aks list -o table

# For a given cluster, show diagnostic settings on the managed resource
AKS_RG="<RESOURCE_GROUP>"
AKS_NAME="<CLUSTER_NAME>"

# Get the underlying managed resource id
AKS_ID=$(az aks show -g "$AKS_RG" -n "$AKS_NAME" --query id -o tsv)

# List diagnostic settings (includes control-plane / audit categories if enabled)
az monitor diagnostic-settings list --resource "$AKS_ID" -o json
EOF
    echo
    echo "Interpretation (problem indicators):"
    echo "  - If there are NO diagnostic settings for the AKS resource, audit"
    echo "    logs are not being shipped off-cluster."
    echo "  - In the diagnostic settings JSON, if categories like 'kube-audit'"
    echo "    or equivalent are absent or disabled, API audit logging is not"
    echo "    being exported."
    echo "  - Confirm destination is Log Analytics / Event Hub / storage with"
    echo "    appropriate tamper-resistance and retention."
    ;;
  *)
    echo "Provider unknown from heuristics."
    echo
    echo "3.1. MANUAL: Determine your control-plane provider and review:"
    echo "  - Platform documentation for 'Kubernetes API audit logging'"
    echo "  - Control-plane logging / diagnostics settings in the provider console"
    echo "  - IaC definitions (Terraform, CloudFormation, ARM/Bicep, etc.)"
    echo
    echo "Example manual evidence commands (generic; adjust for your platform):"
    cat <<'EOF'
# If using Terraform:
grep -Rni "audit" . | head

# If using Helm or other IaC for logging stacks:
grep -Rni "audit" logging/ manifests/ | head
EOF
    ;;
esac

###############################################################################
# 4. What output indicates a problem?
###############################################################################

echo
echo "Interpretation summary: what indicates a potential PROBLEM"
echo "------------------------------------------------------------"
cat <<'EOF'
Flag as needing remediation if you observe ANY of the following:

1) EKS:
   - In 'aws eks describe-cluster ...':
     - 'audit' is missing from 'cluster.logging.clusterLogging[*].types'
       OR
     - 'audit' exists but 'enabled' is false.
   - No clear CloudWatch Logs / external destination configured for control-plane logs.

2) GKE:
   - 'loggingService' is 'none' for the cluster.
   - 'loggingConfig.componentConfig.enableComponents' does NOT include 'APISERVER'.
   - Logs are not visible in Cloud Logging or retention is too short / not protected.

3) AKS:
   - 'az monitor diagnostic-settings list --resource <AKS_ID>' returns no settings.
   - Diagnostic settings do not include categories for API server / audit logs
     (e.g., 'kube-audit' or equivalent).
   - Destination (Log Analytics, Event Hub, Storage) is missing or clearly
     not tamper-resistant.

4) Any provider:
   - No evidence that API-level audit logs are being produced by the control plane.
   - No evidence that such logs are being shipped to an external, independent,
     tamper-resistant log store.
   - IaC / configuration does not define audit logging or external log sinks.

NOTE:
  This script only gathers evidence; it does NOT and CANNOT fully
  determine compliance automatically. You must review the outputs
  and your provider's configuration to decide if:
    - Kubernetes API audit logging is enabled, AND
    - Audit logs are being forwarded to an external, tamper-resistant store.
EOF

echo
echo "=== End of audit ==="