Audit Logging Should Be Enabled And Shipped Off-Cluster

More Info:

Advisory: Kubernetes API audit logging should be enabled and forwarded to an external, tamper-resistant store so control-plane activity is retained independently of the cluster.

Risk Level

Medium

Address

Security

Compliance Standards

Cloudanix Best Practice

Triage and Remediation

Remediation

Remediation

Manual Steps

Confirm control-plane audit logging is enabled for the cluster
- From any machine with gcloud access:
```
gcloud container clusters describe CLUSTER_NAME \
  --region=CLUSTER_REGION \
  --format="yaml(binaryAuthorization,loggingConfig,loggingService)"
```
- In the output, check that loggingConfig.componentConfig.enableComponents includes APISERVER. If not, or if loggingService is none, API audit logs are not being exported and you should plan to enable logging when creating/updating the cluster (via console, gcloud, or Terraform).
Verify audit logs are being produced in Cloud Logging
- In the Google Cloud console, go to Logging → Logs Explorer.
- Run this query, adjusting the project/cluster/region:
```
resource.type="k8s_cluster"
resource.labels.cluster_name="CLUSTER_NAME"
resource.labels.location="CLUSTER_REGION"
log_id("cloudaudit.googleapis.com/activity") OR
log_id("cloudaudit.googleapis.com/system_event") OR
log_id("k8s.io/apiserver")
```
- Confirm you see recent entries that clearly come from the Kubernetes API server (e.g., verbs like create, update, delete on Kubernetes resources).
Decide and configure which audit logs to retain and export off‑cluster
- In Logs Explorer, refine the query to exactly the API logs you consider security‑relevant (e.g., by protoPayload.methodName, resource.labels.namespace_name, etc.).
- Use the “Create sink” button (or gcloud logging sinks) to define an export based on this filter, targeting one of:
  - Cloud Storage bucket (WORM‑configured if needed),
  - BigQuery dataset, or
  - Pub/Sub topic feeding an external SIEM.
- Ensure the sink is configured at the appropriate scope (project, folder, or organization) to capture all relevant clusters.
Ensure the destination is tamper‑resistant
- For a Cloud Storage sink:
  - Check bucket IAM and confirm only restricted identities can storage.objects.delete or storage.objects.update.
  - If required, enable Object Versioning and/or Bucket Lock (retention policies and holds).
- For BigQuery:
  - Restrict bigquery.tables.update and bigquery.tables.delete.
  - Optionally configure table‑level retention policies.
- For Pub/Sub or external SIEM:
  - Review IAM on topics/subscriptions and the downstream system’s retention/immutability features.
If logging or export is missing/incomplete, adjust cluster and logging configuration
- To enable or adjust Kubernetes logging for an existing cluster using gcloud (any machine with gcloud access):
```
gcloud container clusters update CLUSTER_NAME \
  --region=CLUSTER_REGION \
  --logging=SYSTEM,WORKLOAD,API_SERVER
```
- Revisit Step 3 to (re)create or refine logging sinks so all API server audit logs are exported to the selected external store with appropriate retention.
Re‑verify that audit logs are flowing off‑cluster
- Trigger a simple API action (from any machine with kubectl access):
```
kubectl get pods -A
```
- In Logs Explorer, confirm a new corresponding audit entry appears and that it also shows up in the external destination (Cloud Storage objects, BigQuery rows, or downstream SIEM), validating that Kubernetes API audit logging is enabled and shipped off‑cluster.

Using kubectl

kubectl cannot configure Kubernetes API audit logging or where GKE control-plane logs are sent; this is managed entirely through GKE cluster logging settings in the Google Cloud Console, gcloud CLI, or IaC. To enable and ship audit logs off-cluster, adjust those cloud-provider settings as described in the Manual Steps section.

Automation

#!/usr/bin/env bash
# Audit: GKE API server audit logging and export status
# Requirements:
#   - gcloud installed and authenticated
#   - jq installed
#   - Appropriate IAM permissions on the project(s)

set -euo pipefail

# -------- CONFIGURATION --------
# Space-separated list of projects to check
PROJECTS="<PUT_PROJECT_IDS_HERE>"

# Optional: restrict to specific locations (space-separated)
# Leave empty to scan all locations
LOCATIONS=""

# Optional: restrict to specific cluster types
# Valid values include: ZONAL_REGIONAL_AUTOPILOT
CLUSTER_TYPES=""

echo "Checking GKE API audit logging and export status..."
echo "Timestamp: $(date -Iseconds)"
echo

for PROJECT in $PROJECTS; do
  echo "=== Project: ${PROJECT} ==="

  # 1) Cluster-level: check if Kubernetes API audit logging is enabled
  echo "--- Cluster-level Kubernetes API audit logging ---"
  CLUSTER_CMD=(gcloud container clusters list
    --project "${PROJECT}"
    --format=json)

  # Apply optional filters
  if [ -n "${LOCATIONS}" ]; then
    # gcloud doesn't filter by location directly in list; we'll filter later in jq
    :
  fi
  if [ -n "${CLUSTER_TYPES}" ]; then
    # Same: filter in jq
    :
  fi

  CLUSTERS_JSON="$("${CLUSTER_CMD[@]}" || echo "[]")"

  if [ "${CLUSTERS_JSON}" = "[]" ]; then
    echo "No clusters found in project ${PROJECT}"
  else
    echo "${CLUSTERS_JSON}" | jq -r '
      .[] 
      | . as $c
      | (
          "Cluster: \($c.name)",
          "  Location: \($c.location)",
          "  Logging service: \($c.loggingService // "default")",
          "  Logging config.enabledComponents: \(
             ( $c.loggingConfig.enabledComponents // [] ) 
             | join(", ")
            )",
          "  API server audit logs enabled: \(
             if ($c.loggingConfig.enabledComponents // []) 
                | index("APISERVER_AUDIT") 
             then "YES" else "NO" end
            )",
          ""
        )
    '

    echo "Clusters with API server audit logging DISABLED:"
    echo "${CLUSTERS_JSON}" | jq -r '
      .[]
      | select( (.loggingConfig.enabledComponents // []) | index("APISERVER_AUDIT") | not )
      | "  - " + .name + " (" + .location + ")"
    ' || true
  fi

  echo

  # 2) Project-level: check if audit logs are being exported off-cluster
  #    We consider “off-cluster” as:
  #      - Log sinks that include gke.googleapis.com%2Fmaster (API server logs)
  #      - Destination is not just Cloud Logging default; e.g. BigQuery / Cloud Storage / Pub/Sub
  echo "--- Project-level log sinks for API audit logs (off-cluster export) ---"

  SINKS_JSON="$(gcloud logging sinks list \
    --project "${PROJECT}" \
    --format=json || echo "[]")"

  if [ "${SINKS_JSON}" = "[]" ]; then
    echo "No log sinks defined in project ${PROJECT}"
  else
    echo "${SINKS_JSON}" | jq -r '
      .[]
      | . as $s
      | (
          "Sink: \($s.name)",
          "  Destination: \($s.destination)",
          "  Filter: \($s.filter // "<none>")",
          "  Includes GKE master / API logs: \(
             if ($s.filter // "") 
                | test("gke.googleapis.com%2Fmaster|k8s|kubernetes|container.googleapis.com")
             then "LIKELY" else "UNCLEAR/NO" end
            )",
          ""
        )
    '

    echo "Sinks that LIKELY export GKE master/API logs off-cluster:"
    echo "${SINKS_JSON}" | jq -r '
      .[]
      | select(
          (.filter // "") 
          | test("gke.googleapis.com%2Fmaster|k8s|kubernetes|container.googleapis.com")
        )
      | "  - " + .name + " -> " + .destination
    ' || true
  fi

  echo
done

cat <<'EOF'

INTERPRETING RESULTS
--------------------

1) Cluster-level section:
   - For each cluster, look at:
       API server audit logs enabled: YES/NO
   - PROBLEM:
       - Any cluster showing:
           API server audit logs enabled: NO
       means Kubernetes API audit logging is not enabled for that cluster.

2) Project-level sink section:
   - PROBLEM:
       - No log sinks defined in the project, or
       - Sinks exist but "Includes GKE master / API logs" is "UNCLEAR/NO" for all sinks, or
       - Sinks that include these logs only send to destinations that are not considered
         tamper-resistant for your requirements.
   - You should be able to identify at least one sink where:
       - Includes GKE master / API logs: LIKELY
       - Destination is a controlled, tamper-resistant store (e.g. Cloud Storage with
         restricted IAM, BigQuery dataset with retention policy, or Pub/Sub feeding a
         central SIEM).

This script only reports state. Enabling audit logging and configuring export
must be done via GKE/Cloud Logging configuration (console, gcloud, or IaC).
EOF

More Info:​

Risk Level​

Address​

Compliance Standards​

Triage and Remediation​

Remediation​