Skip to main content

OCI Monitoring Should Have Anomalous GPU Usage Alarm

More Info:​

Monitor for anomalous GPU instance provisioning. Attackers frequently deploy high-cost, high-compute GPU instances for illicit cryptocurrency mining after compromising cloud accounts

Risk Level​

Medium

Address​

Compliance, Security

Compliance Standards​

  • APRA CPS 234 (Australia)
  • AWS Well Architected Framework
  • BSI C5 (Germany)
  • Brazil LGPD
  • CCPA / CPRA (California)
  • CIS Critical Security Controls v8
  • CMMC 2.0
  • CSA Cloud Controls Matrix v4
  • DPDPA
  • Digital Operational Resilience Act (EU)
  • ISO/IEC 27017
  • ISO/IEC 27018
  • ISO/IEC 27701
  • KSA PDPL
  • MAS Technology Risk Management (Singapore)
  • MITRE ATT&CK (Cloud)
  • NIS2 Directive
  • NIST SP 800-171
  • NYDFS 23 NYCRR 500
  • SWIFT Customer Security Controls Framework
  • Sarbanes-Oxley IT General Controls
  • UK NCSC Cyber Assessment Framework

Triage and Remediation​

Remediation​

Using Console

Below are concise, console‑only steps to configure an anomalous GPU usage alarm in OCI Monitoring.

1. Confirm GPU Metrics Are Available​

  1. In OCI Console, open Observability & Management → Monitoring → Service Metrics.
  2. In Compartment, select the compartment where your GPU instances run.
  3. Set:
    • Metric namespace: oci_computeagent
    • Resource group: often gpu (or leave as “All” if unsure).
  4. In Metric name, look for GPU metrics such as:
    • GpuUtilization
    • GpuMemoryUtilization
    • GpuTemperature
  5. Select a metric and verify you see recent data points in the chart.
    If there is no data, ensure the Oracle Cloud Agent (Management Agent) with GPU plugins is enabled on the instance.

2. Create an Alarm for Anomalous GPU Usage​

  1. Go to Observability & Management → Monitoring → Alarms.
  2. Click Create alarm.

Basic configuration:

  1. Name: e.g., gpu-anomaly-usage-alarm.
  2. Description: e.g., Alarm when GPU utilization behaves anomalously.
  3. Compartment: choose the compartment of the GPU metrics.
  4. Metric namespace: oci_computeagent.
  5. Resource group: gpu (or as confirmed in step 1).

Choose the metric and anomaly rule:

  1. Under Metric name, select GpuUtilization (or the specific GPU metric you want to monitor).
  2. For Dimensions, optionally scope to:
    • Specific instance: resourceId = <instance-OCID>
    • Or other relevant dimensions if needed.
  3. Under Trigger rule (Alarm condition), select:
    • Rule type: Anomaly (or Dynamic threshold / Anomaly detection if phrased that way in your tenancy).
  4. Configure anomaly condition:
    • Statistic: avg (or max depending on your policy).
    • Period: e.g., 1 minute or 5 minutes.
    • Anomaly model / Baseline: leave as Automatic unless you have specific training data.
    • Sensitivity: set to Medium (adjust later based on noise).

3. Set Alarm Threshold & Evaluation​

  1. Configure when to fire:
    • Condition: e.g., Anomaly score > 0 or the UI’s default anomaly trigger (often “When metric is anomalous”).
    • Trigger delay / trigger after: e.g., 3 consecutive periods to avoid noise.
  2. Severity: choose (e.g., Critical or Warning as per your policy).

4. Configure Notifications​

  1. Under Destination, select or create an OCI Notifications topic:
    • If needed, click Create topic and then add Subscriptions (email, Slack via HTTPS, PagerDuty, etc.) in Developer Services → Notifications.
  2. In the alarm:
    • Choose Notification topic: select the topic you created.

(Optionally, configure Repeat notification and Suppression windows if supported and needed.)

5. Finalize and Test​

  1. Click Create alarm.
  2. Confirm the alarm appears in Alarms list with OK or FIRING state.
  3. To test:
    • Intentionally increase GPU load (e.g., run a GPU‑heavy workload) to cause anomalous behavior versus normal baseline.
    • Verify that the alarm transitions to FIRING and that you receive the notification.

This completes remediation: OCI Monitoring now has an anomalous GPU usage alarm configured through the OCI Console.

Using CLI

Below is a minimal, CLI‑only way to set up an “anomalous GPU usage” alarm in OCI Monitoring. I’ll assume Linux/macOS shell; on Windows, adjust quoting.

1. Prereqs​

  1. Ensure the instance has GPU metrics published:

    • It must be a GPU shape.
    • Compute Instance Agent must be running (default on OCI images).

  2. Your OCI user must have policies like:

    allow group <group-name> to read metrics in compartment <compartment-name>
    allow group <group-name> to manage alarms in compartment <compartment-name>
    allow group <group-name> to use ons-topics in compartment <compartment-name>

  3. OCI CLI configured:

    oci setup config

2. Identify GPU Metric and Namespace​

List available GPU metrics in the instance’s compartment:

COMPARTMENT_OCID="<compartment-ocid>"

oci monitoring metric list \
  --compartment-id "$COMPARTMENT_OCID" \
  --namespace oci_computeagent \
  --query "data[?contains(name, 'Gpu') || contains(name, 'GPU')]" \
  --all

Typical GPU utilization metric is under namespace oci_computeagent with a name like:

  • GpuUtilization or similar (use exactly what you see in the output).

Note the:

  • namespace (e.g., oci_computeagent)
  • metric name (e.g., GpuUtilization)
  • dimensions (e.g., resourceId, gpuIndex)

3. Create a Notification Topic (for the Alarm)​

If you already have a topic OCID, skip to step 4.

TOPIC_NAME="gpu-alerts-topic"

oci ons topic create \
  --compartment-id "$COMPARTMENT_OCID" \
  --name "$TOPIC_NAME" \
  --description "GPU anomalous usage alerts" \
  --query "data.id" \
  --raw-output

Save the returned OCID as:

TOPIC_OCID="<topic-ocid-from-above>"

Optionally subscribe an email:

oci ons subscription create \
  --compartment-id "$COMPARTMENT_OCID" \
  --topic-id "$TOPIC_OCID" \
  --protocol EMAIL \
  --endpoint you@example.com

Then confirm the email.

4. Build an “Anomalous GPU Usage” Query​

Two common ways (pick one):

4.1. Simple “High GPU Utilization” (practical anomaly proxy)​

Example: GPU utilization > 90% for 5 minutes on any GPU in a compartment:

METRIC_NAMESPACE="oci_computeagent"
METRIC_NAME="GpuUtilization"     # replace with your metric name

QUERY_TEXT="${METRIC_NAME}[1m]{resourceId = \"*\"}.mean() > 90"

You can restrict to a specific instance:

INSTANCE_OCID="<gpu-instance-ocid>"
QUERY_TEXT="${METRIC_NAME}[1m]{resourceId = \"${INSTANCE_OCID}\"}.mean() > 90"

4.2. If you want deviation from baseline (pseudo‑anomaly)​

Some tenants use a rule like “> 2× the 1‑hour rolling average”. OCI Monitoring doesn’t yet have a native anomalyDetection() function generally available in all regions, so you usually approximate anomaly with a high threshold or derivative‑type rules. If your tenancy has advanced query features, adapt accordingly; otherwise use 4.1.

5. Create the Alarm via OCI CLI​

ALARM_NAME="Anomalous-GPU-Usage"
ALARM_DESCRIPTION="Alert when GPU utilization is anomalously high."
SEVERITY="CRITICAL"      # or MAJOR, MINOR, WARNING
PENDING_DURATION="PT5M"  # condition must hold 5 minutes

oci monitoring alarm create \
  --compartment-id "$COMPARTMENT_OCID" \
  --display-name "$ALARM_NAME" \
  --metric-compartment-id "$COMPARTMENT_OCID" \
  --namespace "$METRIC_NAMESPACE" \
  --query-text "$QUERY_TEXT" \
  --severity "$SEVERITY" \
  --is-enabled true \
  --destinations "[\"$TOPIC_OCID\"]" \
  --pending-duration "$PENDING_DURATION" \
  --body '{
    "messageFormat": "RAW",
    "resolution": "1m",
    "notificationVersion": "1.0"
  }' \
  --query "data.id" \
  --raw-output

Notes:

  • --namespace must match the namespace from step 2.
  • --query-text is the full alarm expression (no extra where/select syntax, just Monitoring query).
  • pending-duration is ISO‑8601 (e.g., PT5M = 5 minutes).

6. Update or Fix an Existing Alarm (if misconfigured)​

List alarms:

oci monitoring alarm list \
  --compartment-id "$COMPARTMENT_OCID" \
  --all \
  --query "data[?displayName=='${ALARM_NAME}']"

Get the alarm OCID:

ALARM_OCID="<alarm-ocid>"

Update the query to the correct GPU anomaly condition:

oci monitoring alarm update \
  --alarm-id "$ALARM_OCID" \
  --query-text "$QUERY_TEXT" \
  --namespace "$METRIC_NAMESPACE" \
  --metric-compartment-id "$COMPARTMENT_OCID" \
  --is-enabled true

If you paste your actual metric list output (for GPU) and your tenancy’s region, I can give you the exact QUERY_TEXT line you should use.

Using Python

Below is a practical way to remediate this in OCI using Python by:

  1. Making sure GPU metrics are available.
  2. Creating a Monitoring Alarm that fires on anomalous GPU usage.

1. Ensure GPU metrics are available​

  1. Use a GPU shape (e.g. VM.GPU3.1, BM.GPU4.8, etc.).
  2. Ensure the Oracle Cloud Agent is enabled on the instance.
    • In OCI Console:
      Compute → Instances → your instance → Oracle Cloud Agent → ensure it’s enabled and monitoring plugins are running.
  3. Confirm GPU metrics exist:
    • Monitoring → Metrics Explorer
    • Namespace: usually oci_computeagent
    • Look for metrics like GpuUtilization, GpuMemoryUtilization, etc.

Once metrics are visible, you can create an alarm programmatically.

2. Concept for “anomalous” GPU usage​

OCI doesn’t have a built‑in anomaly engine on GPU metrics. A practical approach for “anomalous GPU usage” is:

  • Alarm if GPU utilization is too high for too long (e.g. potential abuse / runaway job).
  • Optionally add another alarm if utilization is too low during expected workload hours.

Example high-usage rule:

GpuUtilization[1m]{resourceId = "ocid1.instance.oc1..xxxx"} > 90

3. Python setup​

Install and configure the OCI Python SDK:

pip install oci

Configure credentials (config file in ~/.oci/config):

[DEFAULT]
user=ocid1.user.oc1..xxxxx
fingerprint=aa:bb:cc:dd:...
key_file=/path/to/oci_api_key.pem
tenancy=ocid1.tenancy.oc1..xxxxx
region=us-phoenix-1

4. Python code to create a GPU anomaly alarm​

This example creates a high GPU utilization alarm on a specific instance.

import oci
from oci.monitoring.models import CreateAlarmDetails

# 1. Load config
config = oci.config.from_file("~/.oci/config", "DEFAULT")

# 2. Clients
monitoring_client = oci.monitoring.MonitoringClient(config)
notifications_client = oci.ons.NotificationControlPlaneClient(config)

# 3. Variables – UPDATE these for your environment
compartment_id = "ocid1.compartment.oc1..xxxx"
gpu_instance_id = "ocid1.instance.oc1.phx.xxxxx"
topic_id = "ocid1.onstopic.oc1.phx.xxxxx"   # OCI Notifications topic OCID for email/Slack/etc.
display_name = "Anomalous GPU Usage Alarm"
severity = "CRITICAL"
# Namespace for metrics coming from the Compute Agent:
metric_namespace = "oci_computeagent"
metric_name = "GpuUtilization"

# 4. Define the alarm query
# Example: GPU utilization > 90% for 5 out of 5 minutes
# Adjust threshold and duration to your needs.
query = (
    f"{metric_name}[1m]{{resourceId = \"{gpu_instance_id}\"}}.mean() > 90"
)

alarm_details = CreateAlarmDetails(
    display_name=display_name,
    compartment_id=compartment_id,
    metric_compartment_id=compartment_id,
    namespace=metric_namespace,
    query=query,
    severity=severity,
    destinations=[topic_id],
    is_enabled=True,
    # alarm conditions
    pending_duration="PT5M",       # condition must hold for 5 minutes
    resolution="PT1M",             # evaluate every 1 minute
    body="High GPU usage detected on instance.",
    message_format="TEXT",
    repeat_notification_duration="PT30M",  # re-notify every 30 minutes if still firing
)

# 5. Create the alarm
response = monitoring_client.create_alarm(alarm_details)
print("Created alarm OCID:", response.data.id)

5. Optional: “Low GPU usage” (expected workload hours)​

If “anomalous” for you means “GPU is unexpectedly idle during work”, you can add another alarm:

low_gpu_query = (
    f"{metric_name}[5m]{{resourceId = \"{gpu_instance_id}\"}}.mean() < 10"
)

low_usage_alarm = CreateAlarmDetails(
    display_name="Low GPU Usage During Expected Hours",
    compartment_id=compartment_id,
    metric_compartment_id=compartment_id,
    namespace=metric_namespace,
    query=low_gpu_query,
    severity="WARNING",
    destinations=[topic_id],
    is_enabled=True,
    pending_duration="PT15M",
    resolution="PT5M",
    body="Low GPU usage detected during expected workload period.",
    message_format="TEXT",
)

resp = monitoring_client.create_alarm(low_usage_alarm)
print("Created low-usage alarm OCID:", resp.data.id)

You can further filter by availabilityDomain, displayName, or custom tags in the query if you want one alarm for a group of GPU instances instead of a single resourceId.

If you share your exact GPU metrics (namespace/metric names as seen in Metrics Explorer) and how you define “anomaly” (thresholds, times), I can adjust the query and Python code precisely.

Using Terraform
resource "oci_monitoring_alarm" "anomalous_gpu_usage_alarm" {
  # OCID of the compartment where GPU instances run
  compartment_id = "OCI_COMPARTMENT_OCID"

  # OCID of the compartment that owns the metric (often same as compartment_id)
  metric_compartment_id = "OCI_METRIC_COMPARTMENT_OCID"

  display_name = "anomalous-gpu-usage"

  # Example query: alert if average GPU utilization exceeds a threshold over 5 minutes.
  # Replace GPU_METRIC_NAME and DIMENSION_FILTERS to match your GPU metric/shape filters,
  # and GPU_UTILIZATION_THRESHOLD with the numeric threshold you require.
  #
  # Example with a common pattern (adjust as needed for your tenancy):
  #   "GpuUtilization[5m]{shape = \"VM.GPU*\"}.mean() > 80"
  query = "GPU_METRIC_NAME[5m]{DIMENSION_FILTERS}.mean() > GPU_UTILIZATION_THRESHOLD"

  # How long the condition must be met before triggering (e.g., "5m", "10m")
  pending_duration = "5m"

  # Notification topic(s) for the alarm
  destinations = [
    "OCI_NOTIFICATIONS_TOPIC_OCID",
  ]

  severity = "CRITICAL"

  is_enabled  = true
  is_suppressed = false

  # Optional: human-readable description
  body = "Alarm for anomalous GPU usage: triggered when GPU utilization breaches the configured threshold."

  # Optional: keep notifications per metric dimension separate
  is_notifications_per_metric_dimension_enabled = true

  # Optional: how often to repeat notifications while the alarm is firing
  repeat_notification_duration = "60m"

  message_format = "PRETTY_JSON"
}

Changing this alarm’s query, thresholds, or destinations is an in-place update and does not force replacement of other resources (no outage risk beyond normal alarm behavior changes).

To verify, terraform plan should show either creation of oci_monitoring_alarm.anomalous_gpu_usage_alarm or an in-place update (~) of the existing alarm with the new query (and any other changed fields).