Skip to main content

Cloud Monitoring feature is enabled for your Vertex AI

More Info:​

Ensure that Cloud Monitoring feature is enabled for your Vertex AI notebook instances

Risk Level​

Medium

Address​

Monitoring

Compliance Standards​

  • CIS GCP

Triage and Remediation​

Remediation​

Using Console

Below are the concise, console-based steps to ensure Cloud Monitoring is enabled for Vertex AI Workbench notebook instances.

1. Verify Cloud Monitoring & APIs​

  1. Go to Google Cloud console: https://console.cloud.google.com
  2. In the top-left, select the correct Project.
  3. In the left menu, go to APIs & Services → Library.
  4. Confirm the following APIs are Enabled (enable if they are not):
    • Cloud Monitoring API
    • Vertex AI API

2. Enable Monitoring on a New Vertex AI Workbench Notebook​

  1. Go to Vertex AI → Workbench in the console.
  2. Click New notebook → choose the type (e.g., Managed notebooks).
  3. Fill in the basic details (name, region, machine type, etc.).
  4. Expand Advanced Settings (or similar section, wording may slightly differ).
  5. Find the Monitoring or Operations section:
    • Check/enable Cloud Monitoring / Ops Agent / Enable monitoring option.
  6. Complete the rest of the configuration as desired.
  7. Click Create.

Cloud Monitoring will now collect metrics from that notebook VM.

3. Enable Monitoring on an Existing Notebook Instance​

You typically need to stop the instance to edit some settings.

  1. Go to Vertex AI → Workbench.
  2. Locate your notebook instance in the list.
  3. If it’s Running, click the three‑dot menu (⋮) → Stop and wait until status is Stopped.
  4. Again click the three‑dot menu (⋮) → Edit.
  5. In the edit page, expand Advanced Settings.
  6. Locate the Monitoring / Ops Agent / Enable Cloud Monitoring section and:
    • Check/enable the monitoring option.
  7. Save the changes (e.g., Save, Update, or Save & Continue depending on UI).
  8. Once updated, click Start to start the notebook instance again.

4. Confirm Metrics in Cloud Monitoring​

  1. Go to Monitoring → Metrics explorer in the console.
  2. In Resource type, select:
    • VM Instance or Vertex AI Workbench (depending on what appears for your setup).
  3. Check that you see metrics (CPU, memory, disk, etc.) for your notebook VM.

This ensures Cloud Monitoring is enabled and collecting data for your Vertex AI notebook instances via the GCP console.

Using CLI

Below is a practical CLI-based way to ensure Vertex AI notebook instances are sending metrics to Cloud Monitoring. The approach is different for:

  • Managed Notebooks (Vertex AI Workbench managed) – monitoring is on by default; you mainly need APIs enabled.
  • User-Managed Notebooks (Vertex AI Workbench user-managed / legacy AI Platform Notebooks) – these are GCE VMs; you enable monitoring by installing the Ops Agent on the VM.

1. Prerequisites (all Vertex AI notebooks)​

1.1. Set default project and region/zone​

PROJECT_ID="your-project-id"
REGION="us-central1"       # for managed notebooks
ZONE="us-central1-b"       # for user-managed (VM-based) notebooks

gcloud config set project "$PROJECT_ID"

1.2. Enable required APIs​

gcloud services enable \
  notebooks.googleapis.com \
  monitoring.googleapis.com

2. Managed Notebooks (Vertex AI Workbench – Managed)​

For managed notebooks, Google manages the underlying infra and Cloud Monitoring integration is enabled by default. The main remediation is just ensuring the APIs are on (done above).

To verify instances:

gcloud notebooks instances list --location="$REGION"

If you are only using managed notebooks, there is nothing more to “turn on” for Monitoring via CLI.

3. User-Managed Notebooks (VM-based) – Install Ops Agent​

For user-managed notebooks, each notebook is a Compute Engine VM. To enable Cloud Monitoring, install the Ops Agent on those VMs.

3.1. List user-managed notebook instances​

gcloud notebooks instances list --location="$ZONE" --format="table(name, state, proxyUri)"

Note: For user-managed notebooks, the instance name is typically the same as the underlying GCE VM name and uses the same zone.

gcloud compute instances add-metadata "INSTANCE_NAME" \
  --zone="$ZONE" \
  --metadata=enable-oslogin=TRUE

Replace INSTANCE_NAME with your notebook VM name.

3.3. SSH into the notebook VM​

gcloud compute ssh "INSTANCE_NAME" --zone="$ZONE"

3.4. Install the Ops Agent (Cloud Monitoring + Logging)​

Once logged into the VM:

Debian/Ubuntu (most Vertex AI notebook images are Debian-based):

curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
sudo bash add-google-cloud-ops-agent-repo.sh --also-install

RHEL/CentOS:

curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
sudo bash add-google-cloud-ops-agent-repo.sh --also-install --distribution=el

The agent will automatically send system metrics and logs to Cloud Monitoring/Logging.

Exit the VM when done:

exit

3.5. (Optional) Verify the Ops Agent status​

From the VM:

sudo systemctl status google-cloud-ops-agent

You should see the service as active (running).

4. Verify in Cloud Monitoring​

After a few minutes, verify metrics:

# List monitored resource types
gcloud monitoring resource-descriptors list --format="value(type)" | grep notebook

Then check metrics in the console:

  • Go to Monitoring → Metrics explorer
  • Resource type: gce_instance
  • Filter by instance name of your notebook VM.

This ensures Cloud Monitoring is receiving metrics for your Vertex AI notebook instances.

Using Python

Below is a practical way to ensure Vertex AI Workbench (user‑managed) notebook VMs are correctly configured for Cloud Monitoring using Python.

For Vertex AI Workbench user‑managed notebooks, monitoring is essentially:

  • The VM must have the Cloud Monitoring and Cloud Logging OAuth scopes.
  • The Ops Agent / Monitoring agent must be installed (on current images it usually is by default).

Important: Scopes on a VM cannot be changed in-place. If an instance was created without monitoring scopes, you must recreate it with the right config. The code below shows how to:

  1. Check scopes for existing notebook VMs.
  2. Create/recreate a notebook instance with Monitoring & Logging enabled.

1. Setup​

pip install google-cloud-notebooks google-api-python-client
gcloud auth application-default login

Ensure your default credentials have roles/notebooks.admin and roles/compute.viewer at least.

2. Check if existing notebook instances have Monitoring enabled​

Vertex AI Workbench user-managed notebooks are backed by Compute Engine VMs. We’ll:

  • List notebook instances via the Notebooks API.
  • For each, find the underlying VM and inspect its OAuth scopes.
  • Flag instances missing monitoring.write and logging.write.
from google.cloud import notebooks_v1
from googleapiclient import discovery
from google.auth import default

PROJECT_ID = "your-project-id"
LOCATION = "us-central1"   # change as needed
PARENT = f"projects/{PROJECT_ID}/locations/{LOCATION}"

def list_notebook_instances():
    client = notebooks_v1.NotebookServiceClient()
    for inst in client.list_instances(parent=PARENT):
        yield inst

def get_compute_client():
    creds, _ = default()
    return discovery.build("compute", "v1", credentials=creds, cache_discovery=False)

def check_monitoring_for_instance(instance, compute):
    # User-managed notebooks have a single VM behind them.
    # The VM name is usually the same as the notebook instance ID.
    # The zone is in instance.gce_setup.zone (or instance.vm_image.gce_zone on some versions).
    try:
        gce_setup = instance.gce_setup
        zone = gce_setup.zone.split("/")[-1]  # e.g. "projects/.../zones/us-central1-b"
        vm_name = gce_setup.vm_name or instance.name.split("/")[-1]
    except AttributeError:
        # Fallback: parse zone from instance.name if needed
        # Adjust if your instances look different
        print(f"Could not determine VM info for {instance.name}")
        return

    vm = compute.instances().get(project=PROJECT_ID, zone=zone, instance=vm_name).execute()

    service_accounts = vm.get("serviceAccounts", [])
    if not service_accounts:
        print(f"{instance.name}: NO service account configured – monitoring/ logging are disabled")
        return

    scopes = set()
    for sa in service_accounts:
        scopes.update(sa.get("scopes", []))

    needs_monitoring = not any("monitoring.write" in s for s in scopes)
    needs_logging = not any("logging.write" in s for s in scopes)

    if needs_monitoring or needs_logging:
        print(f"{instance.name}: missing scopes -> "
              f"{'monitoring' if needs_monitoring else ''} "
              f"{'logging' if needs_logging else ''}")
    else:
        print(f"{instance.name}: OK (Monitoring & Logging scopes present)")

def main():
    compute = get_compute_client()
    for inst in list_notebook_instances():
        check_monitoring_for_instance(inst, compute)

if __name__ == "__main__":
    main()

Any instance printed as missing scopes should be recreated with the correct scopes.

3. Create (or recreate) a notebook with Cloud Monitoring enabled​

The simplest way to “enable Cloud Monitoring” is to ensure the notebook’s underlying VM has:

https://www.googleapis.com/auth/logging.write
https://www.googleapis.com/auth/monitoring.write

and that you’re using a recent Vertex AI Workbench image (which includes the Ops Agent).

from google.cloud import notebooks_v1
from google.protobuf import field_mask_pb2

PROJECT_ID = "your-project-id"
LOCATION = "us-central1-b"  # zone; user-managed notebooks are zonal
PARENT = f"projects/{PROJECT_ID}/locations/{LOCATION}"
INSTANCE_ID = "my-monitored-notebook"

SERVICE_ACCOUNT_EMAIL = "my-sa@your-project-id.iam.gserviceaccount.com"  # or default

def create_notebook_with_monitoring():
    client = notebooks_v1.NotebookServiceClient()

    instance = notebooks_v1.Instance(
        name=f"{PARENT}/instances/{INSTANCE_ID}",
        machine_type=f"projects/{PROJECT_ID}/zones/{LOCATION}/machineTypes/n1-standard-4",
        service_account=SERVICE_ACCOUNT_EMAIL,
        service_account_scopes=[
            "https://www.googleapis.com/auth/cloud-platform",         # superset
            "https://www.googleapis.com/auth/logging.write",
            "https://www.googleapis.com/auth/monitoring.write",
        ],
        vm_image=notebooks_v1.VmImage(
            project="deeplearning-platform-release",
            image_family="common-cpu-notebooks",  # pick an appropriate family
        ),
        install_gpu_driver=False,
        no_public_ip=True,
    )

    op = client.create_instance(
        parent=PARENT,
        instance_id=INSTANCE_ID,
        instance=instance,
    )
    print("Creating instance...")
    result = op.result()
    print(f"Created instance: {result.name}")

if __name__ == "__main__":
    create_notebook_with_monitoring()

If you want to recreate an existing instance:

  1. Export current settings (machine type, disks, network, etc.).
  2. Delete the old instance.
  3. Call create_instance with the same settings plus the Monitoring & Logging scopes as above.

4. (Optional) Verify in Cloud Monitoring​

After the instance is running:

  • In the console: Monitoring → Metrics Explorer → find agent.googleapis.com metrics for the VM.
  • Or via API, list time series for agent.googleapis.com/agent/uptime for that VM.

If you can share whether you’re using user‑managed or managed notebooks and what your current instance config looks like (or a sample Instance JSON), I can adjust the Python exactly to your environment.

Using Terraform
# Enabling the built‑in “Cloud Monitoring” feature on Vertex AI Workbench
# notebook instances is not currently exposed as a Terraform argument on
# google_notebooks_instance / google_notebooks_runtime, so it cannot be
# remediated directly in Terraform.

# You must turn this on via the GCP Console or API:
# Console:
#   Vertex AI → Workbench → (select notebook instance) → Edit →
#   Monitoring → Enable Cloud Monitoring → Save.
#
# Or via the Notebooks/Vertex AI API using the corresponding field, outside
# of Terraform control.

# No valid Terraform arguments exist today that map to this specific toggle,
# so any Terraform change here would not actually satisfy the check.

After this, terraform plan should show no changes related to Cloud Monitoring for the notebook instances, because the setting is not managed by the provider.