Skip to main content

Vertexai Workbench Default VPC Not Used Remediation

Triage and Remediation​

Remediation​

Using Console

Below are concise step‑by‑step instructions using only the GCP Console.

1. Create (or identify) a non‑default VPC network​

  1. In the Google Cloud console, go to VPC network:
    Navigation menu → Networking → VPC network → VPC networks.
  2. Check if you already have a non‑default VPC (any network that is not named default).
    • If yes, note its Name, Subnets, and Region, then skip to section 2.
  3. To create a new VPC:
    1. Click Create VPC network.
    2. Enter a Name (e.g., vertex-notebooks-vpc).
    3. For Subnet creation mode, select Custom.
    4. Click Add subnet:
      • Name: e.g., vertex-notebooks-subnet
      • Region: choose the same region where you will run Vertex AI notebooks.
      • IP address range: e.g., 10.10.0.0/24 (or per your IP plan).
    5. Configure any Firewall rules as required for your environment (e.g., SSH, HTTPS, internal).
    6. Click Create.

2. Create a new Vertex AI notebook using the non‑default VPC​

Network cannot be changed on an existing instance, so you must create a new one and move your work.

  1. In the console, go to Vertex AI Workbench:
    Navigation menu → Vertex AI → Workbench.
  2. Click New notebook (or Create → User-managed notebooks / Managed notebooks, depending on type).
  3. Choose your Region (match the subnet’s region if possible).
  4. Configure machine type, image, etc.
  5. Expand Advanced options (or Networking section; exact wording may vary by UI version).
  6. Under Network:
    • Network: select your non‑default VPC (e.g., vertex-notebooks-vpc).
    • Subnet: select the custom subnet (e.g., vertex-notebooks-subnet).
  7. Optionally adjust:
    • External IP (decide if you want a public IP or only private).
    • Firewall tags or rules, if exposed.
  8. Click Create.

Migrate your work (e.g., copy notebooks, data) from the old instance to the new one (via Git, Cloud Storage, SCP, etc.).

3. Stop using and remove notebooks on the default VPC​

  1. In Vertex AI → Workbench, list all notebook instances.
  2. For each instance:
    1. Click the instance name.
    2. Go to the Details or Networking section and verify the Network:
      • If the Network is default, it is non‑compliant with your requirement.
    3. After you have migrated workloads to a notebook on a custom VPC:
      • Stop the old instance if it is running.
      • Click Delete to remove it.

4. (Optional) Prevent new notebooks from using the default network​

If you want to enforce this at the org/project level:

  1. In the console, go to IAM & Admin → Organization policies.
  2. Look for VPC‑related constraints (e.g., constraints that limit use of the default network or enforce custom networks).
  3. Configure policies so that users cannot create resources on the default network (or consider deleting the default network entirely if it’s not needed and safe to do so).

After these steps, all Vertex AI notebook instances you actively use will be attached to non‑default VPC networks.

Using CLI

Below are step‑by‑step GCP CLI instructions to ensure Vertex AI Workbench / Notebook instances no longer use the default VPC network.

Assumptions:

  • You have gcloud installed and authenticated.
  • Replace placeholder values (PROJECT_ID, REGION, NETWORK_NAME, etc.) with your own.

1. Identify notebooks using the default VPC​

Vertex AI Workbench user‑managed notebooks run on Compute Engine VMs. We’ll inspect those VMs’ networks.

PROJECT_ID="your-project-id"
REGION="us-central1"   # change as needed

gcloud config set project $PROJECT_ID

# List all notebook instances
gcloud notebooks instances list --location=$REGION

# For each instance, find its VM and check network
for INSTANCE in $(gcloud notebooks instances list --location=$REGION --format="value(name)"); do
  echo "=== Instance: $INSTANCE ==="
  gcloud notebooks instances describe $INSTANCE --location=$REGION \
    --format="value(gceSetup.machineType,gceSetup.network,gceSetup.subnet)"
done

Any instance where gceSetup.network is default (or empty and thus implicitly default) must be remediated.

2. Create a dedicated VPC network and subnet​

NETWORK_NAME="vertex-ai-net"
SUBNET_NAME="vertex-ai-subnet"
REGION="us-central1"
SUBNET_RANGE="10.10.0.0/24"   # adjust as needed

# Create custom mode VPC
gcloud compute networks create $NETWORK_NAME \
  --subnet-mode=custom

# Create subnet for Vertex AI
gcloud compute networks subnets create $SUBNET_NAME \
  --network=$NETWORK_NAME \
  --region=$REGION \
  --range=$SUBNET_RANGE

3. Add minimal firewall rules for notebook access​

Example: allow SSH and HTTPS (adjust to your security policy).

# Allow SSH from your admin IP (replace with your IP/CIDR)
ADMIN_CIDR="X.X.X.X/32"

gcloud compute firewall-rules create ${NETWORK_NAME}-allow-ssh \
  --network=$NETWORK_NAME \
  --allow=tcp:22 \
  --source-ranges=$ADMIN_CIDR \
  --direction=INGRESS

# (Optional) Allow HTTPS for web UIs if needed
gcloud compute firewall-rules create ${NETWORK_NAME}-allow-https \
  --network=$NETWORK_NAME \
  --allow=tcp:443 \
  --source-ranges=$ADMIN_CIDR \
  --direction=INGRESS

If notebooks should be private-only (no external IPs), skip or further restrict these rules and ensure no external IP is assigned when creating instances.

4. Create new Vertex AI notebook instances on the custom VPC​

You cannot change the network of an existing underlying VM; you must create a new instance that uses the custom VPC.

Example: create a new Workbench user‑managed instance:

REGION="us-central1"
INSTANCE_ID="vertex-notebook-custom-net"
MACHINE_TYPE="e2-standard-4"       # adjust
IMAGE_FAMILY="common-cpu-notebooks"
IMAGE_PROJECT="deeplearning-platform-release"

gcloud notebooks instances create $INSTANCE_ID \
  --location=$REGION \
  --machine-type=$MACHINE_TYPE \
  --network=$NETWORK_NAME \
  --subnet=$SUBNET_NAME \
  --vm-image-family=$IMAGE_FAMILY \
  --vm-image-project=$IMAGE_PROJECT \
  --no-public-ip         # recommended for private-only

Key flags:

  • --network=$NETWORK_NAME ensures it is not using default.
  • --subnet=$SUBNET_NAME pins it to the custom subnet.
  • --no-public-ip (optional but recommended for locked‑down environments).

5. Migrate workloads and delete old instances using the default network​

For each old notebook instance on default:

  1. Connect and copy data (e.g., using gsutil to move notebooks to Cloud Storage or scp between instances).
  2. Validate that your workloads run correctly on the new instance.
  3. Delete the old instance:
OLD_INSTANCE_ID="old-notebook-id"
REGION="us-central1"

gcloud notebooks instances delete $OLD_INSTANCE_ID \
  --location=$REGION \
  --quiet

Option A: Remove/lock down the default network​

If no other workloads depend on default, you can delete it:

# Delete default firewall rules
for RULE in $(gcloud compute firewall-rules list --filter="network:default" --format="value(name)"); do
  gcloud compute firewall-rules delete $RULE --quiet
done

# Delete default subnets
for SUBNET in $(gcloud compute networks subnets list --network=default --format="value(name,region)" | awk '{print $1":"$2}'); do
  NAME=$(echo $SUBNET | cut -d: -f1)
  REGION=$(echo $SUBNET | cut -d: -f2)
  gcloud compute networks subnets delete $NAME --region=$REGION --quiet
done

# Finally delete the default network
gcloud compute networks delete default --quiet

Only do this if you are sure nothing else requires the default VPC.

Option B: Use org policies​

At the org/folder/project level, you can set policies to prevent default network use (requires org admin privileges). Example (conceptual; may need org-level permissions):

gcloud org-policies set-policy policy.yaml

Where policy.yaml contains constraints such as:

  • constraints/compute.skipDefaultNetworkCreation
  • constraints/compute.restrictVpcPeering
  • (and any internal policy constraining network choices).

By following these steps, all current and future Vertex AI notebook instances will be created on your custom VPC network instead of the default VPC.

Using Python

Below is a practical remediation approach using Python:

Goal:
No Vertex AI Workbench notebook instance should be attached to the default VPC network. Instead, they should use a dedicated/custom VPC.

1. Prerequisites​

pip install google-cloud-notebooks google-api-python-client
gcloud auth application-default login

Make sure:

  • Your account has permissions: compute.networkAdmin, notebooks.admin (or similar).
  • GOOGLE_CLOUD_PROJECT env var is set, or you’ll pass the project ID into the script.

2. Create / Ensure a Custom VPC and Subnet (Python)​

If you don’t already have a non-default VPC, create one. Example uses google-api-python-client for Compute:

import googleapiclient.discovery
from googleapiclient.errors import HttpError

PROJECT_ID = "YOUR_PROJECT_ID"
REGION = "us-central1"
NETWORK_NAME = "vertex-ai-custom-net"
SUBNET_NAME = "vertex-ai-custom-subnet"
NETWORK_URL = f"projects/{PROJECT_ID}/global/networks/{NETWORK_NAME}"
SUBNET_URL = f"projects/{PROJECT_ID}/regions/{REGION}/subnetworks/{SUBNET_NAME}"

compute = googleapiclient.discovery.build("compute", "v1")

def ensure_network():
    try:
        compute.networks().get(project=PROJECT_ID, network=NETWORK_NAME).execute()
        print("Network already exists:", NETWORK_NAME)
    except HttpError as e:
        if e.resp.status == 404:
            print("Creating network:", NETWORK_NAME)
            body = {
                "name": NETWORK_NAME,
                "autoCreateSubnetworks": False,
            }
            op = compute.networks().insert(project=PROJECT_ID, body=body).execute()
            print("Network create operation:", op["name"])
        else:
            raise

def ensure_subnet():
    try:
        compute.subnetworks().get(
            project=PROJECT_ID, region=REGION, subnetwork=SUBNET_NAME
        ).execute()
        print("Subnet already exists:", SUBNET_NAME)
    except HttpError as e:
        if e.resp.status == 404:
            print("Creating subnet:", SUBNET_NAME)
            body = {
                "name": SUBNET_NAME,
                "network": NETWORK_URL,
                "ipCidrRange": "10.10.0.0/24",
                "region": REGION,
                "privateIpGoogleAccess": True,
            }
            op = compute.subnetworks().insert(
                project=PROJECT_ID, region=REGION, body=body
            ).execute()
            print("Subnet create operation:", op["name"])
        else:
            raise

if __name__ == "__main__":
    ensure_network()
    ensure_subnet()

3. Identify Notebook Instances Using the Default Network​

For Vertex AI Workbench managed notebooks, use google-cloud-notebooks:

from google.cloud import notebooks_v1

PROJECT_ID = "YOUR_PROJECT_ID"
LOCATION = "us-central1"  # or "-" for all locations
DEFAULT_NETWORK_SUFFIX = "/global/networks/default"

client = notebooks_v1.NotebookServiceClient()

parent = f"projects/{PROJECT_ID}/locations/{LOCATION}"
instances = client.list_instances(parent=parent)

instances_using_default = []

for inst in instances:
    # For Workbench managed notebooks, `inst.network` is a full URI-like string
    if inst.network.endswith(DEFAULT_NETWORK_SUFFIX):
        instances_using_default.append(inst)

print("Instances using default network:")
for inst in instances_using_default:
    print(inst.name, "->", inst.network)

If you are also using user-managed notebooks (legacy AI Platform Notebooks), adapt similarly using the same client (they are also exposed via notebooks_v1 but under different resource paths).

4. Recreate Notebooks on the Custom Network​

Network configuration is effectively immutable for Workbench notebook instances, so remediation is:

  1. Capture configuration of existing instance.
  2. Create a new instance with the same settings but with network (and optionally subnet) pointing to your custom VPC.
  3. Migrate data (via attached disks, snapshots, Git, or copying files).
  4. Delete the old instance.

Example: clone basic properties and create a replacement instance:

from google.cloud import notebooks_v1
from google.protobuf import field_mask_pb2

PROJECT_ID = "YOUR_PROJECT_ID"
LOCATION = "us-central1"
NEW_NETWORK = NETWORK_URL      # from step 2
NEW_SUBNET = SUBNET_URL

client = notebooks_v1.NotebookServiceClient()

def create_replacement_instance(old_instance: notebooks_v1.Instance, new_id: str):
    parent = f"projects/{PROJECT_ID}/locations/{LOCATION}"

    # Build a new Instance config copying main fields
    new_inst = notebooks_v1.Instance()
    new_inst.name = f"{parent}/instances/{new_id}"
    new_inst.machine_type = old_instance.machine_type
    new_inst.metadata.update(old_instance.metadata)
    new_inst.post_startup_script = old_instance.post_startup_script
    new_inst.service_account = old_instance.service_account

    # Set custom network (and subnet if used)
    new_inst.network = NEW_NETWORK
    new_inst.subnet = NEW_SUBNET

    # Optional: copy container/image/configuration details
    new_inst.vm_image.CopyFrom(old_instance.vm_image)
    new_inst.container_image.CopyFrom(old_instance.container_image)
    new_inst.install_gpu_driver = old_instance.install_gpu_driver
    new_inst.accelerator_config.CopyFrom(old_instance.accelerator_config)

    op = client.create_instance(
        parent=parent,
        instance_id=new_id,
        instance=new_inst,
    )
    print("Creating replacement instance:", new_id)
    result = op.result()  # waits for completion
    print("Created:", result.name)
    return result

# Example usage: recreate each instance using default network
instances = client.list_instances(parent=f"projects/{PROJECT_ID}/locations/{LOCATION}")

for old in instances:
    if not old.network.endswith("/global/networks/default"):
        continue
    old_name = old.name.split("/")[-1]
    new_id = f"{old_name}-nondefault"
    create_replacement_instance(old, new_id)
    # At this point you’d manually/with additional code:
    # - copy data over (e.g., via disks or files)
    # - validate
    # - then delete the old instance:
    # client.delete_instance(name=old.name).result()

Note: Data migration strategy depends on how notebooks store data (boot disk, extra data disk, Git repos). Automating disk reattachment is possible but more complex; keep that separate if needed.

5. Enforce Going Forward (Prevent New Use of Default VPC)​

For future notebooks:

  • Ensure your automation or infra-as-code (Terraform, Deployment Manager, custom scripts) always sets network (and subnet) explicitly to your custom VPC when calling create_instance.
  • Optionally use Organization Policy / security controls to:
    • Restrict use of the default network.
    • Require private IP / specific networks for Vertex AI Workbench.

If you specify whether you use managed vs. user-managed notebooks and whether you want disk migration automated, I can provide a more exact Python script tailored to that.

Using Terraform
# Non-default network and subnet to use instead of "default"
resource "google_compute_network" "VERTEX_NETWORK" {
  name                    = "VERTEX_NETWORK_NAME"        # e.g. "vertex-notebooks-net"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "VERTEX_SUBNET" {
  name          = "VERTEX_SUBNET_NAME"                  # e.g. "vertex-notebooks-subnet"
  ip_cidr_range = "10.10.0.0/24"
  region        = "GCP_REGION"                          # e.g. "us-central1"
  network       = google_compute_network.VERTEX_NETWORK.id
}

# Vertex AI Workbench instance (user-managed) NOT using the default network
resource "google_notebooks_instance" "VERTEX_NOTEBOOK" {
  name         = "VERTEX_NOTEBOOK_NAME"                 # e.g. "nb-ml-workbench-1"
  location     = "GCP_REGION"                           # e.g. "us-central1-b" or "us-central1"
  machine_type = "projects/PROJECT_ID/zones/GCP_REGION/machineTypes/MACHINE_TYPE"

  network_interface {
    # Critical: do NOT point this at "default"; use your custom network instead
    network    = google_compute_network.VERTEX_NETWORK.id
    subnetwork = google_compute_subnetwork.VERTEX_SUBNET.id
  }

  # ...other required arguments (boot_disk, vm_image, etc.)...
}

Changing the network_interface.network / subnetwork of an existing google_notebooks_instance forces replacement of the instance (recreate and potential downtime), so apply carefully.

To verify, terraform plan should show either:

  • creation of a new google_notebooks_instance attached to VERTEX_NETWORK, or
  • replacement of the existing instance where the only network change is from projects/PROJECT_ID/global/networks/default (or similar) to your custom VERTEX_NETWORK.