Skip to main content

Vertexai Workbench Encrypted With Cmek Remediation

Triage and Remediation​

Remediation​

Using Console

Below are the GCP Console steps to ensure Vertex AI notebook instances are encrypted with Customer-Managed Encryption Keys (CMEK). Note that CMEK can only be set at creation time; you cannot convert an existing notebook instance from Google-managed keys to CMEK. For existing notebooks, you must create a new one with CMEK and migrate your work.

1. Create (or identify) a CMEK key in Cloud KMS​

  1. In the GCP Console, go to:
    Security → Key Management (or search “KMS” in the search bar).
  2. At the top, make sure you are in the same project and region where you will create the Vertex AI notebook.
  3. Click + Create key ring (if you don’t already have one):
    • Name: e.g., vertex-ai-notebooks-kr
    • Location type: Regional
    • Location: select the same region you’ll use for the notebook.
    • Click Create.
  4. In the key ring, click + Create key:
    • Key name: e.g., vertex-ai-notebooks-key
    • Key purpose: Symmetric encrypt/decrypt
    • Protection level: Software
    • Leave default rotation if you don’t have a specific policy.
    • Click Create.

2. Grant the notebook’s service account access to the key​

Vertex AI Notebooks use a service account to access resources. That service account needs Cloud KMS Encrypter/Decrypter on your key.

  1. In the Console, go again to Security → Key Management.
  2. Click your key ring, then your key.
  3. Go to the Permissions tab.
  4. Click Grant access.
  5. In New principals, add:
    • The notebook runtime service account you plan to use (often the Compute Engine default service account):
      PROJECT_NUMBER-compute@developer.gserviceaccount.com
      or a custom service account you attach to the notebook.
  6. In Roles, add:
    • Cloud KMS CryptoKey Encrypter/Decrypter (roles/cloudkms.cryptoKeyEncrypterDecrypter)
  7. Click Save.

If you’re unsure which service account your notebook will use, decide it now (e.g., the project’s default Compute Engine SA) and use that consistently.

3. Create a new Vertex AI notebook with CMEK enabled​

Steps differ slightly depending on the notebook type, but the CMEK selection is similar.

A. For User-Managed Notebooks (legacy instances)​

  1. In the Console, go to:
    Vertex AI → Workbench → User-managed notebooks.
  2. Click New notebook.
  3. Select the environment/image you want (e.g., TensorFlow, PyTorch, etc.).
  4. In the notebook configuration page:
    • Choose Region (must match your KMS key’s region).
    • Under Security, look for Encryption.
    • Select Customer-managed key.
    • Click Browse next to the key field, then choose your KMS key created above.
    • Ensure the Service account field matches the service account you granted KMS permissions to.
  5. Configure other options (machine type, disk, etc.) as needed.
  6. Click Create.

B. For Managed Notebooks (Vertex AI Workbench Managed)​

  1. In the Console, go to:
    Vertex AI → Workbench → Managed notebooks.
  2. Click New notebook.
  3. Choose a Region (must match your KMS key’s region).
  4. Configure environment and machine options.
  5. Under Security (or Advanced options → Security depending on UI version):
    • Find Encryption.
    • Select Customer-managed key.
    • Click Browse and choose your KMS key.
    • Confirm the Runtime service account is the one that has KMS permissions.
  6. Click Create.

4. Migrate from an existing notebook (if needed)​

Because encryption cannot be changed in-place:

  1. Open the old notebook.
  2. Export notebooks and data:
    • Save .ipynb files to a Git repo, Cloud Storage, or download locally.
    • Copy any data from the old persistent disk (e.g., to Cloud Storage).
  3. Open the new CMEK-encrypted notebook.
  4. Import your notebooks and data there.
  5. When confirmed working, stop and delete the old (non-CMEK) notebook instance.

If you tell me your notebook type (Managed vs User-Managed) and region, I can tailor the exact screen labels and service account to use.

Using CLI

Below are the remediation steps to ensure Vertex AI notebook instances use Customer-Managed Encryption Keys (CMEK) with the gcloud CLI.

Important: Encryption cannot be changed on an existing notebook instance. You must create a new CMEK-encrypted instance and migrate your work.

1. Set environment variables​

PROJECT_ID="your-project-id"
LOCATION="us-central1"    # or your Vertex AI region
KEYRING_NAME="vertex-ai-keyring"
KEY_NAME="vertex-ai-notebooks-key"
NOTEBOOK_NAME="vertex-cmek-notebook"
gcloud config set project "$PROJECT_ID"

2. Create a KMS key ring and key​

gcloud kms keyrings create "$KEYRING_NAME" \
  --location="$LOCATION"

gcloud kms keys create "$KEY_NAME" \
  --location="$LOCATION" \
  --keyring="$KEYRING_NAME" \
  --purpose="encryption"

Get the full key resource name:

KMS_KEY="projects/$PROJECT_ID/locations/$LOCATION/keyRings/$KEYRING_NAME/cryptoKeys/$KEY_NAME"
echo "$KMS_KEY"

3. Identify the service account used by the notebook​

For Vertex AI Workbench (User-managed notebooks), the default service account is usually the Compute Engine default:

SA="$(gcloud iam service-accounts list \
  --filter='displayName:Compute Engine default service account' \
  --format='value(email)')"
echo "$SA"

If you use a custom service account for notebooks, set it:

SA="your-notebook-sa@${PROJECT_ID}.iam.gserviceaccount.com"

4. Grant KMS permissions to the notebook service account​

gcloud kms keys add-iam-policy-binding "$KEY_NAME" \
  --location="$LOCATION" \
  --keyring="$KEYRING_NAME" \
  --member="serviceAccount:${SA}" \
  --role="roles/cloudkms.cryptoKeyEncrypterDecrypter"

5. Create a CMEK‑encrypted Vertex AI Workbench notebook​

A. User-managed notebook instance​

gcloud notebooks instances create "$NOTEBOOK_NAME" \
  --location="$LOCATION" \
  --vm-image-project="deeplearning-platform-release" \
  --vm-image-family="tf2-2-11-cpu" \
  --machine-type="n1-standard-4" \
  --service-account="$SA" \
  --boot-disk-type="pd-balanced" \
  --boot-disk-size="100GB" \
  --data-disk-type="pd-balanced" \
  --data-disk-size="100GB" \
  --kms-key="$KMS_KEY"

Key flags:

  • --kms-key – applies CMEK to the boot and data disks.
  • --service-account – must match the account granted KMS permissions.

B. Managed notebook runtime (if using Managed Notebooks)​

gcloud notebooks runtimes create "$NOTEBOOK_NAME" \
  --location="$LOCATION" \
  --machine-type="n1-standard-4" \
  --vm-image-project="deeplearning-platform-release" \
  --vm-image-family="tf2-2-11-cpu" \
  --service-account="$SA" \
  --boot-disk-type="pd-balanced" \
  --boot-disk-size="100GB" \
  --kms-key="$KMS_KEY"

6. Verify the notebook uses CMEK​

gcloud notebooks instances describe "$NOTEBOOK_NAME" \
  --location="$LOCATION" \
  --format="flattened(kmsKey,disks[].kmsKey)"

# Or for managed runtime:
gcloud notebooks runtimes describe "$NOTEBOOK_NAME" \
  --location="$LOCATION" \
  --format="flattened(kmsKey,virtualMachine.virtualMachineConfig.dataDisk.kmsKey)"

Confirm the kmsKey fields match your $KMS_KEY.

7. Migrate from existing non‑CMEK notebooks (manual)​

  1. In the existing notebook, back up notebooks/data (e.g., to Cloud Storage or Git).
  2. Create a new CMEK‑encrypted instance as above.
  3. Restore your notebooks/data to the new instance.
  4. Delete the old non‑CMEK instance:
gcloud notebooks instances delete "old-notebook-name" \
  --location="$LOCATION"
Using Python

Below is a concise, end‑to‑end way to ensure Vertex AI notebook instances use CMEK, using Python.

Key points:

  • CMEK must be set at creation time; you cannot change disk encryption of an existing instance.
  • You’ll typically need to (1) create/choose a KMS key, (2) grant the notebook’s service account access to the key, and (3) create the notebook instance with that CMEK.

1. Prerequisites​

pip install google-cloud-kms google-cloud-notebooks google-api-python-client
gcloud auth application-default login

Assume:

  • PROJECT_ID = your GCP project
  • LOCATION = e.g. us-central1
  • INSTANCE_ID = name for the notebook instance
  • SERVICE_ACCOUNT = SA that will run the notebook (or use default)

2. Create a CMEK key (Cloud KMS) via Python​

from google.cloud import kms

project_id = "YOUR_PROJECT_ID"
location_id = "us-central1"
key_ring_id = "vertex-notebooks-kr"
key_id = "vertex-notebooks-key"

client = kms.KeyManagementServiceClient()

# 1) Create Key Ring (if not exists)
parent = f"projects/{project_id}/locations/{location_id}"
key_ring_name = client.key_ring_path(project_id, location_id, key_ring_id)
try:
    key_ring = client.get_key_ring(name=key_ring_name)
except Exception:
    key_ring = client.create_key_ring(
        request={
            "parent": parent,
            "key_ring_id": key_ring_id,
            "key_ring": {},
        }
    )

# 2) Create Crypto Key (if not exists)
key_name = client.crypto_key_path(project_id, location_id, key_ring_id, key_id)
from google.cloud.kms_v1 import CryptoKey

try:
    crypto_key = client.get_crypto_key(name=key_name)
except Exception:
    crypto_key = client.create_crypto_key(
        request={
            "parent": key_ring.name,
            "crypto_key_id": key_id,
            "crypto_key": CryptoKey(
                purpose=CryptoKey.CryptoKeyPurpose.ENCRYPT_DECRYPT,
                rotation_period={"seconds": 60 * 60 * 24 * 30},  # 30 days
            ),
        }
    )

print("CMEK resource name:", crypto_key.name)

Note the CMEK resource name, e.g.:

projects/PROJECT_ID/locations/us-central1/keyRings/vertex-notebooks-kr/cryptoKeys/vertex-notebooks-key

3. Grant KMS access to the notebook service account​

Vertex AI Notebooks uses a service account (either user‑specified or a Compute Engine default SA). That SA must have roles/cloudkms.cryptoKeyEncrypterDecrypter on the KMS key.

Example using gcloud (easiest):

PROJECT_ID=YOUR_PROJECT_ID
LOCATION=us-central1
KEY_RING=vertex-notebooks-kr
KEY_ID=vertex-notebooks-key
SA=YOUR-NOTEBOOK-SA@${PROJECT_ID}.iam.gserviceaccount.com

gcloud kms keys add-iam-policy-binding ${KEY_ID} \
  --keyring=${KEY_RING} --location=${LOCATION} \
  --member="serviceAccount:${SA}" \
  --role="roles/cloudkms.cryptoKeyEncrypterDecrypter" \
  --project=${PROJECT_ID}

4. Create a Vertex AI Notebook instance with CMEK using Python​

For user-managed notebooks (legacy “AI Platform Notebooks” / NotebooksServiceClient):

from google.cloud import notebooks_v1

project_id = "YOUR_PROJECT_ID"
location = "us-central1-b"  # zone, not region
instance_id = "my-cmek-notebook"
service_account = "YOUR-NOTEBOOK-SA@YOUR_PROJECT_ID.iam.gserviceaccount.com"
kms_key = "projects/YOUR_PROJECT_ID/locations/us-central1/keyRings/vertex-notebooks-kr/cryptoKeys/vertex-notebooks-key"

client = notebooks_v1.NotebookServiceClient()

parent = f"projects/{project_id}/locations/{location}"

instance = notebooks_v1.Instance(
    name=f"{parent}/instances/{instance_id}",
    machine_type=f"projects/{project_id}/zones/{location}/machineTypes/n1-standard-4",
    service_account=service_account,
    # CMEK for boot disk
    disk_encryption="CMEK",  # or notebooks_v1.Instance.DiskEncryption.CMEK
    kms_key=kms_key,
)

operation = client.create_instance(
    request={
        "parent": parent,
        "instance_id": instance_id,
        "instance": instance,
    }
)

print("Creating instance...")
result = operation.result(timeout=1800)
print("Created instance:", result.name)

Notes:

  • disk_encryption="CMEK" tells GCE to use the KMS key for the boot disk.
  • kms_key must be in a region compatible with the zone (e.g. us-central1 for us-central1-b).

5. For Managed Notebooks (Vertex AI Workbench “managed”)​

Managed notebooks use the Vertex AI Notebooks API, but with Runtime/ManagedNotebookService. CMEK is set at the runtime level:

from google.cloud import notebooks_v1

project_id = "YOUR_PROJECT_ID"
location = "us-central1"
runtime_id = "my-cmek-runtime"
service_account = "YOUR-NOTEBOOK-SA@YOUR_PROJECT_ID.iam.gserviceaccount.com"
kms_key = "projects/YOUR_PROJECT_ID/locations/us-central1/keyRings/vertex-notebooks-kr/cryptoKeys/vertex-notebooks-key"

client = notebooks_v1.ManagedNotebookServiceClient()
parent = f"projects/{project_id}/locations/{location}"

runtime = notebooks_v1.Runtime(
    virtual_machine=notebooks_v1.VirtualMachine(
        virtual_machine_config=notebooks_v1.VirtualMachineConfig(
            machine_type="n1-standard-4",
            service_account=service_account,
            encryption_config=notebooks_v1.EncryptionConfig(kms_key=kms_key),
        )
    )
)

operation = client.create_runtime(
    request={
        "parent": parent,
        "runtime_id": runtime_id,
        "runtime": runtime,
    }
)

print("Creating runtime...")
result = operation.result(timeout=1800)
print("Created runtime:", result.name)

6. Migration: existing non‑CMEK notebooks​

Because encryption is fixed at disk creation, to “remediate” existing notebooks:

  1. Create a new notebook / runtime with CMEK (as above).
  2. Migrate data:
    • Copy notebooks and data to Cloud Storage (gs://) from old instance.
    • Download them from new CMEK‑protected instance.
  3. Delete the old, non‑CMEK instance and its disks/snapshots.
Using Terraform
resource "google_kms_crypto_key" "VERTEX_NOTEBOOK_CMEK_KEY" {
  name            = "VERTEX_NOTEBOOK_CMEK_KEY_NAME"   # replace with your key name
  key_ring        = "projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME" # replace
  purpose         = "ENCRYPT_DECRYPT"
  rotation_period = "7776000s" # 90 days; adjust as needed

  # Optional: customize labels, protection_level, etc.
}

resource "google_workbench_instance" "VERTEX_NOTEBOOK_INSTANCE" {
  name     = "VERTEX_NOTEBOOK_INSTANCE_NAME"   # replace with your instance name
  location = "LOCATION"                        # e.g. "us-central1"
  project  = "PROJECT_ID"                      # replace with your project ID

  gce_setup {
    machine_type = "e2-standard-4"

    # This is the CMEK setting the finding is asking for:
    kms_key = google_kms_crypto_key.VERTEX_NOTEBOOK_CMEK_KEY.id

    # ...other gce_setup fields as needed (network, service_account, boot_disk, etc.)
  }

  # ...other configuration as needed (labels, instance_owners, desired_state, etc.)
}

Substitute:

  • PROJECT_ID with your GCP project ID.
  • LOCATION with the region of the Workbench/Vertex AI notebook and KMS key.
  • KEY_RING_NAME and VERTEX_NOTEBOOK_CMEK_KEY_NAME with your actual KMS key ring and key.
  • VERTEX_NOTEBOOK_INSTANCE_NAME with your actual instance name.

Changing an existing Vertex AI Workbench/Notebook instance to use CMEK (kms_key) forces replacement of the instance; Terraform will destroy and recreate it, which can cause downtime and loss of any non‑persisted data on the VM.

To verify, terraform plan should show:

  • kms_key being added (or changed) in the gce_setup block of the google_workbench_instance resource.
  • No other unrelated changes.