> ## Documentation Index > Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Vertexai dataset encrypted with cmek remediation ### Triage and Remediation ### Remediation Below are the console-based steps to ensure Vertex AI datasets are encrypted with Customer-Managed Encryption Keys (CMEK). Note that CMEK can only be specified when creating a dataset; you can’t change the encryption key of an existing dataset. *** ## 1. Create or identify a CMEK key 1. In the Google Cloud Console, go to:\ **Security** → **Key Management** (or search “KMS” / “Key Management”). 2. Make sure you are in the **same project** and **region** that your Vertex AI resources will use. 3. If needed, create a key ring: * Click **Create key ring**. * Give it a **Name** and select the **Location** (match your Vertex AI region, e.g., `us-central1`). * Click **Create**. 4. Create a key: * Inside the key ring, click **Create key**. * Choose **Key purpose** = **Symmetric encrypt/decrypt**. * Configure rotation if desired; accept defaults or customize. * Click **Create**. *** ## 2. Grant Vertex AI permission to use the key Vertex AI uses a service agent to access CMEK. You must grant it `Encrypter/Decrypter` on the CryptoKey. 1. In Cloud Console, still under **Security** → **Key Management**: * Click the **key ring**, then the **key** you will use. 2. Go to the **Permissions** tab. 3. Click **Grant access**. 4. In **New principals**, add the Vertex AI service agent for your project: ```text theme={null} service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com ``` Replace `PROJECT_NUMBER` with your actual project number (not project ID). 5. In **Role**, select: * **Cloud KMS CryptoKey Encrypter/Decrypter**\ (`roles/cloudkms.cryptoKeyEncrypterDecrypter`) 6. Click **Save**. If you also use other Vertex AI features with CMEK, ensure any additional required service accounts have the same role. *** ## 3. Create a new Vertex AI dataset using CMEK 1. In the console, go to: **Vertex AI** → **Datasets**. 2. Click **Create** (or **+ Create dataset**). 3. Choose the **dataset type** (e.g., Image, Tabular, Text, etc.), then click **Next**. 4. Fill in: * **Dataset name** * **Region**: must match the region of your CMEK key (or a supported combination; ideally keep them the same). 5. Look for **Encryption** or **Customer-managed key** (may appear under “Advanced options” or “Encryption” section): * By default, it’s **Google-managed encryption key**. * Change to **Customer-managed key (CMEK)**. 6. In the key selector: * Choose the **Key ring**. * Choose the **Key** you created in KMS. 7. Complete the rest of the dataset creation flow: * Configure data source (e.g., GCS path). * Review settings. * Click **Create**. The dataset will now be encrypted with your Customer-Managed Encryption Key. *** ## 4. Handling existing datasets * Existing Vertex AI datasets that were created without CMEK **cannot be re-encrypted in place**. * To “migrate” to CMEK: 1. Create a **new dataset** following the CMEK steps above. 2. Point it to the **same source data** in Cloud Storage or re-import your data. 3. Update any jobs/pipelines/notebooks to use the new dataset. 4. Decommission the old dataset when you’re done. *** That’s all that’s required via the GCP Console: create/assign a KMS key, grant the Vertex AI service agent permissions, and select that key when creating new Vertex AI datasets. Below are step‑by‑step CLI instructions to enforce Customer‑Managed Encryption Keys (CMEK) for Vertex AI datasets. ## 1. Set environment variables ```bash theme={null} PROJECT_ID="your-project-id" LOCATION="us-central1" # or your Vertex AI region KMS_LOCATION="us-central1" # KMS key location; usually same as Vertex AI location KMS_KEYRING="my-key-ring" KMS_KEY="my-cmek-key" DATASET_DISPLAY_NAME="my-vertex-dataset" ``` ```bash theme={null} gcloud config set project "$PROJECT_ID" ``` *** ## 2. Create (or ensure you have) a KMS key If you do not have an existing key: ```bash theme={null} # Create key ring gcloud kms keyrings create "$KMS_KEYRING" \ --location="$KMS_LOCATION" # Create symmetric encryption key gcloud kms keys create "$KMS_KEY" \ --keyring="$KMS_KEYRING" \ --location="$KMS_LOCATION" \ --purpose="encryption" ``` Full key resource name: ```bash theme={null} KMS_KEY_NAME="projects/$PROJECT_ID/locations/$KMS_LOCATION/keyRings/$KMS_KEYRING/cryptoKeys/$KMS_KEY" ``` *** ## 3. Grant Vertex AI service account access to the key Determine your project number and the Vertex AI service account: ```bash theme={null} PROJECT_NUMBER=$(gcloud projects describe "$PROJECT_ID" --format="value(projectNumber)") VERTEX_SA="service-$PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com" ``` Grant KMS permissions: ```bash theme={null} gcloud kms keys add-iam-policy-binding "$KMS_KEY" \ --keyring="$KMS_KEYRING" \ --location="$KMS_LOCATION" \ --member="serviceAccount:$VERTEX_SA" \ --role="roles/cloudkms.cryptoKeyEncrypterDecrypter" ``` *** ## 4. Create a new Vertex AI dataset with CMEK CMEK must be set at **dataset creation time**; you cannot retroactively add CMEK to an existing dataset. You’ll need to create new datasets with the CMEK key. Example: create an empty tabular dataset: ```bash theme={null} gcloud ai datasets create \ --project="$PROJECT_ID" \ --region="$LOCATION" \ --display-name="$DATASET_DISPLAY_NAME" \ --metadata-schema-uri="gs://google-cloud-aiplatform/schema/dataset/metadata/tabular_1.0.0.yaml" \ --encryption-spec-key-name="$KMS_KEY_NAME" ``` For other dataset types, change `--metadata-schema-uri` accordingly (image, text, etc.). *** ## 5. (Optional) Import data into the CMEK‑protected dataset Example for a tabular CSV in Cloud Storage: ```bash theme={null} DATASET_ID="YOUR_DATASET_ID" # from output of the create command GCS_SOURCE="gs://your-bucket/path/to/data.csv" gcloud ai datasets import-data "$DATASET_ID" \ --project="$PROJECT_ID" \ --region="$LOCATION" \ --gcs-source-uris="$GCS_SOURCE" ``` *** ## 6. (Optional) Migrate from non‑CMEK datasets 1. Export or re‑locate your raw data to Cloud Storage (if needed). 2. Create a new CMEK dataset as in step 4. 3. Import the data into the new dataset (step 5). 4. Update any pipelines or models to use the new dataset ID. 5. Delete the old non‑CMEK dataset when no longer needed: ```bash theme={null} OLD_DATASET_ID="old-dataset-id" gcloud ai datasets delete "$OLD_DATASET_ID" \ --project="$PROJECT_ID" \ --region="$LOCATION" ``` This ensures all new Vertex AI datasets are encrypted with your Customer‑Managed Encryption Key. Below is a concise, step‑by‑step guide to ensure Vertex AI datasets use Customer‑Managed Encryption Keys (CMEK) with Python. > Note: CMEK must be specified at **dataset creation time**. You cannot change the encryption key of an existing Vertex AI dataset; you must recreate the dataset with CMEK. *** ## 1. Prerequisites * `gcloud` installed and configured. * Python 3.7+. * Libraries: ```bash theme={null} pip install google-cloud-aiplatform google-cloud-kms ``` * Environment: ```bash theme={null} export PROJECT_ID="your-project-id" export LOCATION="us-central1" # or your region ``` *** ## 2. Create a CMEK key in Cloud KMS ```bash theme={null} KEY_LOCATION="us-central1" # KMS location; must match Vertex AI region KEY_RING="vertex-ai-keyring" KEY_NAME="vertex-ai-cmek-key" gcloud kms keyrings create $KEY_RING \ --location=$KEY_LOCATION gcloud kms keys create $KEY_NAME \ --location=$KEY_LOCATION \ --keyring=$KEY_RING \ --purpose=encryption ``` The key resource ID will be: ```text theme={null} projects/PROJECT_ID/locations/us-central1/keyRings/vertex-ai-keyring/cryptoKeys/vertex-ai-cmek-key ``` *** ## 3. Grant Vertex AI service account access to the key Vertex AI uses a service agent like: ```text theme={null} service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com ``` Find your project number: ```bash theme={null} gcloud projects describe $PROJECT_ID --format="value(projectNumber)" ``` Then grant it `roles/cloudkms.cryptoKeyEncrypterDecrypter`: ```bash theme={null} PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)") SERVICE_AGENT="service-${PROJECT_NUMBER}@gcp-sa-aiplatform.iam.gserviceaccount.com" gcloud kms keys add-iam-policy-binding $KEY_NAME \ --location=$KEY_LOCATION \ --keyring=$KEY_RING \ --member="serviceAccount:${SERVICE_AGENT}" \ --role="roles/cloudkms.cryptoKeyEncrypterDecrypter" ``` *** ## 4. Python: Initialize Vertex AI with CMEK ```python theme={null} from google.cloud import aiplatform PROJECT_ID = "your-project-id" LOCATION = "us-central1" CMEK_KEY = ( "projects/your-project-id/locations/us-central1/" "keyRings/vertex-ai-keyring/cryptoKeys/vertex-ai-cmek-key" ) aiplatform.init(project=PROJECT_ID, location=LOCATION, encryption_spec_key_name=CMEK_KEY) ``` Setting `encryption_spec_key_name` in `aiplatform.init()` makes this the **default CMEK** for subsequent resources created in this context (including datasets). *** ## 5. Create a dataset using CMEK (Python) ### Example: Tabular dataset from BigQuery ```python theme={null} from google.cloud import aiplatform PROJECT_ID = "your-project-id" LOCATION = "us-central1" CMEK_KEY = ( "projects/your-project-id/locations/us-central1/" "keyRings/vertex-ai-keyring/cryptoKeys/vertex-ai-cmek-key" ) aiplatform.init(project=PROJECT_ID, location=LOCATION, encryption_spec_key_name=CMEK_KEY) bq_source = "bq://your-project-id.your_dataset.your_table" dataset = aiplatform.TabularDataset.create( display_name="my-cmek-tabular-dataset", bq_source=bq_source, # Optional: override init() default with an explicit CMEK encryption_spec_key_name=CMEK_KEY, ) print(f"Dataset created: {dataset.resource_name}") ``` ### Example: Image dataset from GCS ```python theme={null} from google.cloud import aiplatform PROJECT_ID = "your-project-id" LOCATION = "us-central1" CMEK_KEY = ( "projects/your-project-id/locations/us-central1/" "keyRings/vertex-ai-keyring/cryptoKeys/vertex-ai-cmek-key" ) aiplatform.init(project=PROJECT_ID, location=LOCATION, encryption_spec_key_name=CMEK_KEY) gcs_uri = "gs://your-bucket/path/to/images/" dataset = aiplatform.ImageDataset.create( display_name="my-cmek-image-dataset", gcs_source=[gcs_uri], import_schema_uri=aiplatform.schema.dataset.ioformat.image.single_label_classification, encryption_spec_key_name=CMEK_KEY, # explicit CMEK ) print(f"Dataset created: {dataset.resource_name}") ``` *** ## 6. Verify the dataset is using CMEK ```python theme={null} from google.cloud import aiplatform dataset = aiplatform.Dataset("projects/your-project-id/locations/us-central1/datasets/1234567890123456789") print(dataset.encryption_spec) ``` You should see the `kms_key_name` set to your CMEK key. *** ## 7. Handling existing datasets (misconfiguration) Since CMEK cannot be retroactively applied: 1. Export or re-point your data source (e.g., BigQuery table or GCS paths). 2. Recreate the dataset with CMEK as shown above. 3. Update any pipelines/models/jobs to use the new dataset. 4. Delete the old non‑CMEK dataset if no longer needed. ```hcl theme={null} resource "google_kms_crypto_key" "VERTEX_DATASET_CMEK_KEY" { name = "VERTEX_DATASET_CMEK_KEY_NAME" # replace with your key name key_ring = "projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME" # replace purpose = "ENCRYPT_DECRYPT" rotation_period = "2592000s" # 30 days; adjust as needed } resource "google_vertex_ai_dataset" "VERTEX_DATASET" { name = "VERTEX_DATASET_NAME" # replace with your dataset ID if importing, or omit to let Vertex AI generate display_name = "VERTEX_DATASET_DISPLAY" # replace with a human-readable name project = "PROJECT_ID" # replace region = "LOCATION" # e.g. "us-central1" # other required fields for your dataset type: # metadata_schema_uri = "SCHEMA_URI" # metadata = jsonencode({ ... }) encryption_spec { kms_key_name = google_kms_crypto_key.VERTEX_DATASET_CMEK_KEY.id } } ``` Changing or adding `encryption_spec.kms_key_name` on an existing `google_vertex_ai_dataset` forces replacement of the dataset resource; this may be disruptive and data may need to be re-imported, so plan carefully. To verify, run `terraform plan` and confirm it shows either: * creation of a new `google_vertex_ai_dataset` with `encryption_spec.kms_key_name` set to your CMEK, or * a `-/+` replacement of the existing dataset where the new resource includes `encryption_spec.kms_key_name`.