Skip to main content

Vertexai Dataset Encrypted With Cmek Remediation

Triage and Remediation​

Remediation​

Using Console

Below are the console-based steps to ensure Vertex AI datasets are encrypted with Customer-Managed Encryption Keys (CMEK). Note that CMEK can only be specified when creating a dataset; you can’t change the encryption key of an existing dataset.

1. Create or identify a CMEK key​

  1. In the Google Cloud Console, go to:
    Security → Key Management (or search “KMS” / “Key Management”).
  2. Make sure you are in the same project and region that your Vertex AI resources will use.
  3. If needed, create a key ring:
    • Click Create key ring.
    • Give it a Name and select the Location (match your Vertex AI region, e.g., us-central1).
    • Click Create.
  4. Create a key:
    • Inside the key ring, click Create key.
    • Choose Key purpose = Symmetric encrypt/decrypt.
    • Configure rotation if desired; accept defaults or customize.
    • Click Create.

2. Grant Vertex AI permission to use the key​

Vertex AI uses a service agent to access CMEK. You must grant it Encrypter/Decrypter on the CryptoKey.

  1. In Cloud Console, still under Security → Key Management:

    • Click the key ring, then the key you will use.

  2. Go to the Permissions tab.

  3. Click Grant access.

  4. In New principals, add the Vertex AI service agent for your project:

    service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com

    Replace PROJECT_NUMBER with your actual project number (not project ID).

  5. In Role, select:

    • Cloud KMS CryptoKey Encrypter/Decrypter
      (roles/cloudkms.cryptoKeyEncrypterDecrypter)

  6. Click Save.

If you also use other Vertex AI features with CMEK, ensure any additional required service accounts have the same role.

3. Create a new Vertex AI dataset using CMEK​

  1. In the console, go to: Vertex AI → Datasets.
  2. Click Create (or + Create dataset).
  3. Choose the dataset type (e.g., Image, Tabular, Text, etc.), then click Next.
  4. Fill in:
    • Dataset name
    • Region: must match the region of your CMEK key (or a supported combination; ideally keep them the same).
  5. Look for Encryption or Customer-managed key (may appear under “Advanced options” or “Encryption” section):
    • By default, it’s Google-managed encryption key.
    • Change to Customer-managed key (CMEK).
  6. In the key selector:
    • Choose the Key ring.
    • Choose the Key you created in KMS.
  7. Complete the rest of the dataset creation flow:
    • Configure data source (e.g., GCS path).
    • Review settings.
    • Click Create.

The dataset will now be encrypted with your Customer-Managed Encryption Key.

4. Handling existing datasets​

  • Existing Vertex AI datasets that were created without CMEK cannot be re-encrypted in place.
  • To “migrate” to CMEK:
    1. Create a new dataset following the CMEK steps above.
    2. Point it to the same source data in Cloud Storage or re-import your data.
    3. Update any jobs/pipelines/notebooks to use the new dataset.
    4. Decommission the old dataset when you’re done.

That’s all that’s required via the GCP Console: create/assign a KMS key, grant the Vertex AI service agent permissions, and select that key when creating new Vertex AI datasets.

Using CLI

Below are step‑by‑step CLI instructions to enforce Customer‑Managed Encryption Keys (CMEK) for Vertex AI datasets.

1. Set environment variables​

PROJECT_ID="your-project-id"
LOCATION="us-central1"   # or your Vertex AI region
KMS_LOCATION="us-central1"  # KMS key location; usually same as Vertex AI location
KMS_KEYRING="my-key-ring"
KMS_KEY="my-cmek-key"
DATASET_DISPLAY_NAME="my-vertex-dataset"
gcloud config set project "$PROJECT_ID"

2. Create (or ensure you have) a KMS key​

If you do not have an existing key:

# Create key ring
gcloud kms keyrings create "$KMS_KEYRING" \
  --location="$KMS_LOCATION"

# Create symmetric encryption key
gcloud kms keys create "$KMS_KEY" \
  --keyring="$KMS_KEYRING" \
  --location="$KMS_LOCATION" \
  --purpose="encryption"

Full key resource name:

KMS_KEY_NAME="projects/$PROJECT_ID/locations/$KMS_LOCATION/keyRings/$KMS_KEYRING/cryptoKeys/$KMS_KEY"

3. Grant Vertex AI service account access to the key​

Determine your project number and the Vertex AI service account:

PROJECT_NUMBER=$(gcloud projects describe "$PROJECT_ID" --format="value(projectNumber)")
VERTEX_SA="service-$PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com"

Grant KMS permissions:

gcloud kms keys add-iam-policy-binding "$KMS_KEY" \
  --keyring="$KMS_KEYRING" \
  --location="$KMS_LOCATION" \
  --member="serviceAccount:$VERTEX_SA" \
  --role="roles/cloudkms.cryptoKeyEncrypterDecrypter"

4. Create a new Vertex AI dataset with CMEK​

CMEK must be set at dataset creation time; you cannot retroactively add CMEK to an existing dataset. You’ll need to create new datasets with the CMEK key.

Example: create an empty tabular dataset:

gcloud ai datasets create \
  --project="$PROJECT_ID" \
  --region="$LOCATION" \
  --display-name="$DATASET_DISPLAY_NAME" \
  --metadata-schema-uri="gs://google-cloud-aiplatform/schema/dataset/metadata/tabular_1.0.0.yaml" \
  --encryption-spec-key-name="$KMS_KEY_NAME"

For other dataset types, change --metadata-schema-uri accordingly (image, text, etc.).

5. (Optional) Import data into the CMEK‑protected dataset​

Example for a tabular CSV in Cloud Storage:

DATASET_ID="YOUR_DATASET_ID"   # from output of the create command
GCS_SOURCE="gs://your-bucket/path/to/data.csv"

gcloud ai datasets import-data "$DATASET_ID" \
  --project="$PROJECT_ID" \
  --region="$LOCATION" \
  --gcs-source-uris="$GCS_SOURCE"

6. (Optional) Migrate from non‑CMEK datasets​

  1. Export or re‑locate your raw data to Cloud Storage (if needed).
  2. Create a new CMEK dataset as in step 4.
  3. Import the data into the new dataset (step 5).
  4. Update any pipelines or models to use the new dataset ID.
  5. Delete the old non‑CMEK dataset when no longer needed:
OLD_DATASET_ID="old-dataset-id"
gcloud ai datasets delete "$OLD_DATASET_ID" \
  --project="$PROJECT_ID" \
  --region="$LOCATION"

This ensures all new Vertex AI datasets are encrypted with your Customer‑Managed Encryption Key.

Using Python

Below is a concise, step‑by‑step guide to ensure Vertex AI datasets use Customer‑Managed Encryption Keys (CMEK) with Python.

Note: CMEK must be specified at dataset creation time. You cannot change the encryption key of an existing Vertex AI dataset; you must recreate the dataset with CMEK.

1. Prerequisites​

  • gcloud installed and configured.
  • Python 3.7+.
  • Libraries: 
    pip install google-cloud-aiplatform google-cloud-kms
  • Environment: 
    export PROJECT_ID="your-project-id"
    export LOCATION="us-central1"  # or your region

2. Create a CMEK key in Cloud KMS​

KEY_LOCATION="us-central1"          # KMS location; must match Vertex AI region
KEY_RING="vertex-ai-keyring"
KEY_NAME="vertex-ai-cmek-key"

gcloud kms keyrings create $KEY_RING \
  --location=$KEY_LOCATION

gcloud kms keys create $KEY_NAME \
  --location=$KEY_LOCATION \
  --keyring=$KEY_RING \
  --purpose=encryption

The key resource ID will be:

projects/PROJECT_ID/locations/us-central1/keyRings/vertex-ai-keyring/cryptoKeys/vertex-ai-cmek-key

3. Grant Vertex AI service account access to the key​

Vertex AI uses a service agent like:

service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com

Find your project number:

gcloud projects describe $PROJECT_ID --format="value(projectNumber)"

Then grant it roles/cloudkms.cryptoKeyEncrypterDecrypter:

PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")
SERVICE_AGENT="service-${PROJECT_NUMBER}@gcp-sa-aiplatform.iam.gserviceaccount.com"

gcloud kms keys add-iam-policy-binding $KEY_NAME \
  --location=$KEY_LOCATION \
  --keyring=$KEY_RING \
  --member="serviceAccount:${SERVICE_AGENT}" \
  --role="roles/cloudkms.cryptoKeyEncrypterDecrypter"

4. Python: Initialize Vertex AI with CMEK​

from google.cloud import aiplatform

PROJECT_ID = "your-project-id"
LOCATION = "us-central1"
CMEK_KEY = (
    "projects/your-project-id/locations/us-central1/"
    "keyRings/vertex-ai-keyring/cryptoKeys/vertex-ai-cmek-key"
)

aiplatform.init(project=PROJECT_ID, location=LOCATION, encryption_spec_key_name=CMEK_KEY)

Setting encryption_spec_key_name in aiplatform.init() makes this the default CMEK for subsequent resources created in this context (including datasets).

5. Create a dataset using CMEK (Python)​

Example: Tabular dataset from BigQuery​

from google.cloud import aiplatform

PROJECT_ID = "your-project-id"
LOCATION = "us-central1"
CMEK_KEY = (
    "projects/your-project-id/locations/us-central1/"
    "keyRings/vertex-ai-keyring/cryptoKeys/vertex-ai-cmek-key"
)

aiplatform.init(project=PROJECT_ID, location=LOCATION, encryption_spec_key_name=CMEK_KEY)

bq_source = "bq://your-project-id.your_dataset.your_table"

dataset = aiplatform.TabularDataset.create(
    display_name="my-cmek-tabular-dataset",
    bq_source=bq_source,
    # Optional: override init() default with an explicit CMEK
    encryption_spec_key_name=CMEK_KEY,
)

print(f"Dataset created: {dataset.resource_name}")

Example: Image dataset from GCS​

from google.cloud import aiplatform

PROJECT_ID = "your-project-id"
LOCATION = "us-central1"
CMEK_KEY = (
    "projects/your-project-id/locations/us-central1/"
    "keyRings/vertex-ai-keyring/cryptoKeys/vertex-ai-cmek-key"
)

aiplatform.init(project=PROJECT_ID, location=LOCATION, encryption_spec_key_name=CMEK_KEY)

gcs_uri = "gs://your-bucket/path/to/images/"

dataset = aiplatform.ImageDataset.create(
    display_name="my-cmek-image-dataset",
    gcs_source=[gcs_uri],
    import_schema_uri=aiplatform.schema.dataset.ioformat.image.single_label_classification,
    encryption_spec_key_name=CMEK_KEY,  # explicit CMEK
)

print(f"Dataset created: {dataset.resource_name}")

6. Verify the dataset is using CMEK​

from google.cloud import aiplatform

dataset = aiplatform.Dataset("projects/your-project-id/locations/us-central1/datasets/1234567890123456789")
print(dataset.encryption_spec)

You should see the kms_key_name set to your CMEK key.

7. Handling existing datasets (misconfiguration)​

Since CMEK cannot be retroactively applied:

  1. Export or re-point your data source (e.g., BigQuery table or GCS paths).
  2. Recreate the dataset with CMEK as shown above.
  3. Update any pipelines/models/jobs to use the new dataset.
  4. Delete the old non‑CMEK dataset if no longer needed.
Using Terraform
resource "google_kms_crypto_key" "VERTEX_DATASET_CMEK_KEY" {
  name            = "VERTEX_DATASET_CMEK_KEY_NAME" # replace with your key name
  key_ring        = "projects/PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_NAME" # replace
  purpose         = "ENCRYPT_DECRYPT"
  rotation_period = "2592000s" # 30 days; adjust as needed
}

resource "google_vertex_ai_dataset" "VERTEX_DATASET" {
  name         = "VERTEX_DATASET_NAME"    # replace with your dataset ID if importing, or omit to let Vertex AI generate
  display_name = "VERTEX_DATASET_DISPLAY" # replace with a human-readable name
  project      = "PROJECT_ID"             # replace
  region       = "LOCATION"               # e.g. "us-central1"

  # other required fields for your dataset type:
  # metadata_schema_uri = "SCHEMA_URI"
  # metadata            = jsonencode({ ... })

  encryption_spec {
    kms_key_name = google_kms_crypto_key.VERTEX_DATASET_CMEK_KEY.id
  }
}

Changing or adding encryption_spec.kms_key_name on an existing google_vertex_ai_dataset forces replacement of the dataset resource; this may be disruptive and data may need to be re-imported, so plan carefully.

To verify, run terraform plan and confirm it shows either:

  • creation of a new google_vertex_ai_dataset with encryption_spec.kms_key_name set to your CMEK, or
  • a -/+ replacement of the existing dataset where the new resource includes encryption_spec.kms_key_name.