> ## Documentation Index > Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Sagemaker Clusters Should Be Encrypted With KMS Customer Managed Keys ### More Info: Sagemaker Clusters should be encrypted with KMS Customer Managed Keys ### Risk Level High ### Address Security ### Compliance Standards * APRA CPS 234 (Australia) * BSI C5 (Germany) * Brazil LGPD * CCPA / CPRA (California) * CIS Critical Security Controls v8 * CMMC 2.0 * CSA Cloud Controls Matrix v4 * DPDPA * Digital Operational Resilience Act (EU) * ISO 27001 * ISO/IEC 27018 * ISO/IEC 27701 * MAS Technology Risk Management (Singapore) * MITRE ATT\&CK (Cloud) * NIST SP 800-171 * NYDFS 23 NYCRR 500 * SOC2 * SWIFT Customer Security Controls Framework * Sarbanes-Oxley IT General Controls * UK NCSC Cyber Assessment Framework ### Triage and Remediation ### Remediation Below are the console-based steps to ensure SageMaker resources are encrypted with a **KMS Customer Managed Key (CMK)** instead of default/AWS-managed keys. *** ## 1. Create a KMS Customer Managed Key 1. Sign in to AWS Console and go to **Key Management Service (KMS)**. 2. In the left pane, choose **Customer managed keys** → **Create key**. 3. Key type: **Symmetric** → **Next**. 4. Set an alias, e.g., `alias/sagemaker-cmk`. 5. Configure key administrators (IAM roles/users that can manage the key). 6. Configure key usage permissions (principals that SageMaker uses, e.g.: * Your SageMaker execution roles (e.g., `AmazonSageMaker-ExecutionRole-...`) * Any IAM roles/users that create or use SageMaker resources). 7. Review and **Create key**. 8. Copy the **Key ID** or **ARN** (you’ll need it in SageMaker). *** ## 2. Use the CMK for New SageMaker Resources ### A. SageMaker Studio Domain 1. Open **Amazon SageMaker** console → **Domains** (under **Admin configurations**). 2. For a new domain: * Choose **Create domain**. * Under **Encryption settings**: * For **Amazon EFS encryption key**: select your CMK (`alias/sagemaker-cmk`). * For **Amazon S3 encryption key** (if shown): select CMK as well. * Complete the wizard. 3. For an existing domain (if the console allows editing in your region/version): * Go to **Domains** → select your domain. * Choose **Edit** (or **Update**). * Under **Encryption settings**, change the EFS/S3 keys to your CMK. * Save changes. > If encryption key fields are not editable, you must create a **new domain** with the CMK and migrate user profiles. *** ### B. SageMaker Notebook Instances You cannot change the KMS key of an existing notebook instance; you must create a new one with CMK and migrate notebooks. 1. In **SageMaker console** → **Notebook instances** → **Create notebook instance**. 2. Fill basic settings (name, instance type, role). 3. Expand **Additional configuration** (or **Encryption** section depending on UI). 4. For **Encryption key**, select your CMK (`alias/sagemaker-cmk`). 5. Create the instance. 6. Migrate data from the old instance: * Copy notebooks from the old instance to S3 (or Git). * Stop/terminate old unencrypted notebook instance once migration is complete. *** ### C. Training / Processing / Transform Jobs When creating jobs from the console: 1. **Training jobs**: * Go to **Training** → **Training jobs** → **Create training job**. * Under **Input data configuration** / **Output data configuration**, set: * **KMS key for model artifacts / output data**: select your CMK. * Under **Resource configuration**: * **Volume KMS key** (or similar field): select your CMK. 2. **Processing jobs**: * Under **Output configuration** and **Volume configuration**, specify the CMK in any **KMS key** fields. 3. **Batch transform jobs**: * Under **Output data configuration**, set **KMS key** to the CMK. For jobs started from the console preconfigured with defaults, ensure every **Encryption key / KMS key** dropdown is set to the customer-managed KMS key. *** ### D. Endpoints and Models 1. **Models**: * **SageMaker console** → **Models** → **Create model**. * Under **Container definition** → **Model artifacts** (S3), if there is an option for KMS: * Ensure the S3 bucket/object is encrypted with your CMK (set via S3 bucket policy/default encryption, using CMK). 2. **Endpoints / Endpoint configurations**: * When creating an **Endpoint configuration**, look for: * **KMS key for storage volume** or **Encryption key**: select your CMK. * Create the endpoint using that endpoint configuration. For existing endpoints, you generally need to: * Create a new **Endpoint configuration** referencing the CMK. * Update the endpoint to use the new configuration (this will perform a rolling update). *** ## 3. Ensure S3 Buckets Used by SageMaker Use CMK 1. Go to **S3** → select bucket(s) used for SageMaker input/output and model artifacts. 2. **Properties** → **Default encryption** → **Edit**. 3. Choose: * **Server-side encryption**: **AWS KMS keys (SSE-KMS)**. * **AWS KMS key**: select your CMK. 4. Save changes. *** ## 4. IAM Permission Check Confirm that: * SageMaker execution roles have `kms:Encrypt`, `kms:Decrypt`, `kms:GenerateDataKey*`, `kms:DescribeKey` permissions on the CMK. * The CMK key policy allows those roles as **key users**. You can edit the key’s **Key policy** in KMS or use the console wizard to add the SageMaker roles. *** If you tell me which specific SageMaker resources you have (Studio, classic notebooks, endpoints, training jobs), I can give you an exact click-path for each one. Below are the practical AWS CLI steps to ensure SageMaker resources (“clusters”) are encrypted with a KMS customer-managed key (CMK). SageMaker does **not** let you “turn on KMS” in place; you must use a CMK when you create or recreate the resource. *** ## 1. Create (or identify) a KMS CMK If you don’t already have a CMK: ```bash theme={null} aws kms create-key \ --description "CMK for SageMaker data encryption" \ --key-usage ENCRYPT_DECRYPT \ --origin AWS_KMS ``` Note the returned `KeyId` (or ARN). You can also alias it: ```bash theme={null} aws kms create-alias \ --alias-name alias/sagemaker-cmk \ --target-key-id ``` You can use either `alias/sagemaker-cmk` or the full key ARN in SageMaker. Make sure the key policy/IAM permissions allow your SageMaker execution roles to use `kms:Encrypt`, `kms:Decrypt`, `kms:GenerateDataKey*`, etc. *** ## 2. Notebook Instances – use `--kms-key-id` You cannot change the KMS key of an *existing* notebook instance. You must: 1. Stop and delete the old notebook (after backing up data if needed). 2. Recreate it with a CMK. Example: ```bash theme={null} aws sagemaker create-notebook-instance \ --notebook-instance-name my-notebook-encrypted \ --instance-type ml.t3.medium \ --role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole \ --kms-key-id alias/sagemaker-cmk \ --subnet-id subnet-abc123 \ --security-group-ids sg-abc123 ``` *** ## 3. Training Jobs – encrypt volumes and output You can’t modify a running/completed training job, but you can ensure **new jobs** use a CMK via `--resource-config` and (optionally) `--output-data-config`. ```bash theme={null} aws sagemaker create-training-job \ --training-job-name my-training-job-encrypted \ --role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole \ --algorithm-specification TrainingImage=,TrainingInputMode=File \ --input-data-config '[{"ChannelName":"train","DataSource":{"S3DataSource":{"S3DataType":"S3Prefix","S3Uri":"s3://my-bucket/train/","S3DataDistributionType":"FullyReplicated"}}}]' \ --output-data-config '{"S3OutputPath":"s3://my-bucket/output/","KmsKeyId":"alias/sagemaker-cmk"}' \ --resource-config '{"InstanceType":"ml.m5.xlarge","InstanceCount":1,"VolumeSizeInGB":50,"VolumeKmsKeyId":"alias/sagemaker-cmk"}' \ --stopping-condition MaxRuntimeInSeconds=3600 ``` Key bits: * `VolumeKmsKeyId` in `--resource-config` * `KmsKeyId` in `--output-data-config` (for S3 output encryption with that CMK) *** ## 4. Endpoint Configurations – encrypt model data For real-time endpoints, the “cluster” storage is defined in the endpoint configuration: ```bash theme={null} aws sagemaker create-endpoint-config \ --endpoint-config-name my-endpoint-config-encrypted \ --production-variants '[ { "VariantName": "AllTraffic", "ModelName": "my-model", "InitialInstanceCount": 2, "InstanceType": "ml.m5.large", "InitialVariantWeight": 1.0 } ]' \ --kms-key-id alias/sagemaker-cmk ``` Then create or update an endpoint to use that config: ```bash theme={null} aws sagemaker create-endpoint \ --endpoint-name my-endpoint \ --endpoint-config-name my-endpoint-config-encrypted ``` To remediate an existing endpoint, create a new encrypted endpoint config as above, then: ```bash theme={null} aws sagemaker update-endpoint \ --endpoint-name my-endpoint \ --endpoint-config-name my-endpoint-config-encrypted ``` *** ## 5. SageMaker Studio Domains & User Profiles (if applicable) For Studio, you specify KMS keys at domain creation time; for existing domains, some encryption settings are not mutable and require re-creation if your security policy demands CMK use everywhere. Example (simplified): ```bash theme={null} aws sagemaker create-domain \ --domain-name my-studio-domain \ --auth-mode IAM \ --default-user-settings '{ "ExecutionRole":"arn:aws:iam::123456789012:role/SageMakerExecutionRole", "SecurityGroups":["sg-abc123"] }' \ --subnet-ids subnet-abc123 \ --vpc-id vpc-abc123 \ --kms-key-id alias/sagemaker-cmk ``` *** ## 6. Verification For each resource type, use `describe-*` and look for the KMS fields: * Notebook: `aws sagemaker describe-notebook-instance --notebook-instance-name ...`\ → `KmsKeyId` * Training job: `aws sagemaker describe-training-job --training-job-name ...`\ → `ResourceConfig.VolumeKmsKeyId`, `OutputDataConfig.KmsKeyId` * Endpoint config: `aws sagemaker describe-endpoint-config --endpoint-config-name ...`\ → `KmsKeyId` * Domain: `aws sagemaker describe-domain --domain-id ...`\ → `KmsKeyId` Ensure they point to your CMK (alias or ARN), not the default AWS-managed keys. Below is a practical, step‑by‑step way to enforce KMS CMK encryption for SageMaker resources using Python (boto3). Adjust to whatever “cluster” means in your environment (most often: training jobs, notebook instances, endpoint configs, processing jobs, or SageMaker Studio domains). *** ## 1. Prerequisites ```bash theme={null} pip install boto3 aws configure # or otherwise provide credentials/role ``` Ensure your IAM role has permissions for: * `kms:CreateKey`, `kms:DescribeKey`, `kms:ListAliases`, `kms:CreateAlias` * `kms:Encrypt`, `kms:Decrypt`, `kms:GenerateDataKey*` * `sagemaker:*` (or the specific APIs you use) *** ## 2. Create (or re‑use) a KMS Customer Managed Key (CMK) ```python theme={null} import boto3 kms = boto3.client("kms", region_name="us-east-1") # Create a symmetric CMK resp = kms.create_key( Description="CMK for SageMaker encryption", KeyUsage="ENCRYPT_DECRYPT", Origin="AWS_KMS" ) cmk_arn = resp["KeyMetadata"]["Arn"] print("CMK ARN:", cmk_arn) # Optional: create an alias for easier reference kms.create_alias( AliasName="alias/sagemaker-cmk", TargetKeyId=cmk_arn ) ``` You can then use either `cmk_arn` or `"alias/sagemaker-cmk"` in SageMaker APIs. *** ## 3. New SageMaker resources: use CMK in create calls ### 3.1. Notebook instances ```python theme={null} import boto3 sm = boto3.client("sagemaker", region_name="us-east-1") sm.create_notebook_instance( NotebookInstanceName="my-kms-encrypted-notebook", InstanceType="ml.t3.medium", RoleArn="arn:aws:iam::123456789012:role/MySageMakerRole", KmsKeyId="alias/sagemaker-cmk", # or cmk_arn VolumeSizeInGB=10 ) ``` ### 3.2. Training jobs ```python theme={null} sm.create_training_job( TrainingJobName="my-kms-training-job", RoleArn="arn:aws:iam::123456789012:role/MySageMakerRole", AlgorithmSpecification={ "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest", "TrainingInputMode": "File" }, InputDataConfig=[{ "ChannelName": "training", "DataSource": { "S3DataSource": { "S3DataType": "S3Prefix", "S3Uri": "s3://my-bucket/training-data/", "S3DataDistributionType": "FullyReplicated" } } }], OutputDataConfig={ "S3OutputPath": "s3://my-bucket/output/", "KmsKeyId": "alias/sagemaker-cmk" # encrypt model artifacts }, ResourceConfig={ "InstanceType": "ml.m5.large", "InstanceCount": 1, "VolumeSizeInGB": 50, "VolumeKmsKeyId": "alias/sagemaker-cmk" # encrypt EBS volumes }, StoppingCondition={"MaxRuntimeInSeconds": 3600} ) ``` ### 3.3. Endpoint configurations ```python theme={null} sm.create_endpoint_config( EndpointConfigName="my-kms-endpoint-config", ProductionVariants=[{ "VariantName": "AllTraffic", "ModelName": "my-trained-model", "InitialInstanceCount": 1, "InstanceType": "ml.m5.large" }], KmsKeyId="alias/sagemaker-cmk" # encrypt endpoint storage ) ``` ### 3.4. Processing jobs ```python theme={null} sm.create_processing_job( ProcessingJobName="my-kms-processing-job", RoleArn="arn:aws:iam::123456789012:role/MySageMakerRole", AppSpecification={ "ImageUri": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-processing-image:latest" }, ProcessingResources={ "ClusterConfig": { "InstanceCount": 2, "InstanceType": "ml.m5.xlarge", "VolumeSizeInGB": 50, "VolumeKmsKeyId": "alias/sagemaker-cmk" # EBS for processing cluster } }, ProcessingOutputConfig={ "Outputs": [{ "OutputName": "output", "S3Output": { "S3Uri": "s3://my-bucket/processing-output/", "LocalPath": "/opt/ml/processing/output", "S3UploadMode": "EndOfJob" } }], "KmsKeyId": "alias/sagemaker-cmk" # S3 output encryption } ) ``` ### 3.5. SageMaker Studio domain If you’re using Studio “clusters” (Jupyter kernels, apps) you must set encryption at the domain: ```python theme={null} sm.create_domain( DomainName="my-kms-domain", AuthMode="IAM", DefaultUserSettings={ "ExecutionRole": "arn:aws:iam::123456789012:role/MySageMakerRole" }, SubnetIds=["subnet-xxxxxxxx"], VpcId="vpc-xxxxxxxx", KmsKeyId="alias/sagemaker-cmk" # encrypt EFS data for Studio ) ``` For existing domains you must use `update_domain`: ```python theme={null} sm.update_domain( DomainId="d-xxxxxxxxxxxx", DefaultUserSettings={ "ExecutionRole": "arn:aws:iam::123456789012:role/MySageMakerRole", "SecurityGroups": ["sg-xxxxxxxx"], "JupyterServerAppSettings": {}, "KernelGatewayAppSettings": {} }, KmsKeyId="alias/sagemaker-cmk" ) ``` (Only some fields are required; include what your environment needs.) *** ## 4. Existing SageMaker resources without CMK Most SageMaker resources *cannot* be retrofitted with a KMS key in place; you typically must: 1. Discover resources without CMK. 2. Recreate them with `KmsKeyId` / `VolumeKmsKeyId` / `OutputDataConfig.KmsKeyId` set. Example discovery pattern (for training jobs): ```python theme={null} unencrypted = [] paginator = sm.get_paginator("list_training_jobs") for page in paginator.paginate(): for job_summary in page["TrainingJobSummaries"]: job_name = job_summary["TrainingJobName"] desc = sm.describe_training_job(TrainingJobName=job_name) vol_kms = desc["ResourceConfig"].get("VolumeKmsKeyId") out_kms = desc["OutputDataConfig"].get("KmsKeyId") if not vol_kms or not out_kms: unencrypted.append(job_name) print("Unencrypted training jobs:", unencrypted) ``` You can then: * For notebooks: create new notebooks with `KmsKeyId`, move data, then delete old ones. * For endpoints: create new endpoint configs with `KmsKeyId`, update endpoints to use them. * For jobs: enforce CMK for all **new** jobs; old jobs are historical. *** ## 5. KMS key policy (optional but recommended) Ensure your SageMaker execution role can use the CMK: ```python theme={null} key_id = cmk_arn.split("/")[-1] policy = kms.get_key_policy(KeyId=key_id, PolicyName="default")["Policy"] # Optionally edit `policy` JSON to add the SageMaker role principal and then: kms.put_key_policy( KeyId=key_id, PolicyName="default", Policy=policy ) ``` *** If you tell me which exact SageMaker resource type your “clusters” refer to (e.g., Studio domains, processing job clusters, or training clusters), I can give a narrower script tailored precisely to that.