Sagemaker Clusters Should Be Encrypted With KMS Customer Managed Keys

Using Console

Below are the console-based steps to ensure SageMaker resources are encrypted with a KMS Customer Managed Key (CMK) instead of default/AWS-managed keys.

1. Create a KMS Customer Managed Key

Sign in to AWS Console and go to Key Management Service (KMS).
In the left pane, choose Customer managed keys → Create key.
Key type: Symmetric → Next.
Set an alias, e.g., alias/sagemaker-cmk.
Configure key administrators (IAM roles/users that can manage the key).
Configure key usage permissions (principals that SageMaker uses, e.g.:
- Your SageMaker execution roles (e.g., AmazonSageMaker-ExecutionRole-...)
- Any IAM roles/users that create or use SageMaker resources).
Review and Create key.
Copy the Key ID or ARN (you’ll need it in SageMaker).

2. Use the CMK for New SageMaker Resources

A. SageMaker Studio Domain

Open Amazon SageMaker console → Domains (under Admin configurations).
For a new domain:
- Choose Create domain.
- Under Encryption settings:
  - For Amazon EFS encryption key: select your CMK (alias/sagemaker-cmk).
  - For Amazon S3 encryption key (if shown): select CMK as well.
- Complete the wizard.
For an existing domain (if the console allows editing in your region/version):
- Go to Domains → select your domain.
- Choose Edit (or Update).
- Under Encryption settings, change the EFS/S3 keys to your CMK.
- Save changes.
If encryption key fields are not editable, you must create a new domain with the CMK and migrate user profiles.

B. SageMaker Notebook Instances

You cannot change the KMS key of an existing notebook instance; you must create a new one with CMK and migrate notebooks.

In SageMaker console → Notebook instances → Create notebook instance.
Fill basic settings (name, instance type, role).
Expand Additional configuration (or Encryption section depending on UI).
For Encryption key, select your CMK (alias/sagemaker-cmk).
Create the instance.
Migrate data from the old instance:
- Copy notebooks from the old instance to S3 (or Git).
- Stop/terminate old unencrypted notebook instance once migration is complete.

C. Training / Processing / Transform Jobs

When creating jobs from the console:

Training jobs:
- Go to Training → Training jobs → Create training job.
- Under Input data configuration / Output data configuration, set:
  - KMS key for model artifacts / output data: select your CMK.
- Under Resource configuration:
  - Volume KMS key (or similar field): select your CMK.
Processing jobs:
- Under Output configuration and Volume configuration, specify the CMK in any KMS key fields.
Batch transform jobs:
- Under Output data configuration, set KMS key to the CMK.

For jobs started from the console preconfigured with defaults, ensure every Encryption key / KMS key dropdown is set to the customer-managed KMS key.

D. Endpoints and Models

Models:
- SageMaker console → Models → Create model.
- Under Container definition → Model artifacts (S3), if there is an option for KMS:
  - Ensure the S3 bucket/object is encrypted with your CMK (set via S3 bucket policy/default encryption, using CMK).
Endpoints / Endpoint configurations:
- When creating an Endpoint configuration, look for:
  - KMS key for storage volume or Encryption key: select your CMK.
- Create the endpoint using that endpoint configuration.

For existing endpoints, you generally need to:

Create a new Endpoint configuration referencing the CMK.
Update the endpoint to use the new configuration (this will perform a rolling update).

3. Ensure S3 Buckets Used by SageMaker Use CMK

Go to S3 → select bucket(s) used for SageMaker input/output and model artifacts.
Properties → Default encryption → Edit.
Choose:
- Server-side encryption: AWS KMS keys (SSE-KMS).
- AWS KMS key: select your CMK.
Save changes.

4. IAM Permission Check

Confirm that:

SageMaker execution roles have kms:Encrypt, kms:Decrypt, kms:GenerateDataKey*, kms:DescribeKey permissions on the CMK.
The CMK key policy allows those roles as key users.

You can edit the key’s Key policy in KMS or use the console wizard to add the SageMaker roles.

If you tell me which specific SageMaker resources you have (Studio, classic notebooks, endpoints, training jobs), I can give you an exact click-path for each one.

Using CLI

Below are the practical AWS CLI steps to ensure SageMaker resources (“clusters”) are encrypted with a KMS customer-managed key (CMK). SageMaker does not let you “turn on KMS” in place; you must use a CMK when you create or recreate the resource.

1. Create (or identify) a KMS CMK

If you don’t already have a CMK:

aws kms create-key \
  --description "CMK for SageMaker data encryption" \
  --key-usage ENCRYPT_DECRYPT \
  --origin AWS_KMS

Note the returned KeyId (or ARN). You can also alias it:

aws kms create-alias \
  --alias-name alias/sagemaker-cmk \
  --target-key-id <YOUR_KEY_ID>

You can use either alias/sagemaker-cmk or the full key ARN in SageMaker.Make sure the key policy/IAM permissions allow your SageMaker execution roles to use kms:Encrypt, kms:Decrypt, kms:GenerateDataKey*, etc.

2. Notebook Instances – use `--kms-key-id`

You cannot change the KMS key of an existing notebook instance. You must:

Stop and delete the old notebook (after backing up data if needed).
Recreate it with a CMK.

Example:

aws sagemaker create-notebook-instance \
  --notebook-instance-name my-notebook-encrypted \
  --instance-type ml.t3.medium \
  --role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole \
  --kms-key-id alias/sagemaker-cmk \
  --subnet-id subnet-abc123 \
  --security-group-ids sg-abc123

3. Training Jobs – encrypt volumes and output

You can’t modify a running/completed training job, but you can ensure new jobs use a CMK via --resource-config and (optionally) --output-data-config.

aws sagemaker create-training-job \
  --training-job-name my-training-job-encrypted \
  --role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole \
  --algorithm-specification TrainingImage=<IMAGE>,TrainingInputMode=File \
  --input-data-config '[{"ChannelName":"train","DataSource":{"S3DataSource":{"S3DataType":"S3Prefix","S3Uri":"s3://my-bucket/train/","S3DataDistributionType":"FullyReplicated"}}}]' \
  --output-data-config '{"S3OutputPath":"s3://my-bucket/output/","KmsKeyId":"alias/sagemaker-cmk"}' \
  --resource-config '{"InstanceType":"ml.m5.xlarge","InstanceCount":1,"VolumeSizeInGB":50,"VolumeKmsKeyId":"alias/sagemaker-cmk"}' \
  --stopping-condition MaxRuntimeInSeconds=3600

Key bits:

VolumeKmsKeyId in --resource-config
KmsKeyId in --output-data-config (for S3 output encryption with that CMK)

4. Endpoint Configurations – encrypt model data

For real-time endpoints, the “cluster” storage is defined in the endpoint configuration:

aws sagemaker create-endpoint-config \
  --endpoint-config-name my-endpoint-config-encrypted \
  --production-variants '[
    {
      "VariantName": "AllTraffic",
      "ModelName": "my-model",
      "InitialInstanceCount": 2,
      "InstanceType": "ml.m5.large",
      "InitialVariantWeight": 1.0
    }
  ]' \
  --kms-key-id alias/sagemaker-cmk

Then create or update an endpoint to use that config:

aws sagemaker create-endpoint \
  --endpoint-name my-endpoint \
  --endpoint-config-name my-endpoint-config-encrypted

To remediate an existing endpoint, create a new encrypted endpoint config as above, then:

aws sagemaker update-endpoint \
  --endpoint-name my-endpoint \
  --endpoint-config-name my-endpoint-config-encrypted

5. SageMaker Studio Domains & User Profiles (if applicable)

For Studio, you specify KMS keys at domain creation time; for existing domains, some encryption settings are not mutable and require re-creation if your security policy demands CMK use everywhere.Example (simplified):

aws sagemaker create-domain \
  --domain-name my-studio-domain \
  --auth-mode IAM \
  --default-user-settings '{
    "ExecutionRole":"arn:aws:iam::123456789012:role/SageMakerExecutionRole",
    "SecurityGroups":["sg-abc123"]
  }' \
  --subnet-ids subnet-abc123 \
  --vpc-id vpc-abc123 \
  --kms-key-id alias/sagemaker-cmk

6. Verification

For each resource type, use describe-* and look for the KMS fields:

Notebook: aws sagemaker describe-notebook-instance --notebook-instance-name ...
→ KmsKeyId
Training job: aws sagemaker describe-training-job --training-job-name ...
→ ResourceConfig.VolumeKmsKeyId, OutputDataConfig.KmsKeyId
Endpoint config: aws sagemaker describe-endpoint-config --endpoint-config-name ...
→ KmsKeyId
Domain: aws sagemaker describe-domain --domain-id ...
→ KmsKeyId

Ensure they point to your CMK (alias or ARN), not the default AWS-managed keys.

Using Python

Below is a practical, step‑by‑step way to enforce KMS CMK encryption for SageMaker resources using Python (boto3). Adjust to whatever “cluster” means in your environment (most often: training jobs, notebook instances, endpoint configs, processing jobs, or SageMaker Studio domains).

1. Prerequisites

pip install boto3
aws configure  # or otherwise provide credentials/role

Ensure your IAM role has permissions for:

kms:CreateKey, kms:DescribeKey, kms:ListAliases, kms:CreateAlias
kms:Encrypt, kms:Decrypt, kms:GenerateDataKey*
sagemaker:* (or the specific APIs you use)

2. Create (or re‑use) a KMS Customer Managed Key (CMK)

import boto3

kms = boto3.client("kms", region_name="us-east-1")

# Create a symmetric CMK
resp = kms.create_key(
    Description="CMK for SageMaker encryption",
    KeyUsage="ENCRYPT_DECRYPT",
    Origin="AWS_KMS"
)

cmk_arn = resp["KeyMetadata"]["Arn"]
print("CMK ARN:", cmk_arn)

# Optional: create an alias for easier reference
kms.create_alias(
    AliasName="alias/sagemaker-cmk",
    TargetKeyId=cmk_arn
)

You can then use either cmk_arn or "alias/sagemaker-cmk" in SageMaker APIs.

3. New SageMaker resources: use CMK in create calls

3.1. Notebook instances

import boto3
sm = boto3.client("sagemaker", region_name="us-east-1")

sm.create_notebook_instance(
    NotebookInstanceName="my-kms-encrypted-notebook",
    InstanceType="ml.t3.medium",
    RoleArn="arn:aws:iam::123456789012:role/MySageMakerRole",
    KmsKeyId="alias/sagemaker-cmk",  # or cmk_arn
    VolumeSizeInGB=10
)

3.2. Training jobs

sm.create_training_job(
    TrainingJobName="my-kms-training-job",
    RoleArn="arn:aws:iam::123456789012:role/MySageMakerRole",
    AlgorithmSpecification={
        "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",
        "TrainingInputMode": "File"
    },
    InputDataConfig=[{
        "ChannelName": "training",
        "DataSource": {
            "S3DataSource": {
                "S3DataType": "S3Prefix",
                "S3Uri": "s3://my-bucket/training-data/",
                "S3DataDistributionType": "FullyReplicated"
            }
        }
    }],
    OutputDataConfig={
        "S3OutputPath": "s3://my-bucket/output/",
        "KmsKeyId": "alias/sagemaker-cmk"   # encrypt model artifacts
    },
    ResourceConfig={
        "InstanceType": "ml.m5.large",
        "InstanceCount": 1,
        "VolumeSizeInGB": 50,
        "VolumeKmsKeyId": "alias/sagemaker-cmk"  # encrypt EBS volumes
    },
    StoppingCondition={"MaxRuntimeInSeconds": 3600}
)

3.3. Endpoint configurations

sm.create_endpoint_config(
    EndpointConfigName="my-kms-endpoint-config",
    ProductionVariants=[{
        "VariantName": "AllTraffic",
        "ModelName": "my-trained-model",
        "InitialInstanceCount": 1,
        "InstanceType": "ml.m5.large"
    }],
    KmsKeyId="alias/sagemaker-cmk"  # encrypt endpoint storage
)

3.4. Processing jobs

sm.create_processing_job(
    ProcessingJobName="my-kms-processing-job",
    RoleArn="arn:aws:iam::123456789012:role/MySageMakerRole",
    AppSpecification={
        "ImageUri": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-processing-image:latest"
    },
    ProcessingResources={
        "ClusterConfig": {
            "InstanceCount": 2,
            "InstanceType": "ml.m5.xlarge",
            "VolumeSizeInGB": 50,
            "VolumeKmsKeyId": "alias/sagemaker-cmk"  # EBS for processing cluster
        }
    },
    ProcessingOutputConfig={
        "Outputs": [{
            "OutputName": "output",
            "S3Output": {
                "S3Uri": "s3://my-bucket/processing-output/",
                "LocalPath": "/opt/ml/processing/output",
                "S3UploadMode": "EndOfJob"
            }
        }],
        "KmsKeyId": "alias/sagemaker-cmk"  # S3 output encryption
    }
)

3.5. SageMaker Studio domain

If you’re using Studio “clusters” (Jupyter kernels, apps) you must set encryption at the domain:

sm.create_domain(
    DomainName="my-kms-domain",
    AuthMode="IAM",
    DefaultUserSettings={
        "ExecutionRole": "arn:aws:iam::123456789012:role/MySageMakerRole"
    },
    SubnetIds=["subnet-xxxxxxxx"],
    VpcId="vpc-xxxxxxxx",
    KmsKeyId="alias/sagemaker-cmk"  # encrypt EFS data for Studio
)

For existing domains you must use update_domain:

sm.update_domain(
    DomainId="d-xxxxxxxxxxxx",
    DefaultUserSettings={
        "ExecutionRole": "arn:aws:iam::123456789012:role/MySageMakerRole",
        "SecurityGroups": ["sg-xxxxxxxx"],
        "JupyterServerAppSettings": {},
        "KernelGatewayAppSettings": {}
    },
    KmsKeyId="alias/sagemaker-cmk"
)

(Only some fields are required; include what your environment needs.)

4. Existing SageMaker resources without CMK

Most SageMaker resources cannot be retrofitted with a KMS key in place; you typically must:

Discover resources without CMK.
Recreate them with KmsKeyId / VolumeKmsKeyId / OutputDataConfig.KmsKeyId set.

Example discovery pattern (for training jobs):

unencrypted = []
paginator = sm.get_paginator("list_training_jobs")

for page in paginator.paginate():
    for job_summary in page["TrainingJobSummaries"]:
        job_name = job_summary["TrainingJobName"]
        desc = sm.describe_training_job(TrainingJobName=job_name)
        vol_kms = desc["ResourceConfig"].get("VolumeKmsKeyId")
        out_kms = desc["OutputDataConfig"].get("KmsKeyId")
        if not vol_kms or not out_kms:
            unencrypted.append(job_name)

print("Unencrypted training jobs:", unencrypted)

You can then:

For notebooks: create new notebooks with KmsKeyId, move data, then delete old ones.
For endpoints: create new endpoint configs with KmsKeyId, update endpoints to use them.
For jobs: enforce CMK for all new jobs; old jobs are historical.

5. KMS key policy (optional but recommended)

Ensure your SageMaker execution role can use the CMK:

key_id = cmk_arn.split("/")[-1]
policy = kms.get_key_policy(KeyId=key_id, PolicyName="default")["Policy"]

# Optionally edit `policy` JSON to add the SageMaker role principal and then:
kms.put_key_policy(
    KeyId=key_id,
    PolicyName="default",
    Policy=policy
)

If you tell me which exact SageMaker resource type your “clusters” refer to (e.g., Studio domains, processing job clusters, or training clusters), I can give a narrower script tailored precisely to that.

AWS Introduction

AWS Pricing

AWS Threats

AWS Misconfigurations

Sagemaker Clusters Should Be Encrypted With KMS Customer Managed Keys

More Info:

Risk Level

Address

Compliance Standards

Triage and Remediation

Remediation

1. Create a KMS Customer Managed Key

2. Use the CMK for New SageMaker Resources

A. SageMaker Studio Domain

B. SageMaker Notebook Instances

C. Training / Processing / Transform Jobs

D. Endpoints and Models

3. Ensure S3 Buckets Used by SageMaker Use CMK

4. IAM Permission Check

1. Create (or identify) a KMS CMK

2. Notebook Instances – use `--kms-key-id`

3. Training Jobs – encrypt volumes and output

4. Endpoint Configurations – encrypt model data

5. SageMaker Studio Domains & User Profiles (if applicable)

6. Verification

1. Prerequisites

2. Create (or re‑use) a KMS Customer Managed Key (CMK)

3. New SageMaker resources: use CMK in create calls

3.1. Notebook instances

3.2. Training jobs

3.3. Endpoint configurations

3.4. Processing jobs

3.5. SageMaker Studio domain

4. Existing SageMaker resources without CMK

5. KMS key policy (optional but recommended)

​More Info:

​Risk Level

​Address

​Compliance Standards

​Triage and Remediation

​Remediation

​1. Create a KMS Customer Managed Key

​2. Use the CMK for New SageMaker Resources

​A. SageMaker Studio Domain

​B. SageMaker Notebook Instances

​C. Training / Processing / Transform Jobs

​D. Endpoints and Models

​3. Ensure S3 Buckets Used by SageMaker Use CMK

​4. IAM Permission Check

​1. Create (or identify) a KMS CMK

​2. Notebook Instances – use --kms-key-id

​3. Training Jobs – encrypt volumes and output

​4. Endpoint Configurations – encrypt model data

​5. SageMaker Studio Domains & User Profiles (if applicable)

​6. Verification

​1. Prerequisites

​2. Create (or re‑use) a KMS Customer Managed Key (CMK)

​3. New SageMaker resources: use CMK in create calls

​3.1. Notebook instances

​3.2. Training jobs

​3.3. Endpoint configurations

​3.4. Processing jobs

​3.5. SageMaker Studio domain

​4. Existing SageMaker resources without CMK

​5. KMS key policy (optional but recommended)

More Info:

Risk Level

Address

Compliance Standards

Triage and Remediation

Remediation

1. Create a KMS Customer Managed Key

2. Use the CMK for New SageMaker Resources

A. SageMaker Studio Domain

B. SageMaker Notebook Instances

C. Training / Processing / Transform Jobs

D. Endpoints and Models

3. Ensure S3 Buckets Used by SageMaker Use CMK

4. IAM Permission Check

1. Create (or identify) a KMS CMK

2. Notebook Instances – use `--kms-key-id`

3. Training Jobs – encrypt volumes and output

4. Endpoint Configurations – encrypt model data

5. SageMaker Studio Domains & User Profiles (if applicable)

6. Verification

1. Prerequisites

2. Create (or re‑use) a KMS Customer Managed Key (CMK)

3. New SageMaker resources: use CMK in create calls

3.1. Notebook instances

3.2. Training jobs

3.3. Endpoint configurations

3.4. Processing jobs

3.5. SageMaker Studio domain

4. Existing SageMaker resources without CMK

5. KMS key policy (optional but recommended)