Skip to main content

Sagemaker Output And Storage Data Encrypted With KMS

More Info:​

Sagemaker output and storage data encrypted with KMS Customer Managed Keys

Risk Level​

Medium

Address​

Security

Compliance Standards​

  • APRA CPS 234 (Australia)
  • BSI C5 (Germany)
  • Brazil LGPD
  • CCPA / CPRA (California)
  • CIS Critical Security Controls v8
  • CMMC 2.0
  • CSA Cloud Controls Matrix v4
  • DPDPA
  • Digital Operational Resilience Act (EU)
  • ISO 27001
  • ISO/IEC 27018
  • ISO/IEC 27701
  • MAS Technology Risk Management (Singapore)
  • MITRE ATT&CK (Cloud)
  • NIST SP 800-171
  • NYDFS 23 NYCRR 500
  • SOC2
  • SWIFT Customer Security Controls Framework
  • Sarbanes-Oxley IT General Controls
  • UK NCSC Cyber Assessment Framework

Triage and Remediation​

Remediation​

Using Console

Below are step‑by‑step AWS Console instructions to ensure SageMaker output and storage data are encrypted with KMS customer managed keys (CMKs). This covers:

  • S3 output data from jobs
  • Attached storage (EBS volumes) for training/processing
  • Notebook instance volumes (where applicable)

1. Create or Select a Customer Managed KMS Key​

  1. In the AWS Console, go to Key Management Service (KMS).
  2. In the left menu, choose Customer managed keys.
  3. Click Create key.
  4. Key type: Symmetric → Next.
  5. Define Alias (e.g., alias/sagemaker-cmk).
  6. Set Key administrators and Key users:
    • Ensure that the IAM roles used by SageMaker (e.g., SageMaker execution roles) are added as Key users.
  7. Finish the wizard and Enable the key.

You’ll use this CMK ARN (visible in the key details) in SageMaker configurations.

2. Encrypt SageMaker Training Job Output + Storage (EBS)​

When you create a new training job:

  1. Go to Amazon SageMaker → Training → Training jobs.
  2. Click Create training job.
  3. Under Permissions and encryption:
    • KMS key for output data (or Output data configuration section):
      • Choose Enter a KMS key or Choose from your AWS KMS keys.
      • Select your CMK (e.g., alias/sagemaker-cmk).
    • KMS key for storage volume (may appear under Resource configuration or similar):
      • Set Volume KMS key / VolumeKmsKeyId to your CMK.
  4. Complete the rest of the training job configuration and Create training job.

Existing training jobs cannot be edited; create new jobs with the CMK configured.

3. Encrypt SageMaker Processing / Batch Transform Jobs​

For Processing jobs:

  1. SageMaker → Processing → Processing jobs → Create processing job.
  2. Under Permissions and encryption:
    • KMS key for output data: select your CMK.
    • KMS key for volume storage (if shown): select your CMK.
  3. Configure the job and click Create.

For Batch transform jobs:

  1. SageMaker → Inference → Batch transform jobs → Create batch transform job.
  2. Under Output data:
    • KMS key for output data: select your CMK.
  3. If there is a Volume KMS key field, set it to your CMK.
  4. Create the job.

4. Encrypt Model Artifacts (for Endpoints)​

When creating a model:

  1. SageMaker → Inference → Models → Create model.
  2. In Container definition / Primary container:
    • If a KMS key for model artifacts (or similar) field is present, choose your CMK.
  3. Complete and create the model.
  4. Use this model for Endpoints or Batch Transform so artifacts stay encrypted with CMK.

5. Encrypt Notebook Instance Volumes (Where Needed)​

You cannot change the encryption key of an existing notebook instance. Create a new one:

  1. SageMaker → Notebook → Notebook instances → Create notebook instance.
  2. Under Encryption key:
    • Choose your CMK (e.g., alias/sagemaker-cmk).
  3. Migrate notebooks/code from the old instance (via Git, S3 sync, etc.), then delete the old instance.

6. Encrypt SageMaker Studio (If You Use Studio)​

For a new Studio Domain:

  1. SageMaker → SageMaker Domain → Create domain.
  2. In Encryption key, choose your CMK.
  3. Finish the domain creation.

For User profiles under that domain, ensure any storage options present use the domain’s CMK.

Changing the key on an existing domain may not be supported via console; if not, you’d create a new domain using the CMK and migrate workloads.

7. Confirm the Misconfiguration Is Resolved​

  • For a training/processing/batch job, open the job details and ensure:
    • KMS key for output data = your CMK.
    • KMS key for volume storage = your CMK.
  • For notebooks and Studio, check the Encryption key field in their respective configuration pages.
  • Optionally, in KMS → Key usage, verify that your CMK is being used by SageMaker.

If you tell me what exact resource the finding is on (training job, notebook, Studio, or batch), I can give a tailored, minimal set of exact console clicks for that specific case.

Using CLI

Below are concise, CLI‑focused steps to ensure SageMaker output and storage are encrypted with KMS Customer Managed Keys (CMKs).

1. Create (or identify) a KMS CMK​

If you don’t already have a CMK:

aws kms create-key \
  --description "CMK for SageMaker data at rest" \
  --key-usage ENCRYPT_DECRYPT \
  --origin AWS_KMS \
  --region <region>

Get the key ID/ARN:

aws kms list-keys --region <region>
# or
aws kms describe-key --key-id <key-id> --region <region>

Use the CMK ARN in the following steps, e.g.: arn:aws:kms:<region>:<account-id>:key/<key-id>

2. Encrypt Notebook Instance Storage (EBS)​

You cannot modify an existing notebook instance’s KMS key. You must recreate it with --kms-key-id.

  1. Get current notebook config:
aws sagemaker describe-notebook-instance \
  --notebook-instance-name <old-notebook-name> \
  --region <region>

Note:

  • InstanceType
  • RoleArn
  • SubnetId / SecurityGroups
  • VolumeSizeInGB
  • DirectInternetAccess, etc.
  1. Stop and delete old notebook:
aws sagemaker stop-notebook-instance \
  --notebook-instance-name <old-notebook-name> \
  --region <region>

aws sagemaker delete-notebook-instance \
  --notebook-instance-name <old-notebook-name> \
  --region <region>
  1. Recreate with CMK:
aws sagemaker create-notebook-instance \
  --notebook-instance-name <new-notebook-name> \
  --instance-type <instance-type> \
  --role-arn <iam-role-arn> \
  --volume-size-in-gb <size> \
  --subnet-id <subnet-id> \
  --security-group-ids <sg-1> <sg-2> \
  --kms-key-id <cmk-arn> \
  --direct-internet-access Enabled \
  --region <region>

3. Encrypt Training Job Output (S3) and Storage (Volumes)​

When starting new training jobs, specify:

  • --output-data-config KmsKeyId=<cmk-arn>,S3OutputPath=<s3-uri>
  • --resource-config VolumeKmsKeyId=<cmk-arn>,...

Example:

aws sagemaker create-training-job \
  --training-job-name my-job \
  --algorithm-specification TrainingImage=<image>,TrainingInputMode=File \
  --role-arn <iam-role-arn> \
  --input-data-config '[{"ChannelName":"training","DataSource":{"S3DataSource":{"S3DataType":"S3Prefix","S3Uri":"s3://my-bucket/train/","S3DataDistributionType":"FullyReplicated"}}}]' \
  --output-data-config "KmsKeyId=<cmk-arn>,S3OutputPath=s3://my-bucket/output/" \
  --resource-config "InstanceType=ml.m5.xlarge,InstanceCount=1,VolumeSizeInGB=50,VolumeKmsKeyId=<cmk-arn>" \
  --stopping-condition MaxRuntimeInSeconds=3600 \
  --region <region>

Existing training jobs cannot be changed; apply this to all future jobs.

4. Encrypt Model Artifacts and Endpoints​

4.1 Create Model with KMS Key​

If your model artifacts are in S3, create the model with --enable-network-isolation as needed and --primary-container using ModelKmsKeyId for volumes (or use --vpc-config + KMS key for attached volumes):

aws sagemaker create-model \
  --model-name my-model \
  --primary-container "Image=<image>,ModelDataUrl=s3://my-bucket/model.tar.gz" \
  --execution-role-arn <iam-role-arn> \
  --enable-network-isolation \
  --region <region>

Model artifacts in S3 should be encrypted with the same CMK via S3 bucket default encryption or per-object encryption:

aws s3api put-bucket-encryption \
  --bucket my-bucket \
  --server-side-encryption-configuration '{
    "Rules":[{
      "ApplyServerSideEncryptionByDefault":{
        "SSEAlgorithm":"aws:kms",
        "KMSMasterKeyID":"<cmk-arn>"
      }
    }]
  }'

5. Encrypt Endpoint Storage (Production Variants / Volumes)​

When creating or updating an endpoint configuration, specify VolumeSizeInGB and VolumeKmsKeyId in the production variants where supported:

aws sagemaker create-endpoint-config \
  --endpoint-config-name my-endpoint-config \
  --production-variants '[
    {
      "VariantName":"AllTraffic",
      "ModelName":"my-model",
      "InstanceType":"ml.m5.xlarge",
      "InitialInstanceCount":1,
      "InitialVariantWeight":1.0,
      "VolumeSizeInGB":50,
      "VolumeKmsKeyId":"<cmk-arn>"
    }
  ]' \
  --region <region>

Then create or update the endpoint:

aws sagemaker create-endpoint \
  --endpoint-name my-endpoint \
  --endpoint-config-name my-endpoint-config \
  --region <region>

For existing endpoints, create a new endpoint config with VolumeKmsKeyId and:

aws sagemaker update-endpoint \
  --endpoint-name my-endpoint \
  --endpoint-config-name my-new-endpoint-config \
  --region <region>

6. Verify Encryption​

  • Notebook:
aws sagemaker describe-notebook-instance \
  --notebook-instance-name <name> \
  --region <region> \
  --query 'KmsKeyId'
  • Training job output/storage:
aws sagemaker describe-training-job \
  --training-job-name my-job \
  --region <region> \
  --query '[OutputDataConfig.KmsKeyId, ResourceConfig.VolumeKmsKeyId]'
  • S3 bucket:
aws s3api get-bucket-encryption \
  --bucket my-bucket

Apply these patterns to all new SageMaker notebooks, training jobs, models, and endpoints to meet the “Output and Storage Data Encrypted With KMS CMKs” requirement.

Using Python

To remediate this finding, you need to ensure all SageMaker artifacts (notebooks, training jobs, models, endpoints, processing jobs, etc.) use KMS customer-managed keys (CMK) instead of AWS‑managed or unencrypted storage.

Below are concise, step‑by‑step instructions with Python/boto3.

1. Prerequisites​

pip install boto3
import boto3

sagemaker = boto3.client("sagemaker", region_name="us-east-1")
kms = boto3.client("kms", region_name="us-east-1")

2. Create or identify a KMS CMK​

If you don’t already have a CMK for SageMaker:

response = kms.create_key(
    Description="CMK for SageMaker data encryption",
    KeyUsage="ENCRYPT_DECRYPT",
    Origin="AWS_KMS",
    Tags=[{"TagKey": "Purpose", "TagValue": "SageMaker"}]
)
kms_key_id = response["KeyMetadata"]["KeyId"]
print(kms_key_id)

Or use an existing CMK ID/ARN:

kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

Make sure the CMK key policy allows SageMaker and your IAM principals to use it (kms:Encrypt, kms:Decrypt, kms:GenerateDataKey, etc.).

3. Encrypt SageMaker Notebook Instances​

New notebook:

sagemaker.create_notebook_instance(
    NotebookInstanceName="my-secure-notebook",
    InstanceType="ml.t3.medium",
    RoleArn="arn:aws:iam::123456789012:role/SageMakerExecutionRole",
    KmsKeyId=kms_key_id,          # Encrypt EBS volume
    VolumeSizeInGB=20
)

Existing notebook (must stop first; some settings are immutable, so recreate if needed):

sagemaker.stop_notebook_instance(NotebookInstanceName="my-notebook")
sagemaker.get_waiter("notebook_instance_stopped").wait(NotebookInstanceName="my-notebook")

sagemaker.update_notebook_instance(
    NotebookInstanceName="my-notebook",
    KmsKeyId=kms_key_id
)

If update fails (field immutable), create a new notebook with KmsKeyId and migrate.

4. Encrypt Training Jobs (output, model artifacts & volume)​

training_job_name = "my-secure-training-job"

sagemaker.create_training_job(
    TrainingJobName=training_job_name,
    AlgorithmSpecification={
        "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",
        "TrainingInputMode": "File",
    },
    RoleArn="arn:aws:iam::123456789012:role/SageMakerExecutionRole",
    InputDataConfig=[
        {
            "ChannelName": "train",
            "DataSource": {
                "S3DataSource": {
                    "S3DataType": "S3Prefix",
                    "S3Uri": "s3://my-bucket/train/",
                    "S3DataDistributionType": "FullyReplicated",
                }
            }
        }
    ],
    OutputDataConfig={
        "S3OutputPath": "s3://my-bucket/output/",
        "KmsKeyId": kms_key_id,     # Encrypt S3 output
    },
    ResourceConfig={
        "InstanceType": "ml.m5.large",
        "InstanceCount": 1,
        "VolumeSizeInGB": 50,
        "VolumeKmsKeyId": kms_key_id  # Encrypt attached EBS volume
    },
    StoppingCondition={"MaxRuntimeInSeconds": 3600},
    EnableNetworkIsolation=True
)

5. Encrypt Models and Endpoints​

Model (stores container settings & model data location):

sagemaker.create_model(
    ModelName="my-secure-model",
    PrimaryContainer={
        "Image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",
        "ModelDataUrl": "s3://my-bucket/output/my-training-job/output/model.tar.gz"
    },
    ExecutionRoleArn="arn:aws:iam::123456789012:role/SageMakerExecutionRole",
    EnableNetworkIsolation=True,
    # Encrypt model container’s volume
    VpcConfig={
        "SecurityGroupIds": ["sg-xxxxxxxx"],
        "Subnets": ["subnet-xxxxxxxx"],
    },
    # For some APIs, use "EnableNetworkIsolation" + "ContainerStartupHealthCheckTimeout"
    # Encryption is mainly at training/output S3 and attached storage, not model object itself.
)

For endpoints, ensure the endpoint configuration uses volumes with KMS:

sagemaker.create_endpoint_config(
    EndpointConfigName="my-secure-endpoint-config",
    ProductionVariants=[
        {
            "VariantName": "AllTraffic",
            "ModelName": "my-secure-model",
            "InitialInstanceCount": 1,
            "InstanceType": "ml.m5.large",
            "InitialVariantWeight": 1.0,
            # For some instance families, data capture / storage volumes honor KMS keys via model/training config
        }
    ],
    DataCaptureConfig={
        "EnableCapture": True,
        "InitialSamplingPercentage": 100,
        "DestinationS3Uri": "s3://my-bucket/datacapture/",
        "KmsKeyId": kms_key_id,  # Encrypt data capture S3
        "CaptureOptions": [{"CaptureMode": "Input"}, {"CaptureMode": "Output"}],
        "CaptureContentTypeHeader": {
            "JsonContentTypes": ["application/json"]
        }
    }
)

sagemaker.create_endpoint(
    EndpointName="my-secure-endpoint",
    EndpointConfigName="my-secure-endpoint-config"
)

6. Encrypt Processing Jobs​

sagemaker.create_processing_job(
    ProcessingJobName="my-secure-processing-job",
    RoleArn="arn:aws:iam::123456789012:role/SageMakerExecutionRole",
    ProcessingResources={
        "ClusterConfig": {
            "InstanceCount": 1,
            "InstanceType": "ml.m5.large",
            "VolumeSizeInGB": 50,
            "VolumeKmsKeyId": kms_key_id,  # EBS encryption
        }
    },
    AppSpecification={"ImageUri": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-processing-image:latest"},
    ProcessingInputs=[
        {
            "InputName": "input-data",
            "S3Input": {
                "S3Uri": "s3://my-bucket/input/",
                "LocalPath": "/opt/ml/processing/input",
                "S3DataType": "S3Prefix",
                "S3InputMode": "File",
            }
        }
    ],
    ProcessingOutputConfig={
        "Outputs": [
            {
                "OutputName": "output-data",
                "S3Output": {
                    "S3Uri": "s3://my-bucket/processing-output/",
                    "LocalPath": "/opt/ml/processing/output",
                    "S3UploadMode": "EndOfJob",
                }
            }
        ],
        "KmsKeyId": kms_key_id  # Encrypt S3 outputs
    }
)

7. Encrypt Pipelines (Pipeline Execution Storage)​

When creating/updating pipelines, specify PipelineDefinitionS3Location with a bucket encrypted by your CMK, and for each step (training, processing, transform, etc.), pass KmsKeyId/VolumeKmsKeyId/OutputDataConfig.KmsKeyId as above.

If you use the SageMaker Python SDK (sagemaker library), most high‑level classes (Estimator, ProcessingJob, Transformer, etc.) accept output_kms_key, volume_kms_key, or encrypt_inter_container_traffic parameters that map to the same fields.

8. Verify Encryption​

Use describe_* calls to confirm KmsKeyId and VolumeKmsKeyId are set:

resp = sagemaker.describe_training_job(TrainingJobName=training_job_name)
print(resp["OutputDataConfig"].get("KmsKeyId"))
print(resp["ResourceConfig"].get("VolumeKmsKeyId"))

Repeat for notebooks, processing jobs, endpoints, data capture, etc.

If you tell me exactly which SageMaker resource (notebook, training job, endpoint, processing job, pipeline) your security tool flagged, I can give a focused Python snippet tailored to that specific type.

Using Terraform
# Customer managed KMS key for SageMaker data encryption
resource "aws_kms_key" "sagemaker_data" {
  description             = "KMS CMK for SageMaker job storage and output data encryption"
  deletion_window_in_days = 30

  # Optionally scope key usage with key policy; replace THIS_KEY_POLICY_JSON
  policy = jsonencode(THIS_KEY_POLICY_JSON) # <-- replace with your key policy JSON
}

# SageMaker training job with output + storage encrypted using the CMK
resource "aws_sagemaker_training_job" "this" {
  name = "YOUR_TRAINING_JOB_NAME" # <-- replace with your training job name

  role_arn = "YOUR_SAGEMAKER_EXECUTION_ROLE_ARN" # <-- replace with IAM role ARN

  algorithm_specification {
    training_image     = "YOUR_TRAINING_IMAGE"  # <-- replace with training container
    training_input_mode = "File"
  }

  input_data_config {
    channel_name = "training"

    data_source {
      s3_data_source {
        s3_data_type = "S3Prefix"
        s3_uri       = "s3://YOUR_INPUT_BUCKET/YOUR_PREFIX/" # <-- replace
        s3_data_distribution_type = "FullyReplicated"
      }
    }
  }

  output_data_config {
    s3_output_path = "s3://YOUR_OUTPUT_BUCKET/YOUR_PREFIX/" # <-- replace

    # Encrypt training job output with CMK
    kms_key_id = aws_kms_key.sagemaker_data.arn
  }

  resource_config {
    instance_type  = "ml.m5.xlarge" # <-- adjust as needed
    instance_count = 1
    volume_size_in_gb = 50

    # Encrypt attached training volume (storage) with CMK
    volume_kms_key_id = aws_kms_key.sagemaker_data.arn
  }

  stopping_condition {
    max_runtime_in_seconds = 3600
  }

  tags = {
    Name = "YOUR_TRAINING_JOB_TAG" # <-- optional
  }
}

Changing the kms_key_id in output_data_config or volume_kms_key_id in resource_config for an existing aws_sagemaker_training_job forces replacement; Terraform will create a new training job (with a new name if required) rather than updating the existing one.

Verification: terraform plan should show + kms_key_id under output_data_config and + volume_kms_key_id under resource_config for the SageMaker job, referencing your CMK ARN.