> ## Documentation Index > Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Sagemaker Training Jobs Should Have Inter Container Traffic Encryption Enabled ### More Info: Sagemaker Training Jobs should have inter-container traffic encryption enabled ### Risk Level Medium ### Address Monitoring, Security ### Compliance Standards * APRA CPS 234 (Australia) * BSI C5 (Germany) * Brazil LGPD * CCPA / CPRA (California) * CIS Critical Security Controls v8 * CMMC 2.0 * CSA Cloud Controls Matrix v4 * DPDPA * Digital Operational Resilience Act (EU) * ISO 27001 * ISO/IEC 27018 * ISO/IEC 27701 * MAS Technology Risk Management (Singapore) * MITRE ATT\&CK (Cloud) * NIST SP 800-171 * NYDFS 23 NYCRR 500 * SOC2 * SWIFT Customer Security Controls Framework * Sarbanes-Oxley IT General Controls * UK NCSC Cyber Assessment Framework ### Triage and Remediation ### Remediation To remediate this in AWS SageMaker using the **AWS Management Console**, you must enable the setting when creating (or re-creating) the training job. Existing training jobs cannot be edited; you create a new job with the correct security settings. ### 1. Open SageMaker Training Jobs 1. Sign in to the AWS Management Console. 2. In the top bar, choose the correct **Region**. 3. Go to **Amazon SageMaker**. 4. In the left navigation pane, select **Training** → **Training jobs**. ### 2. Re-create the Training Job With Encryption You’ll mirror the settings of the non-compliant job but enable inter-container traffic encryption. 1. In **Training jobs**, click the name of the non-compliant training job. 2. Note down (or screenshot) its configuration: * Training image / algorithm * Input data locations (S3 paths, channels) * Instance type and count * Output S3 location * Hyperparameters * VPC settings, IAM role, etc. 3. Click **Create training job** (or **Create** at the top right). ### 3. Configure the New Training Job Fill in all the same settings as the old job, then enable the encryption: 1. **Job name**: Give a new name (e.g., append `-encrypted`). 2. **IAM role**, **algorithm/image**, **input data configuration**, **output data configuration**, **resource configuration**, etc., as per the original job. 3. Scroll down to the **Security and encryption** or **Additional configuration** section (exact label may vary slightly by console version). 4. Locate **Inter-container traffic encryption**: * Check the box **Enable inter-container traffic encryption**\ (or toggle it **On**). 5. Optionally verify or set: * **VPC** settings. * **KMS key** for output data / volume encryption if you’re also standardizing storage encryption. 6. Review all fields to ensure they match the previous job (except the name and the new encryption setting). 7. Click **Create training job**. ### 4. Clean Up / Enforce Going Forward 1. After confirming the new job runs correctly, stop relying on the old non-compliant job (and optionally delete it). 2. Update any process that creates training jobs (e.g., runbooks, internal docs) to require: * **Inter-container traffic encryption** enabled in the console for all new training jobs. ### 5. Validate Compliance 1. In **Training jobs**, open the new job’s details. 2. In the **Security and encryption** section, confirm that: * **Inter-container traffic encryption** is shown as **Enabled**. This ensures all network traffic between containers in the training cluster is encrypted for that job. To fix this, you must **create (or re-create) SageMaker training jobs with inter-container traffic encryption enabled**. This cannot be retroactively changed on an already running or completed training job. Below are step‑by‑step CLI instructions. *** ## 1. Identify training jobs missing inter-container encryption (optional) List recent training jobs: ```bash theme={null} aws sagemaker list-training-jobs \ --status-equals InProgress \ --max-results 50 ``` (There is no direct flag to check this on an existing job; remediation is to ensure **all new jobs** have this flag enabled.) *** ## 2. Create a training job with inter-container traffic encryption enabled Prepare a minimal config file, e.g. `training-job-config.json`: ```json theme={null} { "TrainingJobName": "my-secure-training-job", "RoleArn": "arn:aws:iam::123456789012:role/service-role/AmazonSageMaker-ExecutionRole", "AlgorithmSpecification": { "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest", "TrainingInputMode": "File" }, "InputDataConfig": [ { "ChannelName": "train", "DataSource": { "S3DataSource": { "S3DataType": "S3Prefix", "S3Uri": "s3://my-bucket/training/", "S3DataDistributionType": "FullyReplicated" } } } ], "OutputDataConfig": { "S3OutputPath": "s3://my-bucket/output/" }, "ResourceConfig": { "InstanceType": "ml.m5.xlarge", "InstanceCount": 1, "VolumeSizeInGB": 50 }, "StoppingCondition": { "MaxRuntimeInSeconds": 3600 }, "EnableInterContainerTrafficEncryption": true } ``` Create the training job: ```bash theme={null} aws sagemaker create-training-job \ --cli-input-json file://training-job-config.json ``` Key line for remediation: ```json theme={null} "EnableInterContainerTrafficEncryption": true ``` You can also pass it inline: ```bash theme={null} aws sagemaker create-training-job \ --training-job-name my-secure-training-job \ --role-arn arn:aws:iam::123456789012:role/service-role/AmazonSageMaker-ExecutionRole \ --algorithm-specification TrainingImage=123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest,TrainingInputMode=File \ --input-data-config '[{"ChannelName":"train","DataSource":{"S3DataSource":{"S3DataType":"S3Prefix","S3Uri":"s3://my-bucket/training/","S3DataDistributionType":"FullyReplicated"}}}]' \ --output-data-config S3OutputPath=s3://my-bucket/output/ \ --resource-config InstanceType=ml.m5.xlarge,InstanceCount=1,VolumeSizeInGB=50 \ --stopping-condition MaxRuntimeInSeconds=3600 \ --enable-inter-container-traffic-encryption ``` Note: the short option is `--enable-inter-container-traffic-encryption`. *** ## 3. Update any automation/templates to always enable it Wherever you define training jobs (CloudFormation, CDK, Step Functions, SageMaker Pipelines, custom scripts), ensure the equivalent flag is set: * CloudFormation: `EnableInterContainerTrafficEncryption: true` * CDK (TypeScript/Java/Python): `enableInterContainerTrafficEncryption: true` * Raw CLI/scripts: always pass `--enable-inter-container-traffic-encryption` or `"EnableInterContainerTrafficEncryption": true`. *** ## 4. Recreate important recurring jobs For scheduled or recurring training: 1. Take the existing training job definition (can be viewed via): ```bash theme={null} aws sagemaker describe-training-job \ --training-job-name existing-job-name ``` 2. Copy the response JSON, add `"EnableInterContainerTrafficEncryption": true`. 3. Change `TrainingJobName`. 4. Call `create-training-job` with the modified JSON. Below are step‑by‑step ways to ensure **inter-container traffic encryption** is enabled for SageMaker training jobs using Python. *** ## 1. Using the SageMaker Python SDK (recommended) ### a. Install / upgrade the SDK ```bash theme={null} pip install --upgrade sagemaker boto3 ``` ### b. Enable `encrypt_inter_container_traffic` on an Estimator ```python theme={null} import sagemaker from sagemaker import Estimator import boto3 session = sagemaker.Session() role = "arn:aws:iam:::role/" estimator = Estimator( image_uri="123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training-image:latest", role=role, instance_count=1, instance_type="ml.m5.xlarge", volume_size=50, max_run=3600, sagemaker_session=session, # This flag enables inter-container traffic encryption encrypt_inter_container_traffic=True ) # Example hyperparameters & inputs estimator.set_hyperparameters(epochs=10, batch_size=32) estimator.fit( inputs={"training": "s3://my-bucket/my-training-data/"}, job_name="my-secure-training-job" ) ``` This ensures that if the job uses multiple containers (e.g., distributed training), traffic between containers is encrypted. *** ## 2. Using `boto3` (low-level API) If you are creating training jobs directly with `boto3`, set the field `EnableInterContainerTrafficEncryption` to `True`. ### a. Create a training job with encryption enabled ```python theme={null} import boto3 import time sm_client = boto3.client("sagemaker", region_name="us-east-1") training_job_name = f"my-secure-training-job-{int(time.time())}" response = sm_client.create_training_job( TrainingJobName=training_job_name, AlgorithmSpecification={ "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training-image:latest", "TrainingInputMode": "File" }, RoleArn="arn:aws:iam:::role/", InputDataConfig=[ { "ChannelName": "training", "DataSource": { "S3DataSource": { "S3DataType": "S3Prefix", "S3Uri": "s3://my-bucket/my-training-data/", "S3DataDistributionType": "FullyReplicated", } }, "ContentType": "text/csv", "InputMode": "File", } ], OutputDataConfig={ "S3OutputPath": "s3://my-bucket/my-training-output/" }, ResourceConfig={ "InstanceType": "ml.m5.xlarge", "InstanceCount": 2, # e.g., distributed training "VolumeSizeInGB": 50, }, StoppingCondition={ "MaxRuntimeInSeconds": 3600 }, EnableInterContainerTrafficEncryption=True # ← critical flag ) print("Training job created:", response["TrainingJobArn"]) ``` *** ## 3. Remediating existing non‑compliant jobs You cannot modify this flag on a running or completed training job. To remediate: 1. **Identify non-compliant jobs** (no encryption enabled). 2. **Recreate** those jobs with `EnableInterContainerTrafficEncryption=True` (or `encrypt_inter_container_traffic=True` via the SDK). ### a. Detect jobs missing inter-container traffic encryption ```python theme={null} import boto3 sm_client = boto3.client("sagemaker", region_name="us-east-1") paginator = sm_client.get_paginator("list_training_jobs") for page in paginator.paginate(): for summary in page["TrainingJobSummaries"]: name = summary["TrainingJobName"] desc = sm_client.describe_training_job(TrainingJobName=name) # Default is False if not provided enabled = desc.get("EnableInterContainerTrafficEncryption", False) if not enabled: print("Non-compliant training job:", name) ``` ### b. Clone a non-compliant job with encryption enabled ```python theme={null} def clone_with_encryption(old_job_name, new_job_name): desc = sm_client.describe_training_job(TrainingJobName=old_job_name) # Build new request, copying configuration but overriding the flag params = { "TrainingJobName": new_job_name, "AlgorithmSpecification": desc["AlgorithmSpecification"], "RoleArn": desc["RoleArn"], "InputDataConfig": desc.get("InputDataConfig", []), "OutputDataConfig": desc["OutputDataConfig"], "ResourceConfig": desc["ResourceConfig"], "StoppingCondition": desc["StoppingCondition"], "EnableInterContainerTrafficEncryption": True, } if "VpcConfig" in desc: params["VpcConfig"] = desc["VpcConfig"] if "HyperParameters" in desc: params["HyperParameters"] = desc["HyperParameters"] if "Environment" in desc: params["Environment"] = desc["Environment"] if "RetryStrategy" in desc: params["RetryStrategy"] = desc["RetryStrategy"] if "ExperimentConfig" in desc: params["ExperimentConfig"] = desc["ExperimentConfig"] resp = sm_client.create_training_job(**params) return resp["TrainingJobArn"] # Example usage: old_job = "my-old-training-job" new_job = f"{old_job}-encrypted" arn = clone_with_encryption(old_job, new_job) print("New encrypted job ARN:", arn) ``` *** ## 4. Make encryption the default in your codebase * For all new SageMaker Estimators, always set `encrypt_inter_container_traffic=True`. * For all direct `boto3.create_training_job` calls, always set `EnableInterContainerTrafficEncryption=True`. * Optionally, add a simple CI/static check to fail builds if the flag is missing.