Sagemaker Training Jobs Should Have Inter Container Traffic Encryption Enabled

Using Console

To remediate this in AWS SageMaker using the AWS Management Console, you must enable the setting when creating (or re-creating) the training job. Existing training jobs cannot be edited; you create a new job with the correct security settings.

1. Open SageMaker Training Jobs

Sign in to the AWS Management Console.
In the top bar, choose the correct Region.
Go to Amazon SageMaker.
In the left navigation pane, select Training → Training jobs.

2. Re-create the Training Job With Encryption

You’ll mirror the settings of the non-compliant job but enable inter-container traffic encryption.

In Training jobs, click the name of the non-compliant training job.
Note down (or screenshot) its configuration:
- Training image / algorithm
- Input data locations (S3 paths, channels)
- Instance type and count
- Output S3 location
- Hyperparameters
- VPC settings, IAM role, etc.
Click Create training job (or Create at the top right).

3. Configure the New Training Job

Fill in all the same settings as the old job, then enable the encryption:

Job name: Give a new name (e.g., append -encrypted).
IAM role, algorithm/image, input data configuration, output data configuration, resource configuration, etc., as per the original job.
Scroll down to the Security and encryption or Additional configuration section (exact label may vary slightly by console version).
Locate Inter-container traffic encryption:
- Check the box Enable inter-container traffic encryption
  (or toggle it On).
Optionally verify or set:
- VPC settings.
- KMS key for output data / volume encryption if you’re also standardizing storage encryption.
Review all fields to ensure they match the previous job (except the name and the new encryption setting).
Click Create training job.

4. Clean Up / Enforce Going Forward

After confirming the new job runs correctly, stop relying on the old non-compliant job (and optionally delete it).
Update any process that creates training jobs (e.g., runbooks, internal docs) to require:
- Inter-container traffic encryption enabled in the console for all new training jobs.

5. Validate Compliance

In Training jobs, open the new job’s details.
In the Security and encryption section, confirm that:
- Inter-container traffic encryption is shown as Enabled.

This ensures all network traffic between containers in the training cluster is encrypted for that job.

Using CLI

To fix this, you must create (or re-create) SageMaker training jobs with inter-container traffic encryption enabled. This cannot be retroactively changed on an already running or completed training job.Below are step‑by‑step CLI instructions.

1. Identify training jobs missing inter-container encryption (optional)

List recent training jobs:

aws sagemaker list-training-jobs \
  --status-equals InProgress \
  --max-results 50

(There is no direct flag to check this on an existing job; remediation is to ensure all new jobs have this flag enabled.)

2. Create a training job with inter-container traffic encryption enabled

Prepare a minimal config file, e.g. training-job-config.json:

{
  "TrainingJobName": "my-secure-training-job",
  "RoleArn": "arn:aws:iam::123456789012:role/service-role/AmazonSageMaker-ExecutionRole",
  "AlgorithmSpecification": {
    "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",
    "TrainingInputMode": "File"
  },
  "InputDataConfig": [
    {
      "ChannelName": "train",
      "DataSource": {
        "S3DataSource": {
          "S3DataType": "S3Prefix",
          "S3Uri": "s3://my-bucket/training/",
          "S3DataDistributionType": "FullyReplicated"
        }
      }
    }
  ],
  "OutputDataConfig": {
    "S3OutputPath": "s3://my-bucket/output/"
  },
  "ResourceConfig": {
    "InstanceType": "ml.m5.xlarge",
    "InstanceCount": 1,
    "VolumeSizeInGB": 50
  },
  "StoppingCondition": {
    "MaxRuntimeInSeconds": 3600
  },
  "EnableInterContainerTrafficEncryption": true
}

Create the training job:

aws sagemaker create-training-job \
  --cli-input-json file://training-job-config.json

Key line for remediation:

"EnableInterContainerTrafficEncryption": true

You can also pass it inline:

aws sagemaker create-training-job \
  --training-job-name my-secure-training-job \
  --role-arn arn:aws:iam::123456789012:role/service-role/AmazonSageMaker-ExecutionRole \
  --algorithm-specification TrainingImage=123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest,TrainingInputMode=File \
  --input-data-config '[{"ChannelName":"train","DataSource":{"S3DataSource":{"S3DataType":"S3Prefix","S3Uri":"s3://my-bucket/training/","S3DataDistributionType":"FullyReplicated"}}}]' \
  --output-data-config S3OutputPath=s3://my-bucket/output/ \
  --resource-config InstanceType=ml.m5.xlarge,InstanceCount=1,VolumeSizeInGB=50 \
  --stopping-condition MaxRuntimeInSeconds=3600 \
  --enable-inter-container-traffic-encryption

Note: the short option is --enable-inter-container-traffic-encryption.

3. Update any automation/templates to always enable it

Wherever you define training jobs (CloudFormation, CDK, Step Functions, SageMaker Pipelines, custom scripts), ensure the equivalent flag is set:

CloudFormation: EnableInterContainerTrafficEncryption: true
CDK (TypeScript/Java/Python): enableInterContainerTrafficEncryption: true
Raw CLI/scripts: always pass --enable-inter-container-traffic-encryption or "EnableInterContainerTrafficEncryption": true.

4. Recreate important recurring jobs

For scheduled or recurring training:

Take the existing training job definition (can be viewed via):

aws sagemaker describe-training-job \
  --training-job-name existing-job-name

Copy the response JSON, add "EnableInterContainerTrafficEncryption": true.
Change TrainingJobName.
Call create-training-job with the modified JSON.

Using Python

Below are step‑by‑step ways to ensure inter-container traffic encryption is enabled for SageMaker training jobs using Python.

1. Using the SageMaker Python SDK (recommended)

a. Install / upgrade the SDK

pip install --upgrade sagemaker boto3

b. Enable `encrypt_inter_container_traffic` on an Estimator

import sagemaker
from sagemaker import Estimator
import boto3

session = sagemaker.Session()
role = "arn:aws:iam::<ACCOUNT_ID>:role/<SAGEMAKER_EXECUTION_ROLE>"

estimator = Estimator(
    image_uri="123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training-image:latest",
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge",
    volume_size=50,
    max_run=3600,
    sagemaker_session=session,
    # This flag enables inter-container traffic encryption
    encrypt_inter_container_traffic=True
)

# Example hyperparameters & inputs
estimator.set_hyperparameters(epochs=10, batch_size=32)

estimator.fit(
    inputs={"training": "s3://my-bucket/my-training-data/"},
    job_name="my-secure-training-job"
)

This ensures that if the job uses multiple containers (e.g., distributed training), traffic between containers is encrypted.

2. Using `boto3` (low-level API)

If you are creating training jobs directly with boto3, set the field EnableInterContainerTrafficEncryption to True.

a. Create a training job with encryption enabled

import boto3
import time

sm_client = boto3.client("sagemaker", region_name="us-east-1")

training_job_name = f"my-secure-training-job-{int(time.time())}"

response = sm_client.create_training_job(
    TrainingJobName=training_job_name,
    AlgorithmSpecification={
        "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training-image:latest",
        "TrainingInputMode": "File"
    },
    RoleArn="arn:aws:iam::<ACCOUNT_ID>:role/<SAGEMAKER_EXECUTION_ROLE>",
    InputDataConfig=[
        {
            "ChannelName": "training",
            "DataSource": {
                "S3DataSource": {
                    "S3DataType": "S3Prefix",
                    "S3Uri": "s3://my-bucket/my-training-data/",
                    "S3DataDistributionType": "FullyReplicated",
                }
            },
            "ContentType": "text/csv",
            "InputMode": "File",
        }
    ],
    OutputDataConfig={
        "S3OutputPath": "s3://my-bucket/my-training-output/"
    },
    ResourceConfig={
        "InstanceType": "ml.m5.xlarge",
        "InstanceCount": 2,  # e.g., distributed training
        "VolumeSizeInGB": 50,
    },
    StoppingCondition={
        "MaxRuntimeInSeconds": 3600
    },
    EnableInterContainerTrafficEncryption=True  # ← critical flag
)
print("Training job created:", response["TrainingJobArn"])

3. Remediating existing non‑compliant jobs

You cannot modify this flag on a running or completed training job. To remediate:

Identify non-compliant jobs (no encryption enabled).
Recreate those jobs with EnableInterContainerTrafficEncryption=True (or encrypt_inter_container_traffic=True via the SDK).

a. Detect jobs missing inter-container traffic encryption

import boto3

sm_client = boto3.client("sagemaker", region_name="us-east-1")

paginator = sm_client.get_paginator("list_training_jobs")
for page in paginator.paginate():
    for summary in page["TrainingJobSummaries"]:
        name = summary["TrainingJobName"]
        desc = sm_client.describe_training_job(TrainingJobName=name)
        # Default is False if not provided
        enabled = desc.get("EnableInterContainerTrafficEncryption", False)
        if not enabled:
            print("Non-compliant training job:", name)

b. Clone a non-compliant job with encryption enabled

def clone_with_encryption(old_job_name, new_job_name):
    desc = sm_client.describe_training_job(TrainingJobName=old_job_name)

    # Build new request, copying configuration but overriding the flag
    params = {
        "TrainingJobName": new_job_name,
        "AlgorithmSpecification": desc["AlgorithmSpecification"],
        "RoleArn": desc["RoleArn"],
        "InputDataConfig": desc.get("InputDataConfig", []),
        "OutputDataConfig": desc["OutputDataConfig"],
        "ResourceConfig": desc["ResourceConfig"],
        "StoppingCondition": desc["StoppingCondition"],
        "EnableInterContainerTrafficEncryption": True,
    }

    if "VpcConfig" in desc:
        params["VpcConfig"] = desc["VpcConfig"]
    if "HyperParameters" in desc:
        params["HyperParameters"] = desc["HyperParameters"]
    if "Environment" in desc:
        params["Environment"] = desc["Environment"]
    if "RetryStrategy" in desc:
        params["RetryStrategy"] = desc["RetryStrategy"]
    if "ExperimentConfig" in desc:
        params["ExperimentConfig"] = desc["ExperimentConfig"]

    resp = sm_client.create_training_job(**params)
    return resp["TrainingJobArn"]

# Example usage:
old_job = "my-old-training-job"
new_job = f"{old_job}-encrypted"
arn = clone_with_encryption(old_job, new_job)
print("New encrypted job ARN:", arn)

4. Make encryption the default in your codebase

For all new SageMaker Estimators, always set encrypt_inter_container_traffic=True.
For all direct boto3.create_training_job calls, always set EnableInterContainerTrafficEncryption=True.
Optionally, add a simple CI/static check to fail builds if the flag is missing.

AWS Introduction

AWS Pricing

AWS Threats

AWS Misconfigurations

Sagemaker Training Jobs Should Have Inter Container Traffic Encryption Enabled

More Info:

Risk Level

Address

Compliance Standards

Triage and Remediation

Remediation

1. Open SageMaker Training Jobs

2. Re-create the Training Job With Encryption

3. Configure the New Training Job

4. Clean Up / Enforce Going Forward

5. Validate Compliance

1. Identify training jobs missing inter-container encryption (optional)

2. Create a training job with inter-container traffic encryption enabled

3. Update any automation/templates to always enable it

4. Recreate important recurring jobs

1. Using the SageMaker Python SDK (recommended)

a. Install / upgrade the SDK

b. Enable `encrypt_inter_container_traffic` on an Estimator

2. Using `boto3` (low-level API)

a. Create a training job with encryption enabled

3. Remediating existing non‑compliant jobs

a. Detect jobs missing inter-container traffic encryption

b. Clone a non-compliant job with encryption enabled

4. Make encryption the default in your codebase

​More Info:

​Risk Level

​Address

​Compliance Standards

​Triage and Remediation

​Remediation

​1. Open SageMaker Training Jobs

​2. Re-create the Training Job With Encryption

​3. Configure the New Training Job

​4. Clean Up / Enforce Going Forward

​5. Validate Compliance

​1. Identify training jobs missing inter-container encryption (optional)

​2. Create a training job with inter-container traffic encryption enabled

​3. Update any automation/templates to always enable it

​4. Recreate important recurring jobs

​1. Using the SageMaker Python SDK (recommended)

​a. Install / upgrade the SDK

​b. Enable encrypt_inter_container_traffic on an Estimator

​2. Using boto3 (low-level API)

​a. Create a training job with encryption enabled

​3. Remediating existing non‑compliant jobs

​a. Detect jobs missing inter-container traffic encryption

​b. Clone a non-compliant job with encryption enabled

​4. Make encryption the default in your codebase

More Info:

Risk Level

Address

Compliance Standards

Triage and Remediation

Remediation

1. Open SageMaker Training Jobs

2. Re-create the Training Job With Encryption

3. Configure the New Training Job

4. Clean Up / Enforce Going Forward

5. Validate Compliance

1. Identify training jobs missing inter-container encryption (optional)

2. Create a training job with inter-container traffic encryption enabled

3. Update any automation/templates to always enable it

4. Recreate important recurring jobs

1. Using the SageMaker Python SDK (recommended)

a. Install / upgrade the SDK

b. Enable `encrypt_inter_container_traffic` on an Estimator

2. Using `boto3` (low-level API)

a. Create a training job with encryption enabled

3. Remediating existing non‑compliant jobs

a. Detect jobs missing inter-container traffic encryption

b. Clone a non-compliant job with encryption enabled

4. Make encryption the default in your codebase