Skip to main content

Sagemaker Training Jobs Should Have Inter Container Traffic

More Info:​

Sagemaker Training Jobs should have inter-container traffic encryption enabled

Risk Level​

Medium

Address​

Monitoring, Security

Compliance Standards​

  • APRA CPS 234 (Australia)
  • BSI C5 (Germany)
  • Brazil LGPD
  • CCPA / CPRA (California)
  • CIS Critical Security Controls v8
  • CMMC 2.0
  • CSA Cloud Controls Matrix v4
  • DPDPA
  • Digital Operational Resilience Act (EU)
  • ISO 27001
  • ISO/IEC 27018
  • ISO/IEC 27701
  • MAS Technology Risk Management (Singapore)
  • MITRE ATT&CK (Cloud)
  • NIST SP 800-171
  • NYDFS 23 NYCRR 500
  • SOC2
  • SWIFT Customer Security Controls Framework
  • Sarbanes-Oxley IT General Controls
  • UK NCSC Cyber Assessment Framework

Triage and Remediation​

Remediation​

Using Console

To remediate this in AWS SageMaker using the AWS Management Console, you must enable the setting when creating (or re-creating) the training job. Existing training jobs cannot be edited; you create a new job with the correct security settings.

1. Open SageMaker Training Jobs​

  1. Sign in to the AWS Management Console.
  2. In the top bar, choose the correct Region.
  3. Go to Amazon SageMaker.
  4. In the left navigation pane, select Training → Training jobs.

2. Re-create the Training Job With Encryption​

You’ll mirror the settings of the non-compliant job but enable inter-container traffic encryption.

  1. In Training jobs, click the name of the non-compliant training job.
  2. Note down (or screenshot) its configuration:
    • Training image / algorithm
    • Input data locations (S3 paths, channels)
    • Instance type and count
    • Output S3 location
    • Hyperparameters
    • VPC settings, IAM role, etc.
  3. Click Create training job (or Create at the top right).

3. Configure the New Training Job​

Fill in all the same settings as the old job, then enable the encryption:

  1. Job name: Give a new name (e.g., append -encrypted).

  2. IAM role, algorithm/image, input data configuration, output data configuration, resource configuration, etc., as per the original job.

  3. Scroll down to the Security and encryption or Additional configuration section (exact label may vary slightly by console version).

  4. Locate Inter-container traffic encryption:

    • Check the box Enable inter-container traffic encryption
      (or toggle it On).

  5. Optionally verify or set:

    • VPC settings.
    • KMS key for output data / volume encryption if you’re also standardizing storage encryption.

  6. Review all fields to ensure they match the previous job (except the name and the new encryption setting).

  7. Click Create training job.

4. Clean Up / Enforce Going Forward​

  1. After confirming the new job runs correctly, stop relying on the old non-compliant job (and optionally delete it).
  2. Update any process that creates training jobs (e.g., runbooks, internal docs) to require:
    • Inter-container traffic encryption enabled in the console for all new training jobs.

5. Validate Compliance​

  1. In Training jobs, open the new job’s details.
  2. In the Security and encryption section, confirm that:
    • Inter-container traffic encryption is shown as Enabled.

This ensures all network traffic between containers in the training cluster is encrypted for that job.

Using CLI

To fix this, you must create (or re-create) SageMaker training jobs with inter-container traffic encryption enabled. This cannot be retroactively changed on an already running or completed training job.

Below are step‑by‑step CLI instructions.

1. Identify training jobs missing inter-container encryption (optional)​

List recent training jobs:

aws sagemaker list-training-jobs \
  --status-equals InProgress \
  --max-results 50

(There is no direct flag to check this on an existing job; remediation is to ensure all new jobs have this flag enabled.)

2. Create a training job with inter-container traffic encryption enabled​

Prepare a minimal config file, e.g. training-job-config.json:

{
  "TrainingJobName": "my-secure-training-job",
  "RoleArn": "arn:aws:iam::123456789012:role/service-role/AmazonSageMaker-ExecutionRole",
  "AlgorithmSpecification": {
    "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",
    "TrainingInputMode": "File"
  },
  "InputDataConfig": [
    {
      "ChannelName": "train",
      "DataSource": {
        "S3DataSource": {
          "S3DataType": "S3Prefix",
          "S3Uri": "s3://my-bucket/training/",
          "S3DataDistributionType": "FullyReplicated"
        }
      }
    }
  ],
  "OutputDataConfig": {
    "S3OutputPath": "s3://my-bucket/output/"
  },
  "ResourceConfig": {
    "InstanceType": "ml.m5.xlarge",
    "InstanceCount": 1,
    "VolumeSizeInGB": 50
  },
  "StoppingCondition": {
    "MaxRuntimeInSeconds": 3600
  },
  "EnableInterContainerTrafficEncryption": true
}

Create the training job:

aws sagemaker create-training-job \
  --cli-input-json file://training-job-config.json

Key line for remediation:

"EnableInterContainerTrafficEncryption": true

You can also pass it inline:

aws sagemaker create-training-job \
  --training-job-name my-secure-training-job \
  --role-arn arn:aws:iam::123456789012:role/service-role/AmazonSageMaker-ExecutionRole \
  --algorithm-specification TrainingImage=123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest,TrainingInputMode=File \
  --input-data-config '[{"ChannelName":"train","DataSource":{"S3DataSource":{"S3DataType":"S3Prefix","S3Uri":"s3://my-bucket/training/","S3DataDistributionType":"FullyReplicated"}}}]' \
  --output-data-config S3OutputPath=s3://my-bucket/output/ \
  --resource-config InstanceType=ml.m5.xlarge,InstanceCount=1,VolumeSizeInGB=50 \
  --stopping-condition MaxRuntimeInSeconds=3600 \
  --enable-inter-container-traffic-encryption

Note: the short option is --enable-inter-container-traffic-encryption.

3. Update any automation/templates to always enable it​

Wherever you define training jobs (CloudFormation, CDK, Step Functions, SageMaker Pipelines, custom scripts), ensure the equivalent flag is set:

  • CloudFormation: EnableInterContainerTrafficEncryption: true
  • CDK (TypeScript/Java/Python): enableInterContainerTrafficEncryption: true
  • Raw CLI/scripts: always pass --enable-inter-container-traffic-encryption or "EnableInterContainerTrafficEncryption": true.

4. Recreate important recurring jobs​

For scheduled or recurring training:

  1. Take the existing training job definition (can be viewed via): 
    aws sagemaker describe-training-job \
      --training-job-name existing-job-name
  2. Copy the response JSON, add "EnableInterContainerTrafficEncryption": true.
  3. Change TrainingJobName.
  4. Call create-training-job with the modified JSON.
Using Python

Below are step‑by‑step ways to ensure inter-container traffic encryption is enabled for SageMaker training jobs using Python.

a. Install / upgrade the SDK​

pip install --upgrade sagemaker boto3

b. Enable encrypt_inter_container_traffic on an Estimator​

import sagemaker
from sagemaker import Estimator
import boto3

session = sagemaker.Session()
role = "arn:aws:iam::<ACCOUNT_ID>:role/<SAGEMAKER_EXECUTION_ROLE>"

estimator = Estimator(
    image_uri="123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training-image:latest",
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge",
    volume_size=50,
    max_run=3600,
    sagemaker_session=session,
    # This flag enables inter-container traffic encryption
    encrypt_inter_container_traffic=True
)

# Example hyperparameters & inputs
estimator.set_hyperparameters(epochs=10, batch_size=32)

estimator.fit(
    inputs={"training": "s3://my-bucket/my-training-data/"},
    job_name="my-secure-training-job"
)

This ensures that if the job uses multiple containers (e.g., distributed training), traffic between containers is encrypted.

2. Using boto3 (low-level API)​

If you are creating training jobs directly with boto3, set the field EnableInterContainerTrafficEncryption to True.

a. Create a training job with encryption enabled​

import boto3
import time

sm_client = boto3.client("sagemaker", region_name="us-east-1")

training_job_name = f"my-secure-training-job-{int(time.time())}"

response = sm_client.create_training_job(
    TrainingJobName=training_job_name,
    AlgorithmSpecification={
        "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training-image:latest",
        "TrainingInputMode": "File"
    },
    RoleArn="arn:aws:iam::<ACCOUNT_ID>:role/<SAGEMAKER_EXECUTION_ROLE>",
    InputDataConfig=[
        {
            "ChannelName": "training",
            "DataSource": {
                "S3DataSource": {
                    "S3DataType": "S3Prefix",
                    "S3Uri": "s3://my-bucket/my-training-data/",
                    "S3DataDistributionType": "FullyReplicated",
                }
            },
            "ContentType": "text/csv",
            "InputMode": "File",
        }
    ],
    OutputDataConfig={
        "S3OutputPath": "s3://my-bucket/my-training-output/"
    },
    ResourceConfig={
        "InstanceType": "ml.m5.xlarge",
        "InstanceCount": 2,  # e.g., distributed training
        "VolumeSizeInGB": 50,
    },
    StoppingCondition={
        "MaxRuntimeInSeconds": 3600
    },
    EnableInterContainerTrafficEncryption=True  # ← critical flag
)
print("Training job created:", response["TrainingJobArn"])

3. Remediating existing non‑compliant jobs​

You cannot modify this flag on a running or completed training job. To remediate:

  1. Identify non-compliant jobs (no encryption enabled).
  2. Recreate those jobs with EnableInterContainerTrafficEncryption=True (or encrypt_inter_container_traffic=True via the SDK).

a. Detect jobs missing inter-container traffic encryption​

import boto3

sm_client = boto3.client("sagemaker", region_name="us-east-1")

paginator = sm_client.get_paginator("list_training_jobs")
for page in paginator.paginate():
    for summary in page["TrainingJobSummaries"]:
        name = summary["TrainingJobName"]
        desc = sm_client.describe_training_job(TrainingJobName=name)
        # Default is False if not provided
        enabled = desc.get("EnableInterContainerTrafficEncryption", False)
        if not enabled:
            print("Non-compliant training job:", name)

b. Clone a non-compliant job with encryption enabled​

def clone_with_encryption(old_job_name, new_job_name):
    desc = sm_client.describe_training_job(TrainingJobName=old_job_name)

    # Build new request, copying configuration but overriding the flag
    params = {
        "TrainingJobName": new_job_name,
        "AlgorithmSpecification": desc["AlgorithmSpecification"],
        "RoleArn": desc["RoleArn"],
        "InputDataConfig": desc.get("InputDataConfig", []),
        "OutputDataConfig": desc["OutputDataConfig"],
        "ResourceConfig": desc["ResourceConfig"],
        "StoppingCondition": desc["StoppingCondition"],
        "EnableInterContainerTrafficEncryption": True,
    }

    if "VpcConfig" in desc:
        params["VpcConfig"] = desc["VpcConfig"]
    if "HyperParameters" in desc:
        params["HyperParameters"] = desc["HyperParameters"]
    if "Environment" in desc:
        params["Environment"] = desc["Environment"]
    if "RetryStrategy" in desc:
        params["RetryStrategy"] = desc["RetryStrategy"]
    if "ExperimentConfig" in desc:
        params["ExperimentConfig"] = desc["ExperimentConfig"]

    resp = sm_client.create_training_job(**params)
    return resp["TrainingJobArn"]

# Example usage:
old_job = "my-old-training-job"
new_job = f"{old_job}-encrypted"
arn = clone_with_encryption(old_job, new_job)
print("New encrypted job ARN:", arn)

4. Make encryption the default in your codebase​

  • For all new SageMaker Estimators, always set encrypt_inter_container_traffic=True.
  • For all direct boto3.create_training_job calls, always set EnableInterContainerTrafficEncryption=True.
  • Optionally, add a simple CI/static check to fail builds if the flag is missing.
Using Terraform
resource "aws_sagemaker_training_job" "TRAINING_JOB_NAME" {
  name                = "TRAINING_JOB_NAME" # replace with your job name
  role_arn            = aws_iam_role.SAGEMAKER_ROLE.arn
  training_image      = "ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/IMAGE_NAME:TAG"
  training_input_mode = "File"

  resource_config {
    instance_type  = "ml.m5.xlarge"
    instance_count = 1
    volume_size_gb = 50
  }

  output_data_config {
    s3_output_path = "s3://S3_BUCKET_NAME/output/" # replace with your bucket
  }

  stopping_condition {
    max_runtime_in_seconds = 3600
  }

  # Remediation: enable inter-container traffic encryption
  enable_inter_container_traffic_encryption = true
}

Changing enable_inter_container_traffic_encryption on an existing aws_sagemaker_training_job forces replacement because SageMaker training jobs are immutable; Terraform will destroy the old job resource and create a new one instead of updating in place.

For verification, terraform plan should show the existing aws_sagemaker_training_job scheduled for replacement with enable_inter_container_traffic_encryption changing from false (or null) to true.