Skip to main content

Sagemaker Training Jobs Should Have Network Isolation

More Info:​

Sagemaker Training Jobs should have network isolation enabled

Risk Level​

Medium

Address​

Monitoring, Security

Compliance Standards​

  • APRA CPS 234 (Australia)
  • BSI C5 (Germany)
  • Brazil LGPD
  • CCPA / CPRA (California)
  • CIS Critical Security Controls v8
  • CMMC 2.0
  • CSA Cloud Controls Matrix v4
  • DPDPA
  • Digital Operational Resilience Act (EU)
  • ISO 27001
  • ISO/IEC 27018
  • ISO/IEC 27701
  • MAS Technology Risk Management (Singapore)
  • MITRE ATT&CK (Cloud)
  • NIST SP 800-171
  • NYDFS 23 NYCRR 500
  • SOC2
  • SWIFT Customer Security Controls Framework
  • Sarbanes-Oxley IT General Controls
  • UK NCSC Cyber Assessment Framework

Triage and Remediation​

Remediation​

Using Console

Here’s how to ensure SageMaker Training Jobs have Network Isolation enabled using the AWS Management Console:

1. Understand the Limitation​

  • You cannot edit an existing completed/running training job to enable network isolation.
  • You must create a new training job (or update the template/pipeline/notebook code that creates it) with Network isolation turned on.

2. Enable Network Isolation When Creating a Training Job​

  1. Sign in to AWS Console

  2. Open SageMaker

    • In the Services menu, choose Amazon SageMaker.

  3. Go to Training Jobs

    • In the left navigation pane, under Training, click Training jobs.
    • To recreate an existing job with isolation, select that job and click Create similar (if available in your console version), or note down its configuration to re-enter.

  4. Create a New Training Job

    • Click Create training job (or Create → Training job, depending on UI version).

  5. Fill in Basic Settings

    • Job name: Provide a unique name.
    • IAM role: Select a role with needed permissions (S3, logs, etc.).
    • Configure Algorithm / Container, Input data configuration, Output data configuration, etc. as you normally would.

  6. Enable Network Isolation

    • Scroll to the Network or Security section (label may vary slightly by console version).
    • Find the option usually named:
      • Network isolation, or
      • Enable network isolation
    • Check the box or choose Enabled.
    • This corresponds to setting EnableNetworkIsolation = true for the training job.

  7. (Optional) Configure VPC Settings

    • In the same Network section, you can also:
      • Select a VPC, Subnets, and Security groups if you need training to run in a private subnet.
    • This is separate from network isolation, but commonly used together.

  8. Review and Create

    • Review all configurations.
    • Click Create training job.

This new training job will now run with network isolation: the container won’t have outbound network access except for what SageMaker itself needs for the job lifecycle.

3. Ongoing Remediation​

  • For any place you define training jobs (SageMaker Studio UI, notebook code, CI/CD pipelines, SageMaker Pipelines):
    • Ensure EnableNetworkIsolation = true is always set.
  • Over time, deprecate old jobs / code that created training jobs without network isolation.
Using CLI

For SageMaker training jobs, network isolation is enabled per training job and cannot be turned on for an already-running/completed job. You must submit new jobs with network isolation enabled.

Below are the key steps using AWS CLI.

1. Create a training job with network isolation enabled​

You can either use a JSON config file or pass parameters inline. The important flag is:

--enable-network-isolation

Option A: Using a JSON config file​

  1. Create a file training-job-config.json:
{
  "TrainingJobName": "my-training-job-net-isolated",
  "AlgorithmSpecification": {
    "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training-image:latest",
    "TrainingInputMode": "File"
  },
  "RoleArn": "arn:aws:iam::123456789012:role/SageMakerExecutionRole",
  "InputDataConfig": [
    {
      "ChannelName": "training",
      "DataSource": {
        "S3DataSource": {
          "S3DataType": "S3Prefix",
          "S3Uri": "s3://my-bucket/training-data/",
          "S3DataDistributionType": "FullyReplicated"
        }
      },
      "ContentType": "text/csv"
    }
  ],
  "OutputDataConfig": {
    "S3OutputPath": "s3://my-bucket/output/"
  },
  "ResourceConfig": {
    "InstanceType": "ml.m5.large",
    "InstanceCount": 1,
    "VolumeSizeInGB": 50
  },
  "StoppingCondition": {
    "MaxRuntimeInSeconds": 3600
  },
  "EnableNetworkIsolation": true
}
  1. Create the job via CLI:
aws sagemaker create-training-job \
  --cli-input-json file://training-job-config.json

"EnableNetworkIsolation": true in the JSON is equivalent to --enable-network-isolation on the CLI.

Option B: Passing parameters inline​

aws sagemaker create-training-job \
  --training-job-name my-training-job-net-isolated \
  --algorithm-specification TrainingImage=123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training-image:latest,TrainingInputMode=File \
  --role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole \
  --input-data-config '[{
      "ChannelName":"training",
      "DataSource":{
        "S3DataSource":{
          "S3DataType":"S3Prefix",
          "S3Uri":"s3://my-bucket/training-data/",
          "S3DataDistributionType":"FullyReplicated"
        }
      }
  }]' \
  --output-data-config S3OutputPath=s3://my-bucket/output/ \
  --resource-config InstanceType=ml.m5.large,InstanceCount=1,VolumeSizeInGB=50 \
  --stopping-condition MaxRuntimeInSeconds=3600 \
  --enable-network-isolation

2. Ensure pipelines / automations use the flag​

Wherever training jobs are created (scripts, CI/CD, Step Functions, SageMaker Pipelines, etc.), confirm:

  • They use enable_network_isolation=True (SDK)
    or
  • --enable-network-isolation / "EnableNetworkIsolation": true (CLI/JSON).

This makes all future training jobs compliant.

Using Python

To remediate this, you must (a) ensure all new training jobs are created with network isolation, and (b) stop/recreate any existing non‑isolated jobs (you cannot “flip” isolation on a running job).

Below are step‑by‑step instructions using Python.

1. Using the low-level boto3 SageMaker client​

a. Create training jobs with network isolation​

import boto3

sm = boto3.client("sagemaker", region_name="us-east-1")  # adjust region

training_job_name = "my-training-job-with-network-isolation"

response = sm.create_training_job(
    TrainingJobName=training_job_name,
    AlgorithmSpecification={
        "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",
        "TrainingInputMode": "File",
    },
    RoleArn="arn:aws:iam::123456789012:role/SageMakerExecutionRole",
    InputDataConfig=[
        {
            "ChannelName": "training",
            "DataSource": {
                "S3DataSource": {
                    "S3DataType": "S3Prefix",
                    "S3Uri": "s3://my-bucket/training-data/",
                    "S3DataDistributionType": "FullyReplicated",
                }
            },
            "ContentType": "text/csv",
        }
    ],
    OutputDataConfig={
        "S3OutputPath": "s3://my-bucket/output/",
    },
    ResourceConfig={
        "InstanceType": "ml.m5.xlarge",
        "InstanceCount": 1,
        "VolumeSizeInGB": 50,
    },
    StoppingCondition={
        "MaxRuntimeInSeconds": 3600,
    },
    EnableNetworkIsolation=True,  # <<< THIS IS THE KEY SETTING
)
print("Created training job:", response["TrainingJobArn"])

Key field:
EnableNetworkIsolation=True must be set on every create_training_job call.

b. Recreate existing training jobs with network isolation​

import boto3

sm = boto3.client("sagemaker", region_name="us-east-1")

old_job_name = "my-old-training-job"

# 1. Describe old job
desc = sm.describe_training_job(TrainingJobName=old_job_name)

# 2. Build new job request with EnableNetworkIsolation=True
new_job_name = old_job_name + "-isolated"

create_args = {
    "TrainingJobName": new_job_name,
    "AlgorithmSpecification": desc["AlgorithmSpecification"],
    "RoleArn": desc["RoleArn"],
    "InputDataConfig": desc.get("InputDataConfig", []),
    "OutputDataConfig": desc["OutputDataConfig"],
    "ResourceConfig": desc["ResourceConfig"],
    "StoppingCondition": desc["StoppingCondition"],
    "EnableNetworkIsolation": True,
}

# Optional fields to carry over if present
for k in [
    "HyperParameters",
    "VpcConfig",
    "Tags",
    "EnableManagedSpotTraining",
    "CheckpointConfig",
    "DebugHookConfig",
    "DebugRuleConfigurations",
    "ProfilerConfig",
    "ProfilerRuleConfigurations",
    "Environment",
    "RetryStrategy",
]:
    if k in desc:
        create_args[k] = desc[k]

resp = sm.create_training_job(**create_args)
print("Created new isolated job:", resp["TrainingJobArn"])

Then stop or let the original non‑isolated job complete and only use the new isolated one going forward.

2. Using the high-level sagemaker Python SDK​

If you use the SageMaker Python SDK (pip install sagemaker):

import sagemaker
from sagemaker.estimator import Estimator

session = sagemaker.Session()
role = "arn:aws:iam::123456789012:role/SageMakerExecutionRole"

estimator = Estimator(
    image_uri="123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",
    role=role,
    instance_count=1,
    instance_type="ml.m5.xlarge",
    volume_size=50,
    output_path="s3://my-bucket/output/",
    sagemaker_session=session,
    enable_network_isolation=True,  # <<< REQUIRED
)

estimator.fit(
    inputs={
        "training": "s3://my-bucket/training-data/"
    }
)

Here enable_network_isolation=True ensures EnableNetworkIsolation=True in the underlying training job.

3. Policy / guardrail (optional)​

To enforce this org‑wide, use:

  • AWS Config rule sagemaker-training-job-network-isolation-enabled
  • Or a custom rule that checks EnableNetworkIsolation on training jobs
    and alerts/blocks when it’s not True.

But from a Python remediation standpoint, the essential change is always: set EnableNetworkIsolation=True on all training job creations and recreate any existing ones without it.

Using Terraform
resource "aws_sagemaker_training_job" "TRAINING_JOB_NAME" {
  # Replace TRAINING_JOB_NAME with a valid Terraform identifier (e.g. "image_classifier")

  name                = "SAGEMAKER_TRAINING_JOB_NAME" # replace with the actual job name
  role_arn            = "SAGEMAKER_EXECUTION_ROLE_ARN" # replace with your IAM role ARN
  algorithm_specification {
    training_image     = "TRAINING_IMAGE_URI"          # replace with your container image
    training_input_mode = "File"
  }

  input_data_config {
    channel_name = "training"
    data_source {
      s3_data_source {
        s3_data_type        = "S3Prefix"
        s3_uri              = "s3://TRAINING_DATA_BUCKET/PREFIX/" # replace with your data location
        s3_data_distribution_type = "FullyReplicated"
      }
    }
  }

  output_data_config {
    s3_output_path = "s3://OUTPUT_BUCKET/PREFIX/"      # replace with your output location
  }

  resource_config {
    instance_type   = "ml.m5.xlarge"
    instance_count  = 1
    volume_size_gb  = 50
  }

  stopping_condition {
    max_runtime_in_seconds = 3600
  }

  # Fix: enable network isolation for this SageMaker Training Job
  enable_network_isolation = true
}

Changing enable_network_isolation on an existing SageMaker training job forces replacement (a new training job with a new name), because SageMaker training jobs are immutable once created.

To verify, terraform plan should show enable_network_isolation changing from false (or unset) to true on aws_sagemaker_training_job.TRAINING_JOB_NAME, with Terraform indicating that the resource will be destroyed and re-created.