> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Sagemaker Training Jobs Should Have Network Isolation Enabled

### More Info:

Sagemaker Training Jobs should have network isolation enabled

### Risk Level

Medium

### Address

Monitoring, Security

### Compliance Standards

* APRA CPS 234 (Australia)
* BSI C5 (Germany)
* Brazil LGPD
* CCPA / CPRA (California)
* CIS Critical Security Controls v8
* CMMC 2.0
* CSA Cloud Controls Matrix v4
* DPDPA
* Digital Operational Resilience Act (EU)
* ISO 27001
* ISO/IEC 27018
* ISO/IEC 27701
* MAS Technology Risk Management (Singapore)
* MITRE ATT\&CK (Cloud)
* NIST SP 800-171
* NYDFS 23 NYCRR 500
* SOC2
* SWIFT Customer Security Controls Framework
* Sarbanes-Oxley IT General Controls
* UK NCSC Cyber Assessment Framework

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Using Console" defaultOpen="true">
        Here’s how to ensure SageMaker Training Jobs have **Network Isolation** enabled using the AWS Management Console:

        ***

        ### 1. Understand the Limitation

        * **You cannot edit an existing completed/running training job** to enable network isolation.
        * You must **create a new training job** (or update the template/pipeline/notebook code that creates it) with `Network isolation` turned on.

        ***

        ### 2. Enable Network Isolation When Creating a Training Job

        1. **Sign in to AWS Console**
           * Go to: [https://console.aws.amazon.com/](https://console.aws.amazon.com/)
           * Switch to the region where you run SageMaker.

        2. **Open SageMaker**
           * In the Services menu, choose **Amazon SageMaker**.

        3. **Go to Training Jobs**
           * In the left navigation pane, under **Training**, click **Training jobs**.
           * To recreate an existing job with isolation, select that job and click **Create similar** (if available in your console version), or note down its configuration to re-enter.

        4. **Create a New Training Job**
           * Click **Create training job** (or **Create** → **Training job**, depending on UI version).

        5. **Fill in Basic Settings**
           * **Job name**: Provide a unique name.
           * **IAM role**: Select a role with needed permissions (S3, logs, etc.).
           * Configure **Algorithm / Container**, **Input data configuration**, **Output data configuration**, etc. as you normally would.

        6. **Enable Network Isolation**
           * Scroll to the **Network** or **Security** section (label may vary slightly by console version).
           * Find the option usually named:
             * **Network isolation**, or
             * **Enable network isolation**
           * Check the box or choose **Enabled**.
           * This corresponds to setting `EnableNetworkIsolation = true` for the training job.

        7. **(Optional) Configure VPC Settings**
           * In the same Network section, you can also:
             * Select a **VPC**, **Subnets**, and **Security groups** if you need training to run in a private subnet.
           * This is separate from network isolation, but commonly used together.

        8. **Review and Create**
           * Review all configurations.
           * Click **Create training job**.

        This new training job will now run with **network isolation**: the container won’t have outbound network access except for what SageMaker itself needs for the job lifecycle.

        ***

        ### 3. Ongoing Remediation

        * For any place you define training jobs (SageMaker Studio UI, notebook code, CI/CD pipelines, SageMaker Pipelines):
          * Ensure `EnableNetworkIsolation = true` is always set.
        * Over time, deprecate old jobs / code that created training jobs without network isolation.
      </Accordion>

      <Accordion title="Using CLI">
        For SageMaker training jobs, network isolation is enabled **per training job** and cannot be turned on for an already-running/completed job. You must submit new jobs with network isolation enabled.

        Below are the key steps using AWS CLI.

        ***

        ## 1. Create a training job with network isolation enabled

        You can either use a JSON config file or pass parameters inline. The important flag is:

        ```bash theme={null}
        --enable-network-isolation
        ```

        ### Option A: Using a JSON config file

        1. Create a file `training-job-config.json`:

        ```json theme={null}
        {
          "TrainingJobName": "my-training-job-net-isolated",
          "AlgorithmSpecification": {
            "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training-image:latest",
            "TrainingInputMode": "File"
          },
          "RoleArn": "arn:aws:iam::123456789012:role/SageMakerExecutionRole",
          "InputDataConfig": [
            {
              "ChannelName": "training",
              "DataSource": {
                "S3DataSource": {
                  "S3DataType": "S3Prefix",
                  "S3Uri": "s3://my-bucket/training-data/",
                  "S3DataDistributionType": "FullyReplicated"
                }
              },
              "ContentType": "text/csv"
            }
          ],
          "OutputDataConfig": {
            "S3OutputPath": "s3://my-bucket/output/"
          },
          "ResourceConfig": {
            "InstanceType": "ml.m5.large",
            "InstanceCount": 1,
            "VolumeSizeInGB": 50
          },
          "StoppingCondition": {
            "MaxRuntimeInSeconds": 3600
          },
          "EnableNetworkIsolation": true
        }
        ```

        2. Create the job via CLI:

        ```bash theme={null}
        aws sagemaker create-training-job \
          --cli-input-json file://training-job-config.json
        ```

        `"EnableNetworkIsolation": true` in the JSON is equivalent to `--enable-network-isolation` on the CLI.

        ***

        ### Option B: Passing parameters inline

        ```bash theme={null}
        aws sagemaker create-training-job \
          --training-job-name my-training-job-net-isolated \
          --algorithm-specification TrainingImage=123456789012.dkr.ecr.us-east-1.amazonaws.com/my-training-image:latest,TrainingInputMode=File \
          --role-arn arn:aws:iam::123456789012:role/SageMakerExecutionRole \
          --input-data-config '[{
              "ChannelName":"training",
              "DataSource":{
                "S3DataSource":{
                  "S3DataType":"S3Prefix",
                  "S3Uri":"s3://my-bucket/training-data/",
                  "S3DataDistributionType":"FullyReplicated"
                }
              }
          }]' \
          --output-data-config S3OutputPath=s3://my-bucket/output/ \
          --resource-config InstanceType=ml.m5.large,InstanceCount=1,VolumeSizeInGB=50 \
          --stopping-condition MaxRuntimeInSeconds=3600 \
          --enable-network-isolation
        ```

        ***

        ## 2. Ensure pipelines / automations use the flag

        Wherever training jobs are created (scripts, CI/CD, Step Functions, SageMaker Pipelines, etc.), confirm:

        * They use `enable_network_isolation=True` (SDK)\
          or
        * `--enable-network-isolation` / `"EnableNetworkIsolation": true` (CLI/JSON).

        This makes all future training jobs compliant.
      </Accordion>

      <Accordion title="Using Python">
        To remediate this, you must (a) ensure **all new training jobs** are created with network isolation, and (b) **stop/recreate** any existing non‑isolated jobs (you cannot “flip” isolation on a running job).

        Below are step‑by‑step instructions using Python.

        ***

        ## 1. Using the low-level `boto3` SageMaker client

        ### a. Create training jobs with network isolation

        ```python theme={null}
        import boto3

        sm = boto3.client("sagemaker", region_name="us-east-1")  # adjust region

        training_job_name = "my-training-job-with-network-isolation"

        response = sm.create_training_job(
            TrainingJobName=training_job_name,
            AlgorithmSpecification={
                "TrainingImage": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",
                "TrainingInputMode": "File",
            },
            RoleArn="arn:aws:iam::123456789012:role/SageMakerExecutionRole",
            InputDataConfig=[
                {
                    "ChannelName": "training",
                    "DataSource": {
                        "S3DataSource": {
                            "S3DataType": "S3Prefix",
                            "S3Uri": "s3://my-bucket/training-data/",
                            "S3DataDistributionType": "FullyReplicated",
                        }
                    },
                    "ContentType": "text/csv",
                }
            ],
            OutputDataConfig={
                "S3OutputPath": "s3://my-bucket/output/",
            },
            ResourceConfig={
                "InstanceType": "ml.m5.xlarge",
                "InstanceCount": 1,
                "VolumeSizeInGB": 50,
            },
            StoppingCondition={
                "MaxRuntimeInSeconds": 3600,
            },
            EnableNetworkIsolation=True,  # <<< THIS IS THE KEY SETTING
        )
        print("Created training job:", response["TrainingJobArn"])
        ```

        Key field:\
        `EnableNetworkIsolation=True` must be set on every `create_training_job` call.

        ***

        ### b. Recreate existing training jobs with network isolation

        ```python theme={null}
        import boto3

        sm = boto3.client("sagemaker", region_name="us-east-1")

        old_job_name = "my-old-training-job"

        # 1. Describe old job
        desc = sm.describe_training_job(TrainingJobName=old_job_name)

        # 2. Build new job request with EnableNetworkIsolation=True
        new_job_name = old_job_name + "-isolated"

        create_args = {
            "TrainingJobName": new_job_name,
            "AlgorithmSpecification": desc["AlgorithmSpecification"],
            "RoleArn": desc["RoleArn"],
            "InputDataConfig": desc.get("InputDataConfig", []),
            "OutputDataConfig": desc["OutputDataConfig"],
            "ResourceConfig": desc["ResourceConfig"],
            "StoppingCondition": desc["StoppingCondition"],
            "EnableNetworkIsolation": True,
        }

        # Optional fields to carry over if present
        for k in [
            "HyperParameters",
            "VpcConfig",
            "Tags",
            "EnableManagedSpotTraining",
            "CheckpointConfig",
            "DebugHookConfig",
            "DebugRuleConfigurations",
            "ProfilerConfig",
            "ProfilerRuleConfigurations",
            "Environment",
            "RetryStrategy",
        ]:
            if k in desc:
                create_args[k] = desc[k]

        resp = sm.create_training_job(**create_args)
        print("Created new isolated job:", resp["TrainingJobArn"])
        ```

        Then stop or let the original non‑isolated job complete and only use the new isolated one going forward.

        ***

        ## 2. Using the high-level `sagemaker` Python SDK

        If you use the SageMaker Python SDK (`pip install sagemaker`):

        ```python theme={null}
        import sagemaker
        from sagemaker.estimator import Estimator

        session = sagemaker.Session()
        role = "arn:aws:iam::123456789012:role/SageMakerExecutionRole"

        estimator = Estimator(
            image_uri="123456789012.dkr.ecr.us-east-1.amazonaws.com/my-image:latest",
            role=role,
            instance_count=1,
            instance_type="ml.m5.xlarge",
            volume_size=50,
            output_path="s3://my-bucket/output/",
            sagemaker_session=session,
            enable_network_isolation=True,  # <<< REQUIRED
        )

        estimator.fit(
            inputs={
                "training": "s3://my-bucket/training-data/"
            }
        )
        ```

        Here `enable_network_isolation=True` ensures `EnableNetworkIsolation=True` in the underlying training job.

        ***

        ## 3. Policy / guardrail (optional)

        To enforce this org‑wide, use:

        * AWS Config rule `sagemaker-training-job-network-isolation-enabled`
        * Or a custom rule that checks `EnableNetworkIsolation` on training jobs\
          and alerts/blocks when it’s not `True`.

        But from a Python remediation standpoint, the essential change is always: **set `EnableNetworkIsolation=True` on all training job creations and recreate any existing ones without it.**
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>