Skip to main content

Sagemaker Endpoint Should Have Data Capture Enabled

More Info:​

Sagemaker Endpoint should have data capture enabled

Risk Level​

Medium

Address​

Monitoring, Security

Compliance Standards​

  • APRA CPS 234 (Australia)
  • BSI C5 (Germany)
  • Brazil LGPD
  • CCPA / CPRA (California)
  • CIS Critical Security Controls v8
  • CMMC 2.0
  • CSA Cloud Controls Matrix v4
  • DPDPA
  • Digital Operational Resilience Act (EU)
  • ISO 27001
  • ISO/IEC 27018
  • ISO/IEC 27701
  • MAS Technology Risk Management (Singapore)
  • MITRE ATT&CK (Cloud)
  • NIST SP 800-171
  • NYDFS 23 NYCRR 500
  • SOC2
  • SWIFT Customer Security Controls Framework
  • Sarbanes-Oxley IT General Controls
  • UK NCSC Cyber Assessment Framework

Triage and Remediation​

Remediation​

Using Console

Below are step‑by‑step AWS Console instructions to enable Data Capture on a SageMaker endpoint.

1. Prerequisites​

  1. You must have:
    • An existing SageMaker endpoint.
    • An S3 bucket where captured data will be stored (or permission to create one).
    • IAM permissions to update the endpoint configuration and write to the S3 bucket.

2. Identify the Endpoint Configuration​

  1. Sign in to the AWS Management Console.
  2. Go to Amazon SageMaker.
  3. In the left navigation pane, select Inference → Endpoints.
  4. Click your endpoint name.
  5. On the endpoint details page, note the Endpoint configuration name (you’ll need it if you want to clone/modify it).

3. Create a New Endpoint Configuration with Data Capture Enabled​

You can’t usually edit an existing endpoint configuration in-place; you create a new one with Data Capture enabled and then update the endpoint to use it.

  1. In the SageMaker left navigation pane, go to Inference → Endpoint configurations.

  2. Click Create endpoint configuration.

  3. Under Endpoint configuration name, enter a new name (e.g., my-endpoint-config-with-datacapture).

  4. Under Production variants, replicate the settings from your existing configuration:

    • Model name
    • Instance type
    • Initial instance count
    • Any other variant settings needed (weight, variant name, etc.).

  5. Scroll down to Data capture (or Enable data capture section):

    • Check Enable data capture.
    • S3 location: choose or browse to an S3 bucket/prefix, e.g., s3://my-bucket/sagemaker/datacapture/.
    • Sampling percentage: enter a value (e.g., 100 to capture all requests/responses, or a lower percentage).
    • Capture options: select what to capture:
      • Requests
      • Responses
    • Encryption key (optional): specify a KMS key if you want encryption using a customer-managed key.
    • Leave other options as defaults unless you have specific compliance/logging needs.

  6. Review all settings and click Create endpoint configuration.

4. Update the Endpoint to Use the New Configuration​

  1. Go back to Endpoints.
  2. Click your endpoint.
  3. At the top right, click Update (or Modify).
  4. In the Endpoint configuration section, select the new endpoint configuration you just created (e.g., my-endpoint-config-with-datacapture).
  5. Click Update endpoint.

SageMaker will start updating the endpoint (status: Updating). Once status becomes InService, the endpoint will be using the new configuration with Data Capture enabled.

5. Verify Data Capture​

  1. Send some inference requests to the endpoint (via your application, SDK, or SageMaker notebook).
  2. In the S3 console, navigate to the bucket/prefix you configured for data capture.
  3. Confirm that JSON lines files are appearing under paths like:
    • s3://my-bucket/sagemaker/datacapture/EndpointName/AllTraffic/2024/...

6. (Optional) Clean Up Old Endpoint Configuration​

To avoid confusion:

  1. In Endpoint configurations, find the old configuration that no longer has data capture.
  2. Confirm that no other endpoints are using it.
  3. Select it and choose Delete if safe to do so.
Using CLI

To enable data capture on a SageMaker endpoint via AWS CLI, you must (a) have an S3 bucket for captured data, and (b) use an endpoint configuration that includes DataCaptureConfig. Then you either create a new endpoint with that config or update an existing one to use it.

Below are the minimal step‑by‑step commands.

1. Prerequisites​

  • An S3 bucket for data capture (example: s3://my-sagemaker-data-capture/)
  • The IAM role used by the endpoint must have s3:PutObject to that bucket.

Example policy snippet to attach to the endpoint’s execution role:

{
  "Effect": "Allow",
  "Action": ["s3:PutObject"],
  "Resource": "arn:aws:s3:::my-sagemaker-data-capture/*"
}

2. Check current endpoint and its config​

aws sagemaker describe-endpoint \
  --endpoint-name my-endpoint

Note the EndpointConfigName, e.g. my-endpoint-config.

aws sagemaker describe-endpoint-config \
  --endpoint-config-name my-endpoint-config

If DataCaptureConfig is missing, you need a new endpoint config.

3. Create a new endpoint configuration with DataCaptureConfig​

You generally cannot modify an existing endpoint config; you create a new one and then update the endpoint to use it.

Example JSON file production-variants.json (reusing the same variant(s) as existing endpoint):

[
  {
    "VariantName": "AllTraffic",
    "ModelName": "my-model-name",
    "InitialInstanceCount": 1,
    "InstanceType": "ml.m5.xlarge"
  }
]

Now create the new endpoint config:

aws sagemaker create-endpoint-config \
  --endpoint-config-name my-endpoint-config-with-dc \
  --production-variants file://production-variants.json \
  --data-capture-config '{
    "EnableCapture": true,
    "InitialSamplingPercentage": 100,
    "DestinationS3Uri": "s3://my-sagemaker-data-capture/",
    "CaptureOptions": [
      {"CaptureMode": "Input"},
      {"CaptureMode": "Output"}
    ],
    "CaptureContentTypeHeader": {
      "JsonContentTypes": ["application/json"],
      "CsvContentTypes": ["text/csv"]
    }
  }'

Adjust:

  • InitialSamplingPercentage: e.g., 10 for 10% of traffic
  • CaptureOptions: Input, Output, or both
  • DestinationS3Uri: your bucket/prefix

4. Update the endpoint to use the new config​

aws sagemaker update-endpoint \
  --endpoint-name my-endpoint \
  --endpoint-config-name my-endpoint-config-with-dc

Wait for the update to complete:

aws sagemaker describe-endpoint \
  --endpoint-name my-endpoint \
  --query 'EndpointStatus'

When it shows InService, data capture is enabled.

5. Verify data capture​

After sending some traffic to the endpoint, check S3:

aws s3 ls s3://my-sagemaker-data-capture/ --recursive

You should see JSON files in the capture prefix.

Using Python

To enable data capture on a SageMaker endpoint using Python/boto3, you must:

  1. Create (or update) an endpoint configuration with DataCaptureConfig.
  2. Point it to an S3 bucket/prefix.
  3. Update the endpoint to use that new endpoint configuration.

Below is a minimal, step‑by‑step example.

0. Prerequisites​

pip install boto3

You need:

  • An existing SageMaker endpoint name (e.g. my-endpoint)
  • An S3 bucket/prefix to store captured data (e.g. s3://my-bucket/sagemaker/data-capture/)
  • IAM role used by SageMaker with permissions to write to that S3 location

1. Get current endpoint configuration​

import boto3

sm = boto3.client("sagemaker", region_name="us-east-1")

endpoint_name = "my-endpoint"

# Get the current endpoint info
endpoint_desc = sm.describe_endpoint(EndpointName=endpoint_name)
current_config_name = endpoint_desc["EndpointConfigName"]

# Get current endpoint config to reuse settings (production variants, etc.)
current_config = sm.describe_endpoint_config(
    EndpointConfigName=current_config_name
)

2. Build a new EndpointConfig with DataCaptureConfig​

import time

s3_capture_location = "s3://my-bucket/sagemaker/data-capture/"
# Optional: new config name
new_config_name = f"{endpoint_name}-config-with-capture-{int(time.time())}"

# Reuse production variants from the existing config
production_variants = current_config["ProductionVariants"]

# Optional: reuse tags, kms, etc. if needed
kms_key_id = current_config.get("KmsKeyId")

data_capture_config = {
    "EnableCapture": True,
    "InitialSamplingPercentage": 100,  # capture 100% of traffic (adjust as needed)
    "DestinationS3Uri": s3_capture_location,
    "CaptureOptions": [
        {"CaptureMode": "Input"},
        {"CaptureMode": "Output"}
    ],
    # Optional: specify content types/encodings; here capture everything
    "CaptureContentTypeHeader": {
        "CsvContentTypes": ["text/csv"],
        "JsonContentTypes": ["application/json"]
    }
}

create_config_args = {
    "EndpointConfigName": new_config_name,
    "ProductionVariants": production_variants,
    "DataCaptureConfig": data_capture_config
}
if kms_key_id:
    create_config_args["KmsKeyId"] = kms_key_id

sm.create_endpoint_config(**create_config_args)

print(f"Created new endpoint config: {new_config_name}")

3. Update the Endpoint to use the new config​

sm.update_endpoint(
    EndpointName=endpoint_name,
    EndpointConfigName=new_config_name
)

# Optionally wait until update is complete
waiter = sm.get_waiter("endpoint_in_service")
waiter.wait(EndpointName=endpoint_name)
print("Endpoint updated and InService.")

4. Verify data capture is enabled​

updated_desc = sm.describe_endpoint(EndpointName=endpoint_name)
updated_config_name = updated_desc["EndpointConfigName"]
updated_config = sm.describe_endpoint_config(
    EndpointConfigName=updated_config_name
)

print(updated_config.get("DataCaptureConfig"))

You should now see EnableCapture: True and captured inference data appearing in the specified S3 prefix.

Using Terraform
resource "aws_sagemaker_endpoint_configuration" "NEW_ENDPOINT_CONFIG_WITH_DATA_CAPTURE" {
  name = "NEW_ENDPOINT_CONFIG_NAME" # replace with a unique endpoint config name

  production_variants {
    variant_name           = "PRIMARY_VARIANT"          # replace with your variant name
    model_name             = "EXISTING_MODEL_NAME"      # replace with your SageMaker model name
    initial_instance_count = 1                          # match your existing config
    instance_type          = "ml.m5.large"              # match your existing config
  }

  data_capture_config {
    enable_capture             = true
    initial_sampling_percentage = 100

    destination_s3_uri = "s3://YOUR_CAPTURE_BUCKET/YOUR_PREFIX" # replace with your S3 bucket/prefix for captures
    kms_key_id         = "YOUR_KMS_KEY_ID_OPTIONAL"             # optionally replace or remove if not using KMS

    capture_options {
      capture_mode = "Input"
    }

    capture_options {
      capture_mode = "Output"
    }
  }
}

resource "aws_sagemaker_endpoint" "EXISTING_ENDPOINT" {
  endpoint_name = "EXISTING_ENDPOINT_NAME" # replace with your existing endpoint name

  endpoint_config_name = aws_sagemaker_endpoint_configuration.NEW_ENDPOINT_CONFIG_WITH_DATA_CAPTURE.name

  # IMPORTANT:
  # - This will update the endpoint to use the new configuration with data capture enabled.
  # - Updating an endpoint may cause a brief service interruption for single-instance endpoints.
  # - Ensure the SageMaker execution role has s3:PutObject permissions on the capture S3 URI.
}

Updating the endpoint configuration reference forces SageMaker to deploy new instances for the endpoint (brief interruption for single-instance endpoints but no Terraform resource replacement of the endpoint itself).

For verification, terraform plan should show:

  • a new aws_sagemaker_endpoint_configuration created with data_capture_config enabled.
  • an in-place update of aws_sagemaker_endpoint changing endpoint_config_name to the new configuration.