Triage and Remediation
- Remediation
Remediation
Using Console
Using Console
Below are step‑by‑step console instructions to ensure you’re actually using AWS Secrets Manager (with KMS) rather than just KMS alone for secrets.
If using a customer-managed KMS key, ensure the role is also allowed to use it in the KMS key policy.Do this instead of:
1. Choose or Create a KMS Key for Secrets Manager
- Sign in to AWS Management Console.
- Go to IAM > left menu Encryption keys (KMS)
or directly open AWS Key Management Service (KMS). - In the left pane, select Customer managed keys.
- Either:
- Use an existing key: Note the Key ID or Alias you want Secrets Manager to use,
or - Create a new KMS key:
- Click Create key.
- Key type: Symmetric and Encrypt and decrypt.
- Give it an Alias (e.g.
alias/secrets-manager-key). - Configure key administrators and key usage permissions.
- Finish. Note the alias or key ID.
- Use an existing key: Note the Key ID or Alias you want Secrets Manager to use,
2. Create a Secret in AWS Secrets Manager
- In the console, go to AWS Secrets Manager.
- Click Store a new secret.
- Under Secret type, choose the appropriate option:
- Other type of secret for arbitrary key/value pairs, or
- A specific type like Credentials for RDS database, etc.
- In Key/value pairs, add your secret values (for example
username,password,apiKey, etc.). - Under Encryption key, choose:
- aws/secretsmanager (AWS-managed)
or - Your customer-managed KMS key (e.g.
alias/secrets-manager-key).
- aws/secretsmanager (AWS-managed)
- Click Next.
- Enter a Secret name (e.g.
prod/db/credentials). - (Optional) Add a Description and Tags.
- Click Next to pass rotation for now, or configure it (see step 3).
- Review and click Store.
3. (Recommended) Enable Automatic Rotation
- In Secrets Manager, click the secret you just created.
- Go to the Rotation tab.
- Click Edit rotation.
- Check Enable automatic rotation.
- Choose Create a new Lambda function or Use an existing Lambda function to rotate the secret.
- Set the Rotation interval (e.g. every 30 days).
- Save changes.
4. Update Applications to Use Secrets Manager Instead of KMS/Plaintext
Your misconfiguration is typically that apps are:- Hardcoding secrets, or
- Storing them encrypted with KMS directly (e.g. via
Encrypt/DecryptAPI) but not using Secrets Manager.
A. Grant the Application IAM Permissions
Assign an IAM role or user policy like:B. Update Code to Call Secrets Manager
Example patterns:- AWS SDK (pseudo-code):
- Reading a plaintext config file, or
- Calling KMS
Decryptdirectly on some encrypted blob.
5. Decommission Old Secret Storage
Once all consumers have been migrated to Secrets Manager:- Remove old secrets from:
- Plaintext config files / environment variables.
- SSM Parameter Store (if previously used for secrets in plaintext).
- Custom KMS-encrypted blobs in S3, etc.
- If you had separate KMS keys only used for that old pattern:
- Confirm they are no longer needed.
- Schedule key deletion in KMS (optional, and only after verifying everything works).
6. Verify Remediation
- In Secrets Manager, verify:
- All required secrets exist.
- They are encrypted with a KMS key.
- Rotation is configured (if applicable).
- In CloudTrail, you should see
GetSecretValuecalls from your application role. - Confirm your security tool / compliance rule for “Secrets Manager should be in use” is now passing.
Using CLI
Using CLI
Below are CLI-focused steps to get secrets into AWS Secrets Manager and away from direct KMS usage.Assumptions:
Capture the Optionally give it an alias:
If the value is in a file:
Use the returned value in your application (via SDK / environment init script, etc.), instead of directly from KMS or plaintext.
And optionally restrict general KMS access so applications are not expected to manage sensitive secrets using bare KMS calls.
If you tell me how your secret is currently stored (SSM, env var, KMS-encrypted file, etc.), I can give exact CLI commands tailored to that case.
- You currently have some secret (password/API key/etc.) stored outside Secrets Manager (maybe in plaintext, SSM Parameter, environment variables, or encrypted manually with KMS).
- You want to store it in AWS Secrets Manager and encrypt it with a KMS key.
1. Choose or create a KMS key for Secrets Manager
If you already have a suitable customer managed key (CMK), note its key-id/ARN and skip creation.Create a new KMS key:KeyId or Arn from the output, e.g.:2. Create the secret in Secrets Manager
If you already have the secret value available in plaintext locally (e.g. in a shell variable):secret.json could be:3. (Optional) Enable automatic rotation for the secret
You need a Lambda function that knows how to rotate this secret. Assume you already created a rotation Lambda with ARN in$ROTATION_LAMBDA_ARN.Enable rotation every 30 days:4. Update application to read from Secrets Manager (CLI test)
Confirm you can read the secret:5. Remove old KMS-based / plaintext storage
Once your app is confirmed to be using Secrets Manager:-
If you were storing the secret in an SSM parameter:
- If you stored ciphertext via KMS directly (e.g., in a file / config store), delete or overwrite that configuration securely.
- If you had environment variables or config files containing the secret, remove/rotate them.
6. Lock down IAM so secrets must live in Secrets Manager
Example: grant your app role permission to read the secret, not to use KMS directly for secret decryption.Inline policy example (modify as needed):If you tell me how your secret is currently stored (SSM, env var, KMS-encrypted file, etc.), I can give exact CLI commands tailored to that case.
Using Python
Using Python
To remediate “Secrets Manager should be in use” for secrets currently stored/managed via KMS, you essentially need to:
Remediation impact:
You must:
If you share what your current KMS usage pattern is (e.g., using
- Store the secret in AWS Secrets Manager (encrypted with KMS).
- Update applications to read from Secrets Manager instead of wherever they’re getting the secret now.
- Optionally delete/retire the old KMS-based secret storage.
1. Prerequisites
- Python 3.x
- boto3 installed:
- AWS credentials configured (via environment variables, shared credentials file, or IAM role).
- A KMS key you want Secrets Manager to use (can be the AWS-managed default or a customer-managed key).
2. Create a Secret in AWS Secrets Manager (using KMS)
Assume you currently have a secret value (e.g., password or API key) that might be stored in a config file, environment variable, or some other non–Secrets Manager location.- Secret is now stored in Secrets Manager, encrypted with KMS.
- Access is controlled via IAM policies and Secrets Manager resource policies.
3. Retrieve the Secret from Secrets Manager in Your App
Replace existing logic (e.g., reading from env vars, files, or direct KMS Decrypt) with Secrets Manager retrieval.4. (Optional) Enable Automatic Secret Rotation
If the secret is a database password or similar, use a Lambda rotation function. The basic API call:- Implement the Lambda function to actually change the secret in the target system (DB, API, etc.).
- Attach a role to the Lambda allowing it to read/write the secret and update the backend credential.
5. Decommission the Old KMS-Based Storage (If Applicable)
If your prior pattern was something like:- Encrypting secrets manually with KMS and storing ciphertext in files/parameters, or
- Storing plaintext secrets in config with occasional KMS use,
- Verify no application still reads the old data.
- Remove or redact old secrets (e.g., from SSM Parameter Store, config files, S3, etc.).
- Optionally schedule deletion of the KMS key if it was dedicated purely for that legacy pattern and is no longer needed:
6. IAM Permissions (High-level)
Ensure the application role or user has at least:- To read the secret:
secretsmanager:GetSecretValue
- To create/manage the secret (for your provisioning code):
secretsmanager:CreateSecretsecretsmanager:PutSecretValuesecretsmanager:UpdateSecretsecretsmanager:RotateSecret(if using rotation)
KmsKeyId:kms:Encryptkms:Decryptkms:GenerateDataKeykms:DescribeKey
If you share what your current KMS usage pattern is (e.g., using
kms:Decrypt on ciphertext from a file, SSM parameter, etc.), I can give a more exact “before/after” Python example for that specific migration.Using Terraform
Using Terraform
terraform plan should show 3 to add (aws_kms_key.APP_KMS_KEY, aws_secretsmanager_secret.APP_SECRET, aws_secretsmanager_secret_version.APP_SECRET_VALUE) and no changes to existing resources unless you wire this secret into them.
