Skip to main content

EC2 Instance Retirement Remediation

Triage and Remediation​

Remediation​

Using Console

Here’s how to handle (remediate) AWS EC2 Scheduled Events using the AWS Management Console.

1. Find EC2 instances with scheduled events​

  1. Sign in to the AWS Management Console.
  2. Go to EC2.
  3. In the left menu, select Events under Instances (or Scheduled events, depending on console version).
    • Or go to Instances, then use the filter:
      • Status → Instance status → check With scheduled events.

You’ll see which instances have scheduled events, and the event type, start time, and deadline.

2. Understand the event type and choose action​

Typical event types and their common remediations:

  1. Instance reboot
  2. System reboot
  3. Instance stop (or stop/start)
  4. Retirement (instance or underlying host)
  5. Maintenance (e.g., hardware, network, or instance store impact)

3. Remediate by event type (via Console)​

A. Instance/System Reboot​

If the event is a reboot and your workload tolerates a short interruption:

  1. From EC2 → Instances, select the affected instance.
  2. Choose Instance state → Reboot instance.
  3. Confirm the reboot.
  4. Verify:
    • After it comes back, check application health and logs.
    • This typically clears the scheduled reboot event.

If you cannot reboot immediately:

  • Plan a maintenance window before AWS’s scheduled time.
  • Then do the manual reboot as above.

B. Instance Stop / Stop-Start​

For events specifying the instance must be stopped or stopped/started:

  1. From EC2 → Instances, select the instance.
  2. Ensure you can tolerate downtime and the root volume is EBS (not instance store).
  3. Choose Instance state → Stop instance.
  4. Wait for state to become stopped.
  5. If event calls for stop/start, then:
    • Choose Instance state → Start instance.
  6. Verify application and that the scheduled event is cleared.

Note:

  • If your instance uses an ephemeral (instance store) root volume, data is lost on stop. Back up before stopping (AMI, snapshot, data copy).

C. Instance Retirement​

If the event says the instance is scheduled for retirement:

You must migrate or replace the instance.

  1. Create an AMI (if using EBS root):
    • Select the instance → Actions → Image and templates → Create image.
    • Name the AMI and create it.
  2. Launch a new instance from that AMI:
    • Go to AMIs, select your new AMI → Launch instance.
    • Use a similar instance type, same VPC, subnet, security groups, IAM role, etc.
  3. Update dependencies:
    • Update DNS, load balancer target groups, or other components to point to the new instance.
  4. Once traffic and data are fully migrated:
    • Stop/terminate the old instance (according to your decommissioning policy).

You can also:

  • Change to a different instance type or AZ at this time to avoid similar hardware issues.

D. Scheduled Maintenance That Requires No Action​

Sometimes events are informational only (e.g., minor network maintenance) and require no action:

  • Confirm in the event description that no instance stop/reboot is needed.
  • Optionally plan a maintenance window to validate application behavior during/after the window.

4. (Optional) Request reschedule of events​

For some events, AWS allows you to request a different time.

  1. In EC2 → Events:
    • Select the instance and scheduled event.
  2. If available, choose Request reschedule or similar option.
  3. Pick a new time window within the allowed range and submit.

If the console doesn’t show a reschedule option:

  • You must complete the required action (reboot/stop/replace) before the AWS deadline.

5. Verify remediation​

After you act:

  1. Go back to EC2 → Events.
  2. Confirm the event is cleared or marked as completed.
  3. Validate:
    • Instance Status checks are 2/2 checks passed.
    • Application/services are operational.

This completes remediation of scheduled events for EC2 using the AWS console.

Using CLI

For EC2 “Scheduled Events” there isn’t a single magic “fix” command; remediation is to identify affected instances using the CLI and then take the required action (stop/start, reboot, or replace) before AWS performs the scheduled operation.

Below are step‑by‑step AWS CLI instructions.

1. List EC2 instances with scheduled events​

aws ec2 describe-instance-status \
  --include-all-instances \
  --query 'InstanceStatuses[?Events].{InstanceId:InstanceId,Events:Events}' \
  --output table

You’ll see event codes like:

  • instance-stop
  • instance-reboot
  • system-reboot
  • system-maintenance
  • instance-retirement
  • instance-retirement-scheduled

Note the InstanceId and event code(s) per instance.

2. Prioritize by event time​

To sort by the scheduled time:

aws ec2 describe-instance-status \
  --include-all-instances \
  --query 'InstanceStatuses[?Events].{InstanceId:InstanceId,Events:Events[*].{Code:Code,NotBefore:NotBefore}}' \
  --output json

Focus first on events with the nearest NotBefore time.

3. Decide remediation action per event type​

General rule:
Do the disruptive action yourself at a controlled time instead of waiting for AWS.

A. For instance-stop / instance-retirement​

  1. If the instance is in an Auto Scaling Group (ASG) or behind a load balancer, ensure draining / replacement is handled.
  2. Stop and start the instance to move it to new hardware:
INSTANCE_ID=i-0123456789abcdef0

aws ec2 stop-instances --instance-ids "$INSTANCE_ID"
aws ec2 wait instance-stopped --instance-ids "$INSTANCE_ID"

aws ec2 start-instances --instance-ids "$INSTANCE_ID"
aws ec2 wait instance-running --instance-ids "$INSTANCE_ID"
  1. If you must preserve the instance exactly but hardware migration isn’t sufficient, create a replacement instance: 
    # Create AMI
    aws ec2 create-image \
      --instance-id "$INSTANCE_ID" \
      --name "replacement-$(date +%Y%m%d-%H%M%S)" \
      --no-reboot
    Then launch a new instance from that AMI and update DNS/ELB/Target Groups/ASGs as appropriate.

B. For instance-reboot​

Reboot on your own schedule (often enough):

INSTANCE_ID=i-0123456789abcdef0

aws ec2 reboot-instances --instance-ids "$INSTANCE_ID"

Verify application health afterward.

C. For system-reboot / system-maintenance​

Sometimes a simple stop/start also clears the event by moving the instance to new hardware:

INSTANCE_ID=i-0123456789abcdef0

aws ec2 stop-instances --instance-ids "$INSTANCE_ID"
aws ec2 wait instance-stopped --instance-ids "$INSTANCE_ID"

aws ec2 start-instances --instance-ids "$INSTANCE_ID"
aws ec2 wait instance-running --instance-ids "$INSTANCE_ID"

If the event persists, you may have to accept the scheduled maintenance window or migrate workloads to a new instance and decommission the old one.

4. Confirm the event is cleared​

After you’ve taken action:

aws ec2 describe-instance-status \
  --instance-ids "$INSTANCE_ID" \
  --include-all-instances \
  --query 'InstanceStatuses[0].Events' \
  --output json

An empty or missing Events array indicates the scheduled event is cleared.

5. (Optional) Script to remediate all “stop/retirement” events​

Example bash snippet to stop/start all affected instances (use carefully):

# Get all instance IDs with instance-stop/instance-retirement events
INSTANCES=$(aws ec2 describe-instance-status \
  --include-all-instances \
  --query "InstanceStatuses[?Events[?Code=='instance-stop'||Code=='instance-retirement']].InstanceId" \
  --output text)

for ID in $INSTANCES; do
  echo "Remediating $ID..."
  aws ec2 stop-instances --instance-ids "$ID"
  aws ec2 wait instance-stopped --instance-ids "$ID"
  aws ec2 start-instances --instance-ids "$ID"
  aws ec2 wait instance-running --instance-ids "$ID"
done

If you tell me which event code(s) you’re seeing, I can give a narrower CLI sequence tailored to that exact case.

Using Python

Below is a concise, step‑by‑step approach to detect and remediate EC2 scheduled events (like system reboot/retirement) using Python (boto3).

1. Understand the “misconfiguration”​

For EC2, “Scheduled Events” are AWS‑initiated actions (reboot, stop, retire instances, etc.).
Remediation usually means:

  • Detecting upcoming scheduled events.
  • Taking controlled action yourself before AWS does (e.g., stop/start, replace instance, move workload).

2. Prerequisites​

  1. Install boto3: 
    pip install boto3
  2. Configure AWS credentials/region (e.g., using aws configure or environment variables).
  3. IAM role/user must have at least:
    • ec2:DescribeInstanceStatus
    • ec2:StopInstances
    • ec2:StartInstances
    • ec2:RebootInstances
    • ec2:TerminateInstances (only if you choose to terminate/replace)
    • ec2:DescribeInstances

3. Detect scheduled events via Python​

Use describe_instance_status with IncludeAllInstances=True and look at Events and EventCode.

import boto3

ec2 = boto3.client('ec2')

def get_instances_with_scheduled_events():
    instances_with_events = []

    paginator = ec2.get_paginator('describe_instance_status')
    page_iterator = paginator.paginate(IncludeAllInstances=True)

    for page in page_iterator:
        for status in page.get('InstanceStatuses', []):
            events = status.get('Events', [])
            if events:
                instances_with_events.append({
                    'InstanceId': status['InstanceId'],
                    'Events': events
                })

    return instances_with_events

if __name__ == "__main__":
    instances = get_instances_with_scheduled_events()
    for i in instances:
        print(i)

Look for EventCode values such as:

  • instance-reboot
  • system-reboot
  • instance-retirement
  • system-maintenance
  • instance-stop

4. Decide remediation logic​

Typical automated actions:

  • instance-reboot / system-reboot:
    → You may choose to proactively reboot during a maintenance window.
  • instance-stop:
    → Stop/start the instance on your controlled schedule (note: instance-store volumes are lost).
  • instance-retirement:
    → Replace the instance (create a new one from AMI/snapshot, attach EIP, update ASG, etc.).

Below is a simple example that:

  • Detects events.
  • If instance-stop or instance-retirement is scheduled, it stops the instance (to force change under your control) and then starts it back.

5. Example remediation script (simple)​

import boto3
from botocore.exceptions import ClientError

ec2 = boto3.client('ec2')

ACTIONABLE_EVENT_CODES = {
    'instance-stop',
    'instance-retirement',
    'system-reboot',
    'instance-reboot'
}

def get_instances_with_actionable_events():
    affected = []
    paginator = ec2.get_paginator('describe_instance_status')
    page_iterator = paginator.paginate(IncludeAllInstances=True)

    for page in page_iterator:
        for status in page.get('InstanceStatuses', []):
            events = status.get('Events', [])
            if not events:
                continue

            actionable_events = [
                e for e in events if e['Code'] in ACTIONABLE_EVENT_CODES
            ]
            if actionable_events:
                affected.append({
                    'InstanceId': status['InstanceId'],
                    'Events': actionable_events
                })
    return affected

def stop_instance(instance_id):
    print(f"Stopping instance {instance_id}")
    ec2.stop_instances(InstanceIds=[instance_id])
    waiter = ec2.get_waiter('instance_stopped')
    waiter.wait(InstanceIds=[instance_id])
    print(f"Instance {instance_id} stopped")

def start_instance(instance_id):
    print(f"Starting instance {instance_id}")
    ec2.start_instances(InstanceIds=[instance_id])
    waiter = ec2.get_waiter('instance_running')
    waiter.wait(InstanceIds=[instance_id])
    print(f"Instance {instance_id} running")

def remediate_instance(instance):
    instance_id = instance['InstanceId']
    events = instance['Events']

    # Simple strategy:
    # - For any actionable event: stop & start instance within your window
    # Adjust this per event type if needed.
    event_codes = {e['Code'] for e in events}
    print(f"Remediating {instance_id} for events: {event_codes}")

    # Example: for retire/stop/reboot, do controlled stop/start
    try:
        stop_instance(instance_id)
        start_instance(instance_id)
    except ClientError as e:
        print(f"Error remediating {instance_id}: {e}")

def main():
    affected_instances = get_instances_with_actionable_events()
    if not affected_instances:
        print("No instances with actionable scheduled events.")
        return

    for instance in affected_instances:
        remediate_instance(instance)

if __name__ == "__main__":
    main()

6. Hardening as an operational process​

  • Run this script as a scheduled job (e.g., cron, Lambda with CloudWatch Events).
  • Optionally:
    • Filter by tags (DescribeInstances) so you only remediate certain workloads.
    • Integrate notifications (SNS/Slack) before/after remediation.
    • Use different logic per event type (e.g., rebuild instance for instance-retirement).

If you tell me your preferred remediation behavior (e.g., “for instance-retirement, create a new instance from this AMI and move EIP over”), I can give a more specific Python workflow.

Using Terraform
# There is no Terraform argument or resource that can prevent or clear
# AWS EC2 scheduled events; these are initiated by AWS infrastructure.

# Terraform can only manage the instance configuration itself, e.g.:
resource "aws_instance" "example" {
  ami           = "AMI_ID"          # replace with your AMI ID
  instance_type = "INSTANCE_TYPE"   # e.g. "t3.micro"

  tags = {
    Name = "EXAMPLE_NAME"
  }
}

EC2 scheduled events (instance reboot, stop, retirement, hardware maintenance) are operational actions initiated by AWS and cannot be remediated or acknowledged via Terraform; the provider exposes no argument for this.

To remediate:

  • Use the AWS Console: EC2 → Instances → select instance → “Scheduled events” tab → follow the guidance; for stop/retire events, stop/start or replace the instance (possibly via an Auto Scaling Group or launch template managed in Terraform).
  • Or use AWS CLI: for example, stop/start or replace the instance (aws ec2 stop-instances, start-instances, or create a new instance and update any Terraform-managed references to its ID or to an Auto Scaling Group that replaces it).

Verification with Terraform: terraform plan will show no changes related to scheduled events because they are not part of the Terraform state or schema.