AWS EC2 Support
EC2 status check failed: find the cause and recover the instance.
EC2 runs two checks every minute. A failed system check points at the underlying host or network, while a failed instance check points at the operating system inside. The fix is different for each, so we identify which one failed before touching anything.
When You Need This
The situations where this work pays for itself.
Signs it is time
- The console shows 2/2 checks passed turning into 1/2 or 0/2
- An instance is running but unreachable by SSH, RDP and the application
- A CloudWatch alarm on StatusCheckFailed fired overnight
- An instance hangs after a kernel update, fstab change or disk full event
- You want instances to heal themselves without a pager call
What you get
- Root cause write-up for the failed check
- Recovered instance with data and addressing verified
- CloudWatch alarms with recover or reboot actions
- A short runbook for the next failure
How we do it
A clear, safe process you can follow along with.
1. Tell which check failed
In the console Status checks tab, or with aws ec2 describe-instance-status, we separate the system status check (AWS host, power, network) from the instance status check (OS boot, kernel, networking config). CloudWatch has StatusCheckFailed_System and StatusCheckFailed_Instance as separate metrics.
2. Handle a system check failure
These are infrastructure problems. We check the AWS Health Dashboard, then stop and start the instance (not just reboot) so it lands on a healthy host. Instance store data is lost on stop, and a public IP that is not an Elastic IP changes, so we confirm both first.
3. Read the console output and screenshot
For an instance check failure we pull get-console-output and the console screenshot. They usually show a kernel panic, a filesystem check waiting for input, an fstab entry for a missing volume, or Windows stuck in boot repair.
4. Try a reboot, then use the serial console
A reboot fixes hangs and stuck services. If the OS never reaches networking, we use the EC2 Serial Console on supported Nitro instance types to reach a login or GRUB prompt, where we can fix fstab, disable a bad service or roll back a kernel.
5. Repair the volume from a rescue instance
If the serial console is not available, we stop the instance, detach the root EBS volume, attach it as a secondary volume to a healthy instance in the same Availability Zone, fix the files, then reattach it under the original device name.
6. Set up automatic recovery
We create CloudWatch alarms on StatusCheckFailed_System with the EC2 recover action, and on StatusCheckFailed_Instance with a reboot action or a notification. Instances in Auto Scaling groups are better replaced through ELB health checks instead.
7. Verify and prevent the repeat
We confirm both checks return to passed, review the logs for the root cause, and fix the trigger, such as an unsafe fstab option (nofail), full disks, or unattended kernel updates.
Mistakes we help you avoid
Common ways this goes wrong, and what we do instead.
Rebooting for a system check failure
A reboot keeps the instance on the same impaired host. A stop and start moves it.
Stopping an instance that has instance store data
Instance store volumes are wiped on stop. We confirm what lives on them first.
Mounting data volumes in fstab without nofail
A missing or slow volume then blocks boot and fails the instance check. We add nofail and use UUIDs.
Forgetting the public IP will change
Without an Elastic IP, the address changes after a stop and start and DNS or allow lists break.
Frequently Asked Questions
What is the difference between system and instance status checks?
The system check tests the AWS host and network that the instance runs on. The instance check tests the guest operating system, such as whether it boots and answers network probes. AWS fixes the first, you or we fix the second.
Will EC2 recover a failed instance on its own?
Only if you configure it. A CloudWatch alarm on StatusCheckFailed_System with the recover action restarts the instance on new hardware and keeps its instance ID, private IP and Elastic IP.
Does a failed status check mean data loss?
Usually not for EBS-backed instances, since the volumes persist. Data on instance store volumes is lost when the instance is stopped or the host fails, so we check that before acting.
Want Help With EC2 status check failed: find the cause and recover the instance?
Schedule an appointment. Tell us about your environment and we will outline the safest, highest-value approach.
Schedule Appointment › No commitment required