AWS Load Balancer Support
Target group health check failures: decode the reason and fix it.
When a target is marked unhealthy, the load balancer stops sending it traffic, and if all targets fail you get 503 errors. The target health reason code in the console tells you whether the problem is timing, response code, network or registration. We fix the cause behind it.
When You Need This
The situations where this work pays for itself.
Signs it is time
- Targets show Unhealthy or Draining in the target group
- A new deployment never becomes healthy
- ECS or EKS tasks are killed and replaced repeatedly
- Health checks pass from curl but fail from the load balancer
- All targets fail and users see 503
What you get
- Reason for each unhealthy target
- Corrected health check configuration and security group rules
- A purpose-built lightweight health endpoint
- Alarm on unhealthy host count
How we do it
A clear, safe process you can follow along with.
1. Read the reason code
In the target group Targets tab, each unhealthy target has a reason such as Target.Timeout, Target.ResponseCodeMismatch, Target.FailedHealthChecks, Target.NotInUse or Target.InvalidState. The description spells out the details, like the response code received.
2. Check the health check settings
We review protocol, port, path, success code matcher, interval, timeout and the healthy and unhealthy thresholds. The path must exist and return a success code without requiring login. A redirect to HTTPS returning 301 fails a matcher of 200.
3. Check the port
The health check port can be traffic-port or an override. With ECS dynamic port mapping or Kubernetes node ports, the registered port may differ from what the app listens on, which we verify on the instance or task.
4. Check the network path
The target security group must allow the health check port from the load balancer security group. Network ACLs and host firewalls must allow it too. Timeout with working curl from elsewhere almost always means a security group rule.
5. Test as the load balancer would
We call the health path from an instance in the same VPC with the same Host header and protocol, and check response time against the health check timeout. A slow database call in the health endpoint causes sporadic failures.
6. Fix timing for slow starts
We lengthen the health check interval or thresholds, enable slow start on the target group, or set the ECS health check grace period so targets are not killed during startup. We make the health endpoint light and honest about readiness.
7. Verify
We confirm targets reach healthy, simulate a failing instance to see traffic stop correctly, and alarm on UnHealthyHostCount.
Mistakes we help you avoid
Common ways this goes wrong, and what we do instead.
Health endpoint checks every dependency
A database blip takes every target out of service. Use shallow checks for liveness and deeper checks carefully.
Auth required on the health path
The load balancer gets 401 or 302 and marks the target unhealthy.
Timeout longer than interval allows
Settings must be consistent and realistic for the app.
Allowing a CIDR instead of the ALB security group
Load balancer addresses change, so reference the security group.
Frequently Asked Questions
Why are my ALB targets unhealthy?
The probe is failing on the port, path or response code, or is blocked by a security group. The reason code on the target tells you which.
What does Target.ResponseCodeMismatch mean?
The target answered, but with a status code outside the matcher for the target group, for example a 301, 401 or 500 when only 200 is accepted.
What does Target.Timeout mean?
The target did not answer the health check within the timeout, usually due to blocked network traffic, a wrong port or an overloaded application.
Want Help With Target group health check failures: decode the reason and fix it?
Schedule an appointment. Tell us about your environment and we will outline the safest, highest-value approach.
Schedule Appointment › No commitment required