As a follow-up to our incident on 2024-06-19, this is our findings from analyzing the sequence of events. Again, we sincerely apologize for the disruption, downtime and inconvenience.
Our platform is composed of a number of services. Each service with its own responsibility. One of the services is responsible for handling callbacks from ACSs. We use AWS Fargate to host the services in a server-less environment and thus do not control the underlying hosts. Every once in a while AWS patches these hosts and they do so in a manner where they bring up new tasks of our services and take down the old ones when the new ones are healthy.
The services run in multiple availability zones (AZs) and usually, when AWS takes down a task, they bring up a new one in the same AZ. However, they do not make any guarantees, and the algorithm for balancing the new tasks is best-effort according to AWS. This plays a key part in the incident.
When the incident started, we were aware of issues from the ACSs, however, since that is not an uncommon occurrence, we usually just monitor such an issue and wait for it to resolve itself.
One of our partners reached out to us with concerns and we started to mitigate and investigate. We launched a new, separate environment for production, shifted all traffic to it, and left the old one to be investigated later. Shifting traffic to the new environment immediately solved the problem.
On 2024-06-19, a patch of the underlying hosts had been scheduled and we were aware of it—as it has many times before.
We found that upon starting the new tasks, no new tasks were started in AZ B. Instead those intended for AZ B were started in AZ A and AZ C. As a consequence, AZ B was unable to receive callbacks from the ACSs as there was no task in the load balancer target group to receive the request. Since we did not have cross AZ load balancing enabled, we lost a third of the incoming callbacks because one of the three load balancer targets would time out when a request was sent to it. As a result we would respond with an error when a request was sent to our /postauth endpoint.
AWS states that task balancing in AZs is determined by several parameters and does not respect a task placement strategy (which could place tasks evenly) meaning that we could end up with an empty AZ again (a bit of foreshadowing). Why did we not have cross AZ load balancing enabled already you might ask. We had kept cross-AZ load balancing disabled to keep the AZs as isolated as possible to make our services as resilient to AZ outages as possible. However, given the current situation, we have decided to enable cross AZ load balancing to avoid this issue.
As we were deploying our new solution on 2024-06-24 the issue occurred again independently of our deployment. We managed to mitigate this immediately such that it had little to no impact on our service.
Besides enabling cross AZ load balancing, we have added alarms for when the load balancer experiences unhealthy routing flows. With the alarms we are able to instantly detect similar cross-AZ issues in the future, although cross AZ load balancing should prevent these issues from affecting availability to customer-facing services.