RReq failures from all schemes

Incident Report for 3dsecure.io

Postmortem

Introduction

As a follow-up to our incident on 2024-06-19, this is our findings from analyzing the sequence of events. Again, we sincerely apologize for the disruption, downtime and inconvenience.

Architecture of 3dsecure.io

Our platform is composed of a number of services. Each service with its own responsibility. One of the services is responsible for handling callbacks from ACSs. We use AWS Fargate to host the services in a server-less environment and thus do not control the underlying hosts. Every once in a while AWS patches these hosts and they do so in a manner where they bring up new tasks of our services and take down the old ones when the new ones are healthy.

The services run in multiple availability zones (AZs) and usually, when AWS takes down a task, they bring up a new one in the same AZ. However, they do not make any guarantees, and the algorithm for balancing the new tasks is best-effort according to AWS. This plays a key part in the incident.

The issue

When the incident started, we were aware of issues from the ACSs, however, since that is not an uncommon occurrence, we usually just monitor such an issue and wait for it to resolve itself.

One of our partners reached out to us with concerns and we started to mitigate and investigate. We launched a new, separate environment for production, shifted all traffic to it, and left the old one to be investigated later. Shifting traffic to the new environment immediately solved the problem.

Investigation

On 2024-06-19, a patch of the underlying hosts had been scheduled and we were aware of it—as it has many times before.

We found that upon starting the new tasks, no new tasks were started in AZ B. Instead those intended for AZ B were started in AZ A and AZ C. As a consequence, AZ B was unable to receive callbacks from the ACSs as there was no task in the load balancer target group to receive the request. Since we did not have cross AZ load balancing enabled, we lost a third of the incoming callbacks because one of the three load balancer targets would time out when a request was sent to it. As a result we would respond with an error when a request was sent to our /postauth endpoint.

Solution

AWS states that task balancing in AZs is determined by several parameters and does not respect a task placement strategy (which could place tasks evenly) meaning that we could end up with an empty AZ again (a bit of foreshadowing). Why did we not have cross AZ load balancing enabled already you might ask. We had kept cross-AZ load balancing disabled to keep the AZs as isolated as possible to make our services as resilient to AZ outages as possible. However, given the current situation, we have decided to enable cross AZ load balancing to avoid this issue.

As we were deploying our new solution on 2024-06-24 the issue occurred again independently of our deployment. We managed to mitigate this immediately such that it had little to no impact on our service.

Conclusion

Besides enabling cross AZ load balancing, we have added alarms for when the load balancer experiences unhealthy routing flows. With the alarms we are able to instantly detect similar cross-AZ issues in the future, although cross AZ load balancing should prevent these issues from affecting availability to customer-facing services.

Posted Aug 08, 2024 - 21:00 UTC

Resolved

We have identified the issue with our cloud service provider and have mitigated the issue successfully. Scheduled patching of a third party vendor component that we do not control failed to bring all our services online in the correct availability zones. This caused one of the IP addresses on an internal load balancer to time out. We are in contact with our provider to ensure that this issue does not occur again. We are exploring options for alerting.

The issue occurred approximately 2024-06-19T08:14:00Z - 2024-06-19T10:18:00Z.
Posted Jun 19, 2024 - 13:24 UTC

Monitoring

We have deployed a fix and it has mitigated the issue. We do not see errors anymore on new flows. We are investigating what caused the error.
Posted Jun 19, 2024 - 10:40 UTC

Investigating

We are seeing increased error rates for RReqs coming from Directory Servers from all schemes. We are investigating, and trying to mitigate.
You can expect the next update within 2 hours.
Posted Jun 19, 2024 - 10:14 UTC
This incident affected: 3DSv2 (3DSS).