Previous incidents

We have identified the root cause of the incident as coming from a large burst of queries by a small number of Qualcomm users. The query burst has subsided and the application has recovered.

We are actively working to improve our rate limiting logic to protect the availability of the service in these cases. Additionally, we are continuing to monitor usage and application health to prevent the issue from recurring.

Continued monitoring has confirmed application health is fully restored.

The system has fully recovered. Queries are no longer timing out, and TTFT and latency have returned to baseline levels.

Our team is continuing to analyze the root cause of this incident and will inform of relevant corrective actions as a result of the RCA.

The issue has been mitigated. Availability is restored and performance is at normal levels.

Application performance has returned to normal. We are continuing to monitor the health of the system.

We are no longer observing higher than usual timeout rates. Application performance has returned to normal after some remediation actions.

We are continuing to monitor application performance and investigate the root cause of this performance degradation.