Incidents | Contextual AI Incidents reported on status page for Contextual AI https://status.dragon.contextual.ai/ https://d1lppblt9t2x15.cloudfront.net/logos/7cf93c2e42c0011bf0f82227a9abb24d.png Incidents | Contextual AI https://status.dragon.contextual.ai/ en Dragon recovered https://status.dragon.contextual.ai/ Tue, 02 Sep 2025 20:13:25 +0000 https://status.dragon.contextual.ai/#08384eedd240e77e4cc2cd88bdfa0e9c05554022425be691725be030ca125c13 Dragon recovered Dragon went down https://status.dragon.contextual.ai/ Tue, 02 Sep 2025 20:02:23 +0000 https://status.dragon.contextual.ai/#08384eedd240e77e4cc2cd88bdfa0e9c05554022425be691725be030ca125c13 Dragon went down 🚨 Agents Outage https://status.dragon.contextual.ai/incident/705847 Thu, 14 Aug 2025 05:03:00 -0000 https://status.dragon.contextual.ai/incident/705847#c753369cfbd5833b51e330ad0fe2b557b14c3cdf0c1e3b1335ec16657ed51336 The issue has been fully resolved. All systems are now operating normally. Dragon recovered https://status.dragon.contextual.ai/ Thu, 14 Aug 2025 04:59:13 +0000 https://status.dragon.contextual.ai/#caf5750db209da65dcf4c40ca43b4bc7d56dbf2a418038c2cf60bee629557e4b Dragon recovered 🚨 Agents Outage https://status.dragon.contextual.ai/incident/705847 Thu, 14 Aug 2025 04:10:00 -0000 https://status.dragon.contextual.ai/incident/705847#cab26c978d14128f49ecb9824b5b89486f0d845575e53ee980302b6c6f777fb1 We are currently experiencing an unexpected service outage impacting all agents. Our engineering team has identified the root cause and working to restore normal operations as quickly as possible. Dragon went down https://status.dragon.contextual.ai/ Thu, 14 Aug 2025 04:01:31 +0000 https://status.dragon.contextual.ai/#caf5750db209da65dcf4c40ca43b4bc7d56dbf2a418038c2cf60bee629557e4b Dragon went down Degraded service on model https://status.dragon.contextual.ai/incident/703089 Sun, 03 Aug 2025 12:45:00 -0000 https://status.dragon.contextual.ai/incident/703089#62746e50efd7d14c0ae274aaa9dc6e7847e40d5590b7340fcbedbdd72cb2d761 We have migrated to a stable model on Aug 4th. The issue should have been fully resolved. Degraded service on model https://status.dragon.contextual.ai/incident/703089 Sun, 03 Aug 2025 09:27:00 -0000 https://status.dragon.contextual.ai/incident/703089#93ae3a2c9c229cdb21b7ca0b295de2e4ac7655bad15b79d5086064316f42df63 One model used for attribution and generation was unstable on Aug 3rd, from 2:27am to 5:45am. Queries to certain agents might have taken longer or timed out during this period. Degraded performance https://status.dragon.contextual.ai/incident/601892 Thu, 12 Jun 2025 21:02:00 -0000 https://status.dragon.contextual.ai/incident/601892#5bafa6e111e53d5003fed1531b6fd91e04152941e47265a72b75e9affe183c81 We are observing most GCP services recovered from the outage and our services are back to full availability. Our team is continuing to monitor. Degraded performance https://status.dragon.contextual.ai/incident/601892 Thu, 12 Jun 2025 18:45:00 -0000 https://status.dragon.contextual.ai/incident/601892#4f7e60163668ef17f77805fba81df9e32d6b8bc0bd3c7d7fe4c6c0c208275987 We are currently experiencing degraded performance that may be related to a Google Cloud Platform service disruption. Our team is actively investigating the issue. Elevated Error Rates on Agents https://status.dragon.contextual.ai/incident/571708 Mon, 19 May 2025 21:02:00 -0000 https://status.dragon.contextual.ai/incident/571708#6303fcb371a12f8bcc5e8ea1f9a488f15c894458c4b369c600f826adec40b0e2 The fix for the server-side errors of the TCE agent has been successfully deployed and server-side error rates on the TCE agent have returned back to normal. Contextual's team is continuing to monitor system performance and will follow up with a full root cause analysis and corrective actions. Elevated Error Rates on Agents https://status.dragon.contextual.ai/incident/571708 Mon, 19 May 2025 19:25:00 -0000 https://status.dragon.contextual.ai/incident/571708#44da67f4898767e673bef6a0133a3cf6f193292f3aaec2121da2dd28d68a91e7 We have identified a root cause for the majority of failures on the TCE agent and are pushing out a fix. We have also confirmed that other agents are not affected at this time. Elevated Error Rates on Agents https://status.dragon.contextual.ai/incident/571708 Mon, 19 May 2025 18:36:00 -0000 https://status.dragon.contextual.ai/incident/571708#c58a92c1e37d63b8960314ec92bca082eb6d6ac2cf631f75400ea49f1bb88fcd We are observing elevated failure rates on /query requests, mainly affecting the Technical Customer Engineering agent. Our on-call team is investigating the root cause of these errors with high priority and will update as the investigation progresses. Maintenance resulting in downtime https://status.dragon.contextual.ai/incident/555323 Fri, 02 May 2025 05:42:00 -0000 https://status.dragon.contextual.ai/incident/555323#df9bbd15dd3bffe63b8309045d5d879ecf56dce3c5fc7766044de6d8f8423df8 At 9:31 PM PST, cluster maintenance on the Dragon cluster resulted in unexpected downtime and all agents being unavailable. The incident was mitigated by Contextual on-call engineers and all agents returned to proper function at 10:35 PM PST. Maintenance Resulting in Downtime https://status.dragon.contextual.ai/incident/543358 Fri, 11 Apr 2025 00:05:00 -0000 https://status.dragon.contextual.ai/incident/543358#7bf83eefe12065947f65f669dc9c32d3c6f8d2346041da09b4bff56c8fb1833b Incident has been mitigated. Maintenance Resulting in Downtime https://status.dragon.contextual.ai/incident/543358 Thu, 10 Apr 2025 23:43:00 -0000 https://status.dragon.contextual.ai/incident/543358#975243adc1dab6d80fdfa22e2e12c944dfe90ea7dea5623e1c7d21be7cc265ce The issue has been root-caused and mitigation is in progress. We will update here once the issue has been mitigated. Maintenance Resulting in Downtime https://status.dragon.contextual.ai/incident/543358 Thu, 10 Apr 2025 22:57:00 -0000 https://status.dragon.contextual.ai/incident/543358#2c6986b8eb84a1abfc08e401b7e0efae7e4cceb7992cfee8645935474fced059 Cluster maintenance on the Dragon cluster has resulted in unexpected downtime. All agents are currently unavailable. Our on-call engineers are taking mitigating action to resolve the issue. We will keep this page updated as progress is made. Degraded performance https://status.dragon.contextual.ai/incident/537050 Mon, 31 Mar 2025 01:27:00 -0000 https://status.dragon.contextual.ai/incident/537050#8b6354f8bdc6c39798852058c88e66a6a2c01eef8c54c2088694dd28862fd0eb Incident resolved. Degraded performance https://status.dragon.contextual.ai/incident/537050 Mon, 31 Mar 2025 01:11:00 -0000 https://status.dragon.contextual.ai/incident/537050#60877068a17172fd95a259f8304122543745776770f755feda203a1b39874e6c Incident has been mitigated but our on-call teams are continuing to monitor. Degraded performance https://status.dragon.contextual.ai/incident/537050 Mon, 31 Mar 2025 01:01:00 -0000 https://status.dragon.contextual.ai/incident/537050#ae714cb757883f3923e737bfaf4fa02852cdf5900b6f6fe3150c0e9f9ded6bf9 Degraded performance. On-call team is investigating. Frontend Authorization Unavailable https://status.dragon.contextual.ai/incident/534427 Tue, 25 Mar 2025 19:50:00 -0000 https://status.dragon.contextual.ai/incident/534427#5cde1bd7900666ccb18fe7cea30417bcccdc60845e9c9f983056dc972e4d4913 The incident has been mitigated by the Auth0 team and authentication is back to full availability. Frontend Authorization Unavailable https://status.dragon.contextual.ai/incident/534427 Tue, 25 Mar 2025 18:51:00 -0000 https://status.dragon.contextual.ai/incident/534427#fde52c6f23d5b53b83e0137e2d7a9c38f9b725c56eede20c200b7a7e2844232a We are observing most authentication attempts now succeeding. On-call is continuing to monitor the situation. Frontend Authorization Unavailable https://status.dragon.contextual.ai/incident/534427 Tue, 25 Mar 2025 18:25:00 -0000 https://status.dragon.contextual.ai/incident/534427#af5902a846473b0f6d9abbf24635703436d708181189177e0c082f2018780aca Due to upstream issues with Auth0, logging in to the Contextual AI frontend is currently unavailable. API-key based usage is not affected. Contextual on-call is actively investigating the issue and will report back with further updates. Unavailability across CE and QCL Agents https://status.dragon.contextual.ai/incident/522743 Tue, 04 Mar 2025 22:08:00 -0000 https://status.dragon.contextual.ai/incident/522743#917c31d73081ae30f3dd5dd5b0ea2df7edbbd34fee39dd8d8bf5905cd98fd0b5 The issue has been root-caused to degraded model infrastructure. Infrastructure has been reset and restored, and availability has been restored across all agents. We are continuing to monitor agent performance and investigate the full root cause to ensure no recurrences occur in the future. Unavailability across CE and QCL Agents https://status.dragon.contextual.ai/incident/522743 Tue, 04 Mar 2025 21:54:00 -0000 https://status.dragon.contextual.ai/incident/522743#324e1f93a5ae291af908ba94d19150f64976fa4aec068476e8a2e2ae68bad0d1 We are observing unavailability across the Customer Engineering and QC Linux agents. Our team is investigating and will update here as the issue is root caused and mitigated. Degraded Performance on All Applications https://status.dragon.contextual.ai/incident/494903 Mon, 13 Jan 2025 04:20:00 -0000 https://status.dragon.contextual.ai/incident/494903#aecc2394cf6418b08c02a30c17c95e0913cb24d478689b191d44c8816dc5cd03 Performance on all applications has returned to baseline levels after mitigating action by Contextual on-call engineers. On-call engineers are continuing to monitor application health to ensure no recurrences take place. Degraded Performance on All Applications https://status.dragon.contextual.ai/incident/494903 Mon, 13 Jan 2025 03:12:00 -0000 https://status.dragon.contextual.ai/incident/494903#6c36141ef3462a65df8a235fa96aa9c1fc053504374545af5a08c1a7cb8dce94 We are observing degraded Time to First Token (TTFT) and throughput (tokens / sec) on both the Technical Customer Engineering and QC Linux applications. Both applications are still available but response times are degraded. The root cause has been identified as a sudden spike in usage by Qualcomm users. Contextual on-call engineers are actively investigating and mitigating the situation. Timeouts on QC Linux and CE Applications https://status.dragon.contextual.ai/incident/477019 Fri, 13 Dec 2024 06:25:00 -0000 https://status.dragon.contextual.ai/incident/477019#314a03459cb72bef3f1f763ccdcfee9ffd49eab025e72a9a00f321f37886b0db We have identified the root cause of the incident as coming from a large burst of queries by a small number of Qualcomm users. The query burst has subsided and the application has recovered. We are actively working to improve our rate limiting logic to protect the availability of the service in these cases. Additionally, we are continuing to monitor usage and application health to prevent the issue from recurring. Timeouts on QC Linux and CE Applications https://status.dragon.contextual.ai/incident/477019 Fri, 13 Dec 2024 06:08:00 -0000 https://status.dragon.contextual.ai/incident/477019#16f6480f64802adf0e75d1e270587f8bee2e2f313e7ff098ca9e41902b5a26d3 We are observing timeouts occurring on the QC Linux and CE applications. We are investigating the issue and will update as our investigation progresses. Timeouts on QC Linux and CE application https://status.dragon.contextual.ai/incident/475088 Tue, 10 Dec 2024 04:51:00 -0000 https://status.dragon.contextual.ai/incident/475088#6f99e69f0bb65f4c857127a9b4865c5de278e8027c7984c31df1c19a29db0c9a Continued monitoring has confirmed application health is fully restored. Timeouts on QC Linux and CE application https://status.dragon.contextual.ai/incident/475088 Tue, 10 Dec 2024 03:40:00 -0000 https://status.dragon.contextual.ai/incident/475088#0c1c585a75df78d758eaf0282793090eba62f0cb036d3e67bfe4584a73063bc4 We have mitigated the incident and availability and performance is fully restored for all applications. A full Root Cause & Analysis (RCA) document will follow. Timeouts on QC Linux and CE application https://status.dragon.contextual.ai/incident/475088 Tue, 10 Dec 2024 03:04:00 -0000 https://status.dragon.contextual.ai/incident/475088#10ee8c54598a7687b67be200842ab3db2f24fef45c03f5140d71aa2f3a85df09 We have identified a root cause and deployed a mitigation. The Qualcomm applications are available again. We are continuing to monitor the health of the deployment. Timeouts on QC Linux and CE application https://status.dragon.contextual.ai/incident/475088 Tue, 10 Dec 2024 02:07:00 -0000 https://status.dragon.contextual.ai/incident/475088#a89c62f046476ea9f60b71e64dec747ea0816da8b5d8093719ea03793ff25e3f We are observing timeouts on both the QC-Linux and Customer Engineering applications. Our team is actively investigating the issue and will update as investigation progress is made. Timeouts on QC Linux and CE Application https://status.dragon.contextual.ai/incident/465794 Fri, 22 Nov 2024 06:14:00 -0000 https://status.dragon.contextual.ai/incident/465794#b9c45b7a4517e9351664b8f5aedc80935be55302e726063a2403c5d4e392db70 The system has fully recovered. Queries are no longer timing out, and TTFT and latency have returned to baseline levels. Our team is continuing to analyze the root cause of this incident and will inform of relevant corrective actions as a result of the RCA. Timeouts on QC Linux and CE Application https://status.dragon.contextual.ai/incident/465794 Fri, 22 Nov 2024 06:09:00 -0000 https://status.dragon.contextual.ai/incident/465794#ac6066e843bdc24d01c7a7b9f2b222038ffc2c05ed5e6473fdc4e437cc2bc889 We have identified a root cause and our team has taken mitigating action. The system is recovering and queries are generating responses again. Performance may be degraded for the next several minutes as the system continues to recover. Our team is continuing to monitor the system to ensure stability and performance. Timeouts on QC Linux and CE Application https://status.dragon.contextual.ai/incident/465794 Fri, 22 Nov 2024 06:01:00 -0000 https://status.dragon.contextual.ai/incident/465794#af071343705c3004b912313881c271377a82c89d770a89d8350611eeeaaa44a2 We are observing timeouts on the Qualcomm QC Linux 1.2 and and Technical Customer Engineering applications. Our team is actively investigating. Maintenance Causing Unexpected Unavailability https://status.dragon.contextual.ai/incident/445296 Wed, 16 Oct 2024 01:55:00 -0000 https://status.dragon.contextual.ai/incident/445296#570c5efcf23452e76a0eedcccb0fe100e4e4449b33fad76b214f1b12b511d953 The issue has been mitigated. Availability is restored and performance is at normal levels. Maintenance Causing Unexpected Unavailability https://status.dragon.contextual.ai/incident/445296 Wed, 16 Oct 2024 00:47:00 -0000 https://status.dragon.contextual.ai/incident/445296#29d869b70daaf505afc16af29914c127be53cc8ab85277fbc12002e1d0310301 We are performing routine cluster maintenance on the Qualcomm application. This has resulted in unexpected downtime.We are taking mitigating action to resolve the issue. Degraded Performance https://status.dragon.contextual.ai/incident/444455 Mon, 14 Oct 2024 07:25:00 -0000 https://status.dragon.contextual.ai/incident/444455#36073f951e965087376cffd2b4d8b1e8a5217b3b5f2414998d3c91af5b382201 Application performance has returned to normal. We are continuing to monitor the health of the system. Degraded Query Performance https://status.dragon.contextual.ai/incident/442632 Thu, 10 Oct 2024 23:07:00 -0000 https://status.dragon.contextual.ai/incident/442632#f724e92924967a248465b8df2bc5da026b35dc007f3a4fa4d44dda05650ed51a We are no longer observing higher than usual timeout rates. Application performance has returned to normal after some remediation actions. We are continuing to monitor application performance and investigate the root cause of this performance degradation. Degraded Query Performance https://status.dragon.contextual.ai/incident/442632 Thu, 10 Oct 2024 23:00:00 -0000 https://status.dragon.contextual.ai/incident/442632#a4487d1afd96eae3a9dffbbd57cb81591cea8d90b4d5bbbb1be93e0187bf5217 We are observing higher timeout rates than usual on multiple applications. We are investigating the issue and will post updates here.