New Routific outage

Incident Report for Routific

Postmortem

On January 22, 2025 at approximately 10:16 AM PST, Routific experienced an outage that prevented users from accessing the new Routific. The disruption lasted around 45 minutes before full service was restored. Our investigation determined that a bug in our internal messaging system caused a backlog of unacknowledged messages, which overwhelmed the system and led to the outage.
Our engineering team quickly identified the root cause and deployed a fix to clear the message backlog and stabilize the platform. We have also improved our monitoring and alerting systems to detect similar issues sooner and avoid future disruptions.
Thank you for your patience as we worked to resolve this matter.

Posted Jan 24, 2025 - 09:01 PST

Resolved

The new Routific continues to be stable and fully operational. Our team has identified the root cause of the issue. A bug in our internal messaging system caused an overload, resulting in about 45 minutes of downtime. We have since fixed the issue and will improve our monitoring systems to identify these types of issues earlier to avoid future disruptions.

Posted Jan 22, 2025 - 19:07 PST

Monitoring

The new Routific is stable and operational. Our team will continue monitoring and provide a report on the cause of the issue soon.

Posted Jan 22, 2025 - 11:22 PST

Update

Access to the new Routific has been restored. Thank you for your patience!

We are still investigating the cause of the outage. We will report our findings as soon as possible.

Posted Jan 22, 2025 - 11:19 PST

Update

We are actively working to restore user access to the new Routific. This issue is our team's highest priority. We thank you for your patience as we work to resolve this as soon as possible. The next update will come within 30 minutes.

Posted Jan 22, 2025 - 11:11 PST

Investigating

We are actively working to restore user access to the new Routific. The issue is affecting most users and our engineering team is working on a fix at the moment.
More updates will be provided in the next 30 minutes.

Posted Jan 22, 2025 - 10:37 PST

This incident affected: New Routific.