Delivery tracker outage

Incident Report for Routific

Postmortem

On January 22, 2025 at approximately 10:16 AM PST, Routific experienced an outage that prevented users from sending and accessing delivery tracking links. The disruption lasted around 45 minutes before full service was restored. Our investigation determined that a bug in our internal messaging system caused a backlog of unacknowledged messages, which overwhelmed the system and led to the outage.
Our engineering team quickly identified the root cause and deployed a fix to clear the message backlog and stabilize the platform. We have also improved our monitoring and alerting systems to detect similar issues sooner and avoid future disruptions.
Thank you for your patience as we worked to resolve this matter.

Posted Jan 24, 2025 - 09:02 PST

Resolved

The delivery tracker continues to be stable and fully operational. Our team has identified the root cause of the issue. A bug in our internal messaging system caused an overload, resulting in about 45 minutes of downtime. We have since fixed the issue and will improve our monitoring systems to identify these types of issues earlier to avoid future disruptions.

Posted Jan 22, 2025 - 19:07 PST

Monitoring

Access to the delivery tracker has been restored. Thank you for your patience!

We are still investigating the cause of the outage. We will report our findings as soon as possible.

Posted Jan 22, 2025 - 11:32 PST

Investigating

We are currently experiencing an issue where delivery tracking links are not being sent to customers. For those who have already received a tracking link, it may not display accurate information about the status of the delivery.
Our engineering team is actively working to resolve the issue and restore full functionality as soon as possible. We appreciate your patience and will provide more updates within the next 30 minutes.

Posted Jan 22, 2025 - 11:15 PST

This incident affected: Routific Notifications.