Delivery Tracker Unavailable - beta.routific.com impacted
Incident Report for Routific
Postmortem

On June 21st, 2022 at 4:53pm PST, while performing an upgrade to our infrastructure on our Beta site, we experienced a total data loss of that Beta site data.  This impacted some of our Beta products like our Read Only View of projects, and the Delivery Tracker feature.  Service was restored by 6pm PST.  While we were not able to restore all historical data, by approximately 7:30pm PST, we were able to restore customer accounts, and in-flight projects with associated Delivery Tracker links.  As a "Beta" product/site, we had not applied all of our usual best practices to mitigate such an occurrence, but have since remedied that with our usual rigorous backup regimes, monitoring, and automation.  Please be assured that in the future, any "Beta" products that we release, to any single customer, will be treated with our standard best practices to avoid a similar incident.

Posted Jul 06, 2022 - 13:07 PDT

Resolved
This incident was resolved on June 21st, 2022.
Posted Jun 21, 2022 - 23:00 PDT
Monitoring
A fix has been implemented and we are monitoring the situation. Synced V2 projects and delivery tracker are working now.
Posted Jun 21, 2022 - 21:17 PDT
Identified
We have identified the root cause of the outage. We are working on a fix.
Posted Jun 21, 2022 - 18:35 PDT
Investigating
We are experiencing an outage in Beta Routific V3 and are currently looking into the issue. This is affecting synced V2 projects and the delivery tracker.
Posted Jun 21, 2022 - 17:19 PDT
This incident affected: Routific API, Driver Mobile App, and Beta Routific.