Customer Notifications unavailable
Incident Report for Routific
Postmortem

Postmortem

On March 10th around 23:50 PST, Routific’s Customer Notification service became unavailable. It affected and delayed all customer notifications sent until March 13th 10:00 PST.

We're so sorry for all the stress that we've caused you, as we fully appreciate how important Routific's availability is to our customers.

What happened?

Our infrastructure that handles sending customer notifications went to a critical state on March 10th, around 23:50 PST.

During the downtime, we were still able to queue all notifications, which were sent with a delay once the issue was resolved.

What are we doing about it?

Our team has been committed to improving the reliability and stability of our systems. We are working with our service provider in order to better understand the issue and how we can prevent this incident from occurring in the future. We will look into better ways to improve infrastructure monitoring and set up better tooling and protocol to mitigate infrastructure instability should they arise.

Posted Mar 13, 2020 - 16:10 PDT

Resolved
We have confirmed that customer notifications are being processed normally for the past hour and that the issue has been resolved. A post-mortem with more details will be provided soon. We are sorry for the interruptions that this have caused your business.
Posted Mar 13, 2020 - 10:55 PDT
Monitoring
A fix has been implemented and we are monitoring the status of our services.
Posted Mar 13, 2020 - 10:02 PDT
Identified
We have identified the issue and a fix is being implemented.
Posted Mar 13, 2020 - 09:45 PDT
Investigating
We are currently investigating a issue with our customer notification feature not sending notifications properly.
Posted Mar 13, 2020 - 09:26 PDT
This incident affected: Routific SaaS product.