Routific API partially unavailable
Incident Report for Routific
Postmortem

On September 26, part of Routific’s functionality was unavailable between 5:24 a.m. - 7:39 a.m. Pacific Time. The cause has been identified and our systems are now stable.

What happened?

Our Redis caching system became unreachable, which subsequently prevented users from adding new stops and drivers into existing routes. We immediately spun up a new third-party Redis cache server to ensure future scalability and high availability.

What are we doing about it?

We are committed to improving the reliability and stability of our systems. For the next quarter, our team will be dedicated to this cause by auditing our entire stack to be fault-tolerant, enhancing our monitoring systems, and upgrading our systems for reliability, availability, and recoverability.

Posted Sep 26, 2018 - 17:35 PDT

Resolved
We've identified that it was due to a Redis caching instance being unavailable; we spun up a new cluster with a third party and the issue is resolved.
Posted Sep 26, 2018 - 07:39 PDT
Update
We are continuing to investigate this issue.
Posted Sep 26, 2018 - 07:16 PDT
Investigating
Routific API is partly unavailable intermittently down. This affects attempt to insert new stops and drivers into existing routes.
Posted Sep 26, 2018 - 07:15 PDT
This incident affected: Routific API and Routific SaaS product.