Product API stability issues
Incident Report for Routific
Postmortem

At approximately 5:00am pst we discovered an issue with one of our services. At this point we weren't sure of the impact yet and but continued to investigate in order to resolve the issue as soon as possible.

We later discovered, around 7:00am pst that the issue we were investigating was affecting access to customer profiles. We tried a few actions that temporarily restored access but ultimately we did not know the root cause yet at this point and the feature soon reverted to its degraded status.

At 8:am pst we discovered our database was under a high load and determined this is what was causing the disruption to the Customer Profiles feature. Shortly after we reset our database and this resolved the load issue on the database as well as restoring access to customer profiles.

Our action items to help prevent such issues in the future was to increase our monitoring on our database load to be informed much sooner if any issues are occurring.

We are sorry for the impact this may have had on your business.

Posted Feb 15, 2024 - 09:36 PST

Resolved
We have isolated the cause of this issue which was caused by a long running task in our backend and resolved it. Customer profiles are again fully functional. Sorry for any inconvenience caused this morning.
Posted Feb 09, 2024 - 13:59 PST
Monitoring
We have taken some action to resolve this issue. Customer profiles are currently working again at the moment. We are monitoring the situation closely.
Posted Feb 09, 2024 - 08:34 PST
Update
The ongoing issue is currently affecting customer profiles. We are continuing our investigation to resolve this as soon as possible.
Posted Feb 09, 2024 - 07:17 PST
Investigating
We are currently experiencing some stability issues within our application. We are working to determine exactly what is affected at this time and apologize for any issues this may cause.
Posted Feb 09, 2024 - 06:23 PST
This incident affected: Routific API and Routific SaaS product.