Services have remained stable since going back online this morning. We will continue to monitor over the weekend, and implement further infrastructure improvements to prevent this kind of downtime in the future.
Posted Aug 08, 2020 - 18:13 PDT
Monitoring
Service has been restored for notifications and spreadsheet upload. We are now monitoring both spreadsheet upload (geocoding) and our customer notifications.
Posted Aug 08, 2020 - 10:10 PDT
Update
Spreadsheet upload for stops has been restored. Changes to the infrastructure that should restore Customer Notifications will be deployed shortly.
Posted Aug 08, 2020 - 09:30 PDT
Identified
We have taken a number of steps already to attempt to remediate this issue, unfortunately they have not fully restored service. We are now making a change to our infrastructure which we believe will resolve this issue.
Posted Aug 08, 2020 - 09:11 PDT
Update
We are continuing to investigate the issue affecting spreadsheet upload and customer notifications. We have identified a possible solution, which will be pushed within the hour.
Posted Aug 08, 2020 - 08:31 PDT
Investigating
We are currently investigating an outage with RabbitMQ, which manages our geocoder events for spreadsheet-uploaded stops, as well as customer notification messages.