2020-November-25 Service Incident
Postmortem

Dates:

Wednesday, November 25th 05:50 - 08:00 EST

What happened:

Customers were unable to reach REST API endpoints and access the application UI. 

Why it happened:

All requests were mistakenly marked as blocked by our Web Application Firewall (WAF).

How we fixed it:

We disabled and then redeployed our Web Application Firewall configuration.

What we are doing to prevent it from happening again:

We’re introducing automated monitoring and alerting for WAF. Additional alerting and monitoring has also been put in place to alert immediately if and when all requests receive a 406 in the future.

Posted Dec 22, 2020 - 11:38 EST

Resolved
This incident has been resolved.
Posted Nov 25, 2020 - 08:31 EST
Monitoring
We have taken remedial action and our service is recovering. We are monitoring.
Posted Nov 25, 2020 - 08:09 EST
Investigating
We are experiencing a service incident. The UI is unavailable and returns an HTTP 406 error. Our REST API is also affected. Tests are running fine. We are investigating.
Posted Nov 25, 2020 - 07:16 EST