2020-November-25 Service Incident

Incident Report for ZZ [Archive] Sauce Labs US East Data Center

Postmortem

Dates:

Wednesday, November 25th 05:50 - 08:00 EST

What happened:

Customers were unable to reach REST API endpoints and access the application UI.

Why it happened:

All requests were mistakenly marked as blocked by our Web Application Firewall (WAF).

How we fixed it:

We disabled and then redeployed our Web Application Firewall configuration.

What we are doing to prevent it from happening again:

We’re introducing automated monitoring and alerting for WAF. Additional alerting and monitoring has also been put in place to alert immediately if and when all requests receive a 406 in the future.

Posted Dec 22, 2020 - 11:38 EST

Resolved

This incident has been resolved.

Posted Nov 25, 2020 - 08:31 EST

Monitoring

We have taken remedial action and our service is recovering. We are monitoring.

Posted Nov 25, 2020 - 08:09 EST

Investigating

We are experiencing a service incident. The UI is unavailable and returns an HTTP 406 error. Our REST API is also affected. Tests are running fine. We are investigating.

Posted Nov 25, 2020 - 07:16 EST

Dates:

What happened:

Why it happened:

How we fixed it:

What we are doing to prevent it from happening again:

Subscribe to Incident