2020-July-21 Service Incident
Postmortem

Dates:

Wednesday, July 21, 08:08 - 10:58 CDT 

What happened:

The SSL certificates required for serving test assets and a portion of the user interface had expired, making the dashboards in all data centers inaccessible.

Why it happened:

The renewed SSL certificates were not set properly.

How we fixed it:

We immediately re-configured the renewed certificates and re-loaded the affected services. 

What we are doing to prevent it from happening again:

We have increased the automation of our SSL certificate management and improved our monitoring.

Posted Aug 03, 2020 - 12:42 EDT

Resolved
This incident has been resolved.
Posted Jul 21, 2020 - 10:58 EDT
Identified
Automated Headless testing is now available and running normally. The Sauce Labs dashboard at https://app.us-east-1.saucelabs.com is still unavailable. We continue to work on a resolution.
Posted Jul 21, 2020 - 10:25 EDT
Update
Automated Headless tests are also intermittently affected, meaning some tests will fail to start. We continue to investigate.
Posted Jul 21, 2020 - 09:16 EDT
Investigating
The Sauce Labs dashboard at https://app.us-east-1.saucelabs.com is not responding. Automated testing, Sauce Connect tunnels and other services not requiring the UI are unaffected. We are investigating.
Posted Jul 21, 2020 - 08:39 EDT