Dear Headless User, due an unplanned deploy, testing on Headless may be affected with intermittent downtimes, causing some tests to possibly fail. This outage is scheduled for about 3 hours maximum, between 7:00pm PDT - 10:00pm PDT on Thursday, August 8, 2019. Things are expected to be back to normal once the deploy has finished. We apologize for the inconvenience.
2020-April-22 Service Incident
Incident Report for Sauce Labs US East Data Center
Postmortem

Dates:

Wednesday, April 22 9:28 - 16:57 EDT

What happened:

Customers using our US-EAST datacenter were unable to run Headless tests on Sauce Labs.

Why it happened:

We announced a Saturday, April 25th maintenance window to upgrade/update our US-EAST Headless Kubernetes cluster, to be preceded by a dry run on a staging cluster on Wednesday, April 22nd. On April 22nd, we accidentally triggered a production deployment, applying an update that required recreating our Kubernetes cluster.

How we fixed it:

After we recreated the cluster, we re-deployed all services and necessary configurations to make Headless available.

What we are doing to prevent it from happening again:

We have implemented new policies that require additional approvals for any change to the Kubernetes cluster infrastructure. This review includes the maintenance of a detailed changelog for both our staging and production clusters. We have revised the deployment procedure and added additional restrictions that, without exception, ensures any changes intended for production occur first in staging as a safety check before pushing to a live environment.

Posted May 07, 2020 - 05:12 EDT

Resolved
All our services have recovered successfully. All services are now fully operational.
Posted Apr 22, 2020 - 16:57 EDT
Update
We continue to work on restoring all services to full operational status.
Posted Apr 22, 2020 - 15:00 EDT
Update
We are continuing to work on restoring all services to full operational status.
Posted Apr 22, 2020 - 12:33 EDT
Identified
The issue has been identified and we are working on bringing services back online.
Posted Apr 22, 2020 - 10:41 EDT
Investigating
Automated testing, REST API, Sauce Connect tunnels and the Sauce Labs UI are unavailable in our Headless testing cloud. We are investigating this issue.
Posted Apr 22, 2020 - 09:28 EDT
This incident affected: Headless Testing, Sauce Connect, REST API, and Sauce UI.