2020-April-22 Service Incident


Wednesday, April 22 9:28 - 16:57 EDT

What happened:

Customers using our US-EAST datacenter were unable to run Headless tests on Sauce Labs.

Why it happened:

We announced a Saturday, April 25th maintenance window to upgrade/update our US-EAST Headless Kubernetes cluster, to be preceded by a dry run on a staging cluster on Wednesday, April 22nd. On April 22nd, we accidentally triggered a production deployment, applying an update that required recreating our Kubernetes cluster.

How we fixed it:

After we recreated the cluster, we re-deployed all services and necessary configurations to make Headless available.

What we are doing to prevent it from happening again:

We have implemented new policies that require additional approvals for any change to the Kubernetes cluster infrastructure. This review includes the maintenance of a detailed changelog for both our staging and production clusters. We have revised the deployment procedure and added additional restrictions that, without exception, ensures any changes intended for production occur first in staging as a safety check before pushing to a live environment.

Posted May 07, 2020 - 05:12 EDT

All our services have recovered successfully. All services are now fully operational.
Posted Apr 22, 2020 - 16:57 EDT
We continue to work on restoring all services to full operational status.
Posted Apr 22, 2020 - 15:00 EDT
We are continuing to work on restoring all services to full operational status.
Posted Apr 22, 2020 - 12:33 EDT
The issue has been identified and we are working on bringing services back online.
Posted Apr 22, 2020 - 10:41 EDT
Automated testing, REST API, Sauce Connect tunnels and the Sauce Labs UI are unavailable in our Headless testing cloud. We are investigating this issue.
Posted Apr 22, 2020 - 09:28 EDT