Wednesday, April 22 9:28 - 16:57 EDT
Customers using our US-EAST datacenter were unable to run Headless tests on Sauce Labs.
We announced a Saturday, April 25th maintenance window to upgrade/update our US-EAST Headless Kubernetes cluster, to be preceded by a dry run on a staging cluster on Wednesday, April 22nd. On April 22nd, we accidentally triggered a production deployment, applying an update that required recreating our Kubernetes cluster.
After we recreated the cluster, we re-deployed all services and necessary configurations to make Headless available.
We have implemented new policies that require additional approvals for any change to the Kubernetes cluster infrastructure. This review includes the maintenance of a detailed changelog for both our staging and production clusters. We have revised the deployment procedure and added additional restrictions that, without exception, ensures any changes intended for production occur first in staging as a safety check before pushing to a live environment.