Dear Headless User, due an unplanned deploy, testing on Headless may be affected with intermittent downtimes, causing some tests to possibly fail. This outage is scheduled for about 3 hours maximum, between 7:00pm PDT - 10:00pm PDT on Thursday, August 8, 2019. Things are expected to be back to normal once the deploy has finished. We apologize for the inconvenience.
2021-January-7
Incident Report for Sauce Labs US East Data Center
Postmortem

Dates:

Thursday, January 07, 14:30 - 17:30 EST

What happened:

US-East-1 Sauce Connect tunnels were frequently restarted and for periods throughout the incident, new tunnels couldn’t be started.

Why it happened:

One of our backend services was not correctly configured, causing calls to the API to fail.

How we fixed it:

We updated the service that didn’t have the correct configuration to use the new API. 

What we are doing to prevent it from happening again:

We are implementing additional procedures around API changes and alerts to all the services that could affect the Sauce Connect tunnel subsystem. These include additional tunnel capacity monitoring and alerts, as well as additional monitoring of the service that caused an incident.

Posted Jan 18, 2021 - 05:23 EST

Resolved
Remedial action has been taken, Tunnels should function as expected in US-EAST-1
Posted Jan 07, 2021 - 19:51 EST
Update
We have taken remedial measures and have identified the issue. We are seeing an improvement on tunnels booting up and are working towards fixing the affected areas.
Posted Jan 07, 2021 - 18:31 EST
Monitoring
Changes deployed to US-EAST-1, now monitoring.
Posted Jan 07, 2021 - 17:07 EST
Update
Tunnels will still fail to start on US-EAST-1, we are still taking remedial action.
Posted Jan 07, 2021 - 16:37 EST
Identified
Sauce Connect Tunnels in US-EAST-1 are failing to start. Existing tunnels may close or become unresponsive. We are taking remedial action.
Posted Jan 07, 2021 - 15:38 EST
This incident affected: Sauce Connect.