Thursday, January 07, 14:30 - 17:30 EST

What happened:

US-East-1 Sauce Connect tunnels were frequently restarted and for periods throughout the incident, new tunnels couldn’t be started.

Why it happened:

One of our backend services was not correctly configured, causing calls to the API to fail.

How we fixed it:

We updated the service that didn’t have the correct configuration to use the new API. 

What we are doing to prevent it from happening again:

We are implementing additional procedures around API changes and alerts to all the services that could affect the Sauce Connect tunnel subsystem. These include additional tunnel capacity monitoring and alerts, as well as additional monitoring of the service that caused an incident.

Posted Jan 18, 2021 - 05:23 EST

Remedial action has been taken, Tunnels should function as expected in US-EAST-1
Posted Jan 07, 2021 - 19:51 EST
We have taken remedial measures and have identified the issue. We are seeing an improvement on tunnels booting up and are working towards fixing the affected areas.
Posted Jan 07, 2021 - 18:31 EST
Changes deployed to US-EAST-1, now monitoring.
Posted Jan 07, 2021 - 17:07 EST
Tunnels will still fail to start on US-EAST-1, we are still taking remedial action.
Posted Jan 07, 2021 - 16:37 EST
Sauce Connect Tunnels in US-EAST-1 are failing to start. Existing tunnels may close or become unresponsive. We are taking remedial action.
Posted Jan 07, 2021 - 15:38 EST