Thursday, January 07, 14:30 - 17:30 EST
US-East-1 Sauce Connect tunnels were frequently restarted and for periods throughout the incident, new tunnels couldn’t be started.
One of our backend services was not correctly configured, causing calls to the API to fail.
We updated the service that didn’t have the correct configuration to use the new API.
We are implementing additional procedures around API changes and alerts to all the services that could affect the Sauce Connect tunnel subsystem. These include additional tunnel capacity monitoring and alerts, as well as additional monitoring of the service that caused an incident.