2020-November-03 Resolved Service Incident
Postmortem

Dates:

Tuesday November 3 2:33 AM ET - 3:02 PM ET

What happened:

Customers using our US East Data Center (Headless Service) were not able to run tests, login to our application nor start any tunnels.

Why it happened:

Conflicting configuration changes were deployed in sequence resulting in some of the services in our cluster to fail.

How we fixed it:

Latest deployment was rolled back and affected services were able to start again. 

What we are doing to prevent it from happening again:

We will be separating the configuration changes according to their scope into multiple independent deployments.
This will prevent overlapping changes to be pushed simultaneously.

Posted Nov 09, 2020 - 11:19 EST

Resolved
Today between 2:39pm and 3:04pm EST, Headless UI was slow to load, automated tests and tunnels failed to start. We quickly identified and fixed the problem. All services are fully operational.
Posted Nov 03, 2020 - 15:13 EST