Statement on Wasabi Service Degradation Over September / October 2019

Over the past two months, many of our customers have experienced connection problems with Wasabi. While we have tried to convey the current state of the service through our status page, we have not done a great job of explaining the root causes of the problems and what we are doing to fix them.

The number of simultaneous connections to Wasabi has been growing at almost 20% month over month. In early September, our service started to suffer under the increased load. In retrospect, we have not deployed sufficient hardware capacity to handle the load and we have had scaling issues with our connection manager software. 

We are taking the following steps to solve both these problems:

1. On the hardware capacity front, effective 15 October 2019, we have a new us-east service region (us-east-2) available for customer use.  This region has 22 PB of usable storage available for our customers. This new region is performing well. In addition, we are deploying more hardware in existing Wasabi regions to relieve some of the strain.

2. On the connection management front, at the beginning of September, we began the process of evolving and enhancing our connection management engine to better address the load.  While we are optimistic that these changes will resolve the connection management problems we have been experiencing, realistically these changes will take some time to test and deploy.  We are currently projecting these changes will be implemented by mid-November 2019. We are hoping to improve upon this timeframe as it is the top priority of our development team. However, we want to be transparent that the completion and implementation of these changes will not be immediate. These changes will be necessary before we can return to adequate connection capacity.

We want to emphasize that these throughput and connection management challenges in no way impact our data storage capacity, data durability, or data integrity. We are singularly focused on getting these issues solved and we are confident that the changes described above will help us get back to the high performance levels our customers have seen in the past. Thank you for your patience as we address our growing pains.