Cleaning Up Failed Multi-Part Uploads With Wasabi Hot Cloud Storage

When a file larger than 5 GB is uploaded, it does so in parts. The parts of a multi-part upload will range in size from 5 MB to 5 GB (the last part can be < 5 MB).

When you complete a multi-part upload, the Amazon S3 API used by Wasabi creates an object by concatenating the parts in ascending order based on the part number and the object committed. If the object is not committed, Wasabi keeps the parts for 31 days, so you can resume the upload of any missing parts with only the remaining parts being uploaded. The process is completed with the commit. 

The retained parts of your upload are counted as active storage and included in any charges for your account. The following instructions are provided to help with cleaning up failed multi-part uploads.

You can list out the multi-part uploads you have with the AWS CLI command:

aws s3api list-multipart-uploads --bucket  

$ aws s3api list-multipart-uploads --bucket mikeo-status-test-east --endpoint-url=https://s3.wasabisys.com
{
    "Uploads": [
        {
            "Initiator": {
                "DisplayName": "mikeo", 
                "ID": "64FC53EBEA41CD91234567890E793B721167132B88F80CB6D63D39A3C91182CD"
            }, 
            "Initiated": "2019-07-23T19:36:30.000Z", 
            "UploadId": "KCFf6Rkrv2B7aBcDeFgHjlHv8WwC-r-qUV61jMXuMBMh-iObwi6do3xjVX9myirJIhZ2ez0-CnB1knnmpfYpwdSCwPGKzmVxz7oQzQJVPjXkbZdQ0wlEn_FVd_NtMgq", 
            "StorageClass": "STANDARD", 
            "Key": "10GB-samplefile.bin", 
            "Owner": {
                "DisplayName": "mikeo", 
                "ID": "64FC53EBEA41CD91234567890E793B721167132B88F80CB6D63D39A3C91182CD"
            }
        }, 

From that information, obtain the key and upload ID that use the key in a second command to abort the multi-part upload so the object parts are deleted.  

$ aws s3api abort-multipart-upload --bucket mikeo-status-test-east --endpoint-url=https://s3.wasabisys.com --key 10GB-samplefile.bin --upload-idKCFf6Rkrv2B7aBcDeFgHjlHv8WwC-r-qUV61jMXuMBMh-iObwi6do3xjVX9myirJIhZ2ez0-CnB1knnmpfYpwdSCwPGKzmVxz7oQzQJVPjXkbZdQ0wlEn_FVd_NtMgq

In this example we use us-east-1 region and the us-east-1 endpoint URL. For the endpoint URLs of other regions, refer to Service URLs for Wasabi’s Storage Regions.

Unfortunately, there is no AWS CLI command to do that in one step and no way to do this from the Wasabi Console. However, there is a Minio client that indicates that they can do this with one command: Minio Client aka mc is Open Source and compatible with S3. To list all the incomplete uploads on an associated bucket, use the following command (s3 here refers to the name of the host you define in your configuration file):

$ mc ls -I s3/mybucketname

To remove all incomplete uploads to a bucket:

$ mc rm -I -r --force s3/mybucketname

I = incomplete r = recursive f = with force option

For example:

mc ls s3/mikeo-status-test-east -I
[2019-07-23 15:35:21 EDT]      0B UP/10GB-aws.bin
[2019-07-23 16:16:29 EDT]   16MiB UP/10GB-aws.bin
[2019-09-12 17:25:25 EDT]  2.3GiB UP/10GB-aws.bin
[2019-08-04 08:18:56 EDT]   88MiB UP/10GB-aws.bin
[2019-08-20 11:38:40 EDT]  576MiB UP/10GB-aws.bin
[2019-08-24 12:49:12 EDT]  688MiB UP/10GB-aws.bin
[2019-08-04 10:21:37 EDT]  272MiB UP/10GB-packet.bin
[2019-07-18 12:23:42 EDT]   80MiB UP/10GB-packet.bin
[2019-07-23 15:35:25 EDT]      0B UP/10GB-packet.bin
[2019-07-23 16:32:06 EDT]   64MiB UP/10GB-packet.bin
[2019-08-04 08:19:42 EDT]  1.3GiB UP/10GB-packet.bin
[2019-06-05 16:08:51 EDT]      0B UP/10GB.bin

 To remove them: 

mc rm -I -r --force s3/mikeo-status-test-east
Removing `s3/mikeo-status-test-east/UP/10GB-aws.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-aws.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-aws.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-aws.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-aws.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-aws.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-packet.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-packet.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-packet.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-packet.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB-packet.bin`.
Removing `s3/mikeo-status-test-east/UP/10GB.bin`.