How do I mass delete 'Delete-Markers' inside a bucket?
- 28 Aug 2024
- 3 Minutes to read
- Print
- PDF
How do I mass delete 'Delete-Markers' inside a bucket?
- Updated on 28 Aug 2024
- 3 Minutes to read
- Print
- PDF
Article summary
Did you find this summary helpful?
Thank you for your feedback
Sometimes due to the S3 versioning feature, a bucket can accumulate a large number of Delete Markers (DM). These DMs along with current & non-current objects are part of LIST operation calls to any S3 storage system (AWS S3, Wasabi etc.) If a bucket comprises a large number of DMs, the LIST queries can take a longer time to go through all keys in the buckets and can show slower response times.
There are some s3 applications/tools which can recognize versioning feature and allow you to delete one DM at a time as shown here. This is helpful when you have fewer DMs to delete. However, we can also use a scripted method to delete a large set of DMs using the following python code.
Note: You may also use s3undelete which is a simple utility created by ARQ that removes delete markers from S3 (and S3-compatible) objects in buckets that have versioning enabled. The details of this utility are shown at the end of this document.
PRE-REQUISITES
Make sure you have installed AWS SDK boto3 for python on your CLI and turned off the versioning feature on your bucket before running the script
Install Python 3+ version to run this script
Executions and Details of the Script (output & screenshot attached):
When you execute the script, it will prompt you to select the profile or enter the API keys of the admin who is executing this script.
$ starting script...
$ Press 1 and enter to select existing profile
$ Press 2 and enter to enter Access Key and Secret Key
$ Press 3 to exit: (i) If you already have a profile configured on your CLI, you may Press 1
you may configure the AWS CLI profile for the Wasabi account using the Wasabi keys ahead of time
NOTE It is optional for you to use credential files to run your code but it is always a best practice to use such implementation wherein your credential keys are in a file stored on your local machine rather than being part of your actual code or entering Keys at runtime prompt
In this example, we have set the profile name as "profilename" in the "~/.aws/credentials" file.
(ii) If you do not wish to use the existing profile, you may press 2 and enter your API Keys.
You will be prompted to select an option from the menu. Select Option 2:
.png)
Enter your own Bucket Name (The region of the bucket will be automatically detected)
NOTE: If you are specifying a prefix, please be sure to enter FULL PREFIX PATH (bucket name NOT included)
Before the deletion starts, the script will perform complete pagination of your bucket and show you
Total Number of Delete-Markers which will be deleted
Total Number of Current Objects which will NOT be deleted
Total Number of Non-Current Objects which will NOT be deleted
Each log entry will represent the deletion of 1000 DMs at a time
OUTPUT Screenshot:
OUTPUT:
$ Please select what you want to do:
$
$ Versioned bucket
$ 1- Delete only non-current versions (old versions) (Versioned bucket)
$ 2- Delete only Delete markers (Versioned bucket)
$ 3- Delete current and non current versions (this will clear everything) (Versioned bucket)
$
$ Non Versioned bucket
$ 4- Delete all content (Non versioned bucket)
$
$ Bucket operation
$ 5- Delete a bucket (consider clearing the bucket data before)
$
$ 6- Exit the script
$
$ Choice [1, 2, 3, 4, 5, 6]: 2
$ Please enter the name of the bucket: 0-versioning-1
$ Please enter a prefix (leave blank if you don't need one)
$ Location of the bucket: us-east-1
$ Deleting only delete markers
$ Calculating, please wait... this may take a while
$ listing object versions: page 1
$ Analyze completed.
$ Total number of delete markers: 12
$ Are you sure you want to delete these delete markers [y/n]: y
$ Deleting batch N° 1 - 12 object version
$ Press Enter to resume!The script is also attached to this KB document.
------------------------------------------------------------------------------------------------------
You may also make use of the s3undelete utility as an alternative to delete Delete-Markers (DMs). This utility is macOS and Windows Compatible.
Download the packages here:
For macOS: | |
For Windows: |
Configuration
To use s3undelete, set 2 environment variables, AWS_ACCESS_KEY and AWS_SECRET_KEY, to your S3 access key ID and secret access key respectively.
OUTPUT Examples:
List Delete-Marker(s)
s3undelete list us-east-2 https://s3.us-east-2.wasabisys.com/username-delete-marker-demo-bucket
Removing Delete-Marker(s):
s3undelete undelete us-east-2 https://s3.us-east-2.wasabisys.com/username-delete-marker-demo-bucket
Note: This config example discusses the use of Wasabi's us-east-2 storage region. To use other Wasabi storage regions, please use the appropriate Wasabi service URL as described in this article.
