How do I mass delete current and non-current objects inside a bucket?
    • 28 Aug 2024
    • 2 Minutes to read
    • PDF

    How do I mass delete current and non-current objects inside a bucket?

    • PDF

    Article summary

    There are some s3 applications/tools which recognize the versioning feature and allow you to delete one versioned object at a time as shown in the How do I delete old object versions document. 

    This is helpful when you have fewer objects to delete. We can also use a scripted method to delete large object sets (current + non-current old versioned + delete markers) using the following Python code

    • Make sure you have installed AWS SDK boto3 for Python on your CLI and turned off the versioning feature on your bucket before running the script

    • Install Python 3+ version to run this script

    Executions and Details of the Script (output & screenshot attached):

    1. When you execute the script, it will prompt you to select the profile or enter the API keys of the admin who is executing this script

    (i) If you already have a profile configured on your CLI, you may Press 1

    you may configure the AWS CLI profile for the Wasabi account using the Wasabi keys ahead of time

    NOTE that it is optional for you to use credential files to run your code but it is always a best practice to use such implementation wherein your credential keys are in a file stored on your local machine rather than being part of your actual code or entering Keys at runtime prompt

    (ii) If you do not wish to use the existing profile, you may press 2 and enter your API Keys.

    2. You will be prompted to select an option from the menu. Select Option 1:

    3. Enter your own Bucket Name (The region of the bucket will be automatically detected)

    NOTE: If you are specifying a prefix, please be sure to enter FULL PREFIX PATH (bucket name NOT included)

    4. Before the deletion starts, the script will perform complete pagination of your bucket and show you 

    • Total Number of Delete-Markers which will be deleted (if present)

    • Total Number of Current Objects which will be deleted (if present)

    • Total Number of Non-Current Objects which will be deleted from your bucket (if present)

    5. After presenting the bucket statistics above, the script will ask if you need to enable Governance Mode override. This option is for Object Locked immutable buckets with Governance Mode enabled, not Compliance Mode. When a bucket is locked with Governance Mode, you may remove objects only if your IAM entity has the 's3:BypassGovernanceRetention' permission and you include the BypassGovernanceRetention header in the request. If you require this header and your IAM entity has the proper permissions, please enable this option with the 'y' input. Otherwise, input 'n' to bypass.

    Each log entry will represent the deletion of 1000 objects at a time

    NOTE: After executing the script and successfully completing the deletion process, you may run the same script again to confirm your total number statistics 

    • Total Number of Delete-Markers 

    • Total Number of Current Objects

    • Total Number of Non-Current Objects

    This is what the output should look like:

    The script is also attached to this KB document.


    ESC

    Eddy AI, facilitating knowledge discovery through conversational intelligence