Operations on Objects
    • 23 Feb 2024
    • PDF

    Operations on Objects

    • PDF

    Article Summary

    Operations on Objects include renaming, composing, appending, and deleting objects as well as encryption, storage class, object locking, and compliance.

    By default, public use of a bucket, folder, or file is allowed only by certain paid (not trial) accounts. If your account does not have access to this feature, contact Wasabi Support at support@wasabi.com for assistance.

    Renaming Objects

    Wasabi supports functionality to move an object-in effect, renaming the object by changing the key. This eliminates the two-step process of first copying an object and then deleting the original object. The caller must have s3:PutObject policy permission on the bucket to rename objects.

    To rename objects, use the HTTP method MOVE along with the following parameters in the request headers that affect the move operation.

    Parameter

    Description

    Destination

    Used to form the destination key value for moved objects. If the X-Wasabi-Prefix option is true, then all objects (including all versions) that start with the source key in the URL are renamed by first removing the source key prefix and replacing it with this destination value. Otherwise, the objects (including all versions) exactly matching the source key in the URL are renamed to the key given by the destination.


    This parameter must be provided.

    Overwrite

    A Boolean value that, when “true,” allows overwriting destination objects with the same key. Otherwise, an error is generated and the original key is not changed.


    Note that versioned buckets will always allow creating a new object version with the destination key. The default value is “false.”

    X-Wasabi-Quiet

    A Boolean value that, when “true,” causes the XML status return body to only display the keys that encounter errors. Otherwise, all objects renamed are given in the status. The default value is “false.”

    X-Wasabi-Prefix

    A Boolean value that, when “true,” means that the values given for the source in the URL and the destination are prefixes—they match the left-most part of keys. Conceptually, think of the prefix as a folder of objects. Otherwise, only the objects (including all versions) that exactly match the key are renamed. The default value is “false.”

    The call will return the results as an XML body of the response. The results include the following for each object renamed:

    • original source key,

    • renamed destination key,

    • version ID of the object, and

    • any error in the rename operation.

    The quiet option will cause the results to list only the keys that encountered an error.

    Below is an example, which renames all objects that start with the prefix “TestMove-Dir2/” to have the new prefix “TestMove-Dir2-Renamed/”.

    MOVE http://s3.wasabisys.com/my_bucket/TestMove-Dir2/HTTP/1.1 Destination:TestMove-Dir2-Renamed/X-Wasabi-Prefix:true

    Response body:

    <MoveObjectResult>


         <SourcePrefix>TestMove-Dir2/</SourcePrefix>


         <DestinationPrefix>TestMove-Dir2-Renamed/</DestinationPrefix>


         <MoveObject>


              <SourceKey>TestMove-Dir2/</SourceKey>


              <DestinationKey>TestMove-Dir2-Renamed/</DestinationKey>


         </MoveObject>


         <MoveObject>


              <SourceKey>TestMove-Dir2/Dir1/</SourceKey>


              <DestinationKey>TestMove-Dir2-Renamed/Dir1/</DestinationKey>


         </MoveObject>


         <MoveObject>


              <SourceKey>TestMove-Dir2/Dir1/Obj1</SourceKey>


              <DestinationKey>TestMove-Dir2-Renamed/Dir1/Obj1</DestinationKey>


         </MoveObject>


         <MoveObject>


              <SourceKey>TestMove-Dir2/Dir1/Obj2</SourceKey>


              <DestinationKey>TestMove-Dir2-Renamed/Dir1/Obj2</DestinationKey>


         </MoveObject>


         <MoveObject>


              <SourceKey>TestMove-Dir2/Dir2/</SourceKey>


              <DestinationKey>TestMove-Dir2-Renamed/Dir2/</DestinationKey>


         </MoveObject>


         <MoveObject>


              <SourceKey>TestMove-Dir2/Dir2/Obj1</SourceKey>


              <DestinationKey>TestMove-Dir2-Renamed/Dir2/Obj1</DestinationKey>


         </MoveObject>


         <MoveObject>


              <SourceKey>TestMove-Dir2/Dir2/Obj2</SourceKey>


              <DestinationKey>TestMove-Dir2-Renamed/Dir2/Obj2</DestinationKey>


         </MoveObject>


         <MoveCount>7</MoveCount>


    </MoveObjectResult>

    Composing Objects

    Wasabi provides a feature to create a new object that is composed of other objects. A composed object is formed not by copying the original data objects, but rather by linking the composed object to the data objects. This is in contrast to AWS S3, which does not offer a way to compose objects. Wasabi operation is much faster and does not require multiple copies of the original data. The original composed objects may be deleted at any time, and the system will retain the original data as long as there are any links to the data.

    Composing objects is an alternative to using the multi-part upload feature for creating objects larger than 5 GB. The caller must have s3:PutObject policy permission on the bucket to compose objects.

    Composed objects may be created from other composed objects as well as original data objects. However, the total number of original data objects may not exceed 1024 in any one composed object. Composed objects may only link to other objects in the same bucket.

    The compose operation is done using the HTTP method PUT with the query string parameter “?compose” to indicate that the object is composed of objects given in the XML body. The XML body contains a list of object keys and, optionally, the version IDs that form the new object. If the version ID is not given, the latest version of the object is used. The new object will appear to be one object with all the linked data objects concatenated.

    The new composed object does not have an ETag (i.e, MD5) value, which is the MD5 of the concatenated objects, but rather the result of hashing all the data object MD5s together (like the calculation done when multi-part objects are completed).

    Composed objects may not have user provided encryption keys. However, all data is stored encrypted at rest.

    There is a charge only for the metadata used in composed objects. The original data used in linked objects is charged at the normal rate until all composed objects that link to the original data object are deleted.

    Below is an example that composes a new object “TestCompose-Object-1” from three objects: “TestCompose-Data-1”, “TestCompose-Data-2”, and “TestCompose-Data-3”.

    PUT http://s3.wasabisys.com/my_bucket/TestCompose-Object-1?composeHTTP/1.1Content-Type:text/xml


    <ComposeRequest>


          <Component>


         <Key>TestCompose-Data-1</Key>


         </Component>


         <Component>


         <Key>TestCompose-Data-2</Key>


         </Component>


         <Component>


         <Key>TestCompose-Data-3</Key>


         </Component>


    </ComposeRequest>

    Appending to Objects

    AWS S3 does not offer a way to append to objects.

    Appending to an existing object is a different form of composing an object (see Composing Objects). The call uploads the data in the body to a temporary object and then creates a new composed object consisting of the data from the original object with the newly uploaded data appended. If the bucket is versioned, a new version of the composed object is created while keeping the original data. When not versioned, the newly composed object replaces the original object. The caller must have s3:PutObject policy permission on the bucket to append to objects. Additionally for a bucket without versioning, the s3:DeleteObject policy permission may be required to replace the original object.

    Like composed objects, you can append only 1023 times to any object since each creates a link to the uploaded data. Be careful not to simply append small amounts of data many times to an object since each append data uploaded creates a new object subject to minimum size charges, and can slow down the performance due to a small data read.

    The append operation is done using the HTTP method PUT with the query string parameter “?append”. The data to be appended is uploaded in the body of the request.

    Below is an example that appends the string “<appended data>” to an existing object “TestAppend-Object”.

    PUT http://s3.wasabisys.com/my-bucket/TestAppend-Object?appendHTTP/1.1Content-Length:15


    <appended data>

    PUT Object Copy Uses Link

    Normally, to rename objects using AWS S3 protocol, an object is uploaded with one key value. Then, a PUT operation with the new key value copied from the original key value is given, and the object for the original key value is deleted.

    Wasabi automatically optimizes this sequence by linking the new key value object to the original data from the copy key value. This avoids making another copy of the data, which would result in higher storage charges. Because Wasabi uses links to the original data, any size object up to the 5 TB limit may be copied and does not require the use of a multi-part upload.

    Delete Object MFADelete

    If the MFADelete option is turned on in the bucket versioning, the “x-amz-mfa” is not required if the access credentials were signed using MFA. (See also MFA (Multi-Factor Authentication) Delete.)

    Additionally, the compliance feature can automatically delete objects when the DeleteAfterRetention option is true.

    If bucket logging is enabled, deleting multiple objects will create a separate log entry for each object that is deleted.

    GET Object Response Headers

    Any header in the GET object request that starts with “response-” will be returned as a response header (minus the string “response-”).

    Object Server-Side Encryption

    Wasabi encrypts all data stored at rest regardless of the requested encryption. The system will use any caller provided encryption keys when given, or generate a random key encryption key for each object if no customer key is provided. If the customer provides the encryption key, similar to AWS S3, Wasabi will not keep a copy of the customer key in the metadata and the caller must provide the encryption key to read the data. No action on the part of the caller is needed if Wasabi provides the encryption key.

    The caller may provide the encryption key using the headers x-amz-server-side-encryption-customer-algorithm, x-amz-server-side-encryption-customer-key, and x-amz-server-side-encryption-customer-key-MD5. These parameters work identical to AWS S3.

    Wasabi does not support a key management service. Hence, the “x-amz-server-side-encryption” is not supported along with all the “aws:kms” functionality.

    Wasabi stores an MD5 for the data that is always the uploaded data regardless of server-side encryption.

    Object Storage Class

    Wasabi only provides a single storage class which is most like the standard AWS S3 storage class. Where the storage class is returned in any operation, Wasabi will return the standard AWS S3 storage class.

    Object Locking

    You can add or update the object lock settings for an object using the tags described below.

    Tag

    Description

    Mode

    This tag should be either COMPLIANCE or GOVERNANCE, which are not case-sensitive.

    RetainUntilDate

    This defines the retention time for an object. The object cannot be deleted before this time. Note that the retention time must be in the future.

    LegalHold

    This should be either ON or OFF, which is not case-sensitive.

    The following is an example of setting the object locking configuration on an object:

    PUT https://s3.wasabisys.com/qa.objectlock.002/beta_vpn.png?retention&versionId= 001617181245457483475-4ylsTbqqvP


    <Retention>


         <Mode>compliance</Mode>


         <RetainUntilDate>2021-04-06T09:00:45Z</RetainUntilDate>


    </Retention>

    The following is an example of setting legal hold for an object:

    PUT https://s3.wasabisys.com/qa.objectlock.002/beta_vpn.png?legal-hold&versionId=001617181245457483475-4ylsTbqqvP


    <LegalHold>


         <Status>ON</Status>


    </LegalHold>

    The following is an example of setting governance mode for an object:

    PUT https://s3.wasabisys.com/qa.objectlock.002/public/ACL.PNG?retention


    <Retention>


         <Mode>governance</Mode>


         <RetainUntilDate>2021-05-20T09:00:45Z</RetainUntilDate>


    </Retention>

    The “x-amz-bypass-governance-retention =true” header must be included whenever you want to override governance settings for an object.

    Compliance

    The compliance settings for any one object in a bucket with compliance can also be changed within the limits of the compliance on the bucket. Specify the object compliance settings with the following XML tags.

    Tag

    Description

    ConditionalHold

    A Boolean value “false” to release the object from the conditional hold setting in the bucket policy. The retention period in days is started from the point when the conditional hold is released. Once the conditional hold is set false, it may not be returned to conditional hold.

    LegalHold

    A Boolean value “true” or “false” to set the legal hold status. When an object has a legal hold status of true, the object cannot be deleted regardless of the retention period.

    RetentionTime

    An ISO time giving a new retention time for the object in which the object cannot be deleted before this time. Note that the new retention time must be past the retention period given by the bucket policy or an error is returned.

    The following is an example of setting the compliance on an object:

    PUT http://s3.wasabisys.com/my-bucket/my-object?complianceHTTP/1.1


    <ObjectComplianceConfiguration>


         <ConditionalHold>false</ConditionalHold>


         <RetentionTime>2018-03-13T10:45:00Z</RetentionTime>


    </ObjectComplianceConfiguration>

    The compliance settings for any specific object can also be retrieved using the “?compliance” query string. In addition to the object compliance settings above, the query returns the calculated SHA256 hash for the object, which can be used to determine that the object has not been modified. Note that the SHA256 value is only available for objects that are uploaded as a single object and is not available for multi-part or composed objects.

    The following is an example of getting the compliance on an object:

    GET http://s3.wasabisys.com/my-bucket/my-object?complianceHTTP/1.1

    Response body:

    <ObjectComplianceConfiguration xml ns="http://s3.amazonaws.com/doc/2006-03-01/">


         <RetentionTime>2016-10-31T15:08:05Z</RetentionTime>


         <ConditionalHold>false</ConditionalHold>


         <LegalHold>false</LegalHold>


         <SHA256>14b4be3894e92166b508007b6c2e4fb6e88d3d0ad652c76475089a50ebe6e33b</SHA256>


    </ObjectComplianceConfiguration>

    The object compliance settings also appear in the bucket listings when the bucket has compliance enabled.

    Object Tagging

    GET Object Tagging

    Wasabi supports functionality to fetch the tag set of an object when the object:

    • Does not have any tags

    • Does have tags

    When the Object Does Not Have Any Tags

    GET


    https://s3.wasabisys.com/my_bucket/Test-object1.txt?tagging

    Response body:

    <?xml version="1.0" encoding="UTF-8"?>


    <Tagging>


       <TagSet></TagSet>


    </Tagging>

    Status 200 OK

    When the Object Has Tags

    GET


    https://s3.wasabisys.com/my_bucket/Test-object2.pdf?tagging

    Response body:

    <!--?xml version="1.0" encoding="UTF-8"?-->


    <Tagging>


       <TagSet>


          <Tag>


             <Key>key2</Key>


             <Value>value2</Value>


          </Tag>


          <Tag>


             <Key>key1</Key>


             <Value>value1</Value>


          </Tag>


       </TagSet>


    </Tagging>

    Status 200 OK

    PUT Object Tagging

    Wasabi supports functionality for:

    • Adding the tag value of an existing object in a bucket

    • Appending the tag value of an existing object in a bucket

    Adding the Tag Value of an Existing Object in a Bucket

    PUT


    https://s3.wasabisys.com/my_bucket/Test-object3.pdf?tagging


    Content-Type: text/xml

    Response body:

    <Tagging xmlns="http://s3.amazonaws.com/doc/2006-03-01/">


       <TagSet>


          <Tag>


             <Key>tag1</Key>


             <Value>value1</Value>


          </Tag>


       </TagSet>


    </Tagging>

    Status 200 OK

    Appending the Tag Value of an Existing Object in a Bucket

    In this example, an existing object already has the tag value of “key5” –“value5”, You can append a new tag pairs along with the existing pair.

    PUT


    https://s3.wasabisys.com/my_bucket/Test-object4.jpg?tagging


    Content-Type: text/xml

    Response body:

    <Tagging xmlns="http://s3.amazonaws.com/doc/2006-03-01/">


       <TagSet>


          <Tag>


             <Key>key5</Key>


             <Value>value5</Value>


          </Tag>


          <Tag>


             <Key>key6</Key>


             <Value>value6</Value>


          </Tag>


       </TagSet>


    </Tagging>

    Status 200 OK

    To PUT tags of any other version, use the versionId query parameter.

    DELETE Object Tagging

    This removes the entire tag set from the specified object.

    DELETE


    https://s3.wasabisys.com/my_bucket/Test-object5.pdf?tagging

    Response body:

    <empty>

    Status 204 No Content

    To delete tags of any other version, use the versionId query parameter.

    Object Replication

    PUT Object Replication

    Wasabi supports functionality for replication configuration with:

    • One source and one destination bucket (without filters)

    • One source and one destination bucket (with filters)

    • One source and multiple destination buckets

    In addition, Wasabi supports functionality for updating existing an replication configuration.

    Replication Configuration With One Source and One Destination Bucket (Without Filters)

    PUT


    https://s3.<region>.wasabisys.com/my-replication-source?replication

    Response body:

    <?xml version="1.0" encoding="UTF-8"?>


    <ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">


       <Role></Role>


       <Rule>


          <ID>job1</ID>


          <Priority>1</Priority>


          <Status>Enabled</Status>


          <DeleteMarkerReplication>


             <Status>Enabled</Status>


          </DeleteMarkerReplication>


          <Destination>


             <Bucket>arn:aws:s3:::my-replication-destination</Bucket>


          </Destination>


          <Filter></Filter>


          <WCSM>true</WCSM>


       </Rule>


    </ReplicationConfiguration>

    Status 200 OK

    Notes:

    1. Make sure both the source and destination buckets match the same bucket-property while scheduling the replication configuration.

      1. Non-versioned source —> Non-versioned destination

      2. Versioned source —> Versioned destination

    2. Only new objects (uploaded after scheduling the replication configuration) will be replicated to the destination bucket.

    3. To replicate the existing objects from the source to the destination, you need to initiate the following API only after scheduling the replication configuration.

    PUT


    https://s3.<region>.wasabisys.com/my-replication-source?manifestCreateAndProcess

    Response body:

    <?xml version="1.0" encoding="UTF-8"?>


    <ManifestCreateAndProcess>jobId : DHZsWBhwMUg9euZyFg7eahv4</ManifestCreateAndProcess>

    Status 200 OK

    Replication Configuration With One Source and One Destination Bucket (With Filters)

    In this example, objects that satisfy both conditions (prefix as AA- and tag “key1-value1”) will be replicated to the destination bucket.

    PUT


    https://s3.<region>.wasabisys.com/my-replication-source?replication

    Response body:

    <?xml version="1.0" encoding="UTF-8"?>


    <ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">


       <Role></Role>


       <Rule>


          <ID>job1</ID>


          <Priority>1</Priority>


          <Status>Enabled</Status>


          <DeleteMarkerReplication>


             <Status>Disabled</Status>


          </DeleteMarkerReplication>


          <Destination>


             <Bucket>arn:aws:s3:::my-replication-destination</Bucket>


          </Destination>


          <Filter>


               <And>


                    <Prefix>AA-</Prefix>


                    <Tag>


                         <Key>key1</Key>


                         <Value>value1</Value>


                    </Tag>


               </And>


          </Filter>


          <WCSM>true</WCSM>


       </Rule>


    </ReplicationConfiguration>

    Status 200 OK

    Replication Configuration With One Source and Multiple Destinations

    PUT


    https://s3.<region>.wasabisys.com/my-replication-source?replication

    Response body:

    <?xml version="1.0" encoding="UTF-8"?>


    <ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">


       <Role></Role>


       <Rule>


          <ID>job1</ID>


          <Priority>1</Priority>


          <Status>Enabled</Status>


          <DeleteMarkerReplication>


             <Status>Enabled</Status>


          </DeleteMarkerReplication>


          <Destination>


             <Bucket>arn:aws:s3:::my-replication-destination-1</Bucket>


          </Destination>


          <Filter></Filter>


          <WCSM>true</WCSM>


       </Rule>


       <Rule>


          <ID>job2</ID>


          <Priority>1</Priority>


          <Status>Enabled</Status>


          <DeleteMarkerReplication>


             <Status>Enabled</Status>


          </DeleteMarkerReplication>


          <Destination>


             <Bucket>arn:aws:s3:::my-replication-destination-2</Bucket>


          </Destination>


          <Filter></Filter>


          <WCSM>true</WCSM>


       </Rule>


    </ReplicationConfiguration>

    Status 200 OK

    Updating an Existing Replication Configuration

    The replication configuration will replace an existing configuration. In this example, an existing replication configuration maps job2 to my-replication-destination-2, and the update changes the replication configuration to map job2 to my-replication-destination-3.

    PUT


    https://s3.<region>.wasabisys.com/my-replication-source?replication

    Response body:

    <?xml version="1.0" encoding="UTF-8"?>


    <ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">


       <Role></Role>


       <Rule>


          <ID>job1</ID>


          <Priority>1</Priority>


          <Status>Enabled</Status>


          <DeleteMarkerReplication>


             <Status>Enabled</Status>


          </DeleteMarkerReplication>


          <Destination>


             <Bucket>arn:aws:s3:::my-replication-destination-1</Bucket>


          </Destination>


          <Filter></Filter>


          <WCSM>true</WCSM>


       </Rule>


       <Rule>


          <ID>job2</ID>


          <Priority>1</Priority>


          <Status>Enabled</Status>


          <DeleteMarkerReplication>


             <Status>Enabled</Status>


          </DeleteMarkerReplication>


          <Destination>


             <Bucket>arn:aws:s3:::my-replication-destination-3</Bucket>


          </Destination>


          <Filter></Filter>


          <WCSM>true</WCSM>


       </Rule>


    </ReplicationConfiguration>

    Status 200 OK

    GET Object Replication

    GET


    https://s3.<region>.wasabisys.com/my-replication-source?replication

    Response body:

    <?xml version="1.0" encoding="UTF-8"?>


    <ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">


       <Role></Role>


       <Rule>


          <ID>job1</ID>


          <Priority>1</Priority>


          <Status>Enabled</Status>


          <DeleteMarkerReplication>


             <Status>Enabled</Status>


          </DeleteMarkerReplication>


          <Destination>


             <Bucket>arn:aws:s3:::my-replication-destination</Bucket>


          </Destination>


          <Filter></Filter>


          <WCSM>true</WCSM>


       </Rule>


    </ReplicationConfiguration>

    Status 200 OK

    DELETE Object Replication

    You can delete the existing replication configuration of the bucket. This removes the entire replication configuration from the specified bucket.

    DELETE


    https://s3.<region>.wasabisys.com/my-replication-source?replication

    Response body:

    <empty>

    Status 204 No Content

    Operations on Objects Not Supported in Wasabi

    Operation

    Description

    Object BitTorrent

    Wasabi does not support BitTorrent operations on objects.

    Object RESTORE

    Wasabi does not support multiple classes of storage and does not support the POST object RESTORE request used to restore objects from a different class of storage.

    Select Object Content

    Wasabi does not support the operation to filter the contents of an Amazon S3 object based on a simple structured query language (SQL) statement.

    S3 Batch Operations

    Wasabi does not support S3 Batch Operations for object management.





    Was this article helpful?