Operations on Objects

08 Mar 2024

Operations on Objects

Updated on 08 Mar 2024

Article Summary

Share feedback

Thanks for sharing your feedback!

Operations on Objects include renaming, composing, appending, and deleting objects as well as encryption, storage class, object locking, and compliance.

By default, public use of a bucket, folder, or file is allowed only by certain paid (not trial) accounts. If your account does not have access to this feature, contact Wasabi Support at support@wasabi.com for assistance.

Renaming Objects

Wasabi supports functionality to move an object-in effect, renaming the object by changing the key. This eliminates the two-step process of first copying an object and then deleting the original object. The caller must have s3:PutObject policy permission on the bucket to rename objects.

To rename objects, use the HTTP method MOVE along with the following parameters in the request headers that affect the move operation.

Parameter	Description
Destination	Used to form the destination key value for moved objects. If the X-Wasabi-Prefix option is true, then all objects (including all versions) that start with the source key in the URL are renamed by first removing the source key prefix and replacing it with this destination value. Otherwise, the objects (including all versions) exactly matching the source key in the URL are renamed to the key given by the destination. This parameter must be provided.
Overwrite	A Boolean value that, when “true,” allows overwriting destination objects with the same key. Otherwise, an error is generated and the original key is not changed. Note that versioned buckets will always allow creating a new object version with the destination key. The default value is “false.”
X-Wasabi-Quiet	A Boolean value that, when “true,” causes the XML status return body to only display the keys that encounter errors. Otherwise, all objects renamed are given in the status. The default value is “false.”
X-Wasabi-Prefix	A Boolean value that, when “true,” means that the values given for the source in the URL and the destination are prefixes—they match the left-most part of keys. Conceptually, think of the prefix as a folder of objects. Otherwise, only the objects (including all versions) that exactly match the key are renamed. The default value is “false.”

The call will return the results as an XML body of the response. The results include the following for each object renamed:

original source key,
renamed destination key,
version ID of the object, and
any error in the rename operation.

The quiet option will cause the results to list only the keys that encountered an error.

Below is an example, which renames all objects that start with the prefix “TestMove-Dir2/” to have the new prefix “TestMove-Dir2-Renamed/”.

MOVE http://s3.wasabisys.com/my_bucket/TestMove-Dir2/HTTP/1.1 Destination:TestMove-Dir2-Renamed/X-Wasabi-Prefix:true

Response body:

<SourcePrefix>TestMove-Dir2/</SourcePrefix>

<DestinationPrefix>TestMove-Dir2-Renamed/</DestinationPrefix>

<SourceKey>TestMove-Dir2/</SourceKey>

<DestinationKey>TestMove-Dir2-Renamed/</DestinationKey>

</MoveObject>

<SourceKey>TestMove-Dir2/Dir1/</SourceKey>

<DestinationKey>TestMove-Dir2-Renamed/Dir1/</DestinationKey>

</MoveObject>

<SourceKey>TestMove-Dir2/Dir1/Obj1</SourceKey>

<DestinationKey>TestMove-Dir2-Renamed/Dir1/Obj1</DestinationKey>

</MoveObject>

<SourceKey>TestMove-Dir2/Dir1/Obj2</SourceKey>

<DestinationKey>TestMove-Dir2-Renamed/Dir1/Obj2</DestinationKey>

</MoveObject>

<SourceKey>TestMove-Dir2/Dir2/</SourceKey>

<DestinationKey>TestMove-Dir2-Renamed/Dir2/</DestinationKey>

</MoveObject>

<SourceKey>TestMove-Dir2/Dir2/Obj1</SourceKey>

<DestinationKey>TestMove-Dir2-Renamed/Dir2/Obj1</DestinationKey>

</MoveObject>

<SourceKey>TestMove-Dir2/Dir2/Obj2</SourceKey>

<DestinationKey>TestMove-Dir2-Renamed/Dir2/Obj2</DestinationKey>

</MoveObject>

</MoveObjectResult>

Composing Objects

Wasabi provides a feature to create a new object that is composed of other objects. A composed object is formed not by copying the original data objects, but rather by linking the composed object to the data objects. This is in contrast to AWS S3, which does not offer a way to compose objects. Wasabi operation is much faster and does not require multiple copies of the original data. The original composed objects may be deleted at any time, and the system will retain the original data as long as there are any links to the data.

Composing objects is an alternative to using the multi-part upload feature for creating objects larger than 5 GB. The caller must have s3:PutObject policy permission on the bucket to compose objects.

Composed objects may be created from other composed objects as well as original data objects. However, the total number of original data objects may not exceed 1024 in any one composed object. Composed objects may only link to other objects in the same bucket.

The compose operation is done using the HTTP method PUT with the query string parameter “?compose” to indicate that the object is composed of objects given in the XML body. The XML body contains a list of object keys and, optionally, the version IDs that form the new object. If the version ID is not given, the latest version of the object is used. The new object will appear to be one object with all the linked data objects concatenated.

The new composed object does not have an ETag (i.e, MD5) value, which is the MD5 of the concatenated objects, but rather the result of hashing all the data object MD5s together (like the calculation done when multi-part objects are completed).

Composed objects may not have user provided encryption keys. However, all data is stored encrypted at rest.

There is a charge only for the metadata used in composed objects. The original data used in linked objects is charged at the normal rate until all composed objects that link to the original data object are deleted.

Below is an example that composes a new object “TestCompose-Object-1” from three objects: “TestCompose-Data-1”, “TestCompose-Data-2”, and “TestCompose-Data-3”.

PUT http://s3.wasabisys.com/my_bucket/TestCompose-Object-1?composeHTTP/1.1Content-Type:text/xml

<Key>TestCompose-Data-1</Key>

</Component>

<Key>TestCompose-Data-2</Key>

</Component>

<Key>TestCompose-Data-3</Key>

</Component>

</ComposeRequest>

Appending to Objects

AWS S3 does not offer a way to append to objects.

Appending to an existing object is a different form of composing an object (see Composing Objects). The call uploads the data in the body to a temporary object and then creates a new composed object consisting of the data from the original object with the newly uploaded data appended. If the bucket is versioned, a new version of the composed object is created while keeping the original data. When not versioned, the newly composed object replaces the original object. The caller must have s3:PutObject policy permission on the bucket to append to objects. Additionally for a bucket without versioning, the s3:DeleteObject policy permission may be required to replace the original object.

Like composed objects, you can append only 1023 times to any object since each creates a link to the uploaded data. Be careful not to simply append small amounts of data many times to an object since each append data uploaded creates a new object subject to minimum size charges, and can slow down the performance due to a small data read.

The append operation is done using the HTTP method PUT with the query string parameter “?append”. The data to be appended is uploaded in the body of the request.

Below is an example that appends the string “<appended data>” to an existing object “TestAppend-Object”.

PUT http://s3.wasabisys.com/my-bucket/TestAppend-Object?appendHTTP/1.1Content-Length:15

PUT Object Copy Uses Link

Normally, to rename objects using AWS S3 protocol, an object is uploaded with one key value. Then, a PUT operation with the new key value copied from the original key value is given, and the object for the original key value is deleted.

Wasabi automatically optimizes this sequence by linking the new key value object to the original data from the copy key value. This avoids making another copy of the data, which would result in higher storage charges. Because Wasabi uses links to the original data, any size object up to the 5 TB limit may be copied and does not require the use of a multi-part upload.

Delete Object MFADelete

If the MFADelete option is turned on in the bucket versioning, the “x-amz-mfa” is not required if the access credentials were signed using MFA. (See also MFA (Multi-Factor Authentication) Delete.)

If bucket logging is enabled, deleting multiple objects will create a separate log entry for each object that is deleted.

GET Object Response Headers

Any header in the GET object request that starts with “response-” will be returned as a response header (minus the string “response-”).

Object Server-Side Encryption

Wasabi encrypts all data stored at rest regardless of the requested encryption. The system will use any caller provided encryption keys when given, or generate a random key encryption key for each object if no customer key is provided. If the customer provides the encryption key, similar to AWS S3, Wasabi will not keep a copy of the customer key in the metadata and the caller must provide the encryption key to read the data. No action on the part of the caller is needed if Wasabi provides the encryption key.

The caller may provide the encryption key using the headers x-amz-server-side-encryption-customer-algorithm, x-amz-server-side-encryption-customer-key, and x-amz-server-side-encryption-customer-key-MD5. These parameters work identical to AWS S3.

Wasabi does not support a key management service. Hence, the “x-amz-server-side-encryption” is not supported along with all the “aws:kms” functionality.

Wasabi stores an MD5 for the data that is always the uploaded data regardless of server-side encryption.

Object Storage Class

Wasabi only provides a single storage class which is most like the standard AWS S3 storage class. Where the storage class is returned in any operation, Wasabi will return the standard AWS S3 storage class.

Object Locking

You can add or update the object lock settings for an object using the tags described below.

Tag	Description
Mode	This tag should be either COMPLIANCE or GOVERNANCE, which are not case-sensitive.
RetainUntilDate	This defines the retention time for an object. The object cannot be deleted before this time. Note that the retention time must be in the future.
LegalHold	This should be either ON or OFF, which is not case-sensitive.

The following is an example of setting the object locking configuration on an object:

PUT https://s3.wasabisys.com/qa.objectlock.002/beta_vpn.png?retention&versionId= 001617181245457483475-4ylsTbqqvP

<Mode>compliance</Mode>

</Retention>

The following is an example of setting legal hold for an object:

PUT https://s3.wasabisys.com/qa.objectlock.002/beta_vpn.png?legal-hold&versionId=001617181245457483475-4ylsTbqqvP

</LegalHold>

The following is an example of setting governance mode for an object:

PUT https://s3.wasabisys.com/qa.objectlock.002/public/ACL.PNG?retention

<Mode>governance</Mode>

</Retention>

The “x-amz-bypass-governance-retention =true” header must be included whenever you want to override governance settings for an object.

Compliance

The compliance settings for any one object in a bucket with compliance can also be changed within the limits of the compliance on the bucket. Specify the object compliance settings with the following XML tags.

Tag	Description
ConditionalHold	A Boolean value “false” to release the object from the conditional hold setting in the bucket policy. The retention period in days is started from the point when the conditional hold is released. Once the conditional hold is set false, it may not be returned to conditional hold.
LegalHold	A Boolean value “true” or “false” to set the legal hold status. When an object has a legal hold status of true, the object cannot be deleted regardless of the retention period.
RetentionTime	An ISO time giving a new retention time for the object in which the object cannot be deleted before this time. Note that the new retention time must be past the retention period given by the bucket policy or an error is returned.

The following is an example of setting the compliance on an object:

PUT http://s3.wasabisys.com/my-bucket/my-object?complianceHTTP/1.1

<ConditionalHold>false</ConditionalHold>

</ObjectComplianceConfiguration>

The compliance settings for any specific object can also be retrieved using the “?compliance” query string. In addition to the object compliance settings above, the query returns the calculated SHA256 hash for the object, which can be used to determine that the object has not been modified. Note that the SHA256 value is only available for objects that are uploaded as a single object and is not available for multi-part or composed objects.

The following is an example of getting the compliance on an object:

GET http://s3.wasabisys.com/my-bucket/my-object?complianceHTTP/1.1

Response body:

<ConditionalHold>false</ConditionalHold>

<LegalHold>false</LegalHold>

</ObjectComplianceConfiguration>

The object compliance settings also appear in the bucket listings when the bucket has compliance enabled.

Object Tagging

GET Object Tagging

Wasabi supports functionality to fetch the tag set of an object when the object:

Does not have any tags
Does have tags

When the Object Does Not Have Any Tags

GET

https://s3.wasabisys.com/my_bucket/Test-object1.txt?tagging

Response body:

<?xml version="1.0" encoding="UTF-8"?>

</Tagging>

Status 200 OK

When the Object Has Tags

GET

https://s3.wasabisys.com/my_bucket/Test-object2.pdf?tagging

Response body:

<Tag>

<Value>value2</Value>

</Tag>

<Tag>

<Value>value1</Value>

</Tag>

</TagSet>

</Tagging>

Status 200 OK

PUT Object Tagging

Wasabi supports functionality for:

Adding the tag value of an existing object in a bucket
Appending the tag value of an existing object in a bucket

Adding the Tag Value of an Existing Object in a Bucket

PUT

https://s3.wasabisys.com/my_bucket/Test-object3.pdf?tagging

Content-Type: text/xml

Response body:

<Tag>

<Value>value1</Value>

</Tag>

</TagSet>

</Tagging>

Status 200 OK

Appending the Tag Value of an Existing Object in a Bucket

In this example, an existing object already has the tag value of “key5” –“value5”, You can append a new tag pairs along with the existing pair.

PUT

https://s3.wasabisys.com/my_bucket/Test-object4.jpg?tagging

Content-Type: text/xml

Response body:

<Tag>

<Value>value5</Value>

</Tag>

<Tag>

<Value>value6</Value>

</Tag>

</TagSet>

</Tagging>

Status 200 OK

To PUT tags of any other version, use the versionId query parameter.

DELETE Object Tagging

This removes the entire tag set from the specified object.

DELETE

https://s3.wasabisys.com/my_bucket/Test-object5.pdf?tagging

Response body:

<empty>

Status 204 No Content

To delete tags of any other version, use the versionId query parameter.

Object Replication

PUT Object Replication

Wasabi supports functionality for replication configuration with:

One source and one destination bucket (without filters)
One source and one destination bucket (with filters)
One source and multiple destination buckets

In addition, Wasabi supports functionality for updating existing an replication configuration.

Replication Configuration With One Source and One Destination Bucket (Without Filters)

PUT

https://s3.<region>.wasabisys.com/my-replication-source?replication

Response body:

<?xml version="1.0" encoding="UTF-8"?>

<Rule>

<Status>Enabled</Status>

<Status>Enabled</Status>

</DeleteMarkerReplication>

<Bucket>arn:aws:s3:::my-replication-destination</Bucket>

</Destination>

</Rule>

</ReplicationConfiguration>

Status 200 OK

Notes:

Make sure both the source and destination buckets match the same bucket-property while scheduling the replication configuration.
1. Non-versioned source —> Non-versioned destination
2. Versioned source —> Versioned destination
Only new objects (uploaded after scheduling the replication configuration) will be replicated to the destination bucket.
To replicate the existing objects from the source to the destination, you need to initiate the following API only after scheduling the replication configuration.

PUT

https://s3.<region>.wasabisys.com/my-replication-source?manifestCreateAndProcess

Response body:

<?xml version="1.0" encoding="UTF-8"?>

<ManifestCreateAndProcess>jobId : DHZsWBhwMUg9euZyFg7eahv4</ManifestCreateAndProcess>

Status 200 OK

Replication Configuration With One Source and One Destination Bucket (With Filters)

In this example, objects that satisfy both conditions (prefix as AA- and tag “key1-value1”) will be replicated to the destination bucket.

PUT

https://s3.<region>.wasabisys.com/my-replication-source?replication

Response body:

<?xml version="1.0" encoding="UTF-8"?>

<Rule>

<Status>Enabled</Status>

<Status>Disabled</Status>

</DeleteMarkerReplication>

<Bucket>arn:aws:s3:::my-replication-destination</Bucket>

</Destination>

<And>

<Tag>

<Value>value1</Value>

</Tag>

</And>

</Filter>

</Rule>

</ReplicationConfiguration>

Status 200 OK

Replication Configuration With One Source and Multiple Destinations

PUT

https://s3.<region>.wasabisys.com/my-replication-source?replication

Response body:

<?xml version="1.0" encoding="UTF-8"?>

<Rule>

<Status>Enabled</Status>

<Status>Enabled</Status>

</DeleteMarkerReplication>

<Bucket>arn:aws:s3:::my-replication-destination-1</Bucket>

</Destination>

</Rule>

<Rule>

<Status>Enabled</Status>

<Status>Enabled</Status>

</DeleteMarkerReplication>

<Bucket>arn:aws:s3:::my-replication-destination-2</Bucket>

</Destination>

</Rule>

</ReplicationConfiguration>

Status 200 OK

Updating an Existing Replication Configuration

The replication configuration will replace an existing configuration. In this example, an existing replication configuration maps job2 to my-replication-destination-2, and the update changes the replication configuration to map job2 to my-replication-destination-3.

PUT

https://s3.<region>.wasabisys.com/my-replication-source?replication

Response body:

<?xml version="1.0" encoding="UTF-8"?>

<Rule>

<Status>Enabled</Status>

<Status>Enabled</Status>

</DeleteMarkerReplication>

<Bucket>arn:aws:s3:::my-replication-destination-1</Bucket>

</Destination>

</Rule>

<Rule>

<Status>Enabled</Status>

<Status>Enabled</Status>

</DeleteMarkerReplication>

<Bucket>arn:aws:s3:::my-replication-destination-3</Bucket>

</Destination>

</Rule>

</ReplicationConfiguration>

Status 200 OK

GET Object Replication

GET

https://s3.<region>.wasabisys.com/my-replication-source?replication

Response body:

<?xml version="1.0" encoding="UTF-8"?>

<Rule>

<Status>Enabled</Status>

<Status>Enabled</Status>

</DeleteMarkerReplication>

<Bucket>arn:aws:s3:::my-replication-destination</Bucket>

</Destination>

</Rule>

</ReplicationConfiguration>

Status 200 OK

DELETE Object Replication

You can delete the existing replication configuration of the bucket. This removes the entire replication configuration from the specified bucket.

DELETE

https://s3.<region>.wasabisys.com/my-replication-source?replication

Response body:

<empty>

Status 204 No Content

Operations on Objects Not Supported in Wasabi

Operation	Description
Object BitTorrent	Wasabi does not support BitTorrent operations on objects.
Object RESTORE	Wasabi does not support multiple classes of storage and does not support the POST object RESTORE request used to restore objects from a different class of storage.
Select Object Content	Wasabi does not support the operation to filter the contents of an Amazon S3 object based on a simple structured query language (SQL) statement.
S3 Batch Operations	Wasabi does not support S3 Batch Operations for object management.

What's Next

IAM and STS Support

Table of contents

Renaming Objects
Composing Objects
Appending to Objects
PUT Object Copy Uses Link
Delete Object MFADelete
GET Object Response Headers
Object Server-Side Encryption
Object Storage Class
Object Locking
Compliance
Object Tagging
Object Replication
Operations on Objects Not Supported in Wasabi

WASABI and the WASABI logo are trademarks of Wasabi Technologies LLC and may not be used without permission of Wasabi Technologies. All other names are used for identification purposes only and are trademarks or registered trademarks of their respective companies.

Information presented herein is subject to change without notice. Companies, names, and data used in examples are fictitious unless otherwise noted. No part of this information may be reproduced or transmitted in any form by means electronic or mechanical, for any purpose, without express written permission of Wasabi Technologies.