Operations on Objects With the Wasabi S3 API

Operations on Objects include renaming, composing, appending, and deleting objects as well as encryption, storage class, object locking, and compliance.

Renaming Objects

Wasabi supports functionality to move an object-in effect, renaming the object by changing the key. This eliminates the two-step process of first copying an object and then deleting the original object. The caller must have s3:PutObject policy permission on the bucket to rename objects.

To rename objects, use the HTTP method MOVE along with the following parameters in the request headers that affect the move operation.

The call will return the results as an XML body of the response. The results include the following for each object renamed:

• original source key,

• renamed destination key,

• version ID of the object, and

• any error in the rename operation.

The quiet option will cause the results to list only the keys that encountered an error.

Below is an example, which renames all objects that start with the prefix “TestMove-Dir2/” to have the new prefix “TestMove-Dir2-Renamed/”.

MOVE http://s3.wasabisys.com/my_bucket/TestMove-Dir2/HTTP/1.1 Destination:TestMove-Dir2-Renamed/X-Wasabi-Prefix:true

Response body:

<MoveObjectResult>

     <SourcePrefix>TestMove-Dir2/</SourcePrefix>

     <DestinationPrefix>TestMove-Dir2-Renamed/</DestinationPrefix>

     <MoveObject>

          <SourceKey>TestMove-Dir2/</SourceKey>

          <DestinationKey>TestMove-Dir2-Renamed/</DestinationKey>

     </MoveObject>

     <MoveObject>

          <SourceKey>TestMove-Dir2/Dir1/</SourceKey>

          <DestinationKey>TestMove-Dir2-Renamed/Dir1/</DestinationKey>

     </MoveObject>

     <MoveObject>

          <SourceKey>TestMove-Dir2/Dir1/Obj1</SourceKey>

          <DestinationKey>TestMove-Dir2-Renamed/Dir1/Obj1</DestinationKey>

     </MoveObject>

     <MoveObject>

          <SourceKey>TestMove-Dir2/Dir1/Obj2</SourceKey>

          <DestinationKey>TestMove-Dir2-Renamed/Dir1/Obj2</DestinationKey>

     </MoveObject>

     <MoveObject>

          <SourceKey>TestMove-Dir2/Dir2/</SourceKey>

          <DestinationKey>TestMove-Dir2-Renamed/Dir2/</DestinationKey>

     </MoveObject>

     <MoveObject>

          <SourceKey>TestMove-Dir2/Dir2/Obj1</SourceKey>

          <DestinationKey>TestMove-Dir2-Renamed/Dir2/Obj1</DestinationKey>

     </MoveObject>

     <MoveObject>

          <SourceKey>TestMove-Dir2/Dir2/Obj2</SourceKey>

          <DestinationKey>TestMove-Dir2-Renamed/Dir2/Obj2</DestinationKey>

     </MoveObject>

     <MoveCount>7</MoveCount>

</MoveObjectResult>

Composing Objects

Wasabi provides a feature to create a new object that is composed of other objects. A composed object is formed not by copying the original data objects, but rather by linking the composed object to the data objects. This is in contrast to AWS S3, which does not offer a way to compose objects. Wasabi operation is much faster and does not require multiple copies of the original data. The original composed objects may be deleted at any time, and the system will retain the original data as long as there are any links to the data.

Composing objects is an alternative to using the multi-part upload feature for creating objects larger than 5 GB. The caller must have s3:PutObject policy permission on the bucket to compose objects.

Composed objects may be created from other composed objects as well as original data objects. However, the total number of original data objects may not exceed 1024 in any one composed object. Composed objects may only link to other objects in the same bucket.

The compose operation is done using the HTTP method PUT with the query string parameter “?compose” to indicate that the object is composed of objects given in the XML body. The XML body contains a list of object keys and, optionally, the version IDs that form the new object. If the version ID is not given, the latest version of the object is used. The new object will appear to be one object with all the linked data objects concatenated.

The new composed object does not have an ETag (i.e, MD5) value, which is the MD5 of the concatenated objects, but rather the result of hashing all the data object MD5s together (like the calculation done when multi-part objects are completed).

Composed objects may not have user provided encryption keys. However, all data is stored encrypted at rest.

There is a charge only for the metadata used in composed objects. The original data used in linked objects is charged at the normal rate until all composed objects that link to the original data object are deleted.

Below is an example that composes a new object “TestCompose-Object-1” from three objects: “TestCompose-Data-1”, “TestCompose-Data-2”, and “TestCompose-Data-3”.

PUT http://s3.wasabisys.com/my_bucket/TestCompose-Object-1?composeHTTP/1.1Content-Type:text/xml

<ComposeRequest>

      <Component>

     <Key>TestCompose-Data-1</Key>

     </Component>

     <Component>

     <Key>TestCompose-Data-2</Key>

     </Component>

     <Component>

     <Key>TestCompose-Data-3</Key>

     </Component>

</ComposeRequest>

Appending to Objects

AWS S3 does not offer a way to append to objects.

Appending to an existing object is a different form of composing an object (see Composing Objects). The call uploads the data in the body to a temporary object and then creates a new composed object consisting of the data from the original object with the newly uploaded data appended. If the bucket is versioned, a new version of the composed object is created while keeping the original data. When not versioned, the newly composed object replaces the original object. The caller must have s3:PutObject policy permission on the bucket to append to objects. Additionally for a bucket without versioning, the s3:DeleteObject policy permission may be required to replace the original object.

Like composed objects, you can append only 1023 times to any object since each creates a link to the uploaded data. Be careful not to simply append small amounts of data many times to an object since each append data uploaded creates a new object subject to minimum size charges, and can slow down the performance due to a small data read.

The append operation is done using the HTTP method PUT with the query string parameter “?append”. The data to be appended is uploaded in the body of the request.

Below is an example that appends the string “<appended data>” to an existing object “TestAppend-Object”.

PUT http://s3.wasabisys.com/my-bucket/TestAppend-Object?appendHTTP/1.1Content-Length:15

<appended data>

PUT Object Copy Uses Link

Normally, to rename objects using AWS S3 protocol, an object is uploaded with one key value. Then, a PUT operation with the new key value copied from the original key value is given, and the object for the original key value is deleted.

Wasabi automatically optimizes this sequence by linking the new key value object to the original data from the copy key value. This avoids making another copy of the data, which would result in higher storage charges. Because Wasabi uses links to the original data, any size object up to the 5 TB limit may be copied and does not require the use of a multi-part upload.

Delete Object MFADelete

If the MFADelete option is turned on in the bucket versioning, the “x-amz-mfa” is not required if the access credentials were signed using MFA (see also MFA (Multi-Factor Authentication) Delete).

If bucket logging is enabled, deleting multiple objects will create a separate log entry for each object that is deleted.

GET Object Response Headers

Any header in the GET object request that starts with “response-” will be returned as a response header (minus the string “response-”).

Object Server-Side Encryption

Wasabi encrypts all data stored at rest regardless of the requested encryption. The system will use any caller provided encryption keys when given, or generate a random key encryption key for each object if no customer key is provided. If the customer provides the encryption key, similar to AWS S3, Wasabi will not keep a copy of the customer key in the metadata and the caller must provide the encryption key to read the data. No action on the part of the caller is needed if Wasabi provides the encryption key.

The caller may provide the encryption key using the headers x-amz-server-side-encryption-customer-algorithm, x-amz-server-side-encryption-customer-key, and x-amz-server-side-encryption-customer-key-MD5. These parameters work identical to AWS S3.

Wasabi does not support a key management service. Hence, the “x-amz-server-side-encryption” is not supported along with all the “aws:kms” functionality.

Wasabi stores an MD5 for the data that is always the uploaded data regardless of server-side encryption.

Object Storage Class

Wasabi only provides a single storage class which is most like the standard AWS S3 storage class. Where the storage class is returned in any operation, Wasabi will return the standard AWS S3 storage class.

Operations on Objects Not Supported in Wasabi