Spectra Vail With Wasabi

4. Configuring Cloud Storage

• Data Path Endpoint - endpoint URL of the region where the bucket is located

• Access Key 

• Secret Key 

5. Configuring Vail Lifecycle rules

A Vail lifecycle rule can now be created, which can be attached to any Vail bucket, to direct the initial placement and actions over time for any object placed in that bucket through any Vail endpoint.

As an example, in this section, we will create a Vail bucket called “cloud-and-denver” and a lifecycle rule to move the objects “put” into this bucket to S3-Standard disk storage on BlackPearl S3 in Colorado and to the Wasabi bucket created above. In this case, we will also create a rule that consolidates all copies of that object to live exclusively on Wasabi after 6 months.

We will create the lifecycle rule first and then create the Vail bucket and attach the new lifecycle rule to that bucket.

5.1. On the left hand pane, click on "Lifecycles". In the Lifecycles page, click on " + CREATE".

5.2. The Create Lifecycle dialog box is displayed. In the Parameters tab, provide the following details to configure the lifecycle policy.

• Name - provide a name for the lifecycle policy

• Multipart Upload Expiration - how long multipart uploads should remain open in the case of a slow connection

• Delete Marker Expiration - configure how the deletes are to be handled

• Description - provide an appropriate description that would be useful for other administrators

5.3. In the Rules tab, we will enter two “rules” for this lifecycle. The first rule will be configured to put two clone copies of any uploaded object – one in Boulder and one in Wasabi. Click on "Add Move Rule".

5.4. Provide a name – in this case “Wasabi and Boulder”, specify it should apply immediately, in this case after 0 days, it should apply to all versions of the object, and select the locations, in this case new-vail-bucket just created in Wasabi, and 1.1 s3 share (a location in Boulder in a BlackPearl).

5.5. We will then click “Add Move rule” and add a second rule to this lifecycle that will consolidate the copies to Wasabi only after 6 months.

In this case the name “Consolidate to wasabi”, that it triggers in 6 months (180 days), it should apply to all versions, and the destination of new-vail-bucket is the target. Note that include and exclude filters can be applied limiting specific file types or locations to/from this rule.

Please note that a move rule was used as opposed to a copy rule – a copy rule creates new clone copies, a move rule makes the specified copies and then releases all other clone copies for deletion. An expire rule is also available if these functions need to be split up in time or trigger.

5.6. On the Vail create bucket screen, create a new Vail bucket.

5.7. The Create Bucket dialog box is displayed.  In the Parameters tab, provide a name for the bucket and configure the required settings and click "Next".

5.8. In the Policy tab, configure specific json permissions as per the policy shown below and click "Next".

{
"Version": "2019-08-23",
"Id": "ExamplePolocy01",
"Statement": [
{
"Sid": "ExampleStatement01",
"Effect": "Allow",
"Principal": "*",
"Action": [
"*"
],
"Resource": [
"arn:aws:s3:::examplebucket"
]
}
]
}

5.9. In the Lifecycle tab, select the "Wasabi life" lifecycle  created earlier and click "Next". On the next screen, review the config and click "Submit".

5.10. The lifecycle rule attached to this Vail bucket, as well as any relevant settings are displayed on the bucket info screen.

6. Using Vail to transfer data

6.1. Using a client such as Mountain duck, connecting to any Vail endpoint in the system, the newly created bucket can be seen and mounted as a network drive. Dragging a file into the "cloud-and-denver" bucket results in the object being put/uploaded to the Vail node, the lifecycle rule is triggered, and the appropriate clone copies are created.

6.2. in the bucket menu of the Vail UI, the object (a .pdf) is shown with all its storage locations – a copy in 1.1 s3-share (The Boulder disk), in the wasabi new-vail-bucket and the just uploaded, released for delete version on the London VM – the endpoint to which the file was uploaded. Notice that that copy shows cached, meaning released for delete.

6.3. Logging into the Wasabi web portal, the new “new-vail-bucket” can be seen as well as the subfolder created with the Vail bucket name “cloud-and-denver” as well as the .pdf that was uploaded to London and a copy made directly on Wasabi.