AWS CLI With Wasabi

How do I use AWS CLI with Wasabi?

The AWS CLI (Command Line Interface) tool is certified for use with Wasabi. To use AWS CLI with Wasabi, follow the configuration tips below. Note that this article assumes that you have the awscli tool already installed. For more information on how to install it, refer to https://aws.amazon.com/cli/.

This example discusses the use of Wasabi's us-east-1 storage region. To use other Wasabi storage regions, use the appropriate Wasabi service URL, as described in Service URLs for Wasabi's Storage Regions.

Configuring AWS CLI Using the configure Command

To access your wasabi buckets with the CLI, utilize the configure command.

Typing $ aws configure will bring up a series of prompts, calling for an access key and secret access key. Enter your Wasabi access key and your Wasabi secret key. For the region, type "us-east-1". Leave the last setting blank.

If you follow options 2 or 3 below, you will have to configure the [wasabi] profile after you create it. To do so, simply type$ aws configure --profile wasabi.

Using Wasabi With AWS CLI

There are three methods for utilizing Wasabi with the CLI, ranking from requiring the least amount of configuration to the most.

Add the --endpoint-url Subcommand After Every Command

Add the following after every command you make in the CLI. Be sure to use the appropriate service URL as noted in Service URLs for Wasabi's Storage Regions. The URL may be different from s3.wasabisys.com.

 $ --endpoint-url=https://s3.wasabisys.com 

Example:

To list buckets in the cli, type:

$ aws s3 ls --endpoint-url=https://s3.wasabisys.com

To make a new bucket in the eu-central-1 region, type:

$ aws s3 mb s3://newbucketnamehere --endpoint-url=https://s3.eu-central-1.wasabisys.com

To make a new bucket with object lock enabled in the us-central-1 region, type:

$ aws s3api create-bucket --bucket yournewbucketnamehere --object-lock-enabled-for-bucket --endpoint-url=https://s3.us-central-1.wasabisys.com

Example:

aws s3api create-bucket --bucket yournewbucketnamehere --object-lock-enabled-for-bucket --endpoint-url=https://s3.us-central-1.wasabisys.com

{

    "Location": "/yournewbucketnamehere"

}

Using the awscli-plugin-endpoint Plugin

awscli-plugin-endpoint is a plugin used to easily access third-party S3 providers, such as Wasabi. Using this plugin requires creating a profile in addition to the default one.

1. Open the config file in the .aws directory. Add a profile line at the end of the configuration file, as shown below.

   [profile wasabi]
   

2. Save the file.

3. Once you have a profile created, follow the instructions detailed on the plugin's git page.

You should then be able to use --profile wasabi after every command instead of typing out the entire URL.

Example:

To list buckets in the cli, type:

$ aws s3 ls --profile wasabi

Using export AWS_PROFILE with awscli-plugin-endpoint

Once you have set up Option 2, you can default the aws cli to point to the new profile at the beginning of your session. To do so, follow the instructions below provided by AWS.

For Linux, macOS, or Unix:

$ export AWS_PROFILE=wasabi

For Windows:

> set AWS_PROFILE=wasabi

Setting the environment variable either changes the default profile until the end of your shell session or until you set the variable to a different value.

Example:

To list buckets in the cli, type:

$ aws s3 ls

Using AWS CLI for Cloud-to-Cloud Migration Scenarios

1. Install and configure AWS CLI using your AWS Access Key and Secret Key. For more information, refer to Get started with the AWS CLI.

2. Configure an additional AWS CLI profile for the Wasabi account using the Wasabi keys.

3. Transfer all files from the AWS bucket to a local directory by running the following command.

   $ aws s3 cp s3:/// --recursive

4. Transfer all files from the local directory to the destination bucket at Wasabi.

   $ aws s3 cp / s3:/// --recursive --profile wasabi

Uploading Large Files with AWS CLI

When uploading large files to Wasabi, multipart uploads are used to improve reliability, performance and efficiency. By default AWS CLI is configured to use 8MB of chunksize and depending on the file size, the multipart could generate thousand of parts causing overhead in connections.

Wasabi recommends to change the default chunksize from 8MB to 32MB. To change this configuration use the "aws configure set" command.

$ aws configure set default.s3.multipart_threshold 64MB (This will configure the CLI to start using multipart only if the file is large than 64MB)

$ aws configure set default.s3.multipart_chunksize 32MB (This will configure the CLI to use 32MB as the chunksize for each multipart)

Note:
Huge files: Use 64MB or 128MB to reduce the overhead, and tweak it based on your needs.
Unstable networks: Use 16MB to minimize retries.
Memory: When increasing the chunksize, you might need to adjust the amount of memory in your system.

Using AWS CLI v2 with Wasabi

AWS has implemented a new checksum algorithm CRC64NVME for Data Integrity (https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html), and this is causing an error to Wasabi customers when uploading to their buckets.

"The algorithm type you specified in x-amz-checksum- header is invalid."

The CRC64NVME now is the default checksum algorithm in the AWS CLI v2.  Wasabi is aware about this issue, and we are working to implement this algorithm in our infrastructure.

In the meantime if you want to use the AWS CLI v2 with Wasabi Services, you should force the checksum algorithm to CRC32 as specified below.

$ aws s3 cp <FILENAME> s3://<YOURBUCKETNAME> --endpoint-url https://s3.<WASABI-REGION>.wasabisys.com --checksum-algorithm=CRC32

For PowerShell and SDKs please refer to the links below.

PowerShell - https://docs.aws.amazon.com/powershell/latest/reference/
AWS SDK - https://aws.amazon.com/developer/tools/

Additional Information

For additional information regarding AWS CLI, refer to the following resources.

• How does AWS CLI Credentials & Config chain work?

• Using AWS CLI commands to create a user

• How do I use MFA to authenticate access to Wasabi using AWS CLI?

• How do I delete old object versions?

• How can I see the size of my buckets?

• How do I clean up my failed multipart uploads?

• What cloud-to-cloud migration tools does Wasabi recommend?

• How do I generate pre-signed URLs for temporary access with Wasabi?

• How to download ALL versions of all files within a Bucket

• How do I mass delete current and old object versions inside a bucket?

• How to use Pre-signed URLs to upload files to Wasabi