Google Cloud Command Line Interface With Wasabi

Using the Google Cloud Command Line Interface to Copy Objects From the Google Cloud Platform to Wasabi

The Google Cloud Command Line Interface (gcloud CLI) can be used to copy objects from Google Cloud Platform (GCP) to Wasabi and list the contents of GCP and Wasabi buckets. This article details the required commands to perform these functions.

Note that the gcloud CLI will download objects to the system running the command before sending it to Wasabi. For cloud-to-cloud transfers, consider using services such as Wasabi Cloud Sync Manager (WCSM).

Requirements

An active Wasabi Hot Cloud Storage account.
An active Google account with GCP storage.
Google authentication performed.
The gcloud CLI is installed.
Data stored in GCP bucket(s).
Wasabi access and secret keys. It is recommended to create a sub-user with their own set of keys for this purpose rather than using your root keys. See Creating a User for more details. You may then restrict what access the sub-user has, such as access to a specific bucket, using IAM policies. See IAM and Bucket Policies for details.
Wasabi bucket(s). See Creating a Bucket for details on this procedure.

Configuring Wasabi URL and Credentials

To set the Wasabi URL used by the gcloud CLI, issue the following command.

gcloud config set storage/s3_endpoint_url https://s3.us-east-1.wasabisys.com

This configuration example discusses the use of Wasabi's us-east-1 storage region. Use the region your bucket is located in. For a list of regions, see Service URLs for Wasabi's Storage Regions.

The gcloud CLI by default uses the Wasabi credentials in the AWS credentials file ~/.aws/credentials . The AWS CLI is not required to be installed. You may just create the file with the following contents:

[default]
aws_access_key_id = YOUR_WASABI_ACCESS_KEY
aws_secret_access_key = YOUR_WASABI_SECRET_KEY

Listing Contents of Buckets

The gcloud CLI may be used list the contents of GCP buckets and Wasabi buckets. The command format for GCP buckets is:

gcloud storage ls 'gs://YOUR_GCP_BUCKET'

You may use the --recursive option to recursively list the contents of all folders in the bucket.

Here is an example used to recursively list the contents of the “mt-transfer-bucket” GCP bucket.

Similarly, the command format for Wasabi buckets is:

gcloud storage ls 's3://YOUR_WASABI_BUCKET'

The --recursive options may be used here as well.

An example listing the contents of the “mt-transfer-bucket” Wasabi bucket without the recursive option. We chose the same bucket name for both GCP and Wasabi to make testing easier, although this may not be possible all the time.

Copying Objects from GCP to Wasabi

To copy a single object from a GCP bucket to a Wasabi bucket, use the following format, replacing the GCP bucket (specified with gs://) and the Wasabi bucket (specified with s3://) with your own bucket names and the test.txt object with your own object name.

gcloud storage cp gs://mt-transfer-bucket/test.txt s3://mt-transfer-bucket

Here is an example of copying multiple objects.

gcloud storage cp 'gs://mt-transfer-bucket/*' s3://mt-transfer-bucket

An example of copying from a GCP bucket folder to an existing folder in a Wasabi bucket is as follows.

gcloud storage cp 'gs://mt-transfer-bucket/test/*' s3://mt-transfer-bucket/test

To copy recursively, use the following command format. This copies all folders and creates new folders if needed in your Wasabi bucket. The example shows a “test” folder being created in an empty Wasabi bucket.

gcloud storage cp --recursive 'gs://mt-transfer-bucket/*' s3://mt-transfer-bucket/