Rclone With Wasabi

How do I use Rclone with Wasabi?

The Rclone software package is a command-line tool which allows a variety of modes of operation for managing data in two or more locations. Rclone has been certified for use with Wasabi. This article describes how to use Rclone to manage data with the Wasabi service. For more information about Rclone, please visit the Rclone project page. 

Note: This example discusses the use of Wasabi's us-east-1 storage region. To use other Wasabi storage regions, please use the appropriate Wasabi service URL as described in our Service URLs KB. 

1. Configure Wasabi as an Rclone Remote Location

1.1 Run “rclone config”, Input “n” for “New Remote”,  press “Enter”, input a name for the remote “wasabi-us-east-1”, then press “Enter”.

# rclone config
2022/10/17 10:36:53 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
1. n) New remote
2. s) Set configuration password
3. q) Quit config
n/s/q> n
name> wasabi-us-east-1

1.2 Configure Storage Type. Input “4” for “S3 Compliant”, press “Enter”, input “12” for Wasabi, then press “Enter”.

Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, Tencent COS, etc)
   \ "s3"
Storage> 4
** See help for s3 backend at: https://rclone.org/s3/ **
Choose your S3 provider.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 12 / Wasabi Object Storage
   \ "Wasabi"
provider> 12

1.3 Configure Authentication. Input “1”, press “Enter”, Input your Wasabi Access and Secret key when prompted. See our KB article for Generating Access/Secret Keys if you have not yet created these yet.

Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth> 1
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
access_key_id> E0WXU6X75BEAQLRZ7DCG
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> nW293nIZwbIvPXUU08L4fW7Fvbq6UZnHE5a3NEeS

1.4 Configure Region. Input “1” for “Region”, press “Enter”, input the service/endpoint URL for the region in which you have created your bucket. See our Service URLs KB for a list of possible URL endpoints to use. Leave “Location Constraint” blank, then press “Enter”.

Region to connect to.
Leave blank if you are using an S3 clone and you don't have a region.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Use this if unsure. Will use v4 signatures and an empty region.
   \ ""
 2 / Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH.
   \ "other-v2-signature"
region> 1
Endpoint for S3 API.
Required when using an S3 clone.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Wasabi US East endpoint
   \ "s3.wasabisys.com"
 2 / Wasabi US West endpoint
   \ "s3.us-west-1.wasabisys.com"
 3 / Wasabi EU Central endpoint
   \ "s3.eu-central-1.wasabisys.com"
endpoint> s3.us-east-1.wasabisys.com
Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a string value. Press Enter for the default ("").
location_constraint>

1.5 Configure ACL. Input “1” to set uploaded objects to private, then press “Enter”.

Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server side copying objects as S3 
doesn't copy the ACL from the source but rather writes a fresh one.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
   \ "private"
 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
   \ "public-read"
   / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
 3 | Granting this on a bucket is generally not recommended.
   \ "public-read-write"
 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access.
   \ "authenticated-read"
   / Object owner gets FULL_CONTROL. Bucket owner gets READ access.
 5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
   \ "bucket-owner-read"
   / Both the object owner and the bucket owner get FULL_CONTROL over the object.
 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
   \ "bucket-owner-full-control"
acl> 1

1.6 Finalize Configuration. Input “n” to decline editing advanced config, press “Enter”, then confirm choices. Input “y”, then press “Enter” to finalize the configuration. Finally input “q”, then press “Enter” to quit configuration.

Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
--------------------
[wasabi-us-east-1]
provider = Wasabi
env_auth = false
access_key_id = E0WXU6X75BEAQLRZ7DCG
secret_access_key = nW293nIZwbIvPXUU08L4fW7Fvbq6UZnHE5a3NEeS
endpoint = s3.us-east-1.wasabisys.com
acl = private
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Current remotes:
Name                 Type
====                 ====
wasabi-us-east-1     s3

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

2. Rclone Operation Examples

2.1 Copy Data

Copy a source directory to a remote directory in your Wasabi bucket. The files will exist in both locations.

Basic Command Construction

rclone copy source:path destination:path

Example Usage

Copy local “rclone-demo-directory”, and its contents to remote “wasabi-us-east-1:rclone-demo-bucket/rclone-demo-directory/”.

# ls -alh rclone-demo-directory/
total 28K
drwxr-xr-x 2 root root 4.0K Oct 17 11:32 .
drwx------ 6 root root 4.0K Oct 17 11:31 ..
-rw-r--r-- 1 root root   11 Oct 17 11:17 demo-object-01.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-02.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-03.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-04.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-05.txt
# rclone copy rclone-demo-directory/ wasabi-us-east-1:rclone-demo-bucket/rclone-demo-directory/
# rclone ls wasabi-us-east-1:rclone-demo-bucket/
       11 rclone-demo-directory/demo-object-01.txt
       11 rclone-demo-directory/demo-object-02.txt
       11 rclone-demo-directory/demo-object-03.txt
       11 rclone-demo-directory/demo-object-04.txt
       11 rclone-demo-directory/demo-object-05.txt

2.2 Move Data

Move a source directory to a remote directory in your Wasabi bucket. The files will exist only in the destination location.

Basic Command Construction

rclone move source:path destination:path

Example Usage

Move the contents of local “rclone-demo-directory” to remote “wasabi-us-east-1:rclone-demo-bucket/rclone-demo-directory/”.

# ls -alh rclone-demo-directory/
total 28K
drwxr-xr-x 2 root root 4.0K Oct 17 11:32 .
drwx------ 6 root root 4.0K Oct 17 11:31 ..
-rw-r--r-- 1 root root   11 Oct 17 11:17 demo-object-01.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-02.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-03.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-04.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-05.txt
# rclone move rclone-demo-directory/ wasabi-us-east-1:rclone-demo-bucket/rclone-demo-directory/
# ls -alh rclone-demo-directory/
total 8.0K
drwxr-xr-x 2 root root 4.0K Oct 17 11:53 .
drwx------ 6 root root 4.0K Oct 17 11:31 ..
# rclone ls wasabi-us-east-1:rclone-demo-bucket/
       11 rclone-demo-directory/demo-object-01.txt
       11 rclone-demo-directory/demo-object-02.txt
       11 rclone-demo-directory/demo-object-03.txt
       11 rclone-demo-directory/demo-object-04.txt
       11 rclone-demo-directory/demo-object-05.txt

2.3 Move Data Older Than 90 Days

Move data from the local source directory to the remote Wasabi bucket, only if the data is older than 90 days. The 90 days and older files will only exist in the destination location, and the newer files will only exist in the source location.

Basic Command Construction

rclone --min-age 90d move source:path destination:path

Example Usage

Move the contents of local “rclone-demo-directory” older than 90 days to remote “wasabi-us-east-1:rclone-demo-bucket/rclone-demo-directory/”.

# ls -alh rclone-demo-directory/
total 28K
drwxr-xr-x 2 root root 4.0K Oct 17 11:55 .
drwx------ 6 root root 4.0K Oct 17 11:31 ..
-rw-r--r-- 1 root root   11 Jan  1  2022 demo-object-01.txt
-rw-r--r-- 1 root root   11 Jan  2  2022 demo-object-02.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-03.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-04.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-05.txt
# rclone --min-age 90d move rclone-demo-directory/ wasabi-us-east-1:rclone-demo-bucket/rclone-demo-directory
# ls -alh rclone-demo-directory/
total 20K
drwxr-xr-x 2 root root 4.0K Oct 17 12:56 .
drwx------ 6 root root 4.0K Oct 17 11:31 ..
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-03.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-04.txt
-rw-r--r-- 1 root root   11 Oct 17 11:32 demo-object-05.txt
# rclone ls wasabi-us-east-1:rclone-demo-bucket/
       11 rclone-demo-directory/demo-object-01.txt
       11 rclone-demo-directory/demo-object-02.txt

2.4 Mount a Wasabi Bucket in a Local Directory

Links a local file system path and a remote Rclone location together, so that file system operations (read/write/delete/browse/etc) can be performed on the data that resides in the remote location, from the context of a local file browser.

Basic Command Construction

rclone mount source:path destination:path

Example Usage

1. Create a new empty directory.

2. Mount using rclone mount, and background the process.

3. Confirm mount by listing the local directory path.

4. Unmount with “fusermount -u mountpath”. (removes the mount)

# mkdir rclone-mount
# ls -alh rclone-mount/
total 8.0K
drwxr-xr-x 2 root root 4.0K Oct 17 13:02 .
drwx------ 7 root root 4.0K Oct 17 13:02 ..
# rclone mount wasabi-us-east-1:rclone-demo-bucket/ rclone-mount/ &
[1] 1007146
# ls -alh rclone-mount/
total 0
drwxr-xr-x 1 root root 0 Oct 17 13:02 rclone-demo-directory
# ls -alh rclone-mount/rclone-demo-directory/
total 1.0K
-rw-r--r-- 1 root root 11 Jan  1  2022 demo-object-01.txt
-rw-r--r-- 1 root root 11 Jan  2  2022 demo-object-02.txt
# ls -alh rclone-mount/
total 8.0K
drwxr-xr-x 2 root root 4.0K Oct 17 13:02 .
drwx------ 7 root root 4.0K Oct 17 13:02 ..
[1]+  Done                    rclone mount wasabi-us-east-1:rclone-demo-bucket/ rclone-mount/

2.5 Additional Examples
Please see Rclone's Subcommand Documentation for more examples.