Dell Powerscale SmartSync With Wasabi Dell ObjectScale

Prev Next

How do I use Dell Powerscale SmartSync with Wasabi Dell ObjectScale?

Dell PowerScale SmartSync is a data replication and mobility solution integrated into Dell’s OneFS operating system. It facilitates efficient and secure data movement between PowerScale clusters and Wasabi cloud object storage. SmartSync is beneficial for organizations aiming to implement hybrid cloud strategies, enhance disaster recovery, or stramline data archiving processes.

Prerequisites

  • An active Wasabi with Dell Object Storage Account

  • A Wasabi bucket created in the appropriate Wasabi OBS Storage Vault along with a user with available Access/Secret Keys.

  • Minimum PowerScale software release 9.11 and above. Please check with your Dell representative on recommended release for SmartSync

  • PowerScale’s external network is publicly routable

  • Port 9021 (HTTPS) for S3 API

  • SyncIQ license for your PowerScale OneFS.

  • A CA configured on the Cluster

Reference Architecture Diagram

SmartSync Configuration

In order to use the SmartSync feature you will need to activate the SmartSync Daemon. A CA must be configured on the PowerScale cluster following the steps documented in the Dell PowerScale SmartSync KB Configuration Encryption section.

If a CA is not properly installed you will get an error as described in the Troubleshooting Unconfigured CA section.

Enable the SmartSync (Datamover) Service

After your CA is properly configured as per Dell’s documentation. Activate the Datamover service on your PowerScale cluster

isi services -a isi_dm_d enable

Create a SmartSync Cloud Account for Wasabi

Create an ECS_S3 Account type pointing to your Wasabi ObjectScale bucket.

Note - For the Wasabi URI you will need to ensure to include nns in the string to indicate NoNameSpace

isi dm accounts create ECS_S3 https://s3-nns.us-east-1-dell-obs.wasabisys.com:9021/(BUCKET NAME) --name (UNIQUE ACCOUNT NAME) --auth-mode CLOUD --access-id (ACCESS KEY) --secret-key (SECRET KEY)

isi dm accounts view (ACCOUNT NUMBER)

Create Policies


Create a Dataset Creation Policy

isi dm policies create (UNIQUE POLICY NAME) --priority='NORMAL' --enabled='true' --policy-type='CREATION' --creation-account-id=(LOCAL ACCOUNT ID) --creation-base-path=(SOURCE PATH) --creation-dataset-retention-period=600 --creation-dataset-reserve=2 --creation-dataset-expiry-action=DELETE

Create a Repeat-Copy Policy

isi dm policies create (UNIQUE POLICY NAME) --policy-type='REPEAT_COPY' --priority='NORMAL' --enabled='true' --repeat-copy-source-base-path=(SOURCE PATH) --repeat-copy-base-account-id=(LOCAL ACCOUNT ID) --repeat-copy-base-source-accont-id=(LOCAL ACCOUNT ID) --repeat-copy-base-target-account-id=(CLOUD ACCOUNT ID) --repeat-copy-base-target-base-path=(BUCKET NAME) --repeat-copy-base-target-dataset-type='FILE_ON_OBJECT_BACKUP' --repeat-copy-base-dataset-retention-period=1800 --repeat-copy-base-dataset-reserve=5 --repeat-copy-base-dataset-expiry-action=DELETE

Enable the Dataset Creation policy to run now so that it creates a dataset/snapshot.

isi dm policies modify (POLICY NUMBER) --run-now=true
isi dm jobs list

Enable the Repeat Copy policy to run now to start a backup job to your Wasabi bucket.

isi dm policies modify (POLICY NUMBER) --run-now=true
isi dm jobs list

isi dm jobs list

You can view the job by running “isi dm jobs view xx” and verify that the data was transferred.

isi dm jobs view (JOB NUMBER)

When using the Repeat Copy job the data is transferred to Wasabi ObjectScale storage in proprietary format. Using Cyberduck, S3 Browser or any other application you can view your OBS bucket and see the data is now successfully in your bucket. The data however will not be in a readable format and you will need to use SmartSync to view or restore the data.

Incremental Copy Job

After having run the initial baseline copy/backup job with the initial dataset moving forward any new jobs will now be an incremental type comparing the data against the original snapshot in the series and only uploading the changed files.

isi dm policies modify (POLICY NUMBER) --run-now=true
isi dm policies modify (POLICY NUMBER) --run-now=true
isi dm jobs list

Run “isi dm reports list” to show all the jobs ran and their job type.

isi dm reports list

SmartSync Restore

Restore data from Cloud to the Powerscale Cluster with either Partial or Full Restoration. Partial Restoration is to restore a subset of the directory structure that was specified under the source path. Full restoration is to restore the entire dataset.

Check and view the dataset id for restoration

isi_dm browse
list-accounts
connect-account (Cloud Account ID)
list-datasets
connect-dataset (Dataset ID)

Create a Partial Restoration Policy

Note: --copy-source-subpaths supports multiple listing = “hr”, “finance” and --copy-base-target-base-path which is a Target directory must be empty

isi dm policies create (UNIQUE POLICY NAME) --policy-type='COPY' --enabled='true' --priority='NORMAL' --copy-base-target-account-id=(LOCAL ACCOUNT ID) --copy-base-source-account-id=(SOURCE ACCOUNT ID) --copy-source-base-path=(SOURCE PATH) --copy-create-dataset-on-target='false' --copy-base-target-dataset-type='FILE' --copy-base-target-base-path=(TARGET PATH ON CLUSTER) --copy-dataset-id=(DATASET ID FOR RESTORATION) --copy-source-subpaths=(SUB PATH FOR RESTORATION)

Run the Partial Restoration Policy

isi dm policies list
isi dm policies modify (RESTORATION POLICY ID) --run-now=true

View the Data Restoration Job Status

isi dm jobs list

Once complete you can validate a successful restore by browsing to that local source path and seeing that the file(s) is now restored.

Create a Full Restoration Policy

If you want to restore a full dataset you will follow the same steps as above for Partial Restore just removing the Subpath command.

Note: --copy-base-target-base-path which is a Target directory must be empty

isi dm policies create (UNIQUE POLICY NAME) --policy-type'COPY' --enabled='true' --priority='NORMAL' --copy-base-target-account-id=(LOCAL ACCOUNT ID) --copy-base-source-account-id=(SOURCE ACCOUNT ID) --copy-source-base-path=(SOURCE PATH) --copy-create-dataset-on-target=(TRUE OR FALSE) --copy-base-target-dataset-type='FILE' --copy-base-target-base-path=(TARGET PATH ON CLUSTER) --copy-dataset-id=(DATASET ID FOR RESTORATION) 

Run the Partial Restoration Policy

isi dm policies list
isi dm policies modify (RESTORATION POLICY ID) --run-now=true

View the Data Restoration Job Status

isi dm jobs list

Remote Browse Datasets for Restore

Remote Browsing the datasets allows you to manually view the datasets and restore the data object by object.

Run “isi_dm browse” to enter the dataset commands.

Run “connect-accounts xxxx” where you input your ECS_S3 account number.

Run “list-datasets” to view all datasets created.

Run “connect-dataset x” and specify the number associated with the dataset you want to view.

Run “ls” to list all the files in that dataset.

isi_dm browse
$ connect-account (CLOUD ACCOUNT NUMBER)
$ list-datasets
$ connect-dataset (DATASET ID)
$ ls

For testing purposes if you were to delete a file from one of your shares.

Now run the “download” command while connected to the dataset and restore it to the share.

Specify the file that you are wanting to download and specify the path that you want to restore it to.

When you go back to your share you will see the file back in place.

SmartSync Expiration Policy

Expiration Policies are needed to clean up expired datasets locally on the cluster (Source DM Account) and on the Target platform (Target DM Account)

isi dm policies create [Expiration Policy Name] --policy-type='EXPIRATION' --enabled='true' --priority='NORMAL' --copy-base-target-account-id=[Local account id] –-expiration-account-id [Source DM Account or Target DM Account] ––recurrence="cron expression" --start-time="YYYY-MM-DD HH:MM:SS"

Note: “dataset-reserve" field specifies how many datasets to keep in reserve and protected from expiration, irrespective of the "dataset-retention-period“ specified.