How do I use AWS Lambda with Wasabi?

AWS Lambda is validated for use with Wasabi S3 as the backend. Lambda can be used as a virtual function that you can run on demand and they can scale automatically.

You can integrate this service and use Wasabi S3 to store and process your data. You can create a Lambda function to trigger any events in many programming languages. As the Lambda function requires IAM roles to operate, below is a demonstration of how one can create an IAM role and Lambda function to talk to Wasabi.

------------------------------------------------------------------------------------------------------

Inserting a JSON file in Wasabi S3 using Lambda function

1. Login into Wasabi account and create a bucket on Wasabi (you can also use pre-existing buckets). For this example, we have created a new bucket called "pankaj-lambda-test-with-wasabi" in the us-east-2 region. You can choose any name and region

2. Create IAM Role in AWS account - since this is a PUT operation, we need to make sure that the Lambda function has at least PutObject permission to perform this operation.

(Note: If you have not worked with IAM Policies and Roles previously, it is highly recommended that you go through the official documentation and practice on the console.)

We can create this role using the policy editor in AWS

• Log into your AWS account, go to the IAM service and click on Role

• Then click on "create role"

• Choose "Lambda" and hit "Next: Permissions" 

• click on "Create Policy" and it will open a separate window

Note: You can also attach an existing s3 policy like "AmazonS3FullAccess" if you don't wish to create one, but it is highly recommended to always give the least privileges/permissions to any resources to get the job done. Depending on your Lambda function requirements, you can allow appropriate permission.

Select Service as "S3"

Action: PutObject

Resource: Wasabi Bucket ARN or select All resources (depending upon your use case) and hit "Review Policy"

Name the Policy and hit "Create Policy". In this example we are using a name for this policy as "S3PutPolicy", you can choose any name

For reference, here's how this policy looks like:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "VisualEditor0",
 "Effect": "Allow",
 "Action": "s3:PutObject",
 "Resource": "arn:aws:s3:::pankaj-lambda-test-with-wasabi/*"
 }
 ]
}

Now once the policy is created, close that window and go back to Create role page, type S3 in the filter and refresh this page to see the newly created policy. Select the check box for that policy and hit "Next:Tags".

You can optionally give a tag but not necessary, hit "Next: Review", give a name to the Role, and hit "Create Role"

Note: In this example, we are naming our lambda role as "S3PutObjectRole", you can choose any name

Once the role is created, navigate to the role and click on "Attach policies" to attach one more policy called "AWSLambdaBasicExecutionRole" for debugging purposes (optional)

3. Creating the Lambda Function:

Go to Lambda service in your AWS account and click on "create functions"

Give a function name, we are naming "PutObjectTest", you can choose any name

Runtime: we are using Python 3.6, you can choose any language are you writing lambda function in

Execution Role: select from the existing role which we specifically created for this test and then hit "Create function"

Now, upload your own lambda function code or you can paste your code in the box

Below is a sample code example which we are using:

Actual code to test:

Note that this code example discusses the use of Wasabi's us-east-2 storage region. To use other Wasabi storage regions, please use the appropriate Wasabi service URL as described here.

import json
import boto3

s3 = boto3.client('s3',
endpoint_url = 'https://s3.us-east-2.wasabisys.com',
aws_access_key_id='Wasabi-Access-Key',
aws_secret_access_key='Wasabi-Secret-Key')

def lambda_handler(event, context):
 bucket ='Wasabi-Bucket-Name'

Company = {}
 Company['Company Name'] = 'Wasabi Technologies LLC'
 Company['Address'] = '111 Huntington Avenue Boston, MA 02199'
 Company['Phone'] = '+1 617-307-7912'

fileName = 'Wasabi' + '.json'
uploadByteStream = bytes(json.dumps(Company).encode('UTF-8'))
s3.put_object(Bucket=bucket, Key=fileName, Body=uploadByteStream)

print('Put Complete')

Once function code is uploaded/typed/pasted, go ahead and save this (top right button)

Now click on "Test" button as shown below to create a test event

Give any Event Name and click on "Create"

Now if you click "Test" again, you can see that the Lambda function has been executed and you can check complete logs

This function has successfully triggered an event to Wasabi S3 and inserted a JSON document specified by the code (screenshot below)

------------------------------------------------------------------------------------------------------

Reading from a file stored in Wasabi S3 using Lambda function

Consider a scenario wherein you have a large file with data and you are looking to parse through the body of that file and read-only particular data that you need for your use case through a Lambda function. Here how's you can trigger this event:

1. Start with creating a bucket in the Wasabi account and upload that large data file. 

For this example, we are using the same bucket from earlier called "pankaj-lambda-test-with-wasabi" in the us-east-2 region. Our sample file is shown below, you can download this file at the bottom of this article for reference

Our goal with this Lambda function is to only get:

• Total Deleted Storage Bytes

• Total Raw Storage Bytes

2. Create Lambda function:

Go to Lambda service in your AWS account and click on "create functions"

Give this function a name: we are naming it "QueryingWasabiS3"

Runtime: we are using Python 3.6, you can choose any language are you writing lambda function in

Permissions: Note that this time, instead of creating a role prior to the Lambda function, we are creating a new role from AWS policy templates because S3 object read-only permission is already an in-build template in AWS. You can either use that or make a role with sufficient read permissions as shown earlier and attach that here.

Give a role name: we are naming this role as "S3ReadOnlyRole", you can choose any name and hit "Create function"

Now, upload your own lambda function code or you can paste your code in the box

Below is a sample code example which we are using:

Actual code to test:

Note that this code example discusses the use of Wasabi's us-east-2 storage region. To use other Wasabi storage regions, please use the appropriate Wasabi service URL as described here

import json
import boto3

s3 = boto3.client('s3',
endpoint_url = 'https://s3.us-east-2.wasabisys.com',
aws_access_key_id='Wasabi-Access-Key',
aws_secret_access_key='Wasabi-Secret-Access-Key')

def lambda_handler(event, context):
 bucket = 'Wasabi-Bucket-Name'
 key = 'file-name'

response = s3.get_object(Bucket=bucket, Key=key)

content = response['Body']

jsonObject = json.loads(content.read())

billing = jsonObject['billing']

for record in billing:
 print("Total Deleted Storage Bytes are: " + str(record['DeletedStorageSizeBytes']))
 print("Total Raw Storage Bytes are: " + str(record['RawStorageSizeBytes']))
 print("---")

Once function code is uploaded/typed/pasted, go ahead and save this (top right button)

Now click on the "Test" button as shown below to create a test event

Give any Event Name and click on "Create"

Now if you click "Test" again, you can see that Lambda function has been executed and you can check complete logs

Below is the execution results that shows Lambda has fetched the particular data defined in the function: