How to write a Dictionary to JSON file in S3 Bucket using boto3 and Python

If you want to write a python dictionary to a JSON file in S3 then you can use the code examples below.

There are two code examples doing the same thing below because boto3 provides a client method and a resource method to edit and access AWS S3.

Writing Python Dictionary to an S3 Object using boto3 Client

import boto3
import json
from datetime import date

data_dict = {
    'Name': 'Daikon Retek',
    'Birthdate': date(2000, 4, 7),
    'Subjects': ['Math', 'Science', 'History']
}

# Convert Dictionary to JSON String
data_string = json.dumps(data_dict, indent=2, default=str)


# Upload JSON String to an S3 Object
client = boto3.client('s3')

client.put_object(
    Bucket='radishlogic-bucket', 
    Key='s3_folder/client_data.json',
    Body=data_string
)
Continue reading How to write a Dictionary to JSON file in S3 Bucket using boto3 and Python

How to upload a file to S3 Bucket using boto3 and Python

There are 3 ways to upload or copy a file from your local computer to an Amazon Web Services (AWS) S3 Bucket using boto3. All of these will be discussed in this post including multipart uploads.

The codes below will work if you are Windows, Mac, and Linux. Will also work if you working in a Lambda function using Python.

Each method will have an example using boto3 S3 client and S3 resource so you can use whatever method you are comfortable with.


Continue reading How to upload a file to S3 Bucket using boto3 and Python

How to write Python string to a file in S3 Bucket using boto3

To write a file from a Python string directly to an S3 bucket we need to use the boto3 package.

There are 2 ways to write a file in S3 using boto3. The first is via the boto3 client, and the second is via the boto3 resource. Both of these methods will be shown below.

S3 objects and keys

If you are new to AWS S3, you might be confused with some of the terms. So we’ll define some of them here. If you already know what objects and keys are then you can skip this section.

S3 objects are the same as files. When we run the method put_object what it means is that we are putting a file into S3.

S3 keys are the same as the filename with its full path. So if we want to create an object in S3 with the name of filename.txt within the foobar folder then the key is foobar/filename.txt.

Now that we have clarified some of the AWS S3 terms, follow the details below to start writing Python strings directly to objects in S3.

Continue reading How to write Python string to a file in S3 Bucket using boto3

How to get the ARN of an S3 Bucket

Each resource in AWS has an Amazon Resource Name (ARN). An ARN is a unique identifier of your resource. Its value has no duplicate in other accounts and only exists in your account.

It’s used especially in IAM policies where you set which resources you will allow access to.

You can actually predict the ARN of an S3 Bucket since it has a standard format of arn:aws:s3:::S3_BUCKET_NAME.

But if you are like me who is afraid of making a mistake typing the S3 bucket ARN, I prefer going to the AWS Console, searching for the S3 Bucket ARN, and copy-pasting it.

Follow the instructions below to get the S3 Bucket ARN.

Continue reading How to get the ARN of an S3 Bucket

Minimum IAM Permission to create S3 presigned URLs

If you wanted to publicly share a file or an object inside a private S3 bucket you will need to create an S3 presigned URL. This will create a temporary link to the S3 file which you can share and access publicly.

As best practice, we must apply the least privileged permission to the IAM user or IAM role that will create the S3 presigned URL. Which brings us to the question, what is the minimum IAM permission to create an S3 presigned URL?

Continue reading Minimum IAM Permission to create S3 presigned URLs

How to download all files in an S3 Bucket using AWS CLI

There are many ways to download files from an S3 Bucket, but if you are downloading an entire S3 Bucket then I would recommend using AWS CLI and running the command aws s3 sync s3://SOURCE_BUCKET LOCAL_DESTINATION.

In the examples below, I’m going to download the contents of my S3 Bucket named radishlogic-bucket.

My S3 Bucket in the AWS Console
My S3 Bucket in the AWS management console


Example 1: Download S3 Bucket to Current Local Folder

If you want to download the whole S3 Bucket in the same folder that you are in, then you should use the command aws s3 sync s3://SOURCE_BUCKET ..

In our example S3 Bucket above, the AWS CLI will be like this.

Continue reading How to download all files in an S3 Bucket using AWS CLI

How to create IAM User Access Keys via AWS CLI

To create programmatic Access Keys for an AWS IAM User using AWS CLI, run the command aws iam create-access-key.

On the command below change MyUser with the username of your target IAM User.

aws iam create-access-key --user-name MyUser

This will return the following JSON formatted string.

Continue reading How to create IAM User Access Keys via AWS CLI

How to create IAM User Access Keys using AWS Console

If you want to be able to control your AWS resources on your local computer you will either use AWS CLI or AWS SDK. To use those tools, you will need to have an Access Key ID and a Secret Access Key.

In this post, we will show you how you can generate your own Access Keys so you can programmatically access your AWS resources.

For the instructions later the target username that I want to create Access Keys is rabano. Yours will be different.

Continue reading How to create IAM User Access Keys using AWS Console

How to get the remaining time of a running AWS Lambda Function using Node.js

To get the remaining time of a running Lambda Function using Node.js, we will use the context object’s getRemainingTimeInMillis() function. This returns the number of milliseconds that the Lambda Function can still run before it times out.

Below is a simple code that fetches the remaining time inside a Lambda Function. I have set the timeout to be only 3 seconds, that is why the output is 2,999 milliseconds or approximately 3 seconds.

Continue reading How to get the remaining time of a running AWS Lambda Function using Node.js