How to download files from S3 Bucket using boto3 and Python

If you want to download a file from an AWS S3 Bucket using Python, then you can use the sample codes below.

The codes below use AWS SDK for Python named boto3.

boto3 provides three methods to download a file.

download_file()
download_fileobj() – with multipart upload
get_object()

Then for each method, you can use the client class or the resource class of boto3.

Both of the classes will be used for each of the methods above.

Note: All examples will work with any Python3 environment running in Windows, MacOS or Linux operating systems.

How to upload a file to S3 Bucket using boto3 and Python

There are 3 ways to upload or copy a file from your local computer to an Amazon Web Services (AWS) S3 Bucket using boto3. All of these will be discussed in this post including multipart uploads.

The codes below will work if you are Windows, Mac, and Linux. Will also work if you working in a Lambda function using Python.

Each method will have an example using boto3 S3 client and S3 resource so you can use whatever method you are comfortable with.

upload_file method
upload_fileobj method (supports multipart upload)
put_object method

Convert boto3 AMI Creation Date from string to Python datetime

When retrieving the AMI Creation Date from boto3 it returns a string data type. Visually, this is okay but it is challenging to do operations and comparisons to the AMI Creation Date in this format.

To solve the issue we need to convert the AMI Creation Date from type string to datetime before we could do some operations.

The AMI Creation Date string looks like 2019-09-18T07:34:34.000Z. To convert this we need to use the strptime function from the datetime.datetime library.

How to open Folder in Visual Studio Code using Ubuntu

There are two ways you can do to open a folder in Ubuntu and open it in Visual Studio Code.

Prerequisites

The steps below assume that you have installed the following prerequisites.

Ubuntu operating system. I tested using Ubuntu 22.04 but this will work in most latest Ubuntu Versions.
Visual Studio Code is already installed.

How to install PostgreSQL 15 on Ubuntu

If you try to run the command sudo apt install postgresql in Ubuntu it will install Postgresql version 14 instead of version 15. As of writing the Ubuntu repository does not include version 15.

Follow the tutorial below for a detailed guide to successfully installing PostgreSQL 15.

Step-by-Step Instructions on Configuring Postgresql 15 in Ubuntu

The following steps were tested in Ubuntu 22.04 LTS. If there are new versions of Ubuntu, I will test the steps again and update this post.

1. Installation

1.1 Add the Postgresql Package Repository to Ubuntu

Run the commands below to add the official Postgresql package repository to Ubuntu.

How to install a specific version of PostgreSQL in Ubuntu

If you run the command below in Ubuntu it will install the latest version of PostgreSQL in the Ubuntu repository.

# Installs the latest version of PostgreSQL
sudo apt install postgresql postgresql-client

As of writing it will install PostgreSQL version 14.

EC2 with IAM Role: CloudFormation Sample Template

Creating an EC2 Instance with an IAM Role is easy when you do it via the AWS Console but doing this with CloudFormation is not as direct. You will need an Instance Profile to connect an EC2 with an IAM Role.

TL;DR: See the CloudFormation Template below.

How to get the ARN of an S3 Bucket

Each resource in AWS has an Amazon Resource Name (ARN). An ARN is a unique identifier of your resource. Its value has no duplicate in other accounts and only exists in your account.

It’s used especially in IAM policies where you set which resources you will allow access to.

You can actually predict the ARN of an S3 Bucket since it has a standard format of arn:aws:s3:::S3_BUCKET_NAME.

But if you are like me who is afraid of making a mistake typing the S3 bucket ARN, I prefer going to the AWS Console, searching for the S3 Bucket ARN, and copy-pasting it.

Follow the instructions below to get the S3 Bucket ARN.

Minimum IAM Permission to create S3 presigned URLs

If you wanted to publicly share a file or an object inside a private S3 bucket you will need to create an S3 presigned URL. This will create a temporary link to the S3 file which you can share and access publicly.

As best practice, we must apply the least privileged permission to the IAM user or IAM role that will create the S3 presigned URL. Which brings us to the question, what is the minimum IAM permission to create an S3 presigned URL?

How to download all files in an S3 Bucket using AWS CLI

There are many ways to download files from an S3 Bucket, but if you are downloading an entire S3 Bucket then I would recommend using AWS CLI and running the command aws s3 sync s3://SOURCE_BUCKET LOCAL_DESTINATION.

In the examples below, I’m going to download the contents of my S3 Bucket named radishlogic-bucket.

My S3 Bucket in the AWS Console — My S3 Bucket in the AWS management console

Example 1: Download S3 Bucket to Current Local Folder

If you want to download the whole S3 Bucket in the same folder that you are in, then you should use the command aws s3 sync s3://SOURCE_BUCKET ..

In our example S3 Bucket above, the AWS CLI will be like this.