AWS S3 Archives - Radish Logic

How to list all versions of a single object file in an AWS S3 Bucket using Python boto3

To get the object versions of a single object in AWS S3, we will need to use the list_object_versions() method of boto3. Below are 3 methods to list all the versions of a single S3 object using Python and boto3.

Example 1: Code to list all versions of an S3 object file using boto3 resource
Example 2: Code to list all versions of an S3 object file using boto3 client paginator
Example 3: Code to list all versions of an S3 object file using boto3 client Next Marker

You can scroll to the codes below to quickly access the Python scripts.

All 3 codes do the same thing. The function get_file_versions_list() will accept the bucket name (bucket_name) and the target S3 object key (object_key), then it will use boto3 to get the list of object versions and delete markers of the target S3 object. It will then sort that list from latest to oldest, and then count the number of versions the object has.

Getting the list of versions and the delete markers was not as straight as I thought it would be. In the latter part of this article, I will be discussing the complexity of the prefix parameter and the delete markers.

How to stop object file versioning in an S3 Bucket using Python boto3

To stop versioning in an S3 Bucket, we can use the boto3 method put_bucket_versioning() with the VersioningConfiguration parameter to be set to a Status of Suspended.

Below are 3 methods on how we can stop versioning in an S3 Bucket using AWS-SDK for Python, boto3.

The Python scripts below do the same thing: suspend the versioning of the target S3 Bucket named ‘radishlogic-bucket’.

You may use any method that you like depending on which you are comfortable using.

Interestingly, suspending versioning uses the same boto3 method as enabling versioning, put_bucket_versioning(). The only difference is that the ‘Status’ is ‘Suspended’ instead of ‘Enabled’.

How to enable object file versioning on an S3 Bucket using Python boto3

To enable object versioning in an S3 Bucket using Python boto3, you can use the put_bucket_versioning() function. This guide focuses on utilizing this method to control object versions effectively, ensuring data integrity and secure file handling within your AWS S3 environment.

Below are 3 ways to enable versioning in a target S3 Bucket using AWS-SDK for Python, boto3.

The following examples achieve the same goal: enabling versioning for the AWS S3 Bucket named ‘radishlogic-bucket’.

Feel free to employ any of these methods in your project based on your comfort level.

You can also check if versioning is enabled on your S3 bucket via Python boto3.

How to check if versioning is enabled in an S3 bucket using Python boto3

To check if versioning is enabled in an S3 Bucket using Python boto3, we will need to use the get_bucket_versioning() method of boto3 S3.

Below are 3 ways to code how to get the S3 Bucket versioning status using AWS-SDK for Python, boto3.

The Python scripts below all do the same thing. They check the status of versioning in the target S3 Bucket named radishlogic-bucket.

You can choose whichever method you are comfortable with.

Possible output values for checking versioning status of an S3 Bucket

How to list all objects in an S3 Bucket using boto3 and Python

If you need to list all files/objects inside an AWS S3 Bucket then you will need to use the list_objects_v2 method in boto3.

Below are 3 example codes of how to list all files in a target S3 Bucket.

You can use any of the 3 options since it does the same thing.

It will get all of the files inside the S3 Bucket radishlogic-bucket using Python boto3, put it inside a Python list, then print each object key. It will print the files inside folder recursively, regardless if they are inside a folder or not.

At the end, it will also print the number of items inside the S3 Bucket.

How to delete a file in AWS S3 using boto3 and Python

To delete a file inside an AWS S3 Bucket using Python then you will need to use the delete_object function of boto3.

Below are 3 examples to delete an S3 file.

You can use any of the 3 options since it does the same thing. It will delete the file in S3 with the key of s3_folder/file.txt inside the S3 bucket named radishlogic-bucket using Python boto3.

How to write Python string to a file in S3 Bucket using boto3

To write a file from a Python string directly to an S3 bucket we need to use the boto3 package.

There are 2 ways to write a file in S3 using boto3. The first is via the boto3 client, and the second is via the boto3 resource. Both of these methods will be shown below.

S3 objects and keys

If you are new to AWS S3, you might be confused with some of the terms. So we’ll define some of them here. If you already know what objects and keys are then you can skip this section.

S3 objects are the same as files. When we run the method put_object what it means is that we are putting a file into S3.

S3 keys are the same as the filename with its full path. So if we want to create an object in S3 with the name of filename.txt within the foobar folder then the key is foobar/filename.txt.

Now that we have clarified some of the AWS S3 terms, follow the details below to start writing Python strings directly to objects in S3.

How to read a JSON file in S3 and store it in a Dictionary using boto3 and Python

If you want to get a JSON file from an S3 Bucket and load it into a Python Dictionary then you can use the example codes below.

There are 4 scenarios for the examples scripts below.

AWS boto3 provides 2 ways to access S3 files, the boto3.client('s3') and boto3.resource('s3'). For each of the example scenarios above, a code will be provided for the two methods.

Since both methods will function the same, you can choose whichever method you like.

How to write a Dictionary to JSON file in S3 Bucket using boto3 and Python

If you want to write a python dictionary to a JSON file in S3 then you can use the code examples below.

There are two code examples doing the same thing below because boto3 provides a client method and a resource method to edit and access AWS S3.

Writing Python Dictionary to an S3 Object using boto3 Client

import boto3
import json
from datetime import date

data_dict = {
    'Name': 'Daikon Retek',
    'Birthdate': date(2000, 4, 7),
    'Subjects': ['Math', 'Science', 'History']
}

# Convert Dictionary to JSON String
data_string = json.dumps(data_dict, indent=2, default=str)


# Upload JSON String to an S3 Object
client = boto3.client('s3')

client.put_object(
    Bucket='radishlogic-bucket', 
    Key='s3_folder/client_data.json',
    Body=data_string
)

How to download files from S3 Bucket using boto3 and Python

If you want to download a file from an AWS S3 Bucket using Python, then you can use the sample codes below.

The codes below use AWS SDK for Python named boto3.

boto3 provides three methods to download a file.

download_file()
download_fileobj() – with multipart upload
get_object()

Then for each method, you can use the client class or the resource class of boto3.

Both of the classes will be used for each of the methods above.

Note: All examples will work with any Python3 environment running in Windows, MacOS or Linux operating systems.