How to load a JSON file from S3 to a Python dictionary using boto3

If you want to get a JSON file from an S3 Bucket and load it into a Python Dictionary then you can use the example codes below.

There are 4 scenarios for the examples scripts below.

Basic JSON file from S3 to Python Dictionary
With Try/Except block
With datetime, date, and time conversions
Running the code in a Lambda Function

AWS boto3 provides 2 ways to access S3 files, the boto3.client('s3') and boto3.resource('s3'). For each of the example scenarios above, a code will be provided for the two methods.

Since both method will function the same, you can choose whichever method you like.

How to set the default screen resolution for VNC Viewer when Raspberry Pi is not connected to a monitor

A headless setup is when your Raspberry Pi is turned on without a monitor.

The display is not needed usually, especially if your access is only via SSH. The issue would be if you connect via VNC Viewer, the display will look like this.

The icons are too big but the screen is too small.

What’s happening here is that Raspberry Pi will use a very small screen resolution when it is not connected to a monitor. So when you connect via VNC Viewer, VNC will get the same small screen resolution.

To solve this, we need to change the default headless screen resolution of the Raspberry Pi. You can follow the instructions below to set this.

Changing the default screen resolution of a headless Raspberry Pi

How to connect to WiFi using nmcli

Below are commands that you can use to connect and check the status of your WiFi connection using nmcli.

I usually use this when connecting my Rock Pi running Ubuntu to the WiFi Network. But commands below will also work as long as you have the nmcli in your Linux System.

Commands

Turn On WiFi

# Turn on WiFi
nmcli r wifi on

Scan WiFi Networks

# Scan WiFi Networks
nmcli dev wifi

How to read a file in S3 and store it in a String using Python and boto3

If you want to get a file from an S3 Bucket and then put it in a Python string, try the examples below.

boto3, the AWS SDK for Python, offers two distinct methods for accessing files or objects in Amazon S3: client method and the resource method.

Option 1 uses the boto3.client('s3') method, while options 2 and 3 use the boto3.resource('s3') method.

All 3 options do the exact same thing so get the one that you feel comfortable with or the one that will fit your use case.

How to open Folder in Visual Studio Code using Ubuntu

There are two ways you can do to open a folder in Ubuntu and open it in Visual Studio Code.

Prerequisites

The steps below assume that you have installed the following prerequisites.

Ubuntu operating system. I tested using Ubuntu 22.04 but this will work in most latest Ubuntu Versions.
Visual Studio Code is already installed.

How to install PostgreSQL 15 on Ubuntu

If you try to run the command sudo apt install postgresql in Ubuntu it will install Postgresql version 14 instead of version 15. As of writing the Ubuntu repository does not include version 15.

Follow the tutorial below for a detailed guide to successfully installing PostgreSQL 15.

Step-by-Step Instructions on Configuring Postgresql 15 in Ubuntu

The following steps were tested in Ubuntu 22.04 LTS. If there are new versions of Ubuntu, I will test the steps again and update this post.

1. Installation

1.1 Add the Postgresql Package Repository to Ubuntu

Run the commands below to add the official Postgresql package repository to Ubuntu.

How to install a specific version of PostgreSQL in Ubuntu

If you run the command below in Ubuntu it will install the latest version of PostgreSQL in the Ubuntu repository.

# Installs the latest version of PostgreSQL
sudo apt install postgresql postgresql-client

As of writing it will install PostgreSQL version 14.

How to write a Dictionary to JSON file in S3 Bucket using boto3 and Python

If you want to write a python dictionary to a JSON file in S3 then you can use the code examples below.

There are two code examples doing the same thing below because boto3 provides a client method and a resource method to edit and access AWS S3.

Writing Python Dictionary to an S3 Object using boto3 Client

import boto3
import json
from datetime import date

data_dict = {
    'Name': 'Daikon Retek',
    'Birthdate': date(2000, 4, 7),
    'Subjects': ['Math', 'Science', 'History']
}

# Convert Dictionary to JSON String
data_string = json.dumps(data_dict, indent=2, default=str)


# Upload JSON String to an S3 Object
client = boto3.client('s3')

client.put_object(
    Bucket='radishlogic-bucket', 
    Key='s3_folder/client_data.json',
    Body=data_string
)

How to upload a file to S3 Bucket using boto3 and Python

There are 3 ways to upload or copy a file from your local computer to an Amazon Web Services (AWS) S3 Bucket using boto3. All of these will be discussed in this post including multipart uploads.

The codes below will work if you are Windows, Mac, and Linux. Will also work if you working in a Lambda function using Python.

Each method will have an example using boto3 S3 client and S3 resource so you can use whatever method you are comfortable with.

upload_file method
upload_fileobj method (supports multipart upload)
put_object method

How to write Python string to a file in S3 Bucket using boto3

To write a file from a Python string directly to an S3 bucket we need to use the boto3 package.

There are 2 ways to write a file in S3 using boto3. The first is via the boto3 client, and the second is via the boto3 resource. Both of these methods will be shown below.

S3 objects and keys

If you are new to AWS S3, you might be confused with some of the terms. So we’ll define some of them here. If you already know what objects and keys are then you can skip this section.

S3 objects are the same as files. When we run the method put_object what it means is that we are putting a file into S3.

S3 keys are the same as the filename with its full path. So if we want to create an object in S3 with the name of filename.txt within the foobar folder then the key is foobar/filename.txt.

Now that we have clarified some of the AWS S3 terms, follow the details below to start writing Python strings directly to objects in S3.