Getting Started With Boto3 - AWS SDK for Python

Boto3.jpg

Python is da best! Python makes automating daily repetitive tasks and creating free time for more interesting and meaningful activities possible. While python is awesome as a programming language, what makes super useful is large selection of libraries, tools and frameworks that provide solutions for any problems we may have regarding automating various tasks and projects. Boto3 is one of the super useful tools I have recently discovered.

You use the AWS SDK for Python (Boto3) to create, configure, and manage AWS services, such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3). The SDK provides an object-oriented API as well as low-level access to AWS services.

My favorite python web framework is Streamlit, because it makes publishing and sharing web apps for simple tasks easy without a need for spending countless hours in web development. Since my introduction to Streamlit I managed to create more than a dozen apps specific to certain tasks and projects. A couple of them are Hive based apps. But most are for other projects that involve data manipulation, data analysis and automating. Some of these projects involve working with a lot of files in pdf and excel format, and creating reports. Python has solutions for these tasks as well with libraries like pdfplumber, openpyxl, etc.

Combining all of these tools and heroku, creating shareable apps becomes super easy. One problem I haven't been able to solve until now was storing files used for these projects in the cloud, so that they are accessible anywhere and can be shared with teams. There are many file sharing options in general. However, what I was looking for was the ability for the scripts to have access to these files, create new files, and also store them in a way that they are available whenever needed.

Unfortunately, turns out Heroku doesn't offer storing files options. So, the apps deployed on Heroku wouldn't be able to also store these files. What Heroku does offer it seems an easy integration with AWS's S3 cloud storage. I haven't used Amazon's AWS or S3 before and had no clue how to get started. After quick research I came across Boto3 and finally I think I will be able to add improvements to the projects I am working on.

Streamlit app running on Heroku actually does have uploading and downloading files solutions, which I already use. What I couldn't do was to store them for future use. Store could only be done on a local machine, which wouldn't help because it would only be accessible to the user of the machine. By integrating Amazon's S3 cloud storage to the app, I think this problem will be solved. For this I would need to create an AWS S3 account. They offer a free tier options. That's great to experiment while I am trying to learn how it works. Boto3 will allow accessing, uploading, downloading files stores in AWS S3 programmatically within the python script.

Before we do any programming, we would need to create AWS S3 account, which I just did a couple of hours ago. It was super fast. They offer free tier option. However, they also say free tier can be used for next 12 months. I am not sure if after 12 months it will need to be upgraded. Full year of use is actually not bad, and plenty of time to get familiar with how everything works. If everything works as expected, paying for the services wouldn't be a big deal.

After creating an AWS S3 account, we will need to create a "Bucket", which is a container where we can store all of our files or objects. That was simple to do as well. Next, to be able to connect to our S3 we will need public and private keys or as they word it access_key_id and secret_access_key. The root user doesn't seem to have these keys for security purposes, or I wasn't able to find it. But we can create new users, assign permissions to the new user and create the new user with these keys. Once new user for our S3 created, it displays the keys and we will need to store them somewhere for future use within our python script. After creating S3 account, new bucket and a new user with keys, I uploaded a file. Everything is ready in S3 side, it is time to write a script. Nothing fancy, nothing complicated. At this point I just want to make sure Boto3 works, can actually connect to S3 and do some actions.

Before starting with a python script, I had to pip3 install boto3, so that it can be used within the script. Everything went without any problems, at the first try. Below is the simple script I tested to see if I could download the file I uploaded earlier.

from secrets import access_key_id, secret_access_key, bucket_name
import boto3

session = boto3.Session(aws_access_key_id = access_key_id, 
            aws_secret_access_key = secret_access_key)

s3 = session.resource('s3')
my_bucket = s3.Bucket(bucket_name)
for s3_object in my_bucket.objects.all():
    my_bucket.download_file(s3_object.key, 'Desktop/myfile.pdf')

I ran the script, and the file appeared on my desktop. Success!

I know there isn't much going on in the code above. It is simply downloading the file from the bucket in my S3 account. But the ability to do this simple action programmatically opens the possibilities for more interesting, meaningful, and useful scripts. When learning how to use a new library or a tool, the most frustrating and annoying part is to get things working at the beginning. Because when they don't work at the start, it is more difficult to figure out the reasons. Setting up an S3 account and getting started with Boto3 went smooth and everything worked at the first time.

Next step is to learn more the functionalities of Boto3 and integrating these functions to my existing projects. Hopefully, things will continue to progress smoothly. Boto3 documentation is very useful and easily to follow. If you have used Boto3, make sure to share some tips and tricks in the comments.

Posted Using LeoFinance Beta



0
0
0.000
25 comments
avatar

Indeed very helpful post, python is my favourite programming language!

0
0
0.000
avatar

Sorry, what does AWS mean? Lately I see it written everywhere

0
0
0.000
avatar

Amazon Web Services

0
0
0.000
avatar

thanks for the reply… ankther question, with this service they sell things like virtual computers?

0
0
0.000
avatar

As the name suggests they offer web services like cloud storage and computing. I don’t know much about them, first time trying to use.

0
0
0.000
avatar

Python seems good to learn. There's an app that you can play like a game. But it's actually teaching you to code !

0
0
0.000
avatar

Yes, there are many educational resources and even games for python.

0
0
0.000
avatar

I think python makes things a lot easier and I have used AWS a few times. It does solve the issues as you say but it's still annoying to combine all the applications together.

Posted Using LeoFinance Beta

0
0
0.000
avatar

As long as the end result works, they are worth using.

0
0
0.000
avatar

Python makes automating daily repetitive tasks and creating free time for more interesting and meaningful activities possible.

Like what? finding more tasks to automate with python? ;)

0
0
0.000
avatar

Like working on PineScript projects. :)

0
0
0.000
avatar

I will just comment on the very first sentence of this blog, as I have not any opinion on Boto3 as I don't need to use such a service for my projects. Therefore, I will go with a kind of off-topic comment, if you don't mind. If you do, I hope you will apologise me ;)

Python is da best!

I disagree. Python is great, clearly, but for certain things only. For others things, for instance heavy computing tasks, Python is among the worst. An interesting table with this respect is provided here, in which comparisons of programming languages are made in terms of energy, time, and memory usage.

0
0
0.000
avatar

You are right. It was more of a hyperbole.

0
0
0.000
avatar
(Edited)

I wish I could relate to all this programming stuff!

0
0
0.000
avatar

Yes, Python can make work more interesting and simple.

0
0
0.000
avatar

Nice. Sounds all good. When did u get your start with python and other coding?

0
0
0.000
avatar

I started coding here on Hive. It started with ideas of creating bots to help with curation. Some of the curators I interacted with daily were/are programmers, they helped me getting started. Then my interest in coding continued as I wanted to learn more how the blockchain worked. I still don't know much. Just the basics.

0
0
0.000
avatar

This is a nice article, do you organize tutorial for people who may wish to learn python?

0
0
0.000
avatar

There is abundance of free resources online to learn python.

0
0
0.000