Use boto3 to mess with S3

Introduction

In IoT related projects, it was necessary to download and upload files in S3 from the gateway (hereinafter GW), so try implementing it using python (2.7) and boto3 (AWS SDK for python). I did. I will keep a memorandum of the procedure at that time.

final goals

  1. You will be able to download the files in your bucket
  2. You will be able to upload files to a specific directory

Let's get started

installation of boto3

Log in to GW with ssh connection and install boto3 with pip. (GW has python2.7 pip installed)

sudo pip install boto3

If you want to specify a specific version, install as follows.

sudo pip install boto3==1.0.0

IAM creation

When using boto3, it is necessary to set a policy in IAM. This time I created an IAM user with S3FullAccess set. If necessary, set the minimum necessary policies such as reading only or narrowing down the buckets that can be accessed.

AWS CLI settings

Set the access key information for the IAM user you created earlier.

AWS CLI installation

sudo pip install awscli

Configure settings

aws configure
AWS Access Key ID: xxxxxxxxxxxxxxxxx
AWS Secret Access Key: xxxxxxxxxxxxxxxxx
Default region name: ap-northeast-1
Default output format:

This completes the settings. You can check the settings in "~ / .aws / credentials" and "~ / .aws / config".

Import boto3 in python

In order to import bote3 and mess with S3 files, start writing as follows.

boto3_test.py


# -*- coding: utf-8 -*-

import boto3

s3 = boto3.resource('s3')

If you want to use other services, write as follows. (DynamoDB example)

boto3_test.py


# -*- coding: utf-8 -*-

import boto3

# S3
s3 = boto3.resource('s3')

# DynamoDB
dynamo = bot3.resource('dynamodb')

Actually mess with S3 files

Download from S3

Use download_file (). When downloading fuga.txt in the bucket name hoge

boto3_test.py


# -*- coding: utf-8 -*-

import boto3

s3 = boto3.resource('s3')

bucket = s3.Bucket('hoge')
bucket.download_file('fuga.txt', 'fuga.txt')

The bucket is the same, if you want to download fuga.txt in the folder hogehoge

boto3_test.py


#---abridgement---

bucket = s3.Bucket('hoge')
bucket.download_file('hogehoge/fuga.txt', 'fuga.txt')

Upload to S3

Use upload_file (). When uploading fuga.txt in the bucket name hoge

boto3_test.py


#---abridgement---

bucket = s3.Bucket('hoge')
bucket.upload_file('fuga.txt', 'fuga.txt')

The bucket is the same, if you upload fuga.txt in the folder hogehoge

boto3_test.py


#---abridgement---

bucket = s3.Bucket('hoge')
bucket.upload_file('fuga.txt', 'hogehoge/fuga.txt')

You can now download and upload files in S3. By the way, it is to display each object information in the bucket

boto3_test.py


#---abridgement---

bucket = s3.Bucket('hoge')
print bucket.name

A list is displayed with.

Finally

I had a hard time because I hadn't touched python so much before using boto3, but the official documentation was pretty good. It was helpful. On the contrary, those who do not show rejection of English should read the official document. Sweat

Recommended Posts

Use boto3 to mess with S3
Use boto to upload / download files to s3.
S3 uploader with boto
S3 operation with python boto3
Python: How to use async with
How to use virtualenv with PowerShell
How to use FTP with Python
Connect to Elastic MQ with boto
I wanted to delete multiple objects in s3 with boto3
Generate S3 signed URL with boto
Use boto3 with temporary credentials with SAML authentication
How to use ManyToManyField with Django's Admin
How to use OpenVPN with Ubuntu 18.04.3 LTS
How to use Cmder with PyCharm (Windows)
[Introduction to Python] Let's use foreach with Python
How to use Japanese with NLTK plot
Try server-side encryption on S3 with boto3
How to use jupyter notebook with ABCI
How to catch boto3 S3 NoSuchKey error
[AWS] Link Lambda and S3 with boto3
How to use CUT command (with sample)
Connect to s3 with AWS Lambda Python
S3 server-side encryption SSE with Python boto3
How to use SQLAlchemy / Connect with aiomysql
How to use JDBC driver with Redash
How to deal with SSL error when connecting to S3 with boto of Python
Use Bucket (). Objects.filter instead of list_objects_v2 when outputting S3 list with Boto3
Copy data from Amazon S3 to Google Cloud Storage with Python (boto)
How to use GCP trace with open Telemetry
I tried to use lightGBM, xgboost with Boruta
I want to use MATLAB feval with python
Use boto3-stubs to enable code completion on boto3
Specify the Python executable to use with virtualenv
The easiest way to use OpenCV with python
Story of trying to use tensorboard with pytorch
I want to use Temporary Directory with Python2
I don't want to use -inf with np.log
[Python] Summary of S3 file operations with boto3
Export RDS snapshot to S3 with Lambda (Python)
I want to use ip vrf with SONiC
Upload images to S3 with GUI using tkinter
Add / remove kernel to use jupyter with venv
How to get the key on Amazon S3 with Boto 3, implementation example, notes
Try to use up the Raspberry Pi 2's 4-core CPU with Parallel Python
How to use xml.etree.ElementTree
How to use Python-shell
Use mecab-ipadic-neologd with igo-python
Resize multipart.File type image with golang ~ Upload to S3
Use RTX 3090 with PyTorch
How to use tf.data
Use ansible with cygwin
How to use virtualenv
Use pipdeptree with virtualenv
How to use xgboost: Multi-class classification with iris data
How to use image-match
[Python] Use JSON with Python
How to use shogun
Use Mock with pytest
How to use Pandas 2
Convert 202003 to 2020-03 with pandas
Use indicator with pd.merge