I tried uploading / downloading a file to AWS S3 / Azure BlobStorage / GCP CloudStorage with Python

This article is the 10th day article of Pathee Advent Calendar 2019.

Introduction

Hello. I'm Kunii, an engineer.

I've summarized how to upload / download files to ** AWS S3 ** / ** Azure BlobStorage ** / ** GCP Cloud Storage ** with ** Python **.

I used to upload / download files to Azure Blob Storage with Python, and that's why I decided to try other storage services together.

Somehow I chose AWS S3 / GCP Cloud Storage, It's Python because our backend is implemented in Python.

For each storage service, we will summarize in the following items.

·Advance preparation ・ ** Connection information settings ** ・ ** Module installation ** ・ ** Create a box to store files ** ・ ** Upload the file to the created box ** ・ ** Download the file from the created box ** ・ ** Delete the created box **

Every storage service has a module, so it's fairly easy to use.

environment

Python 3.6.4 macOS 10.15.1

AWS S3

Advance preparation

Connection information settings

You will need ** Access Key ID ** and ** Secret Access Key ** as connection information. Create an access key ID and secret access key by referring to Create AWS Access Key.

Set the access key ID and secret access key in the environment variables with the following command.

$ export AWS_ACCESS_KEY_ID="Access key ID"
$ export AWS_SECRET_ACCESS_KEY="Secret access key"

Module installation

Install ** boto3 **.

$ pip install boto3

For each subsequent operation on AWS S3, ** s3_resource ** created with the following code will be the base.

import  boto3

s3_resource = boto3.resource('s3', region_name='ap-northeast-1')

Creating a bucket

In AWS S3, the box that stores files is called a ** bucket **. You can create a bucket with the following code.

#Creating a bucket
s3_client = s3_resource.meta.client
s3_client.create_bucket(Bucket='The name of the bucket to create', CreateBucketConfiguration={'LocationConstraint': 'ap-northeast-1'})

It seems that there are various setting items in CreateBucketConfiguration, but for the time being, it can be created only by LocationConstraint (constraint of the region where the bucket is created).

File upload

#File upload
s3_resource.Bucket('The name of the bucket to upload to').upload_file('Path of the file to upload', 'File name at the upload destination')

File download

#File download
s3_resource.Bucket('The name of the bucket from which you downloaded it').download_file('File name to download', 'Download destination file path')

Delete bucket

#Delete bucket
s3_client = s3_resource.meta.client
s3_client.delete_bucket(Bucket='The name of the bucket to delete')

In order to delete a bucket, you need to delete all the files uploaded to the bucket and leave them empty. If you try to delete a bucket when the bucket is not empty, an error will occur and you will not be able to delete it.

You can delete the uploaded file with the following code, but I only know how to delete it by specifying the file name ... (There is also a delete_objects method, but it seems that you have to specify the file name after all)

#Delete file
s3_client = s3_resource.meta.client
s3_client.delete_object(Bucket='The name of the bucket that contains the files to delete', Key='File name to delete')

Azure BlobStorage

Advance preparation

Connection information settings

You will need the storage account ** connection string ** as the connection information.

Sign in to the Azure portal and do Create Storage Account (https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal) .. Refer to here Get (copy) the connection string of the storage account.

Set the connection string in the environment variable with the following command.

$ export CONNECT_STR="Connection string"

Module installation

Install ** azure-storage-blob **.

$ pip install azure-storage-blob

Subsequent operations on Azure Blob Storage are based on ** blob_service_client ** created with the following code.

import os
from azure.storage.blob import BlobServiceClient

connect_str = os.getenv('CONNECT_STR')
blob_service_client = BlobServiceClient.from_connection_string(connect_str)

Creating a container

In Azure BlobStorage, the box that stores files is called a ** container **. You can create a container with the following code.

#Creating a container
container_client = blob_service_client.create_container('The name of the container to create')

File upload

#File upload
blob_client = blob_service_client.get_blob_client(container='The name of the container to upload to', blob='The name of the file at the upload destination')
with open('Path of the file to upload', 'rb') as uploaded_file:
    blob_client.upload_blob(uploaded_file)

File download

#File download
blob_client = blob_service_client.get_blob_client(container='The name of the container from which it was downloaded', blob='The name of the file to download')
with open('File path at the download destination', 'wb') as downloaded_file:
    downloaded_file.write(blob_client.download_blob().readall())

Delete container

You can delete a container without deleting all the files you uploaded to the container (although you may have deleted them in the method).

#Delete container
container_client = blob_service_client.get_container_client('The name of the container to delete')
container_client.delete_container()

GCP CloudStorage

Advance preparation

Connection information settings

As connection information, you will need the key of the account (** service account key **) to access the cloud storage called the service account.

[Here](https://sleepless-se.net/2018/05/22/googlecloudstorage%E3%81%A7python%E3%81%8B%E3%82%89%E3%83%95%E3%82% A1% E3% 82% A4% E3% 83% AB% E3% 82% 92% E3% 82% 84% E3% 82% 8A% E3% 81% A8% E3% 82% 8A% E3% 81% 99% Create a service account key by referring to E3% 82% 8B% E6% 96% B9% E6% B3% 95 / # API) and download the JSON file of the service account key.

Set the service account key in the environment variable with the following command.

$ export GOOGLE_APPLICATION_CREDENTIALS="Path of downloaded JSON file"

Module installation

Install ** google-cloud **.

$ pip install --upgrade google-cloud

The ** storage_client ** created with the following code will be the basis for each subsequent operation on Google Cloud Storage.

from google.cloud import storage

storage_client = storage.Client()

Creating a bucket

In GCP Cloud Storage, the box that stores your files is called a ** bucket **. You can create a bucket with the following code.

#Creating a bucket
bucket = storage_client.create_bucket('The name of the bucket to create')

File upload

#File upload
bucket = storage_client.get_bucket('The name of the bucket to upload to')
blob = bucket.blob('The name of the file at the upload destination')
blob.upload_from_filename(filename='Path of the file to upload')

File download

#File download
bucket = storage_client.get_bucket('The name of the bucket that contains the file to download')
blob = bucket.blob('The name of the file to download')
blob.download_to_filename('Download destination file path')

Delete bucket

#Delete bucket
bucket = storage_client.get_bucket('The name of the bucket to delete')
bucket.delete(force=True)

In order to delete a bucket (similar to AWS S3), you need to delete all the files you uploaded to the bucket and leave them empty. If you try to delete a bucket when the bucket is not empty, an error will occur and you will not be able to delete it.

By setting ** force = True **, the bucket will be emptied and then deleted.

By the way, deleting the file is as follows.

#Delete file
bucket = storage_client.get_bucket('The name of the bucket that contains the files to delete')
blob = bucket.blob('The name of the file to delete')
blob.delete()

Reference page link

Recommended Posts

I tried uploading / downloading a file to AWS S3 / Azure BlobStorage / GCP CloudStorage with Python
I want to write to a file with Python
[AWS] [GCP] I tried to make cloud services easy to use with Python
I tried to touch the CSV file with Python
I tried to draw a route map with Python
I tried to automatically generate a password with Python3
I tried to divide the file into folders with Python
[5th] I tried to make a certain authenticator-like tool with python
[Python] Convert CSV file uploaded to S3 to JSON file with AWS Lambda
I tried to convert a Python file to EXE (Recursion error supported)
[2nd] I tried to make a certain authenticator-like tool with python
[3rd] I tried to make a certain authenticator-like tool with python
[Python] A memo that I tried to get started with asyncio
I tried to create a list of prime numbers with python
I tried to make a periodical process with Selenium and Python
I tried to make a 2channel post notification application with Python
I tried to make a todo application using bottle with python
[4th] I tried to make a certain authenticator-like tool with python
[1st] I tried to make a certain authenticator-like tool with python
I want to play with aws with python
I tried a functional language with Python
Connect to s3 with AWS Lambda Python
I made a configuration file with Python
Python: I tried to make a flat / flat_map just right with a generator
I tried to communicate with a remote server by Socket communication with Python.
I tried to create a program to convert hexadecimal numbers to decimal numbers with python
I tried to make a traffic light-like with Raspberry Pi 4 (Python edition)
I tried to discriminate a 6-digit number with a number discrimination application made with python
[Outlook] I tried to automatically create a daily report email with Python
I tried to build a Mac Python development environment with pythonz + direnv
I tried to make a url shortening service serverless with AWS CDK
[Python] When I tried to make a decompression tool with a zip file I just knew, I was addicted to sys.exit ()
I want to make a game with Python
I tried to get CloudWatch data with Python
I tried to output LLVM IR with Python
I tried reading a CSV file using Python
I tried to automate sushi making with python
I tried to make a periodical process with CentOS7, Selenium, Python and Chrome
I tried to make a simple mail sending application with tkinter of Python
When I tried to make a VPC with AWS CDK but couldn't make it
[Patent analysis] I tried to make a patent map with Python without spending money
When I tried to create a virtual environment with Python, it didn't work
[ES Lab] I tried to develop a WEB application with Python and Flask ②
I tried to easily create a fully automatic attendance system with Selenium + Python
I tried to create a table only with Django
I tried to implement Minesweeper on terminal with python
I tried running python etc. from a bat file
I tried to solve the soma cube with python
I tried to implement a pseudo pachislot in Python
I tried to get started with blender python script_Part 02
I want to randomly sample a file in Python
I tried to implement an artificial perceptron with python
I want to work with a robot in python.
I want to AWS Lambda with Python on Mac!
I tried to solve the problem with Python Vol.1
I tried to analyze J League data with Python
I want to run a quantum computer with Python
[Introduction to AWS] I tried playing with voice-text conversion ♪
I tried to solve AOJ's number theory with Python
I tried to make a real-time sound source separation mock with Python machine learning
I tried to automatically generate a character string to be input to Mr. Adjustment with Python