Upload file to GCP's Cloud Storage (GCS) ~ Load with local Python

environment

GCP account registration

[Explanation with image] Register an account with a free trial of Google Cloud Platform (GCP)

Install Google Cloud SDK

Install Google Cloud SDK ~ Initialize

Project creation

Create a project with Google Cloud SDK

Bucket creation

$ gsutil is a command to operate storage.

$ gsutil mb -l us-central1 gs://Bucket name

File upload

$gsutil cp local file path gs://Bucket name/Directory name/file name

Service account / service account key creation

Create a service account / service account key so that you can access GCS from Python.

Service account creation

gcloud iam service-accounts create service account name\
                                   --display-name service account display name\
スクリーンショット 2020-02-10 0.10.39.png

Check the permissions currently associated with the project

gcloud projects get-iam-policy mypj-id

# bindings:
# - members:
#   - user:[email protected]
#   role: roles/owner
# etag: BwWeTrntoao=
# version: 1

Grant permissions to service accounts

Grant storage administrator privileges

gcloud projects add-iam-policy-binding project ID\
--member serviceAccount:Service account name@Project ID.iam.gserviceaccount.com \
--role roles/storage.admin
スクリーンショット 2020-02-10 0.16.18.png

role list

https://cloud.google.com/iam/docs/understanding-roles?hl=ja#predefined_roles

Check permissions again

gcloud projects get-iam-policy mypj-id

# bindings:
# - members:
#   - user:[email protected]
#   role: roles/owner
# - members:
#   - serviceAccount:[email protected]
#   role: roles/storage.admin
# etag: BwWeTz6vIBY=
# version: 1

Service account key creation

$ gcloud iam service-accounts keys create ./service_account_keys/anata_no_key.json \
--iam-account service account name@Project ID.iam.gserviceaccount.com

Directory structure

.
├── .env
├── service_account_keys/
│      └── anata_no_key.json
└── working/
        └── main.py

.env

With this description to set the path of the service account key created earlier in the environment variable. ** * Relative path from the loading source file (file loaded_dotenv ()) **

.env


GOOGLE_APPLICATION_CREDENTIALS=./service_account_keys/anata_no_key.json

Library installation

Install google-cloud-storage`` python-dotenv pandas with pip

$ pip install google-cloud-storage python-dotenv pandas

Download with Python

main.py


import os
from io import BytesIO

from dotenv import load_dotenv
from google.cloud import storage
import pandas as pd


# .Set the contents of env to environment variables
load_dotenv('./.env')

PROJECT_NAME = 'anata_no_project'
BUCKET_NAME = 'anata_no_bucket'
FILE_NAME = 'path/to/dir/train.csv' # gs://Bucket name/~The following path

client = storage.Client(PROJECT_NAME)
bucket = client.get_bucket(BUCKET_NAME)
blob = storage.Blob(FILE_NAME, bucket)
data = blob.download_as_string()
df = pd.read_csv(BytesIO(data))
print(df)

It is OK if it is displayed as df.

Recommended Posts

Upload file to GCP's Cloud Storage (GCS) ~ Load with local Python
File upload to Azure Storage (Python)
How to upload files to Cloud Storage with Firebase's python SDK
How to upload a file to Cloud Storage using Python [Make a fixed point camera with Raspberry PI # 1]
[Python] Write to csv file with Python
Output to csv file with Python
Image upload & download to Azure Storage. With Python + requests + REST API
[GCP] Operate Google Cloud Storage with Python
Upload files to Aspera that comes with IBM Cloud Object Storage (ICOS) using SDK (Python version)
Upload images to Google Drive with Python
Upload & download wav file to X server (X-Server) by FTP with Python
Convert the spreadsheet to CSV and upload it to Cloud Storage with Cloud Functions
Copy data from Amazon S3 to Google Cloud Storage with Python (boto)
How to read a CSV file with Python 2/3
[Python] How to read excel file with pandas
Convert svg file to png / ico with Python
Upload files to Google Drive with Lambda (Python)
I want to write to a file with Python
I tried to touch the CSV file with Python
How to measure mp3 file playback time with python
How to convert JSON file to CSV file with Python Pandas
[No need to build local environment] Deploy Python bottle application made with Cloud9 to Heroku
How to upload with Heroku, Flask, Python, Git (Part 1)
File upload with django
How to upload with Heroku, Flask, Python, Git (Part 2)
Use of Google Cloud Storage (GCS) with "GAE / Py"
Local server with python
[Python] Change the Cache-Control of the object uploaded to Cloud Storage
Easy to use Nifty Cloud API with botocore and python
Cloud Functions to resize images using OpenCV with Cloud Storage triggers
[Python Kivy] How to create an exe file with pyinstaller
Try to decipher the garbled attachment file name with Python
Upload and delete files to Google Cloud Storages with django-storage
Connect Raspberry Pi to Alibaba Cloud IoT Platform with Python
Get the source of the page to load infinitely with python.
Connect to Wikipedia with Python
Post to slack with Python 3
Upload a file to Dropbox
Switch python to 2.7 with alternatives
Write to csv with Python
Download csv file with python
Process the gzip file UNLOADed with Redshift with Python of Lambda, gzip it again and upload it to S3
[Python] Convert CSV file uploaded to S3 to JSON file with AWS Lambda
How to connect to Cloud Firestore from Google Cloud Functions with python code
[ROS2] How to play a bag file with python format launch
Try adding a wall to your IFC file with IfcOpenShell python
[For Python] Quickly create an upload file to AWS Lambda Layer
Convert files uploaded to Cloud Storage with Cloud Functions (Python) so that they are not garbled in Excel
[Package cloud] Manage python packages with package cloud
Extract the xz file with python
Python: How to use async with
Link to get started with python
[Automation with python! ] Part 1: Setting file
Implemented file download with Python + Bottle
Create folders from '01' to '12' with python
Nice to meet you with python
Try to operate Facebook with Python
Create an Excel file with Python3
Convert list to DataFrame with python
Output cell to file with Colaboratory
MP3 to WAV conversion with Python