Bucket / file operations (transfer / acquire / delete / read, etc.) to GCS by python Summary

Running first.py


import os
import pandas as pd
from io import BytesIO
from pathlib import Path
from google.cloud import storage
from google.oauth2 import service_account

def get_project_credentials():
    credentials = service_account.Credentials.from_service_account_file(
        "xx/xx/xx/xx.json",#The path where the json file for the service account key is located
        scopes=["https://www.googleapis.com/auth/cloud-platform"],
    )
    return credentials


def get_storage_client(credentials):
    storage_client = storage.Client(
        credentials = credentials,
        project = credentials.project_id,
    )
    return storage_client

Create a bucket

created_bucket.py


created_bucket_name = 'tmp_bucket_qitaqita'

credentials = get_project_credentials()
storage_client = get_storage_client(credentials)
bucket = storage_client.create_bucket(created_bucket_name)

Note that the bucket name must be unique worldwide. (Initially I tried to create it with the name "tmp_bucket", but I got an error because it already exists.)

Transfer local files to bucket

trans_to_bucket.py


bucket_name = 'tmp_bucket_qitaqita'
file_name = 'send_txt.txt' #Name of what to send
destination_blob_name = 'sended_txt.txt' #Name at the destination

credentials = get_project_credentials()
storage_client = get_storage_client(credentials)

bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
blob.upload_from_filename(file_name)

If you name the destination as follows, a directory called make_dir will be created and the files will be transferred to it. destination_blob_name = 'make_dir/sended_txt_2.txt'

Delete files in the bucket

del_bucket_file.py


bucket_name = 'tmp_bucket_qitaqita'
file_name = 'sended_txt.txt'#The file name you want to delete in the bucket

credentials = get_project_credentials()
storage_client = get_storage_client(credentials)

bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(file_name)
blob.delete()

Display a list of bucket names

bucket_name.py


credentials = get_project_credentials()
storage_client = get_storage_client(credentials)

blobs = storage_client.list_buckets()
for blob in blobs:
    print(blob.name)

Display a list of file names in the bucket

filename_in_bucket.py


bucket_name = 'tmp_bucket_qitaqita'

credentials = get_project_credentials()
storage_client = get_storage_client(credentials)

bucket = storage_client.get_bucket(bucket_name)
blobs = bucket.list_blobs()
for blob in blobs:
    print(blob.name)

Delete the bucket

del_bucket.py


bucket_name = 'tmp_bucket_qitaqita'#Bucket name you want to delete

credentials = get_project_credentials()
storage_client = get_storage_client(credentials)

bucket = storage_client.get_bucket(bucket_name)
bucket.delete()

Read the txt file in the bucket & save it locally

Load & Save.py


bucket_name = 'tmp_bucket_qitaqita'
file_name = 'sended_txt.txt'#File name in the bucket you want to read

credentials = get_project_credentials()
storage_client = get_storage_client(credentials)

bucket = storage_client.get_bucket(bucket_name)
blob = storage.Blob(file_name, bucket)

content = blob.download_as_string()
with open('sended_txt.txt', mode='wb') as f:
    f.write(content)

Read csv file on bucket with pandas

Read csv file.py


bucket_name = 'tmp_bucket_qitaqita'
file_name = 'dataset_data.csv'

credentials = get_project_credentials()
storage_client = get_storage_client(credentials)

bucket = storage_client.get_bucket(bucket_name)
blob = storage.Blob(file_name, bucket)
content = blob.download_as_string()
df_csv = pd.read_csv(BytesIO(content))

If there are other operations that are likely to be in demand, add them as appropriate.

Digression: This is the first post to qiita. I hope it helps.

Recommended Posts

Bucket / file operations (transfer / acquire / delete / read, etc.) to GCS by python Summary
Read the xml file by referring to the Python tutorial
Summary of python file operations
How to switch the configuration file to be read by Python
Read the file line by line in Python
Read the file line by line in Python
How to read a CSV file with Python 2/3
[Python] How to read excel file with pandas
[Python] Summary of S3 file operations with boto3
Read line by line from a file with Python
Template of python script to read the contents of the file
[Python] Privately created & used small-scale functions (file operations, etc.)
Folder creation / file move / compress / delete operations with python
How to read text by standard input or file name specification like cat in Python
File operations in Python
[Python] File / directory operations
File operations in Python
Read Python csv file
How to read an Excel file (.xlsx) with Pandas [Python]