Upload and delete files to Google Cloud Storages with django-storage

background

Google Cloud Stoarge using Django's library django-storages I solved the file deletion method by trial and error, so I will summarize the memo including the upload method.

Method

The django-storages page has basic setup instructions.

settings.py


DEFAULT_FILE_STORAGE = 'storages.backends.gcloud.GoogleCloudStorage'
GS_BUCKET_NAME = 'YOUR_BUCKET_NAME_GOES_HERE'

When,

settings.py


from google.oauth2 import service_account

GS_CREDENTIALS = service_account.Credentials.from_service_account_file(
    "path/to/credentials.json"
)

If you describe the two points and define the model as usual, it will work up to upload. The following is a model example of an image file.

models.py


class Image(models.Model):
    image = models.ImageField(upload_to='image/')

    def __str__(self):
        return self.image.name

However, up to this point, even if you delete the record, the Google Cloud Storage file will not be deleted. This seems to be the design concept of Django itself (I just read the secondary information), not only when using Google Cloud Storage, but also to implement file deletion by myself.

Its implementation requires the decorator to define a function that will be called when deleting records. If you delete a file with Google cloud console (when there is no file on Google Cloud Storage side at the time of this process), an error will occur, so exception handling is included.

models.py


from django.dispatch import receiver
from google.cloud import exceptions

@receiver(models.signals.post_delete, sender=Image)
def auto_delete_image(sender, instance, **kargs):
    file = instance.image
    try:
        file.storage.delete(name=file.name)
    except exceptions.NotFound as e:
        print(e)

If you set the environment variable GOOGLE_APPLICATION_CREDENTIALS, you can also implement it in the google.cloud module as follows. However, in this case, an error occurred unless storage.buckets.get was assigned as the authority of google cloud.

model.py


from google.cloud import storage
from django.conf import settings

@receiver(models.signals.post_delete, sender=Image)
def auto_delete_image(sender, instance, **kargs):
    storage_client = storage.Client()
    bucket = storage_client.get_bucket(settings.GS_BUCKET_NAME)
    blob = bucket.blob(instance.image.name)
    try:
        blob.delete()
    except exceptions.NotFound as e:
        print(e)

Supplement

I will supplement the settings on the Google Cloud Storage side. I haven't verified it well, so I may not need everything, but in my environment I issue a service account with the following permissions.

The official link is posted for the actual setting method.

-Create and manage custom roles --Create and manage service accounts

Recommended Posts

Upload and delete files to Google Cloud Storages with django-storage
Upload files to Google Drive with Lambda (Python)
How to upload files to Cloud Storage with Firebase's python SDK
Upload images to Google Drive with Python
Get tweets with Google Cloud Function and automatically save images to Google Photos
Convert the spreadsheet to CSV and upload it to Cloud Storage with Cloud Functions
Upload files with Django
How to load files in Google Drive with Google Colaboratory
Upload to a shared drive with Google Drive API V3
Easy to use Nifty Cloud API with botocore and python
Speech recognition of wav files with Google Cloud Speech API Beta
How to Delete with SQLAlchemy?
Upload files to Aspera that comes with IBM Cloud Object Storage (ICOS) using SDK (Python version)
Apache Beam 2.0.x with Google Cloud Dataflow starting with IntelliJ and Gradle
Get conversions and revenue with Google Analytics API and report to Slack
How to import CSV and TSV files into SQLite with Python
How to connect to Cloud Firestore from Google Cloud Functions with python code
Try to display google map and geospatial information authority map with python
Install Anaconda on Mac and upload Jupyter (IPython) notebook to Anaconda Cloud
The strongest way to use MeCab and CaboCha with Google Colab
Move data to LDAP with python Change / Delete (Writer and Reader)
Regularly upload files to Google Drive using the Google Drive API in Python
Upload file to GCP's Cloud Storage (GCS) ~ Load with local Python
Upload & move GCS files with Go
Save and retrieve files with Pepper
Selenium and python to open google
Upload and download images with falcon
Upload data to s3 of aws with a command and update it, and delete the used data (on the way)
Use shutil to delete all folders with a small number of files
Introduction to Apache Beam with Google Cloud Dataflow (over 2.0.x series) ~ Combine ~
Wrap reading and writing of GCP to Secret Manager with google subcommands
Copy data from Amazon S3 to Google Cloud Storage with Python (boto)
Feel free to knock 100 data sciences with Google Colab and Azure Notebooks!
I automated job can stamping with selenium and deployed it to Google Cloud Functions, but it was quite difficult
GAE --With Python, rotate the image based on the rotation information of EXIF and upload it to Cloud Storage.
Use MeCab and neologd with Google Colab
[Linux] How to subdivide files and folders
Streaming speech recognition with Google Cloud Speech API
Execute Google Translate and DeepL Translate with GUI
Fractal to make and play with Python
Edit and save read-only files with vim
Try using Python with Google Cloud Functions
Read and write csv files with numpy
How to search Google Drive with Google Colaboratory
Reading and writing JSON files with Python
[GCP] Operate Google Cloud Storage with Python
Convert PDF files to PNG files with GIMP
Use TPU and Keras with Google Colaboratory
Don't use rm command to delete files
Convert HEIC files to PNG files with Python
Google Cloud Functions log output (and addicted)
Use boto to upload / download files to s3.
How to automatically upload .gpx files to Strava
Create an API to convert PDF files to TIF images with FastAPI and Docker
Zip-compress any file with the [shell] command to create a file and delete the original file.
Specify the start and end positions of files to be included with qiitap
How to use Service Account OAuth and API with Google API Client for python