[Cloud Functions] Automatically decompress GZIP files placed in Storage

Preamble

Source

I modified the code (reference link-3) that decompresses the Zip file that my predecessor dropped. It is assumed that a single file is Gzip-compressed with Cloud Functions (Python 3.7) launched by a Cloud Storage trigger.

main.py


from google.cloud import storage
import io,gzip

def hello_gcs(event, context):

    client = storage.Client()
    bucket_name = event['bucket']
    bucket = client.get_bucket(bucket_name)
    blob_name = event['name']
    blob = bucket.blob(blob_name)

    data = io.BytesIO(blob.download_as_string())
    with gzip.open(data) as gz:
        file = gz.read()
        blob_decompress = bucket.blob(blob_name.replace('.gz',''))
        blob_decompress.upload_from_string(file)

Big problem

⇒ At the time of the GZIP file to be placed, the capacity could be 2GB or more, so it ended up being a crap. ..

Reference link

  1. Cloud Functions Guide: Google Cloud Storage Trigger
  2. [I created a serverless job that loads data into BigQuery with event drive when a file is placed in GCS](https://dev.classmethod.jp/articles/cloud-functions-gcs-trigger-load- data2bigquery /)
  3. [How do I unzip the .Zip file on Google Cloud Storage? ](Https://www.it-swarm.dev/ja/google-cloud-platform/google%E3%82%AF%E3%83%A9%E3%82%A6%E3%83%89%E3% 82% B9% E3% 83% 88% E3% 83% AC% E3% 83% BC% E3% 82% B8% E3% 81% A7zip% E3% 83% 95% E3% 82% A1% E3% 82% A4% E3% 83% AB% E3% 82% 92% E8% A7% A3% E5% 87% 8D% E3% 81% 99% E3% 82% 8B% E3% 81% AB% E3% 81% AF% E3% 81% A9% E3% 81% 86% E3% 81% 99% E3% 82% 8C% E3% 81% B0% E3% 82% 88% E3% 81% 84% E3% 81% A7% E3% 81% 99% E3% 81% 8B% EF% BC% 9F / 837268170 /)

Recommended Posts

[Cloud Functions] Automatically decompress GZIP files placed in Storage
Convert files uploaded to Cloud Storage with Cloud Functions (Python) so that they are not garbled in Excel
Get Google Cloud Storage object list in Java
Get files, functions, line numbers running in python