Regular export of Google Analytics raw data to BigQuery using cloud functions

Regular export of Google Analytics raw data to BigQuery using cloud functions

The content is almost the same as Article on the 2nd day, but after a while, it turns out that the article on the 2nd day cannot be used from cloud functions. So, I would like to tell you about the problems and how to deal with them.

things to do

  1. Extract raw data to python using Google Analytics API
  2. Upload to Google Cloud Starage using pandas
  3. Export from GCS to BigQuery

Where it got stuck

1. Cannot create file locally

The one on the second day was fine for running locally. However, the cloud function is read only and I can't create a csv file. So you have to either export it directly to BigQuery or upload it to GCS without creating a csv file. From the results, I chose the latter.

There was an article saying that you can create a file in the / tmp / directory, but I tried it and it didn't work, so I gave up on the way.

2. Two variables are given to the function to be executed arbitrarily

It seems that you have to prepare two variables for the function to be executed.

Reason

In the former case, it seems that only letter, number, and under_score can be used for pagePath, and there are various other rules, so I couldn't use it.

How to upload directly to GCS

When I was looking for an article, this person's article was easy to understand. Google Cloud Storage (GCS) with Python, csv save pandas DataFrame to BigQuery class

code

day5.py

The 5th day is over!

I was wondering if it would be a good idea to go around this kind of thing, but it took a long time, so I hope I can help if there are other people who are the same. Please be careful of Corona, too!

Recommended Posts

Regular export of Google Analytics raw data to BigQuery using cloud functions
Export Google Analytics Standard to BigQuery
Export access data for each user of Google Analytics.
Continue to challenge Cyma's challenges using the OCR service of Google Cloud Platform
A story that contributes to new corona analysis using a free trial of Google Cloud Platform
Cloud Functions to resize images using OpenCV with Cloud Storage triggers
Try to determine food photos using Google Cloud Vision API
[Introduction to Data Scientists] Basics of Python ♬ Functions and classes
The story of creating a database using the Google Analytics API
Fourier transform of raw data
Save the results of crawling with Scrapy to the Google Data Store
I tried to make a regular expression of "amount" using Python
I tried to make a regular expression of "time" using Python
I tried to make a regular expression of "date" using Python
How to connect to Cloud Firestore from Google Cloud Functions with python code
I tried to visualize BigQuery data using Jupyter Lab on GCP
[Introduction to Data Scientists] Basics of Python ♬ Functions and anonymous functions, etc.