EC2 (Python3)-> BigQuery

Premise

・ GCP account is ready -There must be a table that can connect to BigQuery

Preparation of execution environment

ec2 -pyenv -naconda3-4.0.0 -pandas -httplib2 -google-api-python-client

Module addition

sudo su -
pyenv global naconda3-4.0.0
pip install httplib2
pip install google-api-python-client

Installation of GoogleCloud SDK (running in python2.x environment) and authentication

pyenv global system
curl https://sdk.cloud.google.com | bash
~/google-cloud-sdk/bin/gcloud auth login

Run on a table with very few records

import pandas as pd
query = 'SELECT * FROM dataset_name.table_name'
pd.read_gbq(query, 'your_project_ID')

→ Authentication runs with pd.read_gbq, but since EC2 is the execution source, I can't receive the authentication URL to listen on localhost

Authentication information generation with Oauth2

from oauth2client.client import OAuth2WebServerFlow
from oauth2client.file import Storage
flow = OAuth2WebServerFlow(client_id='your_client_id',
                           client_secret='your_client_secret',
                           scope='https://www.googleapis.com/auth/bigquery',
                           redirect_uri='urn:ietf:wg:oauth:2.0:oob')
storage = Storage('bigquery_credentials.dat')
authorize_url = flow.step1_get_authorize_url()
print 'Go to the following link in your browser: ' + authorize_url
code = raw_input('Enter verification code: ')
credentials = flow.step2_exchange(code)
storage.put(credentials)

After executing the above process, "bigquery_credentials.dat" is created in the current directory. → pandas.read_gbq uses the above as authentication information

Run on a table with very few records

import pandas as pd
query = 'SELECT * FROM dataset_name.table_name'
pd.read_gbq(query, 'your_project_ID')

reference

https://developers.google.com/api-client-library/python/guide/aaa_oauth http://stackoverflow.com/questions/37792709/accessing-big-query-from-cloud-datalab-using-pandas

Recommended Posts

EC2 (Python3)-> BigQuery
Use BigQuery from python.
Connect to BigQuery with Python
BigQuery integration for Python users
Python --Install MySQLDB on EC2
kafka python
Python basics ⑤
python + lottery 6
Python Summary
Python comprehension
Python technique
Studying python
Python memorandum
Python FlowFishMaster
Python service
python tips
python function ①
Python basics
Python memo
ufo-> python (3)
Python comprehension
install python
Python Singleton
Python basics ④
Python Memorandum 2
python memo
Python Jinja2
Python increment
atCoder 173 Python
[Python] function
Python installation
python tips
Installing Python 3.4.3.
Try python
Python memo
Python iterative
Python algorithm
Python2 + word2vec
[Python] Variables
How to work with BigQuery in Python
Python sys.intern ()
Python tutorial
Python decimals
python underscore
Python summary
Start python
Note: Python
Python basics ③
python log
Python basics
[Scraping] Python scraping
Python update (2.6-> 2.7)
python memo
Python memorandum
Python # sort
ufo-> python
Python nslookup
python learning
Hannari Python 2020
[Rpmbuild] Python 3.7.3.
Prorate Python (1)