I tried to visualize BigQuery data using Jupyter Lab on GCP

Caution

I'm new to GCP, python and jupyter. I was very happy to be able to move it while going around, so I posted it. I would appreciate it if you could let me know if you have any mistakes.

reference

-[GCP] AI Platform Notebooks has been officially released, so try using it. -[BigQuery] Did you know that you can use CREATE statements? Create table with SQL

Create BigQuery data

Open the GCP console

https://console.cloud.google.com/

Select "BigQuery" from the side menu

Screen Shot 2020-08-23 at 18.27.52.png

Click "Create Dataset"

Screen Shot 2020-08-23 at 18.32.48.png

I tried to create it with the dataset ID set to "test_data_set"

Screen Shot 2020-08-23 at 18.33.46.png

Easily create tables and data for testing

Paste the following query into the query editor and click Run

create table test_data_set.t (i INT64);
insert into test_data_set.t values (1),(2),(5),(7),(12);

Screen Shot 2020-08-23 at 18.45.00.png

Now that we have the data, let's take a look at it using Jupyter.

Screen Shot 2020-08-23 at 18.45.59.png

Instantiate a notebook

Select Side Menu> "AI Platform"> "Notebook"

Screen Shot 2020-08-23 at 17.36.45.png

I selected "New Instance"> "Python 2 and 3"

Screen Shot 2020-08-23 at 17.53.28.png

You can select the region etc. and the estimated amount is also displayed, but this time I will delete the instance as soon as I try it, so I pressed "Create" with the default settings

Screen Shot 2020-08-23 at 18.03.53.png

Click "Open JUPYTER LAB"

Screen Shot 2020-08-23 at 18.10.47.png

Jupyter Lab opens

Screen Shot 2020-08-23 at 19.14.42.png


When I first tried to get bigquery data

"UserWarning: Cannot create BigQuery Storage client, the dependency google-cloud-bigquery-storage is not installed."

I got an error, so install google-cloud-bigquery-storage first

Click "Console"

Screen Shot 2020-08-23 at 19.12.40.png

Install google-cloud-bigquery-storage

$ pip install google-cloud-bigquery-storage

Screen Shot 2020-08-23 at 19.09.06.png


Press Python 3 to create a new notebook

Screen Shot 2020-08-23 at 18.55.47.png

By default, JupyterLab has a file called /tutorials/bigquery/BigQuery basics.ipynb and there is an easy-to-understand explanation, so I tried to easily create a process of data acquisition from bigquery by referring to that area.

from google.cloud import bigquery

client = bigquery.Client(location="US")

query = """
    select *
    from test_data_set.t
    order by i
"""

query_job = client.query(
    query,
    # Location must match that of the dataset(s) referenced in the query.
    location="US",
)  # API request - starts the query

dv = query_job.to_dataframe()
print(df)
df.plot()

Press "▷" to execute

Screen Shot 2020-08-23 at 19.33.57.png

I was able to display the data of the bigquery I made first as a graph!


There will be a charge at the end, so stop with "Stop"

Screen Shot 2020-08-23 at 19.41.54.png


Thank you for watching until the end m (_ _) m

Recommended Posts

I tried to visualize BigQuery data using Jupyter Lab on GCP
I tried using Jupyter
[Pythonocc] I tried using CAD on jupyter notebook
I tried using BigQuery ML
I tried using PySpark from Jupyter 4.x on EMR
I tried to touch jupyter
I tried to search videos using Youtube Data API (beginner)
I tried using "Syncthing" to synchronize files on multiple PCs
[TF] I tried to visualize the learning result using Tensorboard
I tried to analyze scRNA-seq data using Topological Data Analysis (TDA)
I tried to get data from AS / 400 quickly using pypyodbc
I tried AdaNet on table data
Somehow I tried using jupyter notebook
I tried to digitize the stamp stamped on paper using OpenCV
[Python] I tried to get various information using YouTube Data API!
I tried to get data from AS / 400 quickly using pypyodbc Preparation 1
I tried to visualize AutoEncoder with TensorFlow
I tried using YOUTUBE Data API V3
I tried to classify text using TensorFlow
I tried using the BigQuery Storage API
I tried to predict Covid-19 using Darts
[Python] I tried to visualize the night on the Galactic Railroad with WordCloud!
I tried using jpholidayp over proxy to execute cron only on weekdays
I want to display an image on Jupyter Notebook using OpenCV (mac)
I tried to perform a cluster analysis of customers using purchasing data
I tried using Remote API on GAE / J
I tried to save the data with discord
I tried to synthesize WAV files using Pydub.
I want to visualize csv files using Vega-Lite!
I tried to get CloudWatch data with Python
I tried launching jupyter nteract on heroku server
I tried to start Jupyter with Amazon lightsail
I tried DBM with Pylearn 2 using artificial data
I tried to make a ○ ✕ game using TensorFlow
I tried to make a translation BOT that works on Discord using googletrans
I tried to rescue the data of the laptop by booting it on Ubuntu
I tried to make a suspicious person MAP quickly using Geolonia address data
Library for "I want to do that" of data science on Jupyter Notebook
I tried using parameterized
I tried using argparse
I tried using mimesis
I tried using anytree
I tried to implement Minesweeper on terminal with python
I tried using Summpy
I tried to predict the J-League match (data analysis)
Python practice 100 knocks I tried to visualize the decision tree of Chapter 5 using graphviz
I tried using coturn
I tried using Pipenv
I tried using matplotlib
I tried using "Anvil".
I tried using Hubot
I tried using ESPCN
I tried using openpyxl
I tried clustering ECG data using the K-Shape method
I tried using Ipython
I tried to debug.
I tried to get an AMI using AWS Lambda
I tried to approximate the sin function using chainer
I tried using PyCaret
I tried using cron
I tried to become an Ann Man using OpenCV