A memo that reads data from dashDB with Python & Spark

Runs on Python2 with Spark 1.6 in IBM Data Scientist Experience (1) Define credentials Since I registered my dashDB in DSX in advance, click "insert to code" to set it. (Usernames and passwords are hidden ..) Screen Shot 2016-11-16 at 17.46.16.png

credentials_2 = {
  'port':'50000',
  'db':'BLUDB',
  'username':'dashXXXXX',
  'ssljdbcurl':'jdbc:db2://dashdb-entry-yp-dal09-07.services.dal.bluemix.net:50001/BLUDB:sslConnection=true;',
  'host':'dashdb-entry-yp-dal09-07.services.dal.bluemix.net',
  'https_url':'https://dashdb-entry-yp-dal09-07.services.dal.bluemix.net:8443',
  'dsn':'DATABASE=BLUDB;HOSTNAME=dashdb-entry-yp-dal09-07.services.dal.bluemix.net;PORT=50000;PROTOCOL=TCPIP;UID=dashXXXX;PWD=XXXXXXXXXXX;',
  'hostname':'dashdb-entry-yp-dal09-07.services.dal.bluemix.net',
  'jdbcurl':'jdbc:db2://dashdb-entry-yp-dal09-07.services.dal.bluemix.net:50000/BLUDB',
  'ssldsn':'DATABASE=BLUDB;HOSTNAME=dashdb-entry-yp-dal09-07.services.dal.bluemix.net;PORT=50001;PROTOCOL=TCPIP;UID=dash7836;PWD=82f9d02e61da;Security=SSL;',
  'uri':'db2://dashXXXX:[email protected]:50000/BLUDB',
  'password':"""XXXXXXXXXXXXX"""
}

(2) Define a function for data acquisition Referenced site ---> http://stackoverflow.com/questions/37688993/how-to-use-pandas-on-spark-notebook-data-on-dashdb-in-python

def getDashData(credentials,schemaName , tableName):
    from pyspark.sql import SQLContext
    sqlContext = SQLContext(sc)
    props = {}
    props['user'] = credentials['username']
    props['password'] = credentials['password']
    table = schemaName + '.' + tableName     
    return sqlContext.read.jdbc(credentials['jdbcurl'],table,properties=props)

(3) Extract data from dashDB & check the first 10 records

df_dash = getDashData(credentials_2 , 'DASH7836', 'TEST1' )
df_dash.toPandas().head(10)
Screen Shot 2016-11-16 at 17.56.05.png

Recommended Posts

A memo that reads data from dashDB with Python & Spark
Extract data from a web page with Python
A server that echoes data POSTed with flask / python
A memo that I touched the Datastore with python
A Python program that aggregates time usage from icalendar data
[Python] A memo that I tried to get started with asyncio
Get data from MySQL on a VPS with Python 3 and SQLAlchemy
Receive textual data from mysql with python
[Python] A program that creates stairs with #
[Note] Get data from PostgreSQL with Python
Add a Python data source with Redash
A memo with Python2.7 and Python3 on CentOS
A typed world that begins with Python
A memo that allows you to change Pineapple's Python environment with pyenv
Extract lines that match the conditions from a text file with python
Make a decision tree from 0 with Python and understand it (4. Data structure)
Create test data like that with Python (Part 1)
A memo that I wrote a quicksort in Python
A memo that made a graph animated with plotly
Create a page that loads infinitely with python
A story stuck with handling Python binary data
Folium: Visualize data on a map with Python
"Python Kit" that calls a Python script from Swift
A memo when creating a python environment with miniconda
Create a decision tree from 0 with Python (1. Overview)
Read line by line from a file with Python
Data analysis with python 2
Data analysis with Python
A memo connected to HiveServer2 of EMR with python
How to scrape image data from flickr with python
Receive dictionary data from a Python program in AppleScript
Get financial data with python (then a little tinkering)
From a book that programmers can learn ... (Python): Pointer
From buying a computer to running a program with python
[Python] A memo to write CSV vertically with Pandas
I tried collecting data from a website with Scrapy
A memo about building a Django (Python) application with Docker
"A book that understands Flask from scratch" Reading memo
[Basics of data science] Collecting data from RSS with python
Get data from database via ODBC with Python (Access)
A Python program that converts ical data into text
A memo that detects and returns an image acquired from a webcam with Django's OpenCV
A memo that uses an interactive display mode like Jupyter notebook with VSCode + Python
Python2 / numpy> Replace only a specific column in a file with column data from another file> numpy.c_
"Getting stock price time series data from k-db.com with Python" Program environment creation memo
A memo to generate a dynamic variable of class from dictionary data (dict) that has only standard type data in Python3
Sample data created with python
Create a decision tree from 0 with Python and understand it (3. Data analysis library Pandas edition)
Published a library that hides character data in Python images
From a book that programmers can learn ... (Python): About sorting
Twitter graphing memo with Python
[Linux] Copy data from Linux to Windows with a shell script
Notes on importing data from MySQL or CSV with Python
Get Youtube data with python
Let's create a script that registers with Ideone.com in Python.
A learning roadmap that allows you to develop and publish services from scratch with Python
python Condition extraction from a list that I often forget
A memo when face is detected with Python + OpenCV quickly
Use a macro that runs when saving python with vscode
From a book that programmers can learn (Python): Decoding messages
Make a fortune with Python