Python code for writing CSV data to DSX object storage

A reminder about how to read files on object storage from Python code on a Notebook in one of Watson Data Platform's core services, Data Science Experience (https://datascience.ibm.com/) (DSX). This is a sequel to Python code for reading CSV data from DSX object storage, and this time it is the code for writing.

In the case of the previous reading, the data to be read was clear and the authentication information was obtained from that data, but this time the purpose is to draw the result of the calculation process, so there is a container corresponding to the project. Even so, I cannot get the credentials from the file name. So what should I do?

When I log in to DSX and look for credentials to write to the container from the Data Service, I can't find it anywhere. However, you can get container credentials by logging in from Bluemix PaaS and looking for object storage information.

From the list of services, go to DSX-Object Storage.

When the DSX-ObjectStorage service credential is displayed, the service credential is created for each container, that is, for each project, so copy and paste the service credential of the corresponding container to the DSX Notebook to use it.

The service credentials on the Bluemix side don't include the container name, so just add that part manually and you're ready to go.

from io import BytesIO  
import requests
import numpy as np
import json


# Object storage credentials
credentials_5 = {
 'auth_url':'https://identity.open.softlayer.com',
 'project':'object_storage_bc6cdc85_586e_4581_8a09_8f01f7bdf3ed',
 'project_id':'2a9de4c1d50944a49f1a46dd53394158',
 'region':'dallas',
 'user_id':'********************************',
 'domain_id':'fb119f3e1bc0469dad2b253b317ec7ea',
 'domain_name':'952993',
 'username':'***********************************************',
 'password':"********************",
'container':'DefaultProjecttakarajpibmcom' <--- Add by hand
}


def Write_CSV_to_ObjectStorage(credentials, npdata,label, fileName):  
   """This functions returns a StringIO object containing
   the file content from Bluemix Object Storage V3."""

   csvData = ""
# Label data written to the first line of CSV
   c = 0
   for col in label:
       if c > 0:
           csvData = csvData + ','
       csvData = csvData + col
       c = c + 1
   csvData = csvData + "\n"
   
# Convert Numpy array data to CSV format text data
   rows, cols = np.shape(npdata)
   for row in a:
       c = 0
       for col in row:
           if c > 0:
               csvData = csvData + ','
           csvData = csvData + str(col)
           c = c + 1
       csvData = csvData + "\n"
   
#Object storage authentication
   url1 = ''.join(['https://identity.open.softlayer.com', '/v3/auth/tokens'])
   data = {'auth': {'identity': {'methods': ['password'],
           'password': {'user': {'name': credentials['username'],'domain': {'id': credentials['domain_id']},
           'password': credentials['password']}}}}}
   headers1 = {'Content-Type': 'application/json'}
   resp1 = requests.post(url=url1, data=json.dumps(data), headers=headers1)
#Exit when authentication error occurs
   if resp1.status_code != 201:
       return False, resp1.status_code
   resp1_body = resp1.json()

# Write to object storage
   for e1 in resp1_body['token']['catalog']:
       if(e1['type']=='object-store'):
           for e2 in e1['endpoints']:
               if(e2['interface']=='public'and e2['region']=='dallas'):
                   url2 = ''.join([e2['url'],'/', credentials['container'], '/', fileName])
   s_subject_token = resp1.headers['x-subject-token']
   headers2 = {'X-Auth-Token': s_subject_token, 'accept': 'application/json'}
   resp2 = requests.put(url=url2, headers=headers2, data = csvData )
   result = True
   if resp2.status_code != 201:
       print "ERROR ", resp2.status_code
       result = False
   return result, resp2.status_code
   

# Main

# Data generation
x = np.linspace(0,1,60)
y = np.linspace(10,11,60)

# Two-dimensional array of changes (x, y) to a single array
a = np.stack((x, y), axis=-1)
print a
# Data label
label = ["Time","Data"]

# Save to object storage
rslt = Write_CSV_to_ObjectStorage(credentials_5, a, label,"testDataSet.csv")
print rslt
ew

Explanation of how to use

The first argument is the credentials. The second argument is an array of num.py in the data array and will be inserted in the first line of the CSV file with the label of the data column. Next is the file name (the name of the object stored in the object storage container).

rslt = Write_CSV_to_ObjectStorage(credentials_5, a, label,"testDataSet.csv")

The return value will be 201 Created if successful. Otherwise it will fail.