[python] Use DataFrame to label arbitrary variables and arrays together and save them in csv [pandas]

I wanted to save variables all at once, so I made such a program. It can be saved by applying a two-dimensional array.

... I noticed after writing, but the method of adding may be better [If you are busy, jump to the postscript](# postscript) If you simply want to save all variables, use dill

With the append method, an error occurred in the type relationship or the data you tried to use. I didn't get an error when using DataFrame, so It may not be meaningless to insert a DataFrame once.

import pandas as pd

First import pandas


# ---Define variables for testing---
a ,b,c,d = 1,2,3,4
xx = [3,6,8]
yy = [5,8,2]
zz = [8,2,8]

Save this function, array

# ---Collect variables into dataframe and save csv---
def getName(obj):
    return  [k for k, v in globals().items() if id(obj) == id(v)][0] #Return variable name with str

Get the variable name as str by using this function It is almost a copy of this article Click here for details

df = pd.DataFrame()

datas1 = [xx,yy,zz] #While using the variable name as a header, put together a one-dimensional array into a dataframe
for data in datas1:
    df = pd.concat([df, pd.DataFrame({getName(data):data})],axis=1) # {}Add dataframe with concat using set

Create a DataFrame based on {name: value} and save more and more to df with concat. By inserting axis = 1, it will be added horizontally

datas2 = [a,b,c,d]#While using the variable name as a header, group the variables into a dataframe
for data in datas2:
    df =  pd.concat([df, pd.DataFrame({getName(data):[data]})],axis=1)# {}Add dataframe with concat using set

Do the same thing as before The value of{name: value}must be an array, so the value is [data]

df.to_csv('test.csv')
print(df)
del df

Save and display. Since df tends to be used in various places, I will delete it for the time being.

You can see that it is displayed and saved properly in the table below.

   xx  yy  zz    a    b    c    d
0   3   5   8  1.0  2.0  3.0  4.0
1   6   8   2  NaN  NaN  NaN  NaN
2   8   2   8  NaN  NaN  NaN  NaN

Source code

import pandas as pd

# ---Define variables for testing---
a ,b,c,d = 1,2,3,4
xx = [3,6,8]
yy = [5,8,2]
zz = [8,2,8]

# ---Collect variables into dataframe and save csv---
def getName(obj):
    return  [k for k, v in globals().items() if id(obj) == id(v)][0] #Return variable name with str

df = pd.DataFrame()

datas1 = [xx,yy,zz] #While using the variable name as a header, put together a one-dimensional array into a dataframe
for data in datas1:
    df = pd.concat([df, pd.DataFrame({getName(data):data})],axis=1) # {}Add dataframe with concat using set
    
datas2 = [a,b,c,d]#While using the variable name as a header, group the variables into a dataframe
for data in datas2:
    df =  pd.concat([df, pd.DataFrame({getName(data):[data]})],axis=1)# {}Add dataframe with concat using set
df.to_csv('test.csv')
print(df)
del df

Postscript

Write directly to csv

# ---Define variables for testing---
a ,b,c,d = 1,2,3,4
xx = [3,6,8]
yy = [5,8,2]
zz = [8,2,8]
import csv
# ---Select a variable and save csv---
def getName(obj):
    return  [k for k, v in globals().items() if id(obj) == id(v)][0] #Return variable name with str

with open('test2.csv','a', newline="") as f:
    datas = [xx,yy,zz,]
    writer = csv.writer(f)
    for data in datas:
        writer.writerow([getName(data)]+data)
    datas = [a,b,c,d]
    for data in datas:
        writer.writerow([getName(data)]+[data])

Case division is to convert a single variable to list type

Output test2.csv

xx,3,6,8
yy,5,8,2
zz,8,2,8
a,1
b,2
c,3
d,4

How to use dill

Save


import dill
dill.dump_session('session.pkl')

Read

import dill
dill.load_session('session.pkl')

All variables can be used as they are.

Recommended Posts

[python] Use DataFrame to label arbitrary variables and arrays together and save them in csv [pandas]
[python] Use DataFrame to label arbitrary variables and arrays together and save them in csv [pandas] Re-UP
How to use is and == in Python
Use os.getenv to get environment variables in Python
I tried to summarize how to use pandas in python
[Introduction to Udemy Python 3 + Application] 36. How to use In and Not
Put the lists together in pandas to make a DataFrame
Display numbers and letters assigned to variables in python print
Collect tweets using tweepy in Python and save them in MongoDB
Comparison of how to use higher-order functions in Python 2 and 3
processing to use notMNIST data in Python (and tried to classify it)
[Python] How to add rows and columns to a table (pandas DataFrame)
How to use functions in separate files Perl and Python versions
How to use Serverless Framework & Python environment variables and manage stages
[Python] When you want to use all variables in another file
[Python3] Save the mean and covariance matrix in json with pandas
I was addicted to confusing class variables and instance variables in Python
[Python] How to use Pandas Series
How to use SQLite in Python
[Introduction to Python] Let's use pandas
How to use Mysql in python
How to use ChemSpider in Python
How to use PubChem in Python
[Introduction to Python] Let's use pandas
[Introduction to Python] Let's use pandas
How to get a specific column name and index name in pandas DataFrame
Data analysis: Easily apply descriptive and inference statistics to CSV data in Python
Connect to postgreSQL from Python and use stored procedures in a loop.
Read CSV file with Python and convert it to DataFrame as it is
[Python] What is pandas Series and DataFrame?
[Python] Summary of how to use pandas
How to split and save a DataFrame
[Python] Use pandas to extract △△ that maximizes ○○
How to access environment variables in Python
[Python] Use and and or when creating variables
How to dynamically define variables in Python
How to install and use pandas_datareader [Python]
Easy way to use Wikipedia in Python
How to reassign index in pandas dataframe
[Python] Pandas to fully understand in 10 minutes
To reference environment variables in Python in Blender
Read Python csv and export to txt
python: How to use locals () and globals ()
How to use __slots__ in Python class
[Python] Add total rows to Pandas DataFrame
How to use Python zip and enumerate
How to use regular expressions in Python
How to read CSV files in Pandas
Adding Series to columns in python pandas
How to use pandas Timestamp and date_range
Tips for those who are wondering how to use is and == in Python
[Python] How to name table data and output it in csv (to_csv method)
Use libsixel to output Sixel in Python and output a Matplotlib graph to the terminal.
[Python learning part 3] Convert pandas DataFrame, Series, and standard List to each other
Save Twitter's tweets with Geo in CSV and plot them on Google Map.
How to use the C library in Python
Scraping tabelog with python and outputting to CSV
Introducing Sinatra-style frameworks and how to use them
Python variables and data types learned in chemoinformatics
How to use Python Image Library in python3 series
[python] Difference between variables and self. Variables in class