Save & load data with joblib, pickle

Save and load data with joblib, pickle

I will use it often, so I will leave it for notes.

joblib and pickle are ** libraries that can store various data in a nice way **. It can be used not only for text and csv, but also for saving trained models. Also, I feel that the reading and writing speeds are fast. (It seems that memory is used quite a bit)

Basically, it seems that using joblib is better in terms of memory than pickle.

Example

import pandas as pd

arr = ['a','b','c','d','e']
df = pd.DataFrame({'data':arr})
df.head(5)

# data
#0	a
#1	b
#2	c
#3	d
#4	e

Save & load data with joblib

import joblib

#Data storage
joblib.dump(df,'test_jb.pkl', compress=3)
#Data reading
load_df = joblib.load('test_jb.pkl')
load_df.head()

# data
#0	a
#1	b
#2	c
#3	d
#4	e

Save & load data with pickle

import pandas as pd

#Data storage
df.to_pickle('test_pk.pkl')
#Data reading
load_df2 = pd.read_pickle('test_pk.pkl')
load_df2.head()

# data
#0	a
#1	b
#2	c
#3	d
#4	e

If you change the "data" part to "learned model", you can save the model as it is.

Recommended Posts

Save & load data with joblib, pickle
LOAD DATA with PyMysql
Save tweet data with Django
Save / load in-memory DB with python sqlite3
I tried to save the data with discord
Save data to flash with STM32 Nucleo Board
Save the object to a file with pickle
Data analysis with python 2
Save memory with `` __slots__``
Visualize data with Streamlit
Reading data with TensorFlow
Data visualization with pandas
Data manipulation with Pandas!
Shuffle data with pandas
Data Augmentation with openCV
Normarize data with Scipy
Data analysis with Python
Embed audio data with Jupyter
Graph Excel data with matplotlib (1)
Load nested json with pandas
Artificial data generation with numpy
Extract Twitter data with CSV
Save images with web scraping
Get Youtube data with python
Clustering ID-POS data with LDA
Learn new data with PaintsChainer
Binarize photo data with OpenCV
Graph Excel data with matplotlib (2)
Load test Websocket with Locust
Data processing tips with Pandas
Interpolate 2D data with scipy.interpolate.griddata
Read json data with python
How to enable Keras Regressor to be saved with pickle or joblib
Save the results of crawling with Scrapy to the Google Data Store
Let's create a tic-tac-toe AI with Pylearn 2-Save and load models-