[Python] How to run Jupyter-notebook + pandas + multiprocessing (Pool) [pandas] Memo
Purpose
- I had to load a lot of data to analyze a lot of csv data
- Turning the loop was too slow and frustrating.
problem
- Cannot parallelize read_csv of pandas using multiprocessing (Pool) in Jupyter-notebook.
- Konohen or [Konohen](https://github.com/ The same thing was written in ipython / ipython / issues / 10894).
- Is it a problem that occurs only on Windows machines?
Conclusion
- The workaround is to save the function as
func.pyand import it with Jupyter-notebook.
NG example
- The last cell does not end. (Isn't it started in the first place?)
OK example
- Copy the whole as
func.pyto the same folder as.ipynb - Place the
__init__.pyfile - Import and run func on Jupyter-notebook
func.py
import pandas as pd
import os,glob
from datetime import datetime as dt
from multiprocessing import Pool
FOLDER_PATH = r'folder_path\\'
FILE_TYPE = r'*.csv'
FILE_FORMAT = 'Report_%Y%m%d.csv'
def read_report_to_dataframe():
#List file paths
csv_pathlist = glob.glob(FOLDER_PATH + FILE_TYPE)
with Pool(os.cpu_count()) as p:
df = pd.concat(p.map(read_report, csv_pathlist))
return df
#report read
def read_report(csv_path):
separator_list = [';',',']
for sep in separator_list:
df = pd.read_csv(filepath_or_buffer=csv_path,
engine='python',
parse_dates=[0],
index_col=[0],
skiprows=[1],
nrows=96,
sep=sep)
#Check if the data frame is empty
if not df.empty:
break
return df
- Read and execute with jupyter-notebook.
jupyter-notebook
import func
func.read_report_to_dataframe()
Summary
- Stop jupyter-notebook and make it with pycharm.
Recommended Posts