Try to process Titanic data with preprocessing library DataLiner (conversion)

Introduction

This is the third article that introduces each process of Python's preprocessing library DataLiner. This time I would like to introduce the conversion system.

Release article: https://qiita.com/shallowdf20/items/36727c9a18f5be365b37

Installation

! pip install -U dataliner

Data preparation

Prepare Titanic data as usual.

import pandas as pd
import dataliner as dl

df = pd.read_csv('train.csv')
target_col = 'Survived'

X = df.drop(target_col, axis=1)
y = df[target_col]

StandardScaling / StandardizeData(deprecated) Converts the data to mean 0 variance 1. Unlike libraries such as Sklearn, even if category columns are included, only numeric columns are automatically determined, and since they are returned by pandas DataFrame, subsequent processing is easy. Since StandardizeData has been renamed to StandardScaling, a deprecation warning will be issued, and it will be deleted in ver.1.3.0.

trans = dl.StandardScaling() 
Xt = trans.fit_transform(X)

MinMaxScaling Converts the data so that it fits between 0 and 1.

trans = dl.MinMaxScaling() 
Xt = trans.fit_transform(X)

BinarizeNaN Finds the column that contains the missing value and creates a new binary column that tells if the column was missing.

trans = dl.BinarizeNaN() 
Xt = trans.fit_transform(X)

CountRowNaN For each data point (row), count how many missing values are included and add the sum of the missing values as a new feature.

trans = dl.CountRowNaN() 
Xt = trans.fit_transform(X)

ImputeNaN Complements missing values. The default arguments are that the numeric column is complemented by the average and the category column is complemented by the mode. It can be changed with num_strategy and cat_strategy.

trans = dl.ImputeNaN() 
Xt = trans.fit_transform(X)

ClipData Define the X quantile and replace the data above and below the upper limit with the upper and lower limits. You can adjust how much you want to clip with the threshold argument, which defaults to 1%: 99%.

trans = dl.ClipData() 
Xt = trans.fit_transform(X)

GroupRareCategory In the categorical variables, the infrequently occurring categories are collectively replaced with the string "RareCategory". Helps reduce cardinality. It is effective to use it before applying OneHotEncoding. With the argument threshold, you can change what percentage or less of the number of data to replace. The default is 1%.

trans = dl.GroupRareCategory() 
Xt = trans.fit_transform(X)

in conclusion

So, this time I introduced the items of the conversion system of DataLiner. As the pre-processing implemented at the moment (ver.1.1.6), it will be the last in the Append system to be introduced next time.

Dataliner release article: https://qiita.com/shallowdf20/items/36727c9a18f5be365b37 GitHub: https://github.com/shallowdf20/dataliner PyPI: https://pypi.org/project/dataliner/