Tuning hyperparameters with LightGBM Tuner

Introduction

I would like to try tuning optuna's hyperparameters using the data from the last time I did kaggle. The data is described in Previous article.

optuna was developed by Preferred Networks and automatically tunes hyperparameters. In particular, regarding the tuning of LightGBM, the search is additively compared to the multiplicative search of grid search. For example, n_estimators = [50, 100, 150], mad_depth = [10, 15, 20] Suppose you want to find the best parameter. For grid search, repeat 9 trials with 3x3. On the other hand, optuna's LightGBM tuner repeats 6 trials of 3 + 3. Instead of having fewer trials, optuna prioritizes the parameters that are considered to be important for accuracy. By doing so, it will search for the optimum parameters less often. For more details, see [[Hyperparameter auto-optimization with Optuna extension LightGBM Tuner]](https://tech.preferred.jp/ja/blog/hyperparameter-tuning-with-optuna-integration-lightgbm- See tuner /). The results of various experiments on the superiority of LightGBM Tiner are written.

Then, I would like to do it immediately.

Tuning hyperparameters with lightgbm tuner

When using LightGBM Tuner, import through optuna instead of importing lightgbm normally.

# import lightbgm as lab (Instead of this, import using the method below)
from optuna.integration import lightgbm as lgb


# optuna.__version__  1.3.0

If you learn normally in this way, it will tune automatically.

ts = time.time()

dtrain = lgb.Dataset(x_train, label=y_train)
eval_data = lgb.Dataset(x_val, label=y_val)

param = {
        'objective': 'regression',
        'metric': 'rmse',
        'verbosity': -1,
        'boosting_type': 'gbdt',
    }

best = lgb.train(param, 
                 dtrain,
                 valid_sets=eval_data,
                 early_stopping_rounds=100)

time.time() - ts

# time: 2945.9576

"""
###Parameters that are actually tuned###

param = {
        'lambda_l1': trial.suggest_loguniform('lambda_l1', 1e-8, 10.0),
        'lambda_l2': trial.suggest_loguniform('lambda_l2', 1e-8, 10.0),
        'num_leaves': trial.suggest_int('num_leaves', 2, 256),
        'feature_fraction': trial.suggest_uniform('feature_fraction', 0.4, 1.0),
        'bagging_fraction': trial.suggest_uniform('bagging_fraction', 0.4, 1.0),
        'bagging_freq': trial.suggest_int('bagging_freq', 1, 7),
        'min_child_samples': trial.suggest_int('min_child_samples', 5, 100),
    }
"""

By the way, the default value of trial is 1000. Why is early_stopping_rounds set to 100?

The execution time took about 50 minutes. It took quite a while, but I don't usually tune too much I personally think that it's okay to spend about an hour if you can search the parameters in detail.

The parameters after tuning are as follows.

best.params
best.best_iteration
best.best_score

"""
     {'objective': 'regression',
      'metric': 'rmse',
      'verbosity': -1,
      'boosting_type': 'gbdt',
      'lambda_l1': 0.0005523588106120283,
      'lambda_l2': 6.72929973145413e-07,
      'num_leaves': 6,
      'feature_fraction': 0.8,
      'bagging_fraction': 1.0,
      'bagging_freq': 0,
      'min_child_samples': 20}

      best.best_iteration: 555

      best.best_score(rmse): 0.93714
"""

Predicting with a model using this parameter gave a kaggle score of 0.94293. It has grown a little more than last time.

I don't think there will be much change in tuning anymore, so I have to do feature engineering elsewhere.

difficult,,,

At the end

This time, we tuned hyperparameters using LightGBM Tuner.

I haven't compared the accuracy with other tuning methods, but the link above We also use a common benchmark to compare it to other tuning techniques, and the results show that the LightGBM Tuner is the best. It seems that there is room for improvement because it is a method that has only been completed, but I feel that it is much easier than tuning through various trials and errors. Also, instead of tuning with rough values like grid search, it tunes finely, so it is recommended when you want to improve the score a little later with kaggle.