Tuning Keras parameters with Keras Tuner

Hyperparameter tuning modules such as Optuna are coming out, but I would like to try the recently introduced hypertuning module for Keras Keras Tuner. ..

Basically, it is a Japanese translation of here.

conditions

The following modules are required to operate Keras Tuner.

--Python 3.6 and above --TensorFlow 2.0 or higher

The Keras used is also tf.keras.

Installation

You can install it with pip.

pip install -U keras-tuner

image.png

By the way, if you have AutoKeras installed, you already have Keras Tuner installed. (Auto Keras uses Keras Tuner for parameter tuning)

You can also install from the source.

Simple sample

First, let's take a simple sample as an example to see how to use it.

from tensorflow import keras
from tensorflow.keras import layers

from kerastuner.tuners import RandomSearch

(x, y), (val_x, val_y) = keras.datasets.mnist.load_data()
x = x.astype('float32') / 255.
val_x = val_x.astype('float32') / 255.

x = x[:10000]
y = y[:10000]

def build_model(hp):
    model = keras.Sequential()
    model.add(layers.Flatten(input_shape=(28, 28)))
    for i in range(hp.Int('num_layers', 2, 20)):
        model.add(layers.Dense(units=hp.Int('units_' + str(i), 32, 512, 32),
                               activation='relu'))
    model.add(layers.Dense(10, activation='softmax'))
    model.compile(
        optimizer=keras.optimizers.Adam(
            hp.Choice('learning_rate', [1e-2, 1e-3, 1e-4])),
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy'])
    return model

tuner = RandomSearch(
    build_model,
    objective='val_accuracy',
    max_trials=5,
    executions_per_trial=3,
    directory='test_dir',
    project_name='helloworld')

tuner.search_space_summary()

tuner.search(x=x,
             y=y,
             epochs=3,
             validation_data=(val_x, val_y))

tuner.results_summary()

The dataset used is the familiar MNIST.

import

To use "Keras Tuner", import "Random Search".

Model building

First, prepare a function for building a model. Prepare the hyperparameter variable "hp" as an argument.

There are two ways to adjust the value in this example.

--Value range --hp.Int (name, start, end, tick) --The default value for ticks is 1 --Specified value --hp.Choice (name, [list of values])

Search algorithm

An instance that performs random search (RandomSearch) is generated as a search algorithm. You can also specify Hyperband (https://arxiv.org/pdf/1603.06560.pdf) (Hyperband).

Example


from kerastuner.tuners import Hyperband

tuner = Hyperband(
    hypermodel,
    objective='val_accuracy',
    max_epochs=40,
    directory='my_dir',
    project_name='helloworld')

Here, in addition to setting the function for building the model, set the number of trials (max_trials) and the number of models for each trial (executions_per_trial). The results are saved in the "test_dir / helloworld" directory. image.png

Display the contents of the search space with search_space_summary ().

[Search space summary]
 |-Default search space size: 4
 > num_layers (Int)
 |-default: None
 |-max_value: 20
 |-min_value: 2
 |-sampling: None
 |-step: 1
 > units_0 (Int)
 |-default: None
 |-max_value: 512
 |-min_value: 32
 |-sampling: None
 |-step: 32
 > units_1 (Int)
 |-default: None
 |-max_value: 512
 |-min_value: 32
 |-sampling: None
 |-step: 32
 > learning_rate (Choice)
 |-default: 0.01
 |-ordered: True
 |-values: [0.01, 0.001, 0.0001]

In this example, the values for each parameter are displayed.

Search

Search for the best parameter with search (). The argument is the same as the so-called "fit ()".

The results are displayed with results_summary ().

[Results summary]
 |-Results in test_dir\helloworld
 |-Showing 10 best trials
 |-Objective(name='val_accuracy', direction='max')
[Trial summary]
 |-Trial ID: 5dae177f590b1ff7f9a549cda6ae9567
 |-Score: 0.9282999634742737
 |-Best step: 0
 > Hyperparameters:
 |-learning_rate: 0.0001
 |-num_layers: 10
 |-units_0: 256
 |-units_1: 352
 |-units_10: 416
 |-units_11: 448
 |-units_12: 480
 |-units_13: 128
 |-units_14: 64
 |-units_15: 32
 |-units_16: 512
 |-units_17: 256
 |-units_18: 96
 |-units_19: 64
 |-units_2: 480
 |-units_3: 320
 |-units_4: 64
 |-units_5: 512
 |-units_6: 320
 |-units_7: 512
 |-units_8: 320
 |-units_9: 64
[Trial summary]
 |-Trial ID: 496aa846dabfafb3c67270e3ce810234
 |-Score: 0.9157333374023438
 |-Best step: 0
 > Hyperparameters:
 |-learning_rate: 0.01
 |-num_layers: 3
 |-units_0: 64
 |-units_1: 416
 |-units_2: 32
[Trial summary]
 |-Trial ID: c516cbd03faf4aa32cf8182ab34eb114
 |-Score: 0.8071333765983582
 |-Best step: 0
 > Hyperparameters:
 |-learning_rate: 0.0001
 |-num_layers: 18
 |-units_0: 160
 |-units_1: 384
 |-units_10: 32
 |-units_11: 32
 |-units_12: 32
 |-units_13: 32
 |-units_14: 32
 |-units_15: 32
 |-units_16: 32
 |-units_17: 32
 |-units_2: 320
 |-units_3: 512
 |-units_4: 416
 |-units_5: 416
 |-units_6: 96
 |-units_7: 128
 |-units_8: 160
 |-units_9: 32
[Trial summary]
 |-Trial ID: 81260e9782e1bc81da957360c6322371
 |-Score: 0.7860667109489441
 |-Best step: 0
 > Hyperparameters:
 |-learning_rate: 0.01
 |-num_layers: 9
 |-units_0: 64
 |-units_1: 512
 |-units_2: 224
 |-units_3: 32
 |-units_4: 32
 |-units_5: 32
 |-units_6: 32
 |-units_7: 32
 |-units_8: 32
[Trial summary]
 |-Trial ID: eb9da62f11d1bb75b11b9d05c79ae7ec
 |-Score: 0.11349999904632568
 |-Best step: 0
 > Hyperparameters:
 |-learning_rate: 0.01
 |-num_layers: 20
 |-units_0: 224
 |-units_1: 288
 |-units_10: 32
 |-units_11: 64
 |-units_12: 448
 |-units_13: 64
 |-units_14: 512
 |-units_15: 96
 |-units_16: 256
 |-units_17: 64
 |-units_18: 32
 |-units_19: 32
 |-units_2: 352
 |-units_3: 480
 |-units_4: 128
 |-units_5: 160
 |-units_6: 224
 |-units_7: 480
 |-units_8: 224
 |-units_9: 352

In this example, 5 trials are displayed. (In order of best grade)

If you want to get it as a model

Example


models = tuner.get_best_models(num_models=2)

Use tuner.get_best_models () like this.

Derivation of class

You can also define the model construction in classes instead of functions. In that case, it will be derived from the HyperModel class. The only function that needs to be implemented is build ().

from tensorflow import keras
from tensorflow.keras import layers

from kerastuner.tuners import RandomSearch
from kerastuner import HyperModel

class MyHyperModel(HyperModel):

    def __init__(self, num_classes):
        self.num_classes = num_classes

    def build(self, hp):
        model = keras.Sequential()
        for i in range(hp.Int('num_layers', 2, 20)):
            model.add(layers.Dense(units=hp.Int('units_' + str(i), 32, 512, 32),
                               activation='relu'))
        model.add(layers.Dense(self.num_classes, activation='softmax'))
        model.compile(
            optimizer=keras.optimizers.Adam(
                hp.Choice('learning_rate',
                          values=[1e-2, 1e-3, 1e-4])),
            loss='sparse_categorical_crossentropy',
            metrics=['accuracy'])
        return model

hypermodel = MyHyperModel(num_classes=10)

tuner = RandomSearch(
    hypermodel,
    objective='val_accuracy',
    max_trials=10,
    directory='my_dir',
    project_name='helloworld')

tuner.search(x, y,
             epochs=5,
             validation_data=(val_x, val_y))

At this time, you can specify the default value for the value to be adjusted.

Example


hp.Int('units',
       min_value=32,
       max_value=512,
       step=32,
       default=128)

Embedded model

Models for ResNet and Xception are already available.

Example


from kerastuner.applications import HyperResNet

hypermodel = HyperResNet(input_shape=(128, 128, 3), num_classes=10)

Example


from kerastuner.applications import HyperXception

hypermodel = HyperXception(input_shape=(128, 128, 3), num_classes=10)

If you want to change the parameters to be adjusted in the built-in model, prepare hyperparameters with names as shown below and set them when creating an instance of the search algorithm. At that time, set tune_new_entries to False.

from kerastuner.applications import HyperXception
from kerastuner import HyperParameters
from kerastuner.tuners import Hyperband

hypermodel = HyperXception(input_shape=(128, 128, 3), num_classes=10)

hp = HyperParameters()
hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4])

tuner = Hyperband(
    hypermodel,
    hyperparameters=hp,
    tune_new_entries=False,
    objective='val_accuracy',
    max_epochs=40,
    directory='my_dir',
    project_name='helloworld')

tuner.search(x, y,
             validation_data=(val_x, val_y))

On the contrary, if you do not want to adjust it, you can set a fixed value. Specify the value with fixed () and set tune_new_entries = True.

Example


hypermodel = HyperXception(input_shape=(128, 128, 3), num_classes=10)

hp = HyperParameters()
hp.Fixed('learning_rate', value=1e-4)

tuner = Hyperband(
    hypermodel,
    hyperparameters=hp,
    tune_new_entries=True,
    objective='val_accuracy',
    max_epochs=40,
    directory='my_dir',
    project_name='helloworld')

tuner.search(x, y,
             validation_data=(val_x, val_y))

In addition, the variables "optimizer", "loss" and "metrics" can be changed directly.

Example


hypermodel = HyperXception(input_shape=(128, 128, 3), num_classes=10)

tuner = Hyperband(
    hypermodel,
    optimizer=keras.optimizers.Adam(1e-3),
    loss='mse',
    metrics=[keras.metrics.Precision(name='precision'),
             keras.metrics.Recall(name='recall')],
    objective='val_precision',
    max_epochs=40,
    directory='my_dir',
    project_name='helloworld')

tuner.search(x, y,
             validation_data=(val_x, val_y))

Other functions

In addition, the following functions are available.

-Distributed Tuning --Perform parameter adjustment in parallel -Customize Tuner --Correspondence to GAN and reinforcement learning --Used outside the model building function (preprocessing, data extension, test time extension)

Summary

I think it can be used when you want to tune parameters a little with Keras.

Reference URL

https://tksmml.hatenablog.com/entry/2020/02/01/093000

Recommended Posts

Tuning Keras parameters with Keras Tuner
Tuning hyperparameters with LightGBM Tuner
Tuning hyperparameters with GridSearch using pipeline with keras
Image recognition with keras
Parameter tuning with luigi (2)
Parameter tuning with luigi
CIFAR-10 tutorial with Keras
Multivariate LSTM with Keras
Install Keras (used with Anaconda)
Multiple regression analysis with Keras
Auto Encodder notes with Keras
Implemented word2vec with Theano + Keras
Sentence generation with GRU (keras)
Adjusting LightGBM parameters with Optuna
Easily build CNN with Keras
Implemented Efficient GAN with keras
4. Circle parameters with neural network!
Image recognition with Keras + OpenCV
MNIST (DCNN) with Keras (TensorFlow backend)
Predict Kaggle's Titanic with keras (kaggle ⑦)
Function parameters with only an asterisk'*'
[TensorFlow] [Keras] Neural network construction with Keras
Implement Keras LSTM feedforward with numpy
Compare DCGAN and pix2pix with keras
Score-CAM implementation with keras. Comparison with Grad-CAM
Prediction of sine wave with keras
Various Fine Tuning with Mobilenet v2
Beginner RNN (LSTM) | Try with Keras
Write Reversi AI with Keras + DQN
4/22 prediction of sine wave with keras