Hyperparameter tuning modules such as Optuna are coming out, but I would like to try the recently introduced hypertuning module for Keras Keras Tuner. ..
Basically, it is a Japanese translation of here.
The following modules are required to operate Keras Tuner.
--Python 3.6 and above --TensorFlow 2.0 or higher
The Keras used is also tf.keras.
You can install it with pip.
pip install -U keras-tuner
By the way, if you have AutoKeras installed, you already have Keras Tuner installed. (Auto Keras uses Keras Tuner for parameter tuning)
You can also install from the source.
First, let's take a simple sample as an example to see how to use it.
from tensorflow import keras
from tensorflow.keras import layers
from kerastuner.tuners import RandomSearch
(x, y), (val_x, val_y) = keras.datasets.mnist.load_data()
x = x.astype('float32') / 255.
val_x = val_x.astype('float32') / 255.
x = x[:10000]
y = y[:10000]
def build_model(hp):
model = keras.Sequential()
model.add(layers.Flatten(input_shape=(28, 28)))
for i in range(hp.Int('num_layers', 2, 20)):
model.add(layers.Dense(units=hp.Int('units_' + str(i), 32, 512, 32),
activation='relu'))
model.add(layers.Dense(10, activation='softmax'))
model.compile(
optimizer=keras.optimizers.Adam(
hp.Choice('learning_rate', [1e-2, 1e-3, 1e-4])),
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
return model
tuner = RandomSearch(
build_model,
objective='val_accuracy',
max_trials=5,
executions_per_trial=3,
directory='test_dir',
project_name='helloworld')
tuner.search_space_summary()
tuner.search(x=x,
y=y,
epochs=3,
validation_data=(val_x, val_y))
tuner.results_summary()
The dataset used is the familiar MNIST.
To use "Keras Tuner", import "Random Search".
First, prepare a function for building a model. Prepare the hyperparameter variable "hp" as an argument.
There are two ways to adjust the value in this example.
--Value range --hp.Int (name, start, end, tick) --The default value for ticks is 1 --Specified value --hp.Choice (name, [list of values])
An instance that performs random search (RandomSearch) is generated as a search algorithm. You can also specify Hyperband (https://arxiv.org/pdf/1603.06560.pdf) (Hyperband).
Example
from kerastuner.tuners import Hyperband
tuner = Hyperband(
hypermodel,
objective='val_accuracy',
max_epochs=40,
directory='my_dir',
project_name='helloworld')
Here, in addition to setting the function for building the model, set the number of trials (max_trials) and the number of models for each trial (executions_per_trial). The results are saved in the "test_dir / helloworld" directory.
Display the contents of the search space with search_space_summary ().
[Search space summary]
|-Default search space size: 4
> num_layers (Int)
|-default: None
|-max_value: 20
|-min_value: 2
|-sampling: None
|-step: 1
> units_0 (Int)
|-default: None
|-max_value: 512
|-min_value: 32
|-sampling: None
|-step: 32
> units_1 (Int)
|-default: None
|-max_value: 512
|-min_value: 32
|-sampling: None
|-step: 32
> learning_rate (Choice)
|-default: 0.01
|-ordered: True
|-values: [0.01, 0.001, 0.0001]
In this example, the values for each parameter are displayed.
Search for the best parameter with search (). The argument is the same as the so-called "fit ()".
The results are displayed with results_summary ().
[Results summary]
|-Results in test_dir\helloworld
|-Showing 10 best trials
|-Objective(name='val_accuracy', direction='max')
[Trial summary]
|-Trial ID: 5dae177f590b1ff7f9a549cda6ae9567
|-Score: 0.9282999634742737
|-Best step: 0
> Hyperparameters:
|-learning_rate: 0.0001
|-num_layers: 10
|-units_0: 256
|-units_1: 352
|-units_10: 416
|-units_11: 448
|-units_12: 480
|-units_13: 128
|-units_14: 64
|-units_15: 32
|-units_16: 512
|-units_17: 256
|-units_18: 96
|-units_19: 64
|-units_2: 480
|-units_3: 320
|-units_4: 64
|-units_5: 512
|-units_6: 320
|-units_7: 512
|-units_8: 320
|-units_9: 64
[Trial summary]
|-Trial ID: 496aa846dabfafb3c67270e3ce810234
|-Score: 0.9157333374023438
|-Best step: 0
> Hyperparameters:
|-learning_rate: 0.01
|-num_layers: 3
|-units_0: 64
|-units_1: 416
|-units_2: 32
[Trial summary]
|-Trial ID: c516cbd03faf4aa32cf8182ab34eb114
|-Score: 0.8071333765983582
|-Best step: 0
> Hyperparameters:
|-learning_rate: 0.0001
|-num_layers: 18
|-units_0: 160
|-units_1: 384
|-units_10: 32
|-units_11: 32
|-units_12: 32
|-units_13: 32
|-units_14: 32
|-units_15: 32
|-units_16: 32
|-units_17: 32
|-units_2: 320
|-units_3: 512
|-units_4: 416
|-units_5: 416
|-units_6: 96
|-units_7: 128
|-units_8: 160
|-units_9: 32
[Trial summary]
|-Trial ID: 81260e9782e1bc81da957360c6322371
|-Score: 0.7860667109489441
|-Best step: 0
> Hyperparameters:
|-learning_rate: 0.01
|-num_layers: 9
|-units_0: 64
|-units_1: 512
|-units_2: 224
|-units_3: 32
|-units_4: 32
|-units_5: 32
|-units_6: 32
|-units_7: 32
|-units_8: 32
[Trial summary]
|-Trial ID: eb9da62f11d1bb75b11b9d05c79ae7ec
|-Score: 0.11349999904632568
|-Best step: 0
> Hyperparameters:
|-learning_rate: 0.01
|-num_layers: 20
|-units_0: 224
|-units_1: 288
|-units_10: 32
|-units_11: 64
|-units_12: 448
|-units_13: 64
|-units_14: 512
|-units_15: 96
|-units_16: 256
|-units_17: 64
|-units_18: 32
|-units_19: 32
|-units_2: 352
|-units_3: 480
|-units_4: 128
|-units_5: 160
|-units_6: 224
|-units_7: 480
|-units_8: 224
|-units_9: 352
In this example, 5 trials are displayed. (In order of best grade)
If you want to get it as a model
Example
models = tuner.get_best_models(num_models=2)
Use tuner.get_best_models () like this.
You can also define the model construction in classes instead of functions. In that case, it will be derived from the HyperModel class. The only function that needs to be implemented is build ().
from tensorflow import keras
from tensorflow.keras import layers
from kerastuner.tuners import RandomSearch
from kerastuner import HyperModel
class MyHyperModel(HyperModel):
def __init__(self, num_classes):
self.num_classes = num_classes
def build(self, hp):
model = keras.Sequential()
for i in range(hp.Int('num_layers', 2, 20)):
model.add(layers.Dense(units=hp.Int('units_' + str(i), 32, 512, 32),
activation='relu'))
model.add(layers.Dense(self.num_classes, activation='softmax'))
model.compile(
optimizer=keras.optimizers.Adam(
hp.Choice('learning_rate',
values=[1e-2, 1e-3, 1e-4])),
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
return model
hypermodel = MyHyperModel(num_classes=10)
tuner = RandomSearch(
hypermodel,
objective='val_accuracy',
max_trials=10,
directory='my_dir',
project_name='helloworld')
tuner.search(x, y,
epochs=5,
validation_data=(val_x, val_y))
At this time, you can specify the default value for the value to be adjusted.
Example
hp.Int('units',
min_value=32,
max_value=512,
step=32,
default=128)
Models for ResNet and Xception are already available.
Example
from kerastuner.applications import HyperResNet
hypermodel = HyperResNet(input_shape=(128, 128, 3), num_classes=10)
Example
from kerastuner.applications import HyperXception
hypermodel = HyperXception(input_shape=(128, 128, 3), num_classes=10)
If you want to change the parameters to be adjusted in the built-in model, prepare hyperparameters with names as shown below and set them when creating an instance of the search algorithm. At that time, set tune_new_entries to False.
from kerastuner.applications import HyperXception
from kerastuner import HyperParameters
from kerastuner.tuners import Hyperband
hypermodel = HyperXception(input_shape=(128, 128, 3), num_classes=10)
hp = HyperParameters()
hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4])
tuner = Hyperband(
hypermodel,
hyperparameters=hp,
tune_new_entries=False,
objective='val_accuracy',
max_epochs=40,
directory='my_dir',
project_name='helloworld')
tuner.search(x, y,
validation_data=(val_x, val_y))
On the contrary, if you do not want to adjust it, you can set a fixed value. Specify the value with fixed () and set tune_new_entries = True.
Example
hypermodel = HyperXception(input_shape=(128, 128, 3), num_classes=10)
hp = HyperParameters()
hp.Fixed('learning_rate', value=1e-4)
tuner = Hyperband(
hypermodel,
hyperparameters=hp,
tune_new_entries=True,
objective='val_accuracy',
max_epochs=40,
directory='my_dir',
project_name='helloworld')
tuner.search(x, y,
validation_data=(val_x, val_y))
In addition, the variables "optimizer", "loss" and "metrics" can be changed directly.
Example
hypermodel = HyperXception(input_shape=(128, 128, 3), num_classes=10)
tuner = Hyperband(
hypermodel,
optimizer=keras.optimizers.Adam(1e-3),
loss='mse',
metrics=[keras.metrics.Precision(name='precision'),
keras.metrics.Recall(name='recall')],
objective='val_precision',
max_epochs=40,
directory='my_dir',
project_name='helloworld')
tuner.search(x, y,
validation_data=(val_x, val_y))
In addition, the following functions are available.
-Distributed Tuning --Perform parameter adjustment in parallel -Customize Tuner --Correspondence to GAN and reinforcement learning --Used outside the model building function (preprocessing, data extension, test time extension)
I think it can be used when you want to tune parameters a little with Keras.
https://tksmml.hatenablog.com/entry/2020/02/01/093000
Recommended Posts