Tuning hyperparameters with GridSearch using pipeline with keras

Keras learns the weights of variables, but there are also fixed parameters such as how many neurons are good, how many layers are needed, and the optimal learning rate. This time, I will pass it to GridSearch of scikit-learn to tune these parameters. In this example, it is written assuming the case of classifying with data such as iris.

Preparation

In addition to the libraries needed to build the model, load the following:

python


from sklearn.grid_search import GridSearchCV
from sklearn.pipeline import Pipeline

Function build the model

Create a function with the parameter you want to change as an argument. In this example, for the sake of simplicity, the number of neurons in the second and subsequent layers is calculated as the nth root of the original number of neurons.

python


def create_model(self, evMethod, neurons, layers, learn_rate):
	# Criate model
	model = Sequential()
	model.add(Dense(neurons, input_dim=self.column_length, kernel_initializer='normal', activation='relu'))
	for i in range(1, layers):
		model.add(Dense(int(numpy.ceil(numpy.power(neurons,1/i))), kernel_initializer='normal', activation='relu'))
	model.add(Dense(3, kernel_initializer='normal', activation='softmax'))
	# Compile model
	adam = optimizers.Adam(lr=learn_rate)
	model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])
	return model

Prepare an array to pass to the model

All must be passed as an array.

python


batch_size = [1000,300,100]
epochs = [10, 50, 100]
neurons = [20, 30, 45]
learn_rate = [0.001, 0.005, 0.01, 0.07]
layers = [1,3,5]

Learning

Pass the model to the pipeline

Add the created model as mlp to the estimator. The initial value is not used, so it is appropriate.

python


estimators = []
estimators.append(('mlp', KerasClassifier(build_fn=self.create_model, epochs=10, batch_size=200, verbose=1)))
pipeline = Pipeline(estimators)

Pass parameters to Pipeline

python


param_grid = dict(mlp__neurons = neurons, mlp__batch_size = batch_size, mlp__epochs=epochs, mlp__learn_rate=learn_rate, mlp__layers=layers)

Grid Search is the same as normal usage

python


grid = GridSearchCV(estimator=pipeline, param_grid=param_grid)
grid_result = grid.fit(self.X, self.Y)

Show the best combination

best_estimator_ will tell you the parameters of the best model. The accuracy is also displayed.

python


clf = grid_result.best_estimator_
print(clf.get_params())
accuracy = clf.score(self.X, self.Y)
print("\nAccuracy: %.2f" % (accuracy))

Save the best model

Using GridSearch takes dozens of times longer learning time, so it is also recommended to save the best model. Please note that the call is special compared to the case of normal model construction.

python


clf.steps[1][1].model.save('file to path')

You can tune other variables such as activation function in this way, so please take advantage of it.

Postscript: Randomized Gridsearch version

Since the memory was over during learning, I changed from just GridSearch to Randomized GridSearch to reduce the number of trials. Almost all you have to do is replace the function name. If you have a large number of parameter combinations, this is more realistic for execution.

python


grid = RandomizedSearchCV(estimator=pipeline, param_distributions=param_grid, n_iter=30)

Recommended Posts

Tuning hyperparameters with GridSearch using pipeline with keras
Tuning hyperparameters with LightGBM Tuner
Tuning Keras parameters with Keras Tuner
Parameter optimization automation with Keras with GridSearch CV
Parameter tuning with GridSearchCV / RandomizedSearchCV while using Voting Classifier
Parameter tuning with luigi
CIFAR-10 tutorial with Keras
Multivariate LSTM with Keras
I tried handwriting recognition of runes with CNN using Keras