Tuning hyperparameters with GridSearch using pipeline with keras

Keras learns the weights of variables, but there are also fixed parameters such as how many neurons are good, how many layers are needed, and the optimal learning rate. This time, I will pass it to GridSearch of scikit-learn to tune these parameters. In this example, it is written assuming the case of classifying with data such as iris.

Preparation

In addition to the libraries needed to build the model, load the following:

python

from sklearn.grid_search import GridSearchCV
from sklearn.pipeline import Pipeline

Function build the model

Create a function with the parameter you want to change as an argument. In this example, for the sake of simplicity, the number of neurons in the second and subsequent layers is calculated as the nth root of the original number of neurons.

python

def create_model(self, evMethod, neurons, layers, learn_rate):
	# Criate model
	model = Sequential()
	model.add(Dense(neurons, input_dim=self.column_length, kernel_initializer='normal', activation='relu'))
	for i in range(1, layers):
		model.add(Dense(int(numpy.ceil(numpy.power(neurons,1/i))), kernel_initializer='normal', activation='relu'))
	model.add(Dense(3, kernel_initializer='normal', activation='softmax'))
	# Compile model
	adam = optimizers.Adam(lr=learn_rate)
	model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])
	return model

Prepare an array to pass to the model

All must be passed as an array.

python

batch_size = [1000,300,100]
epochs = [10, 50, 100]
neurons = [20, 30, 45]
learn_rate = [0.001, 0.005, 0.01, 0.07]
layers = [1,3,5]

Learning

Pass the model to the pipeline

Add the created model as mlp to the estimator. The initial value is not used, so it is appropriate.

python

estimators = []
estimators.append(('mlp', KerasClassifier(build_fn=self.create_model, epochs=10, batch_size=200, verbose=1)))
pipeline = Pipeline(estimators)

Pass parameters to Pipeline

python

param_grid = dict(mlp__neurons = neurons, mlp__batch_size = batch_size, mlp__epochs=epochs, mlp__learn_rate=learn_rate, mlp__layers=layers)

Grid Search is the same as normal usage

python

grid = GridSearchCV(estimator=pipeline, param_grid=param_grid)
grid_result = grid.fit(self.X, self.Y)

Show the best combination

best_estimator_ will tell you the parameters of the best model. The accuracy is also displayed.

python

clf = grid_result.best_estimator_
print(clf.get_params())
accuracy = clf.score(self.X, self.Y)
print("\nAccuracy: %.2f" % (accuracy))

Save the best model

Using GridSearch takes dozens of times longer learning time, so it is also recommended to save the best model. Please note that the call is special compared to the case of normal model construction.

python

clf.steps[1][1].model.save('file to path')

You can tune other variables such as activation function in this way, so please take advantage of it.

Postscript: Randomized Gridsearch version

Since the memory was over during learning, I changed from just GridSearch to Randomized GridSearch to reduce the number of trials. Almost all you have to do is replace the function name. If you have a large number of parameter combinations, this is more realistic for execution.

python

grid = RandomizedSearchCV(estimator=pipeline, param_distributions=param_grid, n_iter=30)