"Deep Learning from scratch" Self-study memo (No. 10-2) Initial value of weight

While reading "Deep Learning from scratch" (written by Yasuki Saito, published by O'Reilly Japan), I will make a note of the sites I referred to. Part 10 ← → Part 11

I changed the source code ch06 / optimizer_compare_mnist.py used in the comparison of the update method using the MNIST data set a little, and tried several ways to set the initial value.

# coding: utf-8
import os
import sys
sys.path.append(os.pardir)  #Settings for importing files in the parent directory
import matplotlib.pyplot as plt
from dataset.mnist import load_mnist
from common.util import smooth_curve
from common.multi_layer_net import MultiLayerNet
from common.optimizer import *

# 0:Read MNIST data==========
(x_train, t_train), (x_test, t_test) = load_mnist(normalize=True)

train_size = x_train.shape[0]
batch_size = 128
max_iterations = 2000

# 1:Experiment settings==========
optimizers = {}
optimizers['SGD'] = SGD()
optimizers['Momentum'] = Momentum()
optimizers['AdaGrad'] = AdaGrad()
optimizers['Adam'] = Adam()
#optimizers['RMSprop'] = RMSprop()

networks = {}
train_loss = {}
for key in optimizers.keys():
    networks[key] = MultiLayerNet(
        input_size=784, hidden_size_list=[100, 100, 100, 100],
        output_size=10,
        activation='relu',weight_init_std='relu', 
        weight_decay_lambda=0)
    train_loss[key] = []    

# 2:Start of training==========
for i in range(max_iterations):
    batch_mask = np.random.choice(train_size, batch_size)
    x_batch = x_train[batch_mask]
    t_batch = t_train[batch_mask]
    
    for key in optimizers.keys():
        grads = networks[key].gradient(x_batch, t_batch)
        optimizers[key].update(networks[key].params, grads)
    
        loss = networks[key].loss(x_batch, t_batch)
        train_loss[key].append(loss)

#Evaluation with test data
x = x_test
t = t_test

for key in optimizers.keys():
    network = networks[key]

    y = network.predict(x)

    accuracy_cnt = 0
    for i in range(len(x)):
        p= np.argmax(y[i])
        if p == t[i]:
            accuracy_cnt += 1

    print(key + " Accuracy:" + str(float(accuracy_cnt) / len(x))) 

Specify'relu'in the activation function and set "Initial value of He"

activation='relu', weight_init_std='he', 

Result of processing test data

SGD Accuracy:0.9325 Momentum Accuracy:0.966 AdaGrad Accuracy:0.9707 Adam Accuracy:0.972

Specify'sigmoid'in the activation function and set "Initial value of Xavier"

activation='sigmoid', weight_init_std='xavier', 

Result of processing test data

SGD Accuracy:0.1135 Momentum Accuracy:0.1028 AdaGrad Accuracy:0.9326 Adam Accuracy:0.9558

SGD and Momentum had a bad recognition rate, so I set the number of batches to 10000.

SGD Accuracy:0.1135 Momentum Accuracy:0.9262 AdaGrad Accuracy:0.9617 Adam Accuracy:0.9673

Momentum's recognition rate has risen so much, but SGD is totally useless.

Specify'relu'as the activation function and set the normal distribution with a standard deviation of 0.01 as the initial value.

activation='relu', weight_init_std=0.01, 

Result of processing test data

SGD Accuracy:0.1135 Momentum Accuracy:0.1135 AdaGrad Accuracy:0.9631 Adam Accuracy:0.9713

SGD and Momentum don't seem to be learning at all.

Part 10 ← → Part 11