Implementation of a two-layer neural network 2

Continuation of the previous article Implementation of two-layer neural network We created a two-layer neural network and learned MNIST. Confirmation of prediction accuracy of the learned result. Refer to Chapter 4 of Deep Learning from scratch

TwoLayerNet.py


import numpy as np

class TwoLayerNet:
    
    def __init__(self,input_size,hidden_size,output_size,weight_init_std=0.01):
        #Weight initialization
        self.params = {}
        #784 *50 weight matrix
        self.params['W1'] = weight_init_std * np.random.randn(input_size,hidden_size)
        #50 *10 weight matrix
        self.params['W2'] = weight_init_std * np.random.randn(hidden_size,output_size)
        #Bias, as many hidden layers
        self.params['b1'] = np.zeros(hidden_size)
        #Bias, number of output layers
        self.params['b2'] = np.zeros(output_size)
    
    def sigmoid(self,x):
        return 1 / (1 + np.exp(-x))
    
    def softmax(self,a):
        c = np.max(a)
        exp_a = np.exp(a - c)#Overflow measures
        sum_exp_a = np.sum(exp_a)
        y = exp_a / sum_exp_a
        return y
    
    def _numerical_gradient_1d(self,f, x):
        h = 1e-4 # 0.0001
        grad = np.zeros_like(x)

        for idx in range(x.size):
            tmp_val = x[idx]
            x[idx] = float(tmp_val) + h
            fxh1 = f(x) # f(x+h)

            x[idx] = tmp_val - h 
            fxh2 = f(x) # f(x-h)
            grad[idx] = (fxh1 - fxh2) / (2*h)

            x[idx] = tmp_val #Restore the value

        return grad


    def numerical_gradient(self,f, X):
        if X.ndim == 1:
            return self._numerical_gradient_1d(f, X)
        else:
            grad = np.zeros_like(X)

            for idx, x in enumerate(X):
                grad[idx] = self._numerical_gradient_1d(f, x)

            return grad

    def cross_entropy_error(self,y,t):
        if y.ndim == 1:
            t = t.reshape(1,t.size)
            y = y.reshape(1,y.size)
        batch_size = y.shape[0]
        return -np.sum(t * np.log(y)) / batch_size
    
    def predict(self,x):
        W1,W2 = self.params['W1'],self.params['W2']
        b1,b2 = self.params['b1'],self.params['b2']
        
        a1 = np.dot(x,W1) + b1 #a = Wx + b
        z1 = self.sigmoid(a1)
        a2 = np.dot(z1,W2) + b2
        z2 = self.softmax(a2)
        
        return z2
    
    def loss(self, x, t):
        y = self.predict(x)
        
        return self.cross_entropy_error(y,t)
    
    def accuracy(self, x, t):
        y = self.predict(x)
        y = np.argmax(y,axis=1)
        t = np.argmax(t,axis=1)
        
        accuracy = np.sum(y == t) / float(x.shape[0])
        return accuracy
    
    def gradient(self,x,t):
        loss_W = lambda W: self.loss(x,t)
        grads = {}
        grads['W1'] = self.numerical_gradient(loss_W,self.params['W1'])
        grads['W2'] = self.numerical_gradient(loss_W,self.params['W2'])
        grads['b1'] = self.numerical_gradient(loss_W,self.params['b1'])
        grads['b2'] = self.numerical_gradient(loss_W,self.params['b2'])
        
        return grads

Similar to the previous time, mini-batch learning (size 50) was performed 500 times from the MNIST data. This time, the accuracy (acc) is calculated with 50 loops as one epoch.

LearningMNIST.py


#coding: utf-8
import numpy as np
from sklearn.datasets import fetch_mldata
from sklearn.preprocessing import OneHotEncoder

mnist = fetch_mldata('MNIST original', data_home=".")

x_train = mnist['data'][:60000]
t_train = mnist['target'][:60000]

t_train = t_train.reshape(1, -1).transpose()
# encode label
encoder = OneHotEncoder(n_values=max(t_train)+1)
t_train = encoder.fit_transform(t_train).toarray()

x_test = mnist['data'][60000:]
t_test = mnist['target'][60000:]

t_test = t_test.reshape(1, -1).transpose()
# encode label
encoder = OneHotEncoder(n_values=max(t_test)+1)
t_test = encoder.fit_transform(t_test).toarray()

x_train = x_train.astype(np.float64)
x_train /= x_train.max()
x_test = x_test.astype(np.float64)
x_test /= x_test.max()

train_loss_list = []
train_acc_list = []
test_acc_list = []

#hyper parameter
iters_num = 2
train_size = x_train.shape[0]
batch_size = 100
learning_rate = 0.1

#Number of iterations per epoch
#iter_per_epoch = max(train_size / batch_size, 1)
iter_per_epoch = 50

network = TwoLayerNet(input_size=784,hidden_size=50,output_size=10)

iters_num = 500
for i in range(iters_num):
    batch_mask = np.random.choice(train_size,batch_size)
    x_batch = x_train[batch_mask]
    t_batch = t_train[batch_mask]
    
    grad = network.gradient(x_batch,t_batch)
    
    for key in ('W1','W2','b1','b2'):
        network.params[key] -= learning_rate * grad[key]
        
    loss = network.loss(x_batch,t_batch)
    train_loss_list.append(loss)
    #Calculate recognition accuracy for each epoch
    if i % iter_per_epoch == 0:
        train_acc = network.accuracy(x_train, t_train)
        test_acc = network.accuracy(x_test, t_test)
        train_acc_list.append(train_acc)
        test_acc_list.append(test_acc)
        print("train acc, test acc | " + str(train_acc) + ", " + str(test_acc))

The result is the graph below, the vertical axis is the accuracy and the horizontal axis is the number of epochs.

MNIST_acc.jpg

The accuracy improves with each epoch.

Recommended Posts

Implementation of a two-layer neural network 2
I implemented a two-layer neural network
Implementation of a convolutional neural network using only Numpy
Implementation of 3-layer neural network (no learning)
Implementation of "blurred" neural network using Chainer
Visualize the inner layer of a neural network
Implement a 3-layer neural network
Neural network implementation in python
Neural network implementation (NumPy only)
The story of making a music generation neural network
Bayesian optimization implementation of neural network hyperparameters (Chainer + GPyOpt)
Basics of PyTorch (2) -How to make a neural network-
Rank learning using neural network (Implementation of RankNet by Chainer)
Simple neural network implementation using Chainer
Implementation of a simple particle filter
PRML Chapter 5 Neural Network Python Implementation
What is a Convolutional Neural Network?
Simple neural network theory and implementation
Touch the object of the neural network
Construction of a neural network that reproduces XOR by Z3
Understand the number of input / output parameters of a convolutional neural network
Compose with a neural network! Run Magenta
Simple neural network implementation using Chainer-Data preparation-
Simple neural network implementation using Chainer-Model description-
Build a classifier with a handwriting recognition rate of 99.2% with a TensorFlow convolutional neural network
Parametric Neural Network
Simple neural network implementation using Chainer-optimization algorithm setting-
Experiment with various optimization algorithms with a neural network
Reinforcement learning 10 Try using a trained neural network.
Verification of Batch Normalization with multi-layer neural network
Recognition of handwritten numbers by multi-layer neural network
Python vs Ruby "Deep Learning from scratch" Chapter 3 Implementation of 3-layer neural network
[Python] Implementation of clustering using a mixed Gaussian model
Train MNIST data with a neural network in PyTorch
Overview of DNC (Differentiable Neural Computers) + Implementation by Chainer
Implement Convolutional Neural Network
A python implementation of the Bayesian linear regression class
Implementation of Fibonacci sequence
Implement Neural Network from 1
Convolutional neural network experience
Basics of network programs?
A reminder about the implementation of recommendations in Python
How to easily draw the structure of a neural network on Google Colaboratory using "convnet-drawer"
I tried a convolutional neural network (CNN) with a tutorial on TensorFlow on Cloud9-Classification of handwritten images-
I tried to implement a basic Recurrent Neural Network model
Create a web application that recognizes numbers with a neural network
Implementation of VGG16 using Keras created without using a trained model
Try to build a deep learning / neural network with scratch
A simple Python implementation of the k-nearest neighbor method (k-NN)
A network diagram was created with the data of COVID-19.
Try building a neural network in Python without using a library
I made a neural network generator that runs on FPGA
CNN Acceleration Series ~ FCNN: Introduction of Fourier Convolutional Neural Network ~
The method of initializing weight parameters in a neural network is summarized with implementation (initial values of Xavier, He et al., Etc.)
Quantum computer implementation of quantum walk 2
Implementation of TF-IDF using gensim
A memorandum of kernel compilation
Neural network with Python (scikit-learn)
3. Normal distribution with neural network!
Explanation and implementation of SocialFoceModel
A small memorandum of openpyxl