This article is an easy-to-understand output of ** Deep Learning from scratch Chapter 6 Error back propagation method **. I was able to understand it myself in the humanities, so I hope you can read it comfortably. Also, I would be more than happy if you could refer to it when studying this book.
In the previous article, I was able to implement all the parts needed to do the backpropagation method. Therefore, this time, I would like to implement the back propagation process in the neural network by actually using the part and performing the error back propagation method.
** 1, Parameter initialization / Layer creation **
from collections import OrderedDict
class LayerNet:
def __init__(self, input_size, hiden_size, output_size, weight_init_std = 0.01):
self.params = {}
self.params['W1'] = weight_init_std * np.random.randn(input_size, hiden_size)
self.params['b1'] = np.zeros(hiden_size)
self.params['W2'] = weight_init_std * np.random.randn(hiden_size, output_size)
self.params['b2'] = np.zeros(output_size)
#Layer generation
self.layers = OrderedDict()#Ordered dictionary
self.layers['Affine1'] = Affine(self.params['W1'],self.params['b1']) #Create all Affine layers on the first layer of the middle layer
self.layers['Relu1'] = Relu()#Create all Relu layers on the first layer of the middle layer
self.layers['Affine2'] = Affine(self.params['W2'], self.params['b2']) #Create all Affine layers in the second layer of the middle layer
self.layers['Relu2'] = Relu() #Create all Relu layers for the second layer of the middle layer
self.lastLayer = SoftmaxWithLoss() #Layer creation from output layer to loss function
Please refer to the parameter initialization as it is the same as for the numerical differentiation neural network.
In the next layer generation part, we first create an ordered dictionary ** OrderDict ** to store the layers and store it in an instance variable.
Next, put the layers in the created OrderDict in order from the left side of the neural network. Layers are managed in order by putting them in OrderDict. By doing so, you can easily perform forward propagation processing by using the for statement to extract layers one by one, and reverse propagation processing by using the for statement in the reverse order.
** 2, implement gradient expression using predict method, loss method, accuracy method, layer **
from collections import OrderedDict
class LayerNet:
def __init__(self, input_size, hiden_size, output_size, weight_init_std = 0.01):
self.params = {}
self.params['W1'] = weight_init_std * np.random.randn(input_size, hiden_size)
self.params['b1'] = np.zeros(hiden_size)
self.params['W2'] = weight_init_std * np.random.randn(hiden_size, output_size)
self.params['b2'] = np.zeros(output_size)
#Layer generation
self.layers = OrderedDict()#Ordered dictionary
self.layers['Affine1'] = Affine(self.params['W1'],self.params['b1']) #Create all Affine layers on the first layer of the middle layer
self.layers['Relu1'] = Relu()#Create all Relu layers on the first layer of the middle layer
self.layers['Affine2'] = Affine(self.params['W2'], self.params['b2']) #Create all Affine layers in the second layer of the middle layer
self.layers['Relu2'] = Relu() #Create all Relu layers for the second layer of the middle layer
self.lastLayer = SoftmaxWithLoss() #Layer creation from output layer to loss function
def predict(self, x): #Performs forward propagation processing to the intermediate layer of the neural network
for layer in self.layers.values():
x = layer.forward(x) #Process the ordered layers in order with a for statement
return x
def loss(self, x, t): #Performs forward propagation processing from the neural network to the loss function
y = self.predict(x) #Produce forecast data
return self.lastLayer.forward(y,t) #Performs forward propagation processing from the output layer to the loss function
def accuracy(self,x, t): #Give the correct answer rate
y = self.predict(x)
y = np.argmax(y, axis=1) #Label the correct answer forecast from the forecast data
if t.ndim != 1 : t = np.argmax(t, axis=1) #Correct answer data is one_If it is hot, change it to a one-dimensional array
accuracy = np.sum(y == t) /float(x.shape[0]) #Give the correct answer rate
return accuracy
def gradient(self, x, t): #Gradient equation using layer backpropagation
#forward
self.loss(x, t)
#backward
dout = 1
dout = self.lastLayer.backward(dout) #Backpropagation of layers from output layer to loss function
layers = list(self.layers.values())
layers.reverse() #Reverse the order
for layer in layers: #Perform backpropagation processing from the layer to the right of the intermediate layer
dout = layer.backward(dout)
#Gradient recovery
grads = {}
grads['W1'] = self.layers['Affine1'].dW
grads['b1'] = self.layers['Affine1'].db
grads['W2'] = self.layers['Affine2'].dW
grads['b2'] = self.layers['Affine2'].db
return grads
The predict method performs forward propagation processing of the neural network. As I wrote a little earlier, forward propagation processing can be performed by using the for statement in OrderDict, performing the forward method in order, and returning the output value with return.
Since the loss method performs the predict method plus the activation function and loss function of the output layer, it can be implemented by using the result for the forward of the lastLayer after the predict method.
The accuracy is the same as for numerical differentiation, so please refer to that.
Gradient formulas using layers can be implemented by reverse propagation processing of the neural network, first reverse the order of OrderDict with the reverse method, then perform backward with the for statement in order, and return the result with return. can.
Recommended Posts