[Deep Learning from scratch] I tried to explain the gradient confirmation in an easy-to-understand manner.

Introduction

This article is an easy-to-understand output of ** Deep Learning from scratch Chapter 6 Error back propagation method **. I was able to understand it myself in the humanities, so I hope you can read it comfortably. Also, I would be more than happy if you could refer to it when studying this book.

What is gradient confirmation?

The gradient formula by the error back propagation method implemented in the previous article is very complicated, so there is a high possibility that mistakes will occur. However, the processing speed is extremely fast, so I have to use it. Therefore, it is important to check whether there is a mistake in the gradient equation by the error back propagation method. This is called gradient confirmation, and a gradient equation of numerical differentiation that is simple to make and has few mistakes is used for the gradient confirmation.

Although the processing speed of the gradient formula of numerical differentiation is slow, there are few mistakes due to its simple construction. Therefore, what is done by gradient confirmation is to confirm the gradient obtained from the gradient equation of numerical differentiation and the gradient error obtained by the gradient equation by the error back propagation method.

Now, I would like to implement it.

#Add the following numerical differentiation gradient equation to the neural network class method
def slopeing_grad_net(self,x,t):#Gradient expression of numerical differentiation to confirm whether the error back propagation method is correct
    loss_c = lambda W: self.loss(x, t) 
        
    grads_t = {}
    grads_t['W1'] = slopeing_grad(loss_c,self.params['W1'])
    grads_t['b1'] = slopeing_grad(loss_c,self.params['b1'])
    grads_t['W2'] = slopeing_grad(loss_c,self.params['W2'])
    grads_t['b2'] = slopeing_grad(loss_c,self.params['b2'])
        
    return grads_t

#Gradient confirmation
#Issue a mini batch
x_train, x_test, t_train, t_test = dataset['train_img'], dataset['test_img'], \
                                            dataset['train_label'], dataset['test_label']

x_batch = x_train[:3]
t_batch = t_train[:3]

netwark = LayerNet(input_size = 784, hiden_size = 50, output_size = 10)
#Put out each gradient
grad_slope = netwark.slopeing_grad_net(x_batch, t_batch)
grad_new = netwark.gradient(x_batch, t_batch)

for key in grad_slope.keys(): #Compare those differences
    diff = np.average(np.abs(grad_new[key] - grad_slope[key]))
    print(key + ':' + str(diff))


W1:0.03976913034251971
b1:0.0008997051177847986
W2:0.3926011094391389
b2:0.04117287920452093

Check the average of the gradient errors with a code like the one above to see if the errors are within range.

Recommended Posts

[Deep Learning from scratch] I tried to explain the gradient confirmation in an easy-to-understand manner.
[Deep Learning from scratch] I tried to explain Dropout
[For beginners] I want to explain the number of learning times in an easy-to-understand manner.
[Python] I tried to summarize the set type (set) in an easy-to-understand manner.
I tried to implement Perceptron Part 1 [Deep Learning from scratch]
I tried to explain how to get the article content with MediaWiki API in an easy-to-understand manner with examples (Python 3)
I will explain how to use Pandas in an easy-to-understand manner.
[Deep Learning from scratch] I tried to implement sigmoid layer and Relu layer.
I tried to summarize Cpaw Level1 & Level2 Write Up in an easy-to-understand manner
I tried to summarize Cpaw Level 3 Write Up in an easy-to-understand manner
I tried to understand supervised learning of machine learning in an easy-to-understand manner even for server engineers 1
[Python] I tried to explain words that are difficult for beginners to understand in an easy-to-understand manner.
I tried to understand supervised learning of machine learning in an easy-to-understand manner even for server engineers 2
Introduction to Deep Learning (1) --Chainer is explained in an easy-to-understand manner for beginners-
[Deep Learning from scratch] I implemented the Affine layer
I tried to display the analysis result of the natural language processing library GiNZA in an easy-to-understand manner
[Python] [Natural language processing] I tried Deep Learning ❷ made from scratch in Japanese ①
I tried to extract a line art from an image with Deep Learning
"Deep Learning from scratch" Self-study memo (No. 16) I tried to build SimpleConvNet with Keras
"Deep Learning from scratch" Self-study memo (No. 17) I tried to build DeepConvNet with Keras
An amateur stumbled in Deep Learning from scratch Note: Chapter 1
An amateur stumbled in Deep Learning from scratch Note: Chapter 3
An amateur stumbled in Deep Learning from scratch Note: Chapter 7
An amateur stumbled in Deep Learning from scratch Note: Chapter 5
An amateur stumbled in Deep Learning from scratch Note: Chapter 4
An amateur stumbled in Deep Learning from scratch Note: Chapter 2
"Deep Learning from scratch" Self-study memo (Part 8) I drew the graph in Chapter 6 with matplotlib
"Deep Learning from scratch" in Haskell (unfinished)
[Deep Learning from scratch] About the layers required to implement backpropagation processing in a neural network
I tried the common story of using Deep Learning to predict the Nikkei 225
Deep Learning from scratch ① Chapter 6 "Techniques related to learning"
I tried to organize the evaluation indexes used in machine learning (regression model)
I'm not sure, but I feel like I understand Deep Learning (I tried Deep Learning from scratch)
I tried running an object detection tutorial using the latest deep learning algorithm
I tried to predict the change in snowfall for 2 years by machine learning
Deep Learning from scratch
I tried deep learning
I tried to graph the packages installed in Python
I tried to detect the iris from the camera image
I tried to compress the image using machine learning
I tried to predict horse racing by doing everything from data collection to deep learning
[Part 4] Use Deep Learning to forecast the weather from weather images
[Machine learning] Let's summarize random forest in an easy-to-understand manner
Deep Learning from scratch 1-3 chapters
[Part 1] Use Deep Learning to forecast the weather from weather images
[Part 3] Use Deep Learning to forecast the weather from weather images
I tried to create API list.csv in Python from swagger.yaml
[Pokemon Sword Shield] I tried to visualize the judgment basis of deep learning using the three family classification as an example.
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 5
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 2
Create an environment for "Deep Learning from scratch" with Docker
I tried to summarize the code often used in Pandas
I tried to illustrate the time and time in C language
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 7
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 1
[Part 2] Use Deep Learning to forecast the weather from weather images
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 4
I tried to summarize the commands often used in business
I tried to implement the mail sending function in Python
[TF] I tried to visualize the learning result using Tensorboard
[Machine learning] I tried to summarize the theory of Adaboost