[Deep Learning from scratch] Speeding up neural networks I explained back propagation processing
Introduction
This article is an easy-to-understand output of ** Deep Learning from scratch Chapter 6 Error back propagation method **. I was able to understand it myself, so I hope you can read it comfortably. Also, I would be more than happy if you could refer to it when studying this book.
What is back propagation processing?
As I talked about a little last time, backpropagation processing is the processing of solving the calculation graph from right to left, and the derivative of each variable can be obtained.
The gradient formula of the neural network implemented in the previous chapter used a method of finding the derivative by adding a minute value to a variable called numerical differentiation. Numerical differentiation is simple and easy to understand, but it has the disadvantage of long processing time. That's where ** backpropagation processing ** is used. By implementing back propagation processing in the processing of the neural network, the gradient can be obtained faster and more efficiently than before.
As a starting point, I would like to implement forward propagation processing and back propagation processing of a simple calculation graph in python.
a = 100
as = 3
o = 50
os = 4
t = 1.1
#Forward propagation processing
az = a * as #Total amount of apples
oz = o * os #Total amount of mandarin oranges
z = az + oz #Total amount of apple oranges
y = z * t #total fee
#Back propagation processing
dy = 1 #Differentiation of answer y of forward propagation processing
# z *Since t is multiplication, the values of each variable are exchanged and the previous derivative is applied.
dz = t * dy = 1.1 #Differentiation of z
dt = z * dy = 500 #Differentiation of t
# az +Since oz is addition, it inherits the previous derivative as it is
daz = dz = 1.1 #Differentiation of az
doz = dz = 1.1 #Differentiation of oz
# o *Since os is multiplication, the values of each variable are exchanged and the previous derivative is applied.
do = os * doz = 4.4 #Differentiation of o
dos = o * doz = 55 #Differentiation of os
# a *Since as is multiplication, the values of each variable are exchanged and the previous derivative is applied.
da = as * daz = 3.3 #Differentiation of a
das = a * adz = 330 #Differentiation of as
Since the backpropagation processing of multiplication and addition can be easily performed as described above, the back propagation processing is implemented in the neural network using this.
Recommended Posts