[Deep Learning from scratch] I implemented the Affine layer

Introduction

This article is an easy-to-understand output of ** Deep Learning from scratch Chapter 6 Error back propagation method **. I was able to understand it myself in the humanities, so I hope you can read it comfortably. Also, I would be more than happy if you could refer to it when studying this book.

Affine layer implementation

The Affine layer is a layer in which the input signal of the neuron is multiplied by the weight, summed, and biased.

class Affine:#Layer implementation of processing that weights the input signal and adds its sum and bias
    def __init__(self, W, b):
        self.W = W
        self.b = b
        self.x = None
        self.dW = None #Create an instance variable to save each derivative obtained by backpropagation processing
        self.db = None #Create an instance variable to save each derivative obtained by backpropagation processing
    
    def forward(self, x):
        self.x = x
        out = np.dot(x, self.W) + self.b
        
        return out
    
    def backward(self, dout):
        dx = np.dot(dout, self.W.T)
        self.dW = np.dot(self.x.T, dout)
        self.db = np.sum(dout, axis=0)#To support multiple data (batch)
        #Find the derivative of the bias by the sum of the previous derivatives
        
        return dx

Since the back propagation process uses multiplication and addition in the forward propagation process of the Affine layer, the derivative of the bias inherits the previous derivative, and the derivative of the weight and the input value is transposed and replaced with the previous derivative. It is calculated by multiplying by. Bias is not just inherited, but batch processing is done as above.

Recommended Posts

[Deep Learning from scratch] I implemented the Affine layer
Deep Learning from scratch
Deep Learning from scratch 1-3 chapters
[Deep Learning from scratch] I tried to implement sigmoid layer and Relu layer.
[Deep Learning from scratch] I tried to explain Dropout
Deep learning from scratch (cost calculation)
Deep Learning memos made from scratch
I tried to implement Perceptron Part 1 [Deep Learning from scratch]
[Learning memo] Deep Learning made from scratch [Chapter 7]
Deep learning from scratch (forward propagation edition)
Deep learning / Deep learning from scratch 2-Try moving GRU
Deep learning / Deep learning made from scratch Chapter 6 Memo
[Learning memo] Deep Learning made from scratch [Chapter 5]
[Deep Learning from scratch] I tried to explain the gradient confirmation in an easy-to-understand manner.
[Learning memo] Deep Learning made from scratch [Chapter 6]
"Deep Learning from scratch" in Haskell (unfinished)
Deep learning / Deep learning made from scratch Chapter 7 Memo
[Windows 10] "Deep Learning from scratch" environment construction
Learning record of reading "Deep Learning from scratch"
"Deep Learning from scratch" Self-study memo (Part 8) I drew the graph in Chapter 6 with matplotlib
[Deep Learning from scratch] About hyperparameter optimization
"Deep Learning from scratch" Self-study memo (Part 12) Deep learning
[Learning memo] Deep Learning made from scratch [~ Chapter 4]
[Deep Learning from scratch] Layer implementation from softmax function to cross entropy error
"Deep Learning from scratch" self-study memo (unreadable glossary)
"Deep Learning from scratch" Self-study memo (9) MultiLayerNet class
Deep Learning from scratch ① Chapter 6 "Techniques related to learning"
Good book "Deep Learning from scratch" on GitHub
Deep Learning from scratch Chapter 2 Perceptron (reading memo)
[Learning memo] Deep Learning from scratch ~ Implementation of Dropout ~
Python vs Ruby "Deep Learning from scratch" Summary
"Deep Learning from scratch" Self-study memo (10) MultiLayerNet class
"Deep Learning from scratch" Self-study memo (No. 11) CNN
I tried deep learning
Python vs Ruby "Deep Learning from scratch" Chapter 3 Implementation of 3-layer neural network
Prepare the environment for O'Reilly's book "Deep Learning from scratch" with apt-get (Debian 8)
I'm not sure, but I feel like I understand Deep Learning (I tried Deep Learning from scratch)
[Python] [Natural language processing] I tried Deep Learning ❷ made from scratch in Japanese ①
Deep Learning from scratch The theory and implementation of deep learning learned with Python Chapter 3
[Deep Learning from scratch] Speeding up neural networks I explained back propagation processing
I installed and used the Deep Learning library Chainer
"Deep Learning from scratch" Self-study memo (No. 19) Data Augmentation
"Deep Learning from scratch 2" Self-study memo (No. 21) Chapters 3 and 4
Application of Deep Learning 2 made from scratch Spam filter
Deep Learning from the mathematical basics Part 2 (during attendance)
Othello ~ From the tic-tac-toe of "Implementation Deep Learning" (4) [End]
"Deep Learning from scratch" Self-study memo (No. 16) I tried to build SimpleConvNet with Keras
"Deep Learning from scratch" Self-study memo (No. 17) I tried to build DeepConvNet with Keras
Deep learning / LSTM scratch code
I tweeted from the terminal!
I implemented Extreme learning machine
[Deep Learning from scratch] Implementation of Momentum method and AdaGrad method
[Part 4] Use Deep Learning to forecast the weather from weather images
An amateur stumbled in Deep Learning from scratch Note: Chapter 1
[Part 1] Use Deep Learning to forecast the weather from weather images
[Part 3] Use Deep Learning to forecast the weather from weather images
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 5
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 2
Create an environment for "Deep Learning from scratch" with Docker
An amateur stumbled in Deep Learning from scratch Note: Chapter 3
An amateur stumbled in Deep Learning from scratch Note: Chapter 7