[Deep Learning from scratch] I tried to explain Dropout

Introduction

This article is an easy-to-understand output of ** Deep Learning from scratch Chapter 7 Learning Techniques **. I was able to understand it myself in the humanities, so I hope you can read it comfortably. Also, I would be more than happy if you could refer to it when studying this book.

What is Dropout?

Do you know what ensemble learning is? Ensemble learning is the one that can produce good learning results by learning using multiple models. Dropout improves the learning results by reproducing the ensemble learning in a simulated manner.

Is Dropout doing the inside specifically? It is to randomly erase neurons during learning. Dropout creates ensemble learning by creating multiple different models by randomly erasing neurons.

Below is a simple implementation example.

class Dropout:#It is generated after the activation function layer and activated every time learning is performed. Do not activate with predict
    
    def __init__(self,dropout_ratio=0.5):
        self.dropout_ratio = dropout_ratio
        self.mask = None #Contains an array of neurons to be erased
    
    def forward(self,x,train_flg=True):
        if train_flg:
            self.mask = np.random.rand(*x.shape) > self.dropout_ratio#Randomly determine neurons to erase
            return x * self.mask
        else:
            return x * (1 - self.dropout_ratio)

    def backward(self,dout):
        return dout * self.mask#Same as Relu
        

Recommended Posts

[Deep Learning from scratch] I tried to explain Dropout
I tried to implement Perceptron Part 1 [Deep Learning from scratch]
[Deep Learning from scratch] I tried to implement sigmoid layer and Relu layer.
[Deep Learning from scratch] I tried to explain the gradient confirmation in an easy-to-understand manner.
Deep Learning from scratch
I tried deep learning
"Deep Learning from scratch" Self-study memo (No. 16) I tried to build SimpleConvNet with Keras
"Deep Learning from scratch" Self-study memo (No. 17) I tried to build DeepConvNet with Keras
Deep Learning from scratch ① Chapter 6 "Techniques related to learning"
[Learning memo] Deep Learning from scratch ~ Implementation of Dropout ~
Deep Learning from scratch 1-3 chapters
[Deep Learning from scratch] I implemented the Affine layer
I'm not sure, but I feel like I understand Deep Learning (I tried Deep Learning from scratch)
[Python] [Natural language processing] I tried Deep Learning ❷ made from scratch in Japanese ①
I tried to extract a line art from an image with Deep Learning
I tried to implement Deep VQE
Deep learning from scratch (cost calculation)
I tried to explain Pytorch dataset
I tried deep learning using Theano
Deep Learning memos made from scratch
I tried to divide with a deep learning language model
I tried to predict horse racing by doing everything from data collection to deep learning
[Learning memo] Deep Learning made from scratch [Chapter 7]
Deep learning from scratch (forward propagation edition)
I tried to make deep learning scalable with Spark × Keras × Docker
Deep learning / Deep learning made from scratch Chapter 6 Memo
[Learning memo] Deep Learning made from scratch [Chapter 5]
[Learning memo] Deep Learning made from scratch [Chapter 6]
Image alignment: from SIFT to deep learning
"Deep Learning from scratch" in Haskell (unfinished)
[Windows 10] "Deep Learning from scratch" environment construction
Learning record of reading "Deep Learning from scratch"
[Deep Learning from scratch] About hyperparameter optimization
"Deep Learning from scratch" Self-study memo (Part 12) Deep learning
[Learning memo] Deep Learning made from scratch [~ Chapter 4]
I tried to implement deep learning that is not deep with only NumPy
[Reinforcement learning] Finally surpassed humans! ?? I tried to explain / implement Agent57 (Keras-RL)
I tried to classify Oba Hana and Emiri Otani by deep learning
[Deep Learning from scratch] Layer implementation from softmax function to cross entropy error
I tried the common story of using Deep Learning to predict the Nikkei 225
"Deep Learning from scratch" self-study memo (unreadable glossary)
"Deep Learning from scratch" Self-study memo (9) MultiLayerNet class
Good book "Deep Learning from scratch" on GitHub
Deep Learning from scratch Chapter 2 Perceptron (reading memo)
Python vs Ruby "Deep Learning from scratch" Summary
"Deep Learning from scratch" Self-study memo (10) MultiLayerNet class
"Deep Learning from scratch" Self-study memo (No. 11) CNN
I tried to debug.
I tried to paste
[Python] Deep Learning: I tried to implement deep learning (DBN, SDA) without using a library.
[Deep Learning from scratch] Speeding up neural networks I explained back propagation processing
I tried to classify Oba Hana and Emiri Otani by deep learning (Part 2)
I tried to make deep learning scalable with Spark × Keras × Docker 2 Multi-host edition
I tried to move machine learning (ObjectDetection) with TouchDesigner
"Deep Learning from scratch" Self-study memo (No. 19) Data Augmentation
"Deep Learning from scratch 2" Self-study memo (No. 21) Chapters 3 and 4
Application of Deep Learning 2 made from scratch Spam filter
I tried to compress the image using machine learning
I tried to learn PredNet
I tried to organize SVM.
Deep learning / LSTM scratch code