Deep Learning Memorandum

It's basically a memorandum of Deep Learning content made from O'Reilly's zero. I have a python tag, but I don't see much python code.

perceptron

To put it simply, a logic circuit

AND gate

A gate that operates as follows

x_1 x_2 y
0 0 0
0 1 0
1 0 0
1 1 1

Change the weight (w_1, w_2) $ of $ b + w_1x_1 + w_2x_2 to realize the above gate Same for other gates

Multilayer perceptron

XOR gate cannot be reproduced with 1-layer perceptron

XOR gate

x_1 x_2 y
0 0 0
0 1 1
1 0 1
1 1 0

I won't go into detail, but this cannot be achieved with a single perceptron. This can be achieved by stacking layers (multilayer perceptron) This is the basis of neural networks

neural network

The basic idea is with Perceptron

Then what's the difference?

Perceptron output 1 if the output of the function mentioned earlier exceeds 0, otherwise it outputs 0. In other words, the output of one layer is output to the next layer using a step function. The step function here is called the ** activation function **.

This activation function replaces in neural networks

Sigmoid function

h(x)=\frac{1}{1+exp(-x)}

A function is a function that returns some output when a certain input is given. The sigmoid function is also just a function

What is better than a step function?

Very simply, the shape of the function is smooth

ReLU function

I haven't done it yet

Simple neural network

If the input is $ X $, the weight of the first layer is $ W $, and the bias of the first layer is $ B $, The weighted sum of the first layer can be expressed as $ A = XW + B $. Give this $ A $ vector as the input of the sigmoid function and give its output as the next input The output of the last layer uses another function instead of the sigmoid function

def sigmoid(x):
  return 1/(1+np.exp(-x))

def init_network():
  network = {}
  network['W1'] = np.array([[0.1, 0.3, 0.5],[0.2, 0.4, 0.6]])
  network['B1'] = np.array([0.1, 0.2, 0.3])
  # ...Initialize the other layers as well
  return network

def forward(network, x):
  W1 = network['W1']
  B1 = network['B1']
  a1 = np.dot(x, W1)
  z1 = sigmoid(a1) 
  # ...Continue
  y = identity_function(a3)
  return y

#I think it will be object-oriented in the future
network = init_network()
x = np.array([1.0, 2.0])
y = forward(network, x)

Output layer

Use identity function or softmax function The identity function is a function that outputs the input as it is The softmax function is expressed by the following formula

y_k = \frac{exp(a_k)}{\sum_{i=1}^{n}exp(a_i)}

However, when implementing the softmax function programmatically, you have to be careful about overflow, so Subtract the maximum value of the vector from the value of a.

What is a softmax function?

It converts the output of the neural network into the probability of classifying it into a certain class. However, since the magnitude relationship of the output does not change, it is generally omitted in the inference phase. The softmax function is used in the learning phase

Recommended Posts

Deep Learning Memorandum
Deep Learning
Start Deep learning
Python Deep Learning
Deep learning × Python
First Deep Learning ~ Struggle ~
Python: Deep Learning Practices
Deep learning / activation functions
Deep learning 1 Practice of deep learning
Deep learning / cross entropy
First Deep Learning ~ Preparation ~
First Deep Learning ~ Solution ~
[AI] Deep Metric Learning
I tried deep learning
Python: Deep Learning Tuning
Deep learning large-scale technology
Deep learning / softmax function
A memorandum of studying and implementing deep learning
Effective Python Learning Memorandum Day 15 [15/100]
[Learning memorandum] Introduction to vim
Deep Learning from scratch 1-3 chapters
Try deep learning with TensorFlow
Deep Learning Gaiden ~ GPU Programming ~
<Course> Deep Learning: Day2 CNN
Effective Python Learning Memorandum Day 12 [12/100]
Effective Python Learning Memorandum Day 9 [9/100]
Deep learning image recognition 1 theory
Effective Python Learning Memorandum Day 8 [8/100]
Deep learning / LSTM scratch code
Rabbit Challenge Deep Learning 1Day
<Course> Deep Learning: Day1 NN
Deep Kernel Learning with Pyro
Learning memorandum for me w
Try Deep Learning with FPGA
Deep learning for compound formation?
Introducing Udacity Deep Learning Nanodegree
Effective Python Learning Memorandum Day 14 [14/100]
Effective Python Learning Memorandum Day 1 [1/100]
Subjects> Deep Learning: Day3 RNN
Introduction to Deep Learning ~ Learning Rules ~
Rabbit Challenge Deep Learning 2Day
Effective Python Learning Memorandum Day 13 [13/100]
Effective Python Learning Memorandum Day 3 [3/100]
Effective Python Learning Memorandum Day 5 [5/100]
Effective Python Learning Memorandum Day 4 [4/100]
Deep Reinforcement Learning 1 Introduction to Reinforcement Learning
Deep reinforcement learning 2 Implementation of reinforcement learning
Effective Python Learning Memorandum Day 7 [7/100]
Generate Pokemon with Deep Learning
Effective Python Learning Memorandum Day 2 [2/100]
Introduction to Deep Learning ~ Backpropagation ~
Deep Learning Model Lightening Library Distiller
Deep Learning / Deep Learning from Zero 2 Chapter 4 Memo
Try Deep Learning with FPGA-Select Cucumbers
Deep Learning / Deep Learning from Zero Chapter 3 Memo
Make ASCII art with deep learning
Deep Learning / Deep Learning from Zero 2 Chapter 5 Memo
Implement Deep Learning / VAE (Variational Autoencoder)
Introduction to Deep Learning ~ Function Approximation ~
Try deep learning with TensorFlow Part 2
Deep learning from scratch (cost calculation)