Deep learning / cross entropy

1.First of all

This time, I will summarize what I learned about cross entropy (cross entropy).

2. Derivation from maximum likelihood estimation

If the sigmoid function is $ \ sigma and $ y = $ \ sigma $ (W ・ x + b), the probability that the neuron will fire (output 1) can be expressed as follows. P (C = 1 | x) = $ \ sigma $ (W · x + b)

On the contrary, the probability of not firing can be expressed as follows. P (C = 0 | x) = 1-$ \ sigma $ (W · x + b)

Expressing these two in one equation, the firing probability of one neuron can be expressed as follows (however, t = 0 or t = 1).   P(C = t|x) = y^t(1 -y)^{1-t}

Since the likelihood L of the entire network is the product of the firing probabilities of all neurons,   L = \prod_{n=1}^N y_n^{t_n}(1-y_n)^{1-t_n}

Maximum likelihood can be obtained by maximizing this equation, but it is easier to optimize by minimizing it, so multiply it by minus. Probability multiplication takes a logarithm (log) because the value becomes smaller and smaller and difficult to handle. And if you divide by N so that you can compare even if N changes,   L = -\frac{1}{N}\sum_{n=1}^N t_nlogy_n+(1-t_n)log(1-y_n)

This is the formula for cross entropy (cross entropy).

3. Specific error calculation

Now, suppose that the correct label $ t_1 $ ~ $ t_3 $ and the prediction probability $ y_1 $ ~ $ y_3 $ are as follows. スクリーンショット 2020-03-28 11.29.28.png Substituting a value into the cross entropy formula above スクリーンショット 2020-03-28 11.54.05.png

4. Code

If you write the code for cross entropy in python,

import numpy as np

def calc_cross_entropy(y_true, y_pred):
    loss = np.mean( -1 * (y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred)), axis=0)
    return loss

y_true =np.array([[1], [0], [0]])
y_pred = np.array([[0.8], [0.1], [0.1]])

answer = calc_cross_entropy(y_true,  y_pred)
print(answer)

#output
# [0.14462153]

Recommended Posts

Deep learning / cross entropy
Deep Learning
Deep Learning Memorandum
Start Deep learning
Python Deep Learning
Deep learning × Python
First Deep Learning ~ Struggle ~
Python: Deep Learning Practices
Deep learning / activation functions
Deep Learning from scratch
Deep learning 1 Practice of deep learning
First Deep Learning ~ Preparation ~
First Deep Learning ~ Solution ~
[AI] Deep Metric Learning
I tried deep learning
Python: Deep Learning Tuning
Deep learning large-scale technology
[Deep Learning from scratch] Layer implementation from softmax function to cross entropy error
Deep learning / softmax function
Deep Learning from scratch 1-3 chapters
Deep Learning Gaiden ~ GPU Programming ~
<Course> Deep Learning: Day2 CNN
Deep running 2 Tuning of deep learning
Rabbit Challenge Deep Learning 1Day
<Course> Deep Learning: Day1 NN
Deep Kernel Learning with Pyro
Try Deep Learning with FPGA
Deep learning for compound formation?
Introducing Udacity Deep Learning Nanodegree
Subjects> Deep Learning: Day3 RNN
Introduction to Deep Learning ~ Learning Rules ~
Rabbit Challenge Deep Learning 2Day
Deep Reinforcement Learning 1 Introduction to Reinforcement Learning
Deep reinforcement learning 2 Implementation of reinforcement learning
Generate Pokemon with Deep Learning
Introduction to Deep Learning ~ Backpropagation ~
Deep Learning Model Lightening Library Distiller
Deep Learning / Deep Learning from Zero 2 Chapter 4 Memo
Cat breed identification with deep learning
Deep Learning / Deep Learning from Zero Chapter 3 Memo
Make ASCII art with deep learning
Deep Learning / Deep Learning from Zero 2 Chapter 5 Memo
Implement Deep Learning / VAE (Variational Autoencoder)
Introduction to Deep Learning ~ Function Approximation ~
Try deep learning with TensorFlow Part 2
Deep learning from scratch (cost calculation)
About Deep Learning (DNN) Project Management
Deep learning to start without GPU
Solve three-dimensional PDEs with deep learning.
Introduction to Deep Learning ~ Coding Preparation ~
Organize machine learning and deep learning platforms
Deep learning learned by implementation 1 (regression)
Deep Learning / Deep Learning from Zero 2 Chapter 7 Memo
Deep Learning / Deep Learning from Zero 2 Chapter 8 Memo
Microsoft's Deep Learning Library "CNTK" Tutorial
Deep Learning / Deep Learning from Zero Chapter 5 Memo
Check squat forms with deep learning
Deep Learning / Deep Learning from Zero Chapter 4 Memo
Deep Reinforcement Learning 3 Practical Edition: Breakout
Deep learning image recognition 2 model implementation
Categorize news articles with deep learning