Roughly think about the loss function

I don't want to think too deeply, and I always want to understand it roughly and sensuously if possible. So, let's take a look at this loss function so that we can understand it intuitively.

What is a loss function?

What is a loss function in the first place? A lot of research will give you a difficult explanation, but it seems to be a function that makes the difference between the two values smaller. In deep learning identification, etc., the main thing is to adjust the weight parameter so that the value is close to the answer at the time of learning, but the part of "to be close to the answer" is responsible. , This is the "loss function". u0o686000002d7d8.jpg Since "value = loss", how to reduce this loss is the "loss function".

Types of loss functions

So what kind of loss function is there? Even if you take a quick look, there are quite a few types.

--Hinge loss function --ε tolerance function --Huber function --Exponential loss function

Deep Learning uses "cross entropy error" and "square error" instead of these difficult ones. The important thing seems to be a function that can do the subsequent processing, "error backpropagation". If you look at the site where it is written that it is difficult to know what it is, what to do with it, etc., I have already decided on the above two here. If other functions come out while studying, I will add them secretly (explosion)

Cross entropy error

No matter how I explain it, it will be a difficult story, so I will post the formula, but in general it is like this. 010.png

So, when this becomes a certain condition (classification problem), it becomes easy. 011.png

If you write this in TensorFlow, it will be like this.

cross_entropy = tensorflow.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y)

"Y_" is the label of the correct answer, and "y" is the result of learning. The process is such that this difference becomes smaller.

In addition, it is bad if it is not normalized even for comparison, so in the case of "softmax_cross_entropy_with_logits ()", it seems that it is softmax in this.

If "y" is already normalized by softmax, it seems to write like this.

cross_entropy = -tensorflow.reduce_sum(y_ * tf.log(y), reduction_indices=[1])

"Reduce_sum ()" is an addition, and "log ()" is a function that finds the natural logarithm by itself.

Square error

The least squares method seems to be this. This is simple: just find the square of the difference and add it to all classes. (Image adjusted to minimize the result) LeastSquareMethod-1.png

There seem to be many ways to write in TensorFlow, but the square of the difference is added by "tensorflow.nn.reduce_sum ()", there is a function called "tensorflow.nn.l2_loss ()" in the first place, and the square is also "tensorflow. Use "square ()", etc.

Summary

So, I don't know what it is, but it seems that the obtained results are averaged. In the case of cross entropy error, it seems to be written like this.

cross_entropy = tensorflow.reduce_mean(tensorflow.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y))

Oh, next I have to understand the error back propagation. .. ..

Recommended Posts

Roughly think about the loss function
Roughly think about the gradient descent method
About the Unfold function
About the enumerate function (python)
Think about the minimum change problem
About the arguments of the setup function of PyCaret
About the test
About the queue
[Python] Seriously think about the M-1 winning method.
Think about the selective interface on the command line
Think about how to program Python on the iPad
Think about the next generation of Rack and WSGI
The first GOLD "Function"
Python: About function arguments
About the service command
About the confusion matrix
About the Visitor pattern
Think about the analysis environment (Part 1: Overview) * As of January 2017
Covector to think in function
Regarding the activation function Gelu
What is the activation function?
About the Python module venv
Think about architecture in python
About python beginner's memorandum function
About the ease of Python
About fork () function and execve () function
About the traveling salesman problem
About understanding the 3-point reader [...]
About the components of Luigi
About the features of Python
Think about dropouts with MNIST
Think about why Kubernetes is described as "Linux in the cloud"
I made AI think about the lyrics of Kenshi Yonezu (pre-processing)
I made AI think about the lyrics of Kenshi Yonezu (implementation)