The inaccuracy of Tensorflow was due to log (0)

I recently started using TensorFlow, but during learning I had a problem that the accuracy suddenly dropped and did not change. If it is below, the accuracy will suddenly decrease from the 70th step.

.
.
.
step:67 train:0.894584 test:0.756296
step:68 train:0.900654 test:0.756944
step:69 train:0.897526 test:0.758796
step:70 train:0.361345 test:0.333333
step:71 train:0.361345 test:0.333333
step:72 train:0.361345 test:0.333333
step:73 train:0.361345 test:0.333333
.
.
.

Looking at the weights, it was as follows.

(pdb) w1
array([[[[ nan,  nan,  nan, ...,  nan,  nan,  nan],
         [ nan,  nan,  nan, ...,  nan,  nan,  nan],
         [ nan,  nan,  nan, ...,  nan,  nan,  nan]],

        [[ nan,  nan,  nan, ...,  nan,  nan,  nan],
         [ nan,  nan,  nan, ...,  nan,  nan,  nan],
         [ nan,  nan,  nan, ...,  nan,  nan,  nan]], 
.
.
.

Since it is NaN, I searched for "tensorflow nan" and found a solution. http://stackoverflow.com/questions/33712178/tensorflow-nan-bug

The problematic part was the calculation part of the cross entropy, which was as follows (y_conv is the probability of each label by the softmax function).

python


cross_entropy = -tf.reduce_sum(labels*tf.log(y_conv))

If this is left as it is, it will become log (0) and NaN may come out. Therefore, it was solved by normalizing to the range of 1e-10 to 1.0 and then taking the log as shown below.

python


cross_entropy = -tf.reduce_sum(labels*tf.log(tf.clip_by_value(y_conv,1e-10,1.0)))

There is a function called ~~ tf.nn.softmax_cross_entropy_with_logits, and it seems better to use it as follows. ~~ → This method didn't work.

Recommended Posts

The inaccuracy of Tensorflow was due to log (0)
Setting to output the log of cron execution
I tried to find the average of the sequence with TensorFlow
Take the execution log of Celery
Supplement to the explanation of vscode
The story that the version of python 3.7.7 was not adapted to Heroku
Conditional branch due to the existence of a shell script file
[TensorFlow 2] How to check the contents of Tensor in graph mode
The first artificial intelligence. How to check the version of Tensorflow installed.
I tried to transform the face image using sparse_image_warp of TensorFlow Addons
The story of trying to reconnect the client
Did the number of store closures increase due to the impact of the new coronavirus?
Script to change the description of fasta
How to check the version of Django
The story of adding MeCab to ubuntu 16.04
I'm stunned by the behavior of filter () due to different versions of Python
Probably the most straightforward introduction to TensorFlow
The story of pep8 changing to pycodestyle
Visualize the export data of Piyo log
P100-PCIE-16GB was added to the GPU of Google Colab before I knew it
AtCoder AGC 041 C --I was addicted to the full search of Domino Quality
I didn't understand the Resize of TensorFlow so I tried to summarize it visually.
How to install the deep learning framework Tensorflow 1.0 in the Anaconda environment of Windows
How to calculate the volatility of a brand
Log in to the remote server with SSH
How to find the area of the Voronoi diagram
Set the range of active strips to the preview range
I tried to touch the API of ebay
I tried to correct the keystone of the image
Change the decimal point of logging from, to.
The shape of the one-dimensional array of numpy was complicated
To get the path of the currently running python.exe
I want to customize the appearance of zabbix
Use pygogo to get the log in json.
From the introduction of pyethapp to the execution of contract
Try to simulate the movement of the solar system
The story of moving from Pipenv to Poetry
Create a command to get the work log
I tried to predict the price of ETF
I tried to vectorize the lyrics of Hinatazaka46!
I tried to make something like a chatbot with the Seq2Seq model of TensorFlow
[Django] Let's try to clarify the part of Django that was somehow through in the test
It was a little difficult to do flask with the docker version of nginx-unit
python beginners tried to predict the number of criminals
The wall of changing the Django service from Python 2.7 to Python 3
Template of python script to read the contents of the file
How to get the number of digits in Python
A memo to visually understand the axis of pandas.Panel
I want to grep the execution result of strace
I tried to summarize the basic form of GPLVM
Add information to the bottom of the figure with Matplotlib
I tried the MNIST tutorial for beginners of tensorflow.
Try to solve the problems / problems of "Matrix Programmer" (Chapter 1)
The idea of Tensorflow learned from potato chip manufacturing
How to visualize the decision tree model of scikit-learn
[python] option to turn off the output of click.progressbar
Let's break down the basics of TensorFlow Python code
Send log data from the server to Splunk Cloud
Try to estimate the number of likes on Twitter
Get to know the feelings of gradient boosting trees
Log the value of SwitchBot thermo-hygrometer with Raspberry Pi