Think about dropouts with MNIST

Let's think about dropout using the article here posted the other day. However, it ends in one word.

What is a dropout?

Overfitting sometimes occurs when deep learning is used for learning. This means that it processes too much training data and becomes a dedicated discriminator, making it impossible to handle other data.

Identification is basically a classification problem, but this classification doesn't work. It's easy to see in the graph, but there are some points, which are not so along the points at first, but eventually a curve close to the points is completed. However, if you learn too much, you will end up with a strange curve, although it is close to each point. You can see that even if you classify it with this, it will not work. dc21319e-6fb5-aa76-34b4-b9e18891553d.png

So, after learning to some extent, we will ignore the parts that are not so relevant. This is a "dropout".

In the case of Deep Learning, each node is connected in a hierarchy, but I will delete the node that seems to be unnecessary. 20160714170858.jpg

Description in TensorFlow

In TensorFlow, set the percentage of this "disappearing node" to remain. The function to use is "tensorflow.nn.dropout ()". The ratio is specified by the second argument.

keep_prob = tensorflow.placeholder(tf.float32)
h_fc1_drop = tensorflow.nn.dropout(h_fc1, keep_prob)

In the tutorial, the percentage left when learning is executed is specified as "0.5". (Leave half the nodes and erase half the nodes)

train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})

Please do not drop out during testing. (Set "1.0")

accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0})

Value adjustment

There is no choice but to make trial and error as to what the value should be, but it seems that not only the value but also the location and number of dropouts are subject to adjustment.

Recommended Posts

Think about dropouts with MNIST
Notes about with
MNIST (DCNN) with skflow
Think yaml with python
About import
Think about architecture in python
Think about dropouts with MNIST
Think about architecture in python
About learning with google colab
Implement blockchain with about 60 lines
Try TensorFlow MNIST with RNN
What beginners think about programming in 2016
Think about the minimum change problem
Inference works with Chainer 2.0 MNIST sample
Roughly think about the loss function
Think about menus by combinatorial optimization