[Explanation for beginners] TensorFlow tutorial Deep MNIST

A tutorial for TensorFlow experts is explained from an introductory perspective

I ran the tutorial for TensorFlow experts Deep MNIST for Experts. I will explain from the perspective of beginners. There are many parts that lack accuracy, but I give priority to clarity. It was very difficult compared to the tutorial for beginners: skull: * Added a code so that image can be output, and posted an image of convolution / pooling processing (2017/7/27) </ sup>

Reference link

-Installing TensorFlow on Windows was easy even for Python beginners -[Explanation for beginners] TensorFlow basic syntax and concept -[Explanation for beginners] TensorFlow tutorial MNIST (for beginners) -TensorFlow API memo -[Introduction to TensorBoard] Visualize TensorFlow processing to deepen understanding -[Introduction to TensorBoard: image] TensorFlow Visualize image processing to deepen understanding -[Introduction to TensorBoard: Projector] Make TensorFlow processing look cool -Visualize TensorFlow tutorial MNIST (for beginners) with TensorBoard -Yuki Kashiwagi's facial features to understand TensorFlow [Part 1]

Outline of processing

The other day, the photo was impressive, though the content was also impressive "I haven't eaten for nearly 80 years and amazed Indian ascetic doctors" I will explain it sensuously without using mathematical formulas, using the photo in the above as an example (I used it because the photo was very easy to understand). The TensorFlow expert tutorial Deep MNIST for Experts is primarily ** convolution **, ** pooling **, ** tightly coupled **. It consists of three processes. How to judge the image by each process is as shown in the figure below. 10.Overview.JPG

A little detailed processing outline

The following two Youtube videos were very easy to understand, so I will post a link (in English).

In addition, as an explanation blog in Japanese, "Mathematical background of TensorFlow Tutorial-Deep MNIST for Experts (Part 1)" I understood while looking at it. Wonderful.

processing

The figure below shows the entire process using a Tensor. 20.ProcessOverview02.JPG This time, you can see that the two-layer (multi-layer) convolution and pooling process is performed. You can see from the amount of Tensor that "find features" using a filter in convolution and "roughly organize features" in pooling.

It looks like this when the actual transition up to the second layer is represented by an image. 30.Tensorboard_images.JPG

The explanation of convolution processing is described in detail in the article "[Explanation for beginners] Introduction to convolution processing (explained in TensorFlow)". The explanation of pooling process is described in detail in the article "[Explanation for beginners] Introduction to pooling process (explained in TensorFlow)". The explanation of image output was written in detail in the article "[Introduction to TensorBoard: image] TensorFlow Visualize image processing to deepen understanding".

Graph of TensorBoard is as shown in the figure below. 20.ProcessOverview.JPG Processing starts from the input below. The exact tightly coupled layer is divided into fc1_layer and fc2_layer.

code

Finally, put the code output to TensorBoard. When executed on a 2-core PC, the CPU was almost used up immediately as shown in the figure below, and other processing could not be performed on the PC during execution. Please note that it will take some time to complete the process. 20.CPU_Full.JPG I'm a little unsure about the accuracy. I would be grateful if you could point out any mistakes.

import tensorflow as tf
sess = tf.InteractiveSession()
#Read MNIST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)

#TensorBoard information output directory
log_dir = '/tmp/tensorflow/mnist/logs/mnist_expert_images'

#Delete the specified directory if it exists and recreate it
if tf.gfile.Exists(log_dir):
    tf.gfile.DeleteRecursively(log_dir)
tf.gfile.MakeDirs(log_dir)

# Input placeholders &Image conversion
with tf.name_scope('input'):
    x  = tf.placeholder(tf.float32, shape=[None, 784], name='x-input')
    y_ = tf.placeholder(tf.float32, shape=[None, 10],  name='y-input')

    #Convert image to 28x28
    x_image = tf.reshape(x, [-1, 28, 28, 1])
    tf.summary.image('preprocess', x_image, 10)

#Function that processes variables and outputs TensorBoard
def variable_summaries(var):

    #Variable Summary
    with tf.name_scope('summaries'):
        mean = tf.reduce_mean(var)                      #Scalar output(average)
        tf.summary.scalar('mean', mean)
        with tf.name_scope('stddev'):
            stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))
        tf.summary.scalar('stddev', stddev)             #Scalar output(standard deviation)
        tf.summary.scalar('max', tf.reduce_max(var))    #Scalar output(Maximum value)
        tf.summary.scalar('min', tf.reduce_min(var))    #Scalar output(minimum value)
        tf.summary.histogram('histogram', var)          #Histogram output

#Weighted value
def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)    #Standard deviation 0.Normal distribution random number of 1
    return tf.Variable(initial)

#Bias value
def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)             #Initial value 0.1 constant
    return tf.Variable(initial)

#Convolution process
def conv2d(x, W):
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

# Max Pooling
def max_pool_2x2(x):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')

#1st layer
with tf.name_scope('1st_layer'):

    #1st convolution layer
    with tf.name_scope('conv1_layer'):
        with tf.name_scope('weight'):
            W_conv1 = weight_variable([5, 5, 1, 32])
            variable_summaries(W_conv1)
            
            #Tensor[5,5,1,32]From[32,5,5,1]Permutation conversion and image output
            tf.summary.image('filter', tf.transpose(W_conv1,perm=[3,0,1,2]), 10)

        with tf.name_scope('bias'):            
            b_conv1 = bias_variable([32])
            variable_summaries(b_conv1)
        with tf.name_scope('activations'):            
            h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
            tf.summary.histogram('activations', h_conv1)

            #Tensor[-1,28,28,32]From[-1,32,28,28]And permutation,[-1]When[-32]Merge and output image
            tf.summary.image('convolved', tf.reshape(tf.transpose(h_conv1,perm=[0,3,1,2]),[-1,28,28,1]), 10)

#1st pooling layer
    with tf.name_scope('pool1_layer'):    
        h_pool1 = max_pool_2x2(h_conv1)

        #Tensor[-1,14,14,32]From[-1,32,14,14]And permutation,[-1]When[32]Merge and output image
        tf.summary.image('pooled', tf.reshape(tf.transpose(h_pool1,perm=[0,3,1,2]),[-1,14,14,1]), 10)

#2nd layer
with tf.name_scope('2nd_layer'):
    
    #2nd convolution layer
    with tf.name_scope('conv2_layer'):
        with tf.name_scope('weight'):
            W_conv2 = weight_variable([5, 5, 32, 64])
            variable_summaries(W_conv2)

            #Tensor[5,5,32,64]From[32*64,5,5,1]Permutation conversion and image output
            tf.summary.image('filter', tf.reshape(tf.transpose(W_conv2,perm=[2,3,0,1]),[-1,5,5,1]), 20)
        with tf.name_scope('bias'):            
            b_conv2 = bias_variable([64])
            variable_summaries(b_conv2)
        with tf.name_scope('activations'):
            h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
            tf.summary.histogram('activations', h_conv2)
            #Tensor[-1,14,14,64]From[-1,64,14,14]And permutation,[-1]When[64]Merge and output image
            tf.summary.image('convolved', tf.reshape(tf.transpose(h_conv2,perm=[0,3,1,2]),[-1,14,14,1]), 10)
    
#2nd pooling layer
    with tf.name_scope('pool2_layer'):
        h_pool2 = max_pool_2x2(h_conv2)
        
        #Tensor[-1,7,7,64]From[-1,64,7,7]And permutation,[-1]When[64]Merge and output image
        tf.summary.image('pooled', tf.reshape(tf.transpose(h_pool2,perm=[0,3,1,2]),[-1,7,7,1]), 10)

#Tightly coupled layer
with tf.name_scope('fc1_layer'):
    h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
    with tf.name_scope('weight'):
        W_fc1 = weight_variable([7 * 7 * 64, 1024])
        variable_summaries(W_fc1)
    with tf.name_scope('bias'):            
        b_fc1 = bias_variable([1024])
        variable_summaries(b_fc1)
    with tf.name_scope('activations'):
        h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
        tf.summary.histogram('activations', h_fc1)

    #Drop out
    with tf.name_scope('dropout'):
        keep_prob  = tf.placeholder(tf.float32)
        tf.summary.scalar('dropout_keep_probability', keep_prob)
        h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)

#Read layer
with tf.name_scope('fc2_layer'):
    with tf.name_scope('weight'):
        W_fc2 = weight_variable([1024, 10])
        variable_summaries(W_fc2)
    with tf.name_scope('bias'):            
        b_fc2 = bias_variable([10])
        variable_summaries(b_fc2)
    with tf.name_scope('preactivations'):
        y_conv = tf.matmul(h_fc1_drop, W_fc2) + b_fc2
        tf.summary.histogram('preactivations', y_conv)

#Cross entropy(Cross entropy)Calculation
with tf.name_scope('cross_entropy'):
    cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))
    tf.summary.scalar("cross_entropy", cross_entropy)

#Training
with tf.name_scope('train'):
    train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)

#Correct answer rate calculation
with tf.name_scope('accuracy'):
    correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
    tf.summary.scalar('accuracy', accuracy)

#Output all Summary
merged = tf.summary.merge_all()
train_writer = tf.summary.FileWriter(log_dir + '/train', sess.graph) #Training data
test_writer  = tf.summary.FileWriter(log_dir + '/test')              #test data

#Initialization
tf.global_variables_initializer().run()

#Repeat training / test
for i in range(3000):
    batch = mnist.train.next_batch(50)
    
    #Training trace details every 100 times
    if i % 100 == 0:
        run_options  = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
        run_metadata = tf.RunMetadata()
        summary, train_accuracy, _   = sess.run([merged, accuracy , train_step],
                                                feed_dict={x: batch[0], y_:batch[1], keep_prob: 1.0},
                                                options=run_options,
                                                run_metadata=run_metadata)
        train_writer.add_run_metadata(run_metadata, 'step%03d' % i)
        train_writer.add_summary(summary, i)
        print('step %d, training accuracy %g' % (i, train_accuracy))
    #Test every 100 times
    elif i % 100 == 99:
        summary_test, train_accuracy = sess.run([merged, accuracy], feed_dict={x: mnist.test.images, y_:mnist.test.labels, keep_prob: 1.0})
        test_writer.add_summary(summary_test, i)
        
    #Training result writing
    summary, _ = sess.run([merged, train_step], feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
    train_writer.add_summary(summary, i)

#Final test result output
print('test accuracy %g' % accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))

#Writer closed
train_writer.close()
test_writer.close()

Recommended Posts

[Explanation for beginners] TensorFlow tutorial Deep MNIST
[Explanation for beginners] TensorFlow tutorial MNIST (for beginners)
TensorFlow Tutorial MNIST For ML Beginners
TensorFlow Tutorial -MNIST For ML Beginners
Conducting the TensorFlow MNIST For ML Beginners Tutorial
I tried the MNIST tutorial for beginners of tensorflow.
TensorFlow MNIST For ML Beginners Translation
TensorFlow Deep MNIST for Experts Translation
[Roughly translate TensorFlow Tutorial into Japanese] 1. MNIST For ML Beginners
[Roughly translate TensorFlow Tutorial into Japanese] 2. Deep MNIST For Experts
Supplementary notes for TensorFlow MNIST For ML Beginners
[Explanation for beginners] TensorFlow basic syntax and concept
I tried a TensorFlow tutorial (MNIST for beginners) on Cloud9-Classification of handwritten images-
[Explanation for beginners] Introduction to convolution processing (explained in TensorFlow)
[Explanation for beginners] Introduction to pooling processing (explained in TensorFlow)
[Deprecated] Chainer v1.24.0 Tutorial for beginners
TensorFlow tutorial tutorial
I tried running the TensorFlow tutorial with comments (_TensorFlow_2_0_Introduction for beginners)
I tried the TensorFlow tutorial MNIST 3rd
Django tutorial summary for beginners by beginners ③ (View)
Implementation and explanation using XGBoost for beginners
Beginners read "Introduction to TensorFlow 2.0 for Experts"
[Deep learning] Nogizaka face detection ~ For beginners ~
Django tutorial summary for beginners by beginners ⑤ (test)
Django tutorial summary for beginners by beginners ⑦ (Customize Admin)
Django tutorial summary for beginners by beginners ⑥ (static file)
Recommended study order for machine learning / deep learning beginners
Roadmap for beginners
Django Tutorial Summary for Beginners by Beginners (Model, Admin)
Django tutorial summary for beginners by beginners ① (project creation ~)
Django tutorial summary for beginners by beginners ④ (Generic View)
Installing TensorFlow on Windows Easy for Python beginners
[Translation] NumPy Official Tutorial "NumPy: the absolute basics for beginners"
MNIST image generation program creation by DCGAN (tensorflow tutorial)
Spacemacs settings (for beginners)
python textbook for beginners
Enable GPU for tensorflow
Dijkstra algorithm for beginners
OpenCV for Python beginners
[For beginners] Summary of standard input in Python (with explanation)
How to learn TensorFlow for liberal arts and Python beginners
"Deep Learning from scratch" self-study memo (No. 15) TensorFlow beginner tutorial
[For beginners] I tried using the Tensorflow Object Detection API