2017-02-19 Python> Link> Redirect to output file immediately> sys.stdout.flush ()

Operating environment


GeForce GTX 1070 (8GB)
ASRock Z170M Pro4S [Intel Z170chipset]
Ubuntu 14.04 LTS desktop amd64
TensorFlow v0.11
cuDNN v5.1 for Linux
CUDA v8.0
Python 2.7.6
IPython 5.1.0 -- An enhanced Interactive Python.
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

The error output of the learning process of TensorFlow is performed by print ().

I'm running TensorFlow code with redirection (> res), but the size of the destination file (res) didn't increase immediately. It was an action that greatly increased at some timing.

I wondered if I should flush and found the following.

http://7rpn.hatenablog.com/entry/2015/11/29/131601

sys.stdout.flush()

By adding sys.stdout.flush () to the 92nd line as shown below, the error output of the learning process can be output in a timely manner.

Thank you for the information.

learn_in100out100.py


#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys
import tensorflow as tf
import tensorflow.contrib.slim as slim
import numpy as np

'''
v0.6 Feb. 19, 2017
    - add sys.stdout.flush() to immediately print 
v0.5 Feb. 18, 2017
    - add fc_drop()
v0.4 Feb. 15, 2017
    - fix bug > ZeroDivisionError: float division by zero @ shuffle_batch()
v0.3 Feb. 15, 2017
    - tweak [batch_size] for shuffle_batch()
v0.2 Feb. 15, 2017
    - fix bug > different dimensions for placeholder and network
v0.1 Feb. 06, 2017
    - read [test_in.csv],[test_out.csv]
'''

'''
codingrule:PEP8
'''


def fc_drop(inputs, *args, **kwargs):
    # Thanks to: http://qiita.com/shngt/items/f532601b4f059ce8584f
    net = slim.fully_connected(inputs, *args, **kwargs)
    return slim.dropout(net, 0.5)

filename_inp = tf.train.string_input_producer(["test_in.csv"])
filename_out = tf.train.string_input_producer(["test_out.csv"])
NUM_INP_NODE = 100
NUM_OUT_NODE = 100

# parse csv
# a. input node
reader = tf.TextLineReader()
key, value = reader.read(filename_inp)
deflist = [[0.] for idx in range(NUM_INP_NODE)]
input1 = tf.decode_csv(value, record_defaults=deflist)
# b. output node
key, value = reader.read(filename_out)
deflist = [[0.] for idx in range(NUM_OUT_NODE)]
output1 = tf.decode_csv(value, record_defaults=deflist)
# c. pack
# inputs = tf.pack([input1])
inputs = input1
# outputs = tf.pack([output1])
outputs = output1

batch_size = 2
inputs_batch, output_batch = tf.train.shuffle_batch(
    [inputs, outputs], batch_size, capacity=10, min_after_dequeue=batch_size)

input_ph = tf.placeholder("float", [None, 100])
output_ph = tf.placeholder("float", [None, 100])

# network
hiddens = slim.stack(input_ph, slim.fully_connected, [7, 7, 7],
                     activation_fn=tf.nn.sigmoid, scope="hidden")
# a. without dropout
# prediction = slim.fully_connected(
#    hiddens, 100, activation_fn=None, scope="output")
# b. with dropout
drpout = slim.stack(hiddens, fc_drop, [100, 100], scope='fc')
prediction = slim.fully_connected(
    drpout, 100, activation_fn=None, scope="output")
loss = tf.contrib.losses.mean_squared_error(prediction, output_ph)

train_op = slim.learning.create_train_op(loss, tf.train.AdamOptimizer(0.001))

init_op = tf.initialize_all_variables()

with tf.Session() as sess:
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(sess=sess, coord=coord)

    try:
        sess.run(init_op)
        # for idx in range(10000):
        for idx in range(100000):
            inpbt, outbt = sess.run([inputs_batch, output_batch])
            _, t_loss = sess.run(
                [train_op, loss],
                feed_dict={input_ph: inpbt, output_ph: outbt})

            if (idx+1) % 100 == 0:
                print("%d,%f" % (idx+1, t_loss))
                sys.stdout.flush()
    finally:
        coord.request_stop()

    coord.join(threads)
$ python learn_in100out100.py > res.learn.N\=1000_dropout@output_longrun_170218

res.learn.N \ = 1000_dropout @ output_longrun_170218 has been updated accordingly, making it easier to check the error graph in Jupyter.

qiita.png

Below is the result during learning. It soon became clear that the result was completely useless. .. ..

qiita.png

Recommended Posts

2017-02-19 Python> Link> Redirect to output file immediately> sys.stdout.flush ()
Output to csv file with Python
Output python log to both console and file
Change the standard output destination to a file in Python
[Python] How to output a pandas table to an excel file
Link to get started with python
Tips on Python file input / output
[Python] Write to csv file with Python
Write standard output to a file
Output cell to file with Colaboratory
Notes for Python file input / output
File upload to Azure Storage (Python)
[Python] How to convert db file to csv
Output color characters to pretty with python
How to convert Python to an exe file
Convert psd file to png in Python
Output Python log to console with GAE
Python> Output numbers from 1 to 100, 501 to 600> For csv
[Road to intermediate Python] Article link summary
Python / subprocess> Symbolic link Implementation to get only the destination file name> os.readlink ()
Output timing is incorrect when standard (error) output is converted to a file in Python
How to read a CSV file with Python 2/3
MessagePack-Try to link Java and Python with RPC
Import Excel file from Python (register to DB)
From file to graph drawing in Python. Elementary elementary
How to create a JSON file in Python
[Python] How to read excel file with pandas
Convert svg file to png / ico with Python
Writing logs to CSV file (Python, C language)
I tried to output LLVM IR with Python
How to erase the characters output by Python
Export Python3 version OpenCV KeyPoint to a file
How to output "Ketsumaimo" as standard output in Python
I want to write to a file with Python
Sample to put Python Kivy in one file
[CentOS8] How to output Python standard output to systemd log
Speed evaluation of CSV file output in Python
Script python file
Python file processing
Output the specified table of Oracle database in Python to Excel for each file
"Backport" to python 2
Parse a JSON string written to a file in Python
Template of python script to read the contents of the file
python> Change file owner> os.chown ("path / to / file", uid, gid)
I tried to touch the CSV file with Python
[python] option to turn off the output of click.progressbar
How to measure mp3 file playback time with python
Read the xml file by referring to the Python tutorial
I want to randomly sample a file in Python
How to convert JSON file to CSV file with Python Pandas
[Python] Change standard input from keyboard to text file
Read json file with Python, format it, and output json
Procedure to exe python file from Ubunts environment construction
[Python-pptx] Output PowerPoint font information to csv with python
Output to "7-segment LED" using python on Raspberry Pi 3!
Output a binary dump in binary and revert to a binary file
[Python] How to output the list values in order