Read the output of subprocess.Popen in real time

Overview

Follow and read the contents output to the standard output with subprocess.Popen. At this time, instead of using readline () to read each line break, use read () to read (?). I often worry about the stream, so make a note.

I have confirmed the operation with Python 2.7.5 on CentOS7.

Character output script

This is the script that is started by Popen.

textout.py


#!/usr/bin/env python
import sys
import time
for i in range(0, 10):
    sys.stdout.write("[Count {0:04d}]".format(i))
    time.sleep(1)

A simple script that outputs [Count 0000] every second. However, since it does not break, it cannot be read by readline ().

Execution script

Call the above script with Popen and read the standard output.

reader.py


#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys, os, time, errno, fcntl
from subprocess import Popen, PIPE

#Textout with Popen.Run py
#At this time to python-Pass u option to avoid buffering
p = Popen([sys.executable, "-u", "textout.py"], bufsize=0, stdout=PIPE)

# p.Put stdout into non-blocking mode
flag = fcntl.fcntl(p.stdout.fileno(), fcntl.F_GETFL)
fcntl.fcntl(p.stdout.fileno(), fcntl.F_SETFL, flag | os.O_NONBLOCK)

while True:
    #Read loop
    try:
        # p.load stdout
        #Since it is a non-blocking mode, read as much as you can read
        buf = p.stdout.read()
        if buf == "": break   # p.Exit when stdout is closed
        sys.stdout.write(buf) #Output what you read
        sys.stdout.flush()    #Flush because there are no line breaks()
    except IOError, e:
        #IOError if there is nothing to read(11,
        # "Resource temporarily unavailable")But
        #Wait because it will be thrown
        if e.errno == errno.EAGAIN:
            time.sleep(0.1)
p.wait()

Execution result

$ ./reader.py
[Count 0000][Count 0001][Count 0002][Count 0003][Count 0004][Count 0005][Count 0006][Count 0007][Count 0008][Count 0009]

[Count 0000] is output every second.

point

--Do not buffer when running textout.py, or flush () after writing to sys.stdout --reader.py reads p.stdout with read (), so read in non-blocking mode --When waiting for output, errno.EAGAIN is sent, so exception handling and waiting --If you wait until textout.py finishes, read () one shot (^^;) without doing such a troublesome thing

Recommended Posts

Read the output of subprocess.Popen in real time
Read the standard output of a subprocess line by line in Python
Output in the form of a python array
Gradually display the output of the command executed by subprocess.Popen
Output the time from the time the program was started in python
Get standard output in real time with Python subprocess
Filter the output of tracemalloc
Read Fortran output in python
Survey on the use of machine learning in real services
Summary of stumbling blocks in Django for the first time
Find the eigenvalues of a real symmetric matrix in Python
The story of participating in AtCoder
Have python read the command output
Process the result of% time,% timeit
The story of the "hole" in the file
The meaning of ".object" in Django
I tried to describe the traffic in real time with WebSocket
I compared the calculation time of the moving average written in Python
Read the image of the puzzle game and output the sequence of each block
Output the contents of ~ .xlsx in the folder to HTML with Python
A function that measures the processing time of a method in python
Verify the compression rate and time of PIXZ used in practice
Part 1 I wrote the answer to the reference problem of how to write offline in real time in Python
[Understanding in 3 minutes] The beginning of Linux
Check the behavior of destructor in Python
The story of an error in PyOCR
Output tree structure of files in Python
Implement part of the process in C ++
Read all csv files in the folder
The result of installing python in Anaconda
Let's claim the possibility of pyenv-virtualenv in 2021
Read the file line by line in Python
Read the file line by line in Python
MongoDB for the first time in Python
The basics of running NoxPlayer in Python
Difference in output of even-length window function
Conversion of time data in 25 o'clock notation
Read all the contents of proc / [pid]
I read the implementation of golang channel
In search of the fastest FizzBuzz in Python
Read the implementation of ARM global timer
Get a datetime instance at any time of the day in Python
[Python] Read the specified line in the file
[IOS] Change the display time for each frame of GIF animation in Pythonista3.
Various ways to read the last line of a csv file in Python
[Linux] Difference in time information depending on the clock ID of the clock_gettime () function
Visualize accelerometer information from the microcomputer board in real time with mbed + Python
To output a value even in the middle of a cell with Jupyter Notebook
Get Unix time of the time specified by JST regardless of the time zone of the server in Python
I want to store the result of% time, %% time, etc. in an object (variable)
[Python] Read the source code of Bottle Part 2
The meaning of {version-number} in the mysql rpm package
[Python] Sort the list of pathlib.Path in natural sort
Shortening the analysis time of Openpose using sound
I read the implementation of range (Objects / rangeobject.c)
Setting to output the log of cron execution
Get the caller of a function in Python
Match the distribution of each group in Python
View the result of geometry processing in Python
Make a copy of the list in Python
Find the number of days in a month