Gradually display the output of the command executed by subprocess.Popen

I often used the communicate method when executing external commands using the subprocess module.

python:subprocess.Popen.communicate()


import subprocess

(stdoutdata, stderrdata) = subprocess.Popen(['some', 'command'], stdout=subprocess.PIPE).communicate()
print(stdoutdata)

However, if you want to output the result asynchronously such as a build script, it cannot be realized by the communicate method. As shown in Library Reference, the communicate method waits until the process ends, and then prints the standard output data. Because I will return it.

If you want to fulfill this requirement, you need to use the io module to create a stream that corresponds to the output of the child process.

(ref) http://docs.python.jp/2/library/subprocess.html#subprocess.Popen.stdout (ref) https://gist.github.com/mattbornski/3299031

Stream corresponding to the output of the child process


def exec_process(commands):
    process = subprocess.Popen(commands, stdout=subprocess.PIPE, bufsize=-1)
    with io.open(process.stdout.fileno(), closefd=False) as stream:
        [print(line.rstrip('\n')) for line in stream]

    #Wait until the end of the process and judge the result
    process.wait()
    if process.returncode != 0:
        print('Build process aborts.')
        sys.exit(1)

Below, a memo of some points I was addicted to

  1. When creating a stream with ʻio.open, you need to give the file path or file descriptor (which can be obtained with (file object) .fileno ()). I don't know the pathname of subprocess.Popen.stdout, so I need to give it a file descriptor. Note that you are not giving the file object (= subprocess.Popen.stdout`) as is.

(ref) http://docs.python.jp/2/library/io.html#io.open

python


# process.io stdout.If given directly to open, it will end with a runtime exception.
def exec_process(commands):
    process = subprocess.Popen(commands, stdout=subprocess.PIPE, bufsize=-1)
    with io.open(process.stdout, closefd=False) as stream:
        [print(line.rstrip('\n')) for line in stream]
  1. When creating a stream with ʻio.open, you need to specify closefd = False. The default is closefd = True`, and in that state the following error occurs.
close failed in file object destructor:
IOError: [Errno 9] Bad file descriptor

It is thought that the cause is trying to close the file object (= subprocess.Popen.stdout) that the child process is holding at the timing when the stream is closed (= in this case, when exiting the with statement).

Recommended Posts

Gradually display the output of the command executed by subprocess.Popen
Get the output value of the command (as received by xargs)
Read the output of subprocess.Popen in real time
Filter the output of tracemalloc
Debug output of chalice command
Save the output of GAN one by one ~ With the implementation of GAN by PyTorch ~
Changed the default style (CSS) of pandas data frame output by display in Google Colab
[Python] Display only the elements of the list side by side [Vertical, horizontal]
Read the standard output of a subprocess line by line in Python
Have python read the command output
How to output the output result of the Linux man command to a file
[Linux] Command to get a list of commands executed in the past
Display the graph of tensorBoard on jupyter
Migemo version of the: find command,: mfind
Pandas of the beginner, by the beginner, for the beginner [Python]
Notice the completion of a time-consuming command
Output all the email body of the email group searched by Gmail and narrowed down
Output the number of CPU cores in Python
Check the operation of OpenCV3 installed by Anaconda
Quickly display the QR code on the command line
Setting to output the log of cron execution
Sort the elements of the array by specifying the conditions
Display output of a list of floating point numbers
Linux: Rename the process displayed by the ps command
The story of misreading the swap line of the top command
How to erase the characters output by Python
Linux: Understand the information displayed by the top command
Minimize the number of polishings by combinatorial optimization
Judging the finish of mahjong by combinatorial optimization
Output in the form of a python array
Search by the value of the instance in the list
Get the location of the file where the exe is placed when the exe created by PyInstaller is executed
Sort files updated within the period specified by the find command in order of size
Save the output of conditional GAN for each class ~ With cGAN implementation by PyTorch ~