Behavior when giving a list with shell = True in subprocess

Origin

If you try to run a file with a jar embedded in a shell script like embulk or digdag using Python's subprocess with shell = False

>>> import subprocess
>>> subprocess.check_call(["digdag"])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 181, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 168, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1024, in _execute_child
    raise child_exception
OSError: [Errno 8] Exec format error

It fails like. This can be avoided by setting shell = True, but I investigated how to pass arguments to the program because it was different when shell was False (default) and when it was True.

What I wondered

If you give a list when shell = False, it will be processed as a command and its arguments. For example

sample.sh


#!/bin/sh
echo "$1", "$2", "$3" > out.txt

Code that calls a shell script called subprocess.check_call

caller_l0.py


import subprocess
cmd = ["./sample.sh", "1starg", "2ndarg", "3rdarg"]
subprocess.check_call(cmd, shell=False)

When you execute

$ python caller_l0.py
$ cat out.txt
1starg, 2ndarg, 3rdarg

You can see that the arguments are passed as intended. Just set it to shell = True and

caller_l1.py


import subprocess
cmd = ["./sample.sh", "1starg", "2ndarg", "3rdarg"]
subprocess.check_call(cmd, shell=True)

When you execute

$ python caller_l1.py
$ cat out.txt
, ,

Where did you go after the second element of the list?

What I looked up

The official documentation (https://docs.python.org/3/library/subprocess.html) describes the args parameter passed to the subprocess as follows:

args is required for all calls and should be a string, or a sequence of program arguments.
Providing a sequence of arguments is generally preferred, as it allows the module to take
care of any required escaping and quoting of arguments (e.g. to permit spaces in file names).
If passing a single string, either shell must be True (see below) or else the string must
simply name the program to be executed without specifying any arguments.

There is no explanation here for the case where list is given with shell = True. If you look a little more

If args is a sequence, the first item specifies the command string, and any additional
items will be treated as additional arguments to the shell itself.
That is to say, Popen does the equivalent of:

  Popen(['/bin/sh', '-c', args[0], args[1], ...])

On MacOS, / bin / sh is bash, so if you do man

-c string If  the  -c  option  is  present, then commands are read from
          string.  If there are arguments after the  string,  they  are
          assigned to the positional parameters, starting with $0.

It says, but I didn't understand it, so I searched for it https://stackoverflow.com/questions/1711970/cant-seem-to-use-bash-c-option-with-arguments-after-the-c I arrived at -option-string and understood that \ $ 0, \ $ 1, ... in the string given to -c would be replaced.

caller_l1_.py


import subprocess
cmd = ["./sample.sh $0 $1 $2", "1starg", "2ndarg", "3rdarg"]
subprocess.check_call(cmd, shell=True)
$ python caller_l1_.py
$ cat out.txt
1starg, 2ndarg, 3rdarg

Arguments after sh -c

When I checked the behavior when multiple arguments were passed after sh -c, other than bash, it was the same behavior on dash on Ubuntu and sh on FreeBSD. From that, I wondered if there was any use for it.

$ bash -c '$0 $1, $2, $3' echo '1-0 1-1' 2 3
1-0 1-1, 2, 3

Then, apply the quoted one, and do the same via subprocess.

$ python -c 'import subprocess;subprocess.check_call(["$0 $1, $2, $3", "echo", "1-0 1-1", "2", "3"], shell=True)'
1-0 1-1, 2, 3

However, if this is the case, you can expand the character string from the beginning, while on the other hand

caller_l1__.py


import subprocess
cmd = ["$0 $1 $2 $3", "./sample.sh", "1-1 1-2", "2", "3"]
subprocess.check_call(cmd, shell=True)

Since the one after being replaced is only passed to sample.sh

$ python caller_l1__.py
$ cat out.txt
1-1, 1-2, 2

After all, I wasn't sure where to give multiple arguments after -c.

Summary

The behavior when passing a list with shell = True in the Python subprocess is similar to the behavior when passing multiple arguments after sh -c. However, I didn't know how to use it (someone tell me).

When shell = True, I felt that it was straightforward to specify a character string instead of a list.

caller_s1.py


import subprocess
cmd = "./sample.sh 1starg 2ndarg 3rdarg"
subprocess.check_call(cmd, shell=True)
$ python2.7 caller_s1.py
$ cat out.txt
1starg, 2ndarg, 3rdarg

Recommended Posts

Behavior when giving a list with shell = True in subprocess
When creating a matrix in a list
Behavior when returning in the with block
Things to note when initializing a list in Python
Use communicate () when receiving output in a Python subprocess
Differences in behavior between append () and "+ =" operators when adding data to a list in Python
Create a flag in settings that will be True only when testing with Django
Create a list in Python with all followers on twitter
Try programming with a shell!
Behavior in each language when coroutines are reused with for
Process the files in the folder in order with a shell script
Execute a command in Go like Python's subprocess.call (~, shell = True)
Flask-Create a Todo list with CSRF measures in Flask with WTF
When I try to divide a list with MeCab, I get'TypeError: in method'Tagger_parse', argument 2 of type'char const *''
Process the contents of the file in order with a shell script
Get a list of files in a folder with python without a path
reload in django shell with ipython
[python] Manage functions in a list
When writing a program in Python
Get a list of packages installed in your current environment with python
Generate a list packed with the number of days in the current month.
Receive a list of the results of parallel processing in Python with starmap
What to do when a warning message is displayed in pip list
Spiral book in Python! Python with a spiral book! (Chapter 14 ~)
Python3> in keyword> True with partial match?
Precautions when pickling a function in python
Display a list of alphabets in Python 3
Draw a heart in Ruby with PyCall
Change the list in a for statement
A workaround when installing pyAudio with pip.
Manage logrotate generations with a shell script.
Implement a model with state and behavior
How to pass the execution result of a shell command in a list in Python
Behavior when SIGEV_THREAD is set in sigev_notify of sigevent with timer_create (C language)
Differences in the behavior of each LL language when the list index is skipped
If you get a long error when tabbing an interactive shell with Anaconda
Character strings placed in GCS with python are garbled when viewed with a browser
How to get a list of files in the same directory with python