Write a batch script with Python3.5 ~

This article summarizes the know-how for replacing Bash scripts with Python scripts, but we plan to make corrections in the future.

Display of character string

Use the print function. This is a variable length argument, and if you give more than one, it will be output separated by a space by default.

Bash

echo Hello, world!

↓

Python3.5

print("Hello,", "world!")

If you want to output other than the standard output, specify it in the form of file = ○○. For example, in the case of a standard error:

Bash

echo Error >&2

↓

Python3.5

import sys

print("Error", file=sys.stderr)

When overwriting / adding to a file, it will be as follows.

Bash

echo foo > file1.txt
echo bar >> file1.txt
echo hoge >> file2.txt
echo fuga >> file2.txt

↓

Python3.5

with open("file1.txt", "w") as f:
    print("foo", file=f)
    print("bar", file=f)
with open("file2.txt", "a") as f:
    print("hoge", file=f)
    print("fuga", file=f)

The space between the displayed arguments and the line feed character at the end of the line can be changed to another character. For example, to set the delimiter to tab and no character at the end of the line:

Python3.5

print("foo", "bar", sep="\t", end="")

Optional analysis

In Bash etc., it is a difficult option analysis, but in Python it can be easily analyzed with the ʻargparse` module. Basically, use it as follows.

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--verbose", "-v", help="Verbose", action="store_true")
parser.add_argument("--quiet", "-q", help="Suppress messages", action="store_true")
parser.add_argument("target", help="target")
parser.add_argument("source", nargs="+", help="source")
args = parser.parse_args()

function(args.option)

Each option can be accessed with ʻargs.verbose, ʻargs.quiet, etc.

See the official commentary for details. https://docs.python.jp/3/library/argparse.html

Command execution

You can use subprocess.run from Python 3.5. Use this if you don't think about compatibility with past versions.

Just run

Bash

g++ sample.cpp -o sample -std=gnu++14 -march=native -O3

↓

Python3.5

import subprocess

subprocess.run(["g++", "sample.cpp", "-o", "sample", "-std=gnu++14", "-march=native", "-O3"])

In addition to the list, iterable is OK. (Tuples, etc.)

Exception if the command ends with a non-zero

Specify check = True. The exception is subprocess.CalledProcessError.

Python3.5

import subprocess

subprocess.run(["git", "push"], check=True)

Capture standard output / error

Specify stdout = subprocess.PIPE ・ stderr = subprocess.PIPE. It can be retrieved with the stdout and stderr attributes of the return value, respectively. Please note that it is a byte type.

Python3

import subprocess

result = subprocess.run(["apt", "list", "--upgradable"], stderr=subprocess.PIPE, stdout=subprocess.PIPE, check=True)
process(result.stdout)

Discard standard output / error

Specify subprocess.DEVNULL instead of subprocess.PIPE.

Specify standard input

Add ʻinput =to the argument ofsubprocess.run`. Specify by byte type.

Python3.5

import subprocess

assert subprocess.run(["cat"], input=b"aaa\n", stdout=subprocess.PIPE).stdout == b"aaa\n"

redirect

Specify the file object opened by the ʻopen function etc. in each argument of subprocess.runargumentstdin (not ʻinput), stdout, stderr. As long as the input and output are text data, the second argument of ʻopen can be without b`, but it is better to have it.

Directory related

PS: It's easier to use pathlib.

from pathlib import Path

python_path = Path("/usr/bin/python3")

The following introduces a legacy method that does not use pathlib.

Get current directory

You can get it with ʻos.getcwd () . Or, when using pathlib, you can get it with Path () (relative path), Path.cwd () or Path (). Resolve () `(absolute path).

Directory change

Use ʻos.chdir`.

Bash

cd directory

↓

Python3.5

import os

os.chdir("directory")

View the contents of the directory

Use ʻos.listdir`. No argument is required to display the contents of the current directory.

Bash

foo=$(ls)

↓

Python3.5

import os

foo = os.listdir()

If you want to see the contents of a particular directory, give that path.

Bash

foo=$(ls bar)

↓

Python3.5

import os

foo = os.listdir("bar")

To use pathlib.Path, use the method ʻiterdir (). Since it is a generator that is returned, either use a for loop or list it with list ()`.

Python3.5+pathlib

from pathlib import Path

foo = Path().iterdir() # foo=$(ls)
bar = Path("baz").iterdir() # bar=$(ls baz)

# ls qux
for path in Path("qux").iterdir():
  print(path)

If you want to use pattern matching, use glob.glob or glob.iglob instead. The former returns a list and the latter returns an iterator, so the latter should be used to pass to a for statement, map, filter, etc.

Bash

audio_files=$(ls *.wav)

ls *.tmp | xargs rm -f

↓

Python3.5

import glob
import os

audio_files = glob.glob("*.wav")

for tmp_file in glob.iglob("*.tmp"):
    os.remove(tmp_file)

If you use pathlib.Path, use the method glob.

Python3.5+pathlib

from pathlib import Path

for tmp_file in Path().glob("*.tmp"):
    tmp_file.unlink()

Creating a directory

If you want to create a single directory, use ʻos.mkdir`.

Bash

mkdir test_dir

↓

Python3.5

import os

os.mkdir("test_dir")

For pathlib.Path, use the method mkdir.

Python3.5+pathlib

from pathlib import Path

Path("test_dir").mkdir()

If you want to limit permissions beyond the default values (777 & umask values), use mode.

Bash

mkdir -m 700 secret_dir

↓

Python3.5

import os

os.mkdir("secret_dir", mode=0o700)

Note that if you try to create a directory with this function but it already exists, the exception FileExitsError will be thrown. This argument can also be used with the pathlib.Path.mkdir method.

If you want to create directories recursively, use ʻos.makedirs. If you don't mind the directory already exists, use the ʻexist_ok option. You can also use the mode option as well as ʻos.mkdir`.

Bash

mkdir -p /usr/local/my_software

↓

Python3.5

import os

os.makedirs("/usr/local/my_software", exist_ok=True)

If you do not specify ʻexist_ok = True and the directory already exists, the exception ʻOSError will be thrown. For pathlib, use the parents option of the mkdir method.

Python3.5+pathlib

from pathlib import Path

Path("/usr/local/my_software").mkdir(parents=True, exist_ok=True)

Search inside the directory

Bash

find . -type d -exec chmod 700 {} \;
find . -type f -exec chmod 600 {} \;

↓

Python

import os

for root, dirs, files in os.walk("."):
    for dir in dirs:
        os.chmod(0o700, os.path.join(root,dir))
    for file in files:
        os.chmod(0o600, os.path.join(root, file))

If you pass an instance of the Path class as an argument, you can pass it as it is if it is Python 3.6 or later. In the case of 3.5, use str to convert it to a character string once.

Permissions / ownership / UID / GID

Change permissions

Use ʻos.chmod`.

Bash

chmod 700 directory

↓

Python3.5

import os

os.chmod("directory", 0o700)

The Path class also has a method with the same name.

Python3.5+pathlib

from pathlib import Path

Path("directory").chmod(0o700)

Also, when specifying in the original permission + α format, such as when assigning execution bits, use it together with ʻos.stat` ・ bit operation.

Bash

chmod +x script.sh

↓

Python3.5

import os

os.chmod("script.sh", os.stat("script.sh").st_mode | 0o111)

For the Path class, use the stat method instead of ʻos.stat`.

Python3.5+pathlib

from pathlib import Path

script_path = Path("script.sh")
script_path.chmod(script_path.stat().st_mode | 0o111)

Obtaining UID / GID

Use ʻos.getuid () and ʻos.getgid () to get the UID and GID of the current user, respectively.

Python3.5

import os, sys

if os.getuid() != 0:
    print("Run this script as root", file=sys.stderr)
    exit(1)

Also, use pwd.getpwnam and grp.getgrnam to get the UID and group GID of the user with the specified name, respectively. The pw_uid and gr_gid attributes of each return value are the UID and GID, respectively. If there is no user group with the specified name, the exception KeyError will be returned, so catch it as appropriate.

Python3.5

import grp
import pwd

assert pwd.getpwnam("root").pw_uid == 0
assert grp.getgrnam("root").gr_gid == 0

Change of ownership

Use the shutil.chown command.

Bash

chown root:www /var/www/html

↓

Python3.5

import shutil

shutil.chown("/var/www/html", "root", "www")

Note: These functions are not implemented on Windows. Also, the pwd and grp modules are not available from Windows.

Copy / move / delete files / directories

copy

Use shutil.copy to copy the file normally. Use it in the same way as the cp command.

import shutil

shutil.copy("old.txt", "new.txt")
shutil.copy("old.txt", "directory")

If you also want to copy the metadata, use shutil.copy2. Usage is the same as copy. If you want to copy between file objects, use shutil.copyfileobj.

import shutil

with open("old.txt", "rb") as old:
    with open("new.txt", "wb") as new:
        shutil.copyfileobj(old, new)

If you want to copy the directory, use shutil.copytree.

Move

Use shutil.move. There is also ʻos.rename`, but it is for files only.

Delete

Use ʻos.remove` to remove the file.

Bash

rm file

↓

Python3.5

import os
from pathlib import Path

os.remove("file") # or Path("file").unlink()

If you want to remove an empty directory, use the ʻos.rmdir or the method ʻunlink of the Path class. Use shutil.rmtree to recursively delete the entire non-empty directory.

Bash

rm -rf /

↓

Python3.5

import shutil

shutil.rmtree("/")

Creating a symbolic link

It can be created with the symlink_to method of the ʻos.symlink or Path` class.

Bash

ln -s foo bar

↓

Python3.5

import os

os.symlink("foo", "bar") # or Path("bar").symlink_to("foo")

File download

You may want to use the requests package. If you want to get the contents of the file and parse it directly in Python later, write as follows.

import requests

req = requests.get("http://server/page.html")
html = req.text

If you want to download a large file, you can write:

import requests
import posixpath
import os
import shutil

req = requests.get("http://server/largefile.deb", stream=True)

with open(os.path.join("~/download",posixpath.basename(req.url)), "wb") as f:
    shutil.copyfileobj(req.raw, f)

If you cannot use all the packages, use the ʻurllib.request` module (https://docs.python.jp/3/library/urllib.request.html) instead.

Alternative to the test command

Path existence check

Bash

[[ -e "file.txt" ]] && echo "Exist" || echo "Not exist"

↓

Python3.5

import os

print("Exist" if os.path.exists("file.txt") else "Not exist")

Python3.5+pathlib

from pathlib import Path

print("Exist" if Path("file.txt").exists() else "Not exist")

File / directory check

Bash

[[ -f "file.txt" ]] && echo "File" || echo "Not file"
[[ -d "directory" ]] && echo "Directory" || echo "Not directory"

↓

Python3.5

import os

print("File" if os.path.isfile("file.txt") else "Not file")
print("Directory" if os.path.isdir("directory") else "Not directory")

Python3.5+pathlib

from pathlib import Path

print("File" if Path("file.txt").is_file() else "Not file")
print("Directory" if Path("directory").is_dir() else "Not directory")

Permission check

Use ʻos.access` to check permissions for the current user.

Bash

[[ -r 'file.txt' ]] && echo 'True' || echo 'False'
[[ -r 'file.txt' -a -w 'file.txt' ]] && echo 'True' || echo 'False'
[[ -r 'directory' -a -x 'directory' ]] && echo 'True' || echo 'False'

↓

Python3.5

import os

print(os.access("file.txt", os.R_OK))
print(os.access("file2.txt", os.R_OK | os.W_OK))
print(os.access("directory", os.R_OK | os.X_OK))

Check update date

The modification date timestamp can be obtained with the st_mtime attribute of the named tuple, which can be obtained with the ʻos.stat or the method statof thePath` class.

File size check

The file size can also be obtained with the st_size attribute of the return value of the method stat of the ʻos.stator thePath` class. Note that in the case of a directory, it is not the total size of the files contained in it.

Archive compression / decompression

Use shutil

Use shutil.unpack_archive to decompress. This is the easiest for various tarballs and zips.

Bash

tar xf foo.tar.xz

↓

Python

import shutil

shutil.unpack_archive("foo.tar.xz")

You can specify the decompression destination with the second argument of this function.

Bash

tar xf foo.tar.gz -C bar

↓

Python

import shutil

shutil.unpack_archive("foo.tar.gz", "bar")

Use shutil.make_archive if you want to compress an entire directory. For example, if you want to compress a directory named bar to create an archive named foo.tar.xz, write:

Bash

tar cf foo.tar.xz bar

↓

Python

import shutil

shutil.make_archive("foo", "xztar", base_dir="bar")

gz ・ bz2 ・ xz

These files can be read and written like ordinary files by using the gzip, bz2, and lzma modules, respectively.

import gzip
import bz2
import lzma
import shutil

with gzip.open("file.gz", "rb") as gz:
    with bz2.open("file.bz2", "wb") as bzip2:
        shutil.copyfileobj(gz, bzip2)
    with lzma.open("file.xz", "wb") as xz:
        shutil.copyfileobj(gz, xz)

tar Use the tarfile module. Use ʻextractall ()` for decompression. For example, use it as follows.

import tarfile

with tarfile.open("tarfile.tar.xz") as f:
    f.extractall()

If you specify a directory as the first argument, it will be expanded there. Get the file list of the tar file with members (). (The list will be returned)

Use ʻadd () when creating a tar file, but be careful about the file mode specified by ʻopen (). Only w is considered uncompressed. For example, if you want to create an archive in tar.xz format, write as follows.

import tarfile

with tarfile.open("archive.tar.xz", "w:xz") as f:
    f.add("directory")

Add : (compressed format name) after w like this.

zip Use zipfile.ZipFile. (http://docs.python.jp/3.6/library/zipfile.html#zipfile-objects) For example, if you want to extract all the contents of the ZIP file in the current directory, write as follows.

import zipfile

with zipfile.ZipFile("foo.zip") as ar:
    ar.extractall()

Since there is a file name encoding problem in the zip file, it may be good to call the OS zip / ʻunzip command with subprocess.run`. (Windows / Ubuntu Japanese Remix)

Temporary files directory

Temporary files and directories use the tempfile module.

temporary file

If you do not want to use the file externally, use tempfile.TemporaryFile.

import tempfile

with tempfile.TemporaryFile() as temp_file_handler:
  temp_file_handler.write(some_byte_array)
  # ...
  some_bytes = temp_file_handler.read(length)
  # ...

For external use, use tempfile.NamedTemporaryFile. It has a name attribute, which allows you to get the filename.

import tempfile
import subprocess

with tempfile.NamedTemporaryFile() as temp_file:
  temp_file.write(some_byte_array)
  # ...
  subprocess.run(("external-process", temp_file.name), check=True)
  # ...

If you want to specify the extension / prefix, use suffix / prefix. For example, the following code

import tempfile

with tempfile.NamedTemporaryFile(prefix="tempfile-", suffix=".deb") as temp_file:
  print(temp_file.name)

Prints a string like /tmp/tempfile-hz1u73cr.deb and exits.

In both cases, the file will be deleted when the with statement is exited. Also, the default open mode for files is w + b, or binary mode.

Temporary directory

Temporary directories are created with tempfile.TemporaryDirectory. Normally, you would use the with statement as follows:

import tempfile
import os
import subprocess

with tempfile.TemporaryDirectory() as temp_dir_name:
  temp_conf_name = os.path.join(temp_dir_name, "hoge.conf")
  with open(temp_conf_name, "w") as temp_conf:
    print("hoge=hogehoge", file=temp_conf)
  subprocess.run(("sudo", "hoge", temp_conf_name), check=True)

In this case, if you exit the with statement, the temporary directory and the files in it will be deleted. Also, in this example, the type of temp_dir_name is str.

Environment variable

You can get and set it in a dictionary (like) named ʻos.environ`.

Python3.5

import os

print("256 colors" if "TERM" in os.environ and "256color" in os.environ["TERM"] else "16 colors")

Python3.5

import os
import pip

if "CUDA_ROOT" not in os.environ:
    os.environ["CUDA_ROOT"] = "/usr/local/cuda"
pip.main(["install", "-U", "chainer"])

If you want to embed environment variables in a string, use ʻos.path.expandvars`.

Python3.5

import os

WORKING_DIR = os.path.join(COMMON_ROOT, os.path.expandvars("work-$USER"))

String manipulation of the path by the method of the Path class

Path concatenation

Python3.5+pathlib

from pathlib import Path

#All Path("/bin/bash")return it
Path("/bin","bash")
Path("/bin") / "bash"
Path("/bin") / Path("bash")
Path("/bin").joinpath("bash")
Path("/bin").joinpath(Path("bash"))

Path split

Python3.5+pathlib

from pathlib import Path

p = Path("dir/dir2/file.tar.gz")
print(p.name) # file.tar.gz
print(p.parent) # dir/dir2
print([str(dir) for dir in p.parents]) # ['dir/dir2', 'dir', '.']

print(p.stem) # file.tar
print(p.suffix) # .gz
print(p.suffixes) # ['.tar', '.gz']

Change the name and extension while keeping the parent directory

Python3.5+pathlib

from pathlib import Path

ico = Path("img/favicon.ico")

gitignore = ico.with_name(".gitignore") # img/.gitignore
png = ico.with_suffix(".png ") # img/favicon.png

String manipulation

Path manipulation (deprecated; use of Path class recommended)

Various delimiters can be obtained with ʻos.sep (directory), ʻos.extsep (extension), and ʻos.pathsep` (environment variable PATH).

Python3.5

import os

print(os.sep, os.extsep, os.pathsep)
# Windows: \ . ;
#other than that: / . :

Use ʻos.path.join` to join paths.

Python3.5

import os

os.makedirs(os.path.join(PROJECT_ROOT, "build"), exist_ok=True)

Use ʻos.path.split, ʻos.path.basename, ʻos.path.dirname` to split the path into basename and dirname.

Python3.5

import os

dirname, basename = os.path.split(FILE_PATH)
assert dirname == os.path.dirname(FILE_PATH)
assert basename == os.path.basename(FILE_PATH)

Use ʻos.path.splitext` to extract the extension / non-extension part of the path.

Python3.5

import os

root, ext = os.path.splitext("/etc/nginx/nginx.conf")
print(root, ext)
# /etc/nginx/nginx .conf

Use ʻos.path.expanduserto handle paths that use~` to represent your home directory.

Bash

echo "Your home directory is: $HOME"

↓

Python3.5

import os

print("Your home directory is:", os.path.expanduser("~"))

If you are conscious of compatibility with Windows, only use it to get such a home directory, and use ʻos.path.join` together.

Also, use ʻos.path.expandvars` to work with paths that include environment variables.

Python3.5

import os

os.makedirs(os.path.join(PROJECT_ROOT, "build", os.path.expandvars("python-$PYTHON_VERSION")), exist_ok=True)

There is also a way to use pathlib, but I will not explain it at this time. https://docs.python.jp/3/library/pathlib.html

Other

Other than that, I plan to write something equivalent to sed, ʻawk, and grep`.

reference

http://qiita.com/supersaiakujin/items/12451cd2b8315fe7d054