Write a batch script with Python3.5 ~

This article summarizes the know-how for replacing Bash scripts with Python scripts, but we plan to make corrections in the future.

Display of character string

Use the print function. This is a variable length argument, and if you give more than one, it will be output separated by a space by default.

Bash


echo Hello, world!

Python3.5


print("Hello,", "world!")

If you want to output other than the standard output, specify it in the form of file = ○○. For example, in the case of a standard error:

Bash


echo Error >&2

Python3.5


import sys

print("Error", file=sys.stderr)

When overwriting / adding to a file, it will be as follows.

Bash


echo foo > file1.txt
echo bar >> file1.txt
echo hoge >> file2.txt
echo fuga >> file2.txt

Python3.5


with open("file1.txt", "w") as f:
    print("foo", file=f)
    print("bar", file=f)
with open("file2.txt", "a") as f:
    print("hoge", file=f)
    print("fuga", file=f)

The space between the displayed arguments and the line feed character at the end of the line can be changed to another character. For example, to set the delimiter to tab and no character at the end of the line:

Python3.5


print("foo", "bar", sep="\t", end="")

Optional analysis

In Bash etc., it is a difficult option analysis, but in Python it can be easily analyzed with the ʻargparse` module. Basically, use it as follows.

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--verbose", "-v", help="Verbose", action="store_true")
parser.add_argument("--quiet", "-q", help="Suppress messages", action="store_true")
parser.add_argument("target", help="target")
parser.add_argument("source", nargs="+", help="source")
args = parser.parse_args()

function(args.option)

Each option can be accessed with ʻargs.verbose, ʻargs.quiet, etc.

See the official commentary for details. https://docs.python.jp/3/library/argparse.html

Command execution

You can use subprocess.run from Python 3.5. Use this if you don't think about compatibility with past versions.

Just run

Bash


g++ sample.cpp -o sample -std=gnu++14 -march=native -O3

Python3.5


import subprocess

subprocess.run(["g++", "sample.cpp", "-o", "sample", "-std=gnu++14", "-march=native", "-O3"])

In addition to the list, iterable is OK. (Tuples, etc.)

Exception if the command ends with a non-zero

Specify check = True. The exception is subprocess.CalledProcessError.

Python3.5


import subprocess

subprocess.run(["git", "push"], check=True)

Capture standard output / error

Specify stdout = subprocess.PIPE stderr = subprocess.PIPE. It can be retrieved with the stdout and stderr attributes of the return value, respectively. Please note that it is a byte type.

Python3


import subprocess

result = subprocess.run(["apt", "list", "--upgradable"], stderr=subprocess.PIPE, stdout=subprocess.PIPE, check=True)
process(result.stdout)

Discard standard output / error

Specify subprocess.DEVNULL instead of subprocess.PIPE.

Specify standard input

Add ʻinput =to the argument ofsubprocess.run`. Specify by byte type.

Python3.5


import subprocess

assert subprocess.run(["cat"], input=b"aaa\n", stdout=subprocess.PIPE).stdout == b"aaa\n"

redirect

Specify the file object opened by the ʻopen function etc. in each argument of subprocess.runargumentstdin (not ʻinput), stdout, stderr. As long as the input and output are text data, the second argument of ʻopen can be without b`, but it is better to have it.

Directory related

PS: It's easier to use pathlib.

from pathlib import Path

python_path = Path("/usr/bin/python3")

The following introduces a legacy method that does not use pathlib.

Get current directory

You can get it with ʻos.getcwd () . Or, when using pathlib, you can get it with Path () (relative path), Path.cwd () or Path (). Resolve () `(absolute path).

Directory change

Use ʻos.chdir`.

Bash


cd directory

Python3.5


import os

os.chdir("directory")

View the contents of the directory

Use ʻos.listdir`. No argument is required to display the contents of the current directory.

Bash


foo=$(ls)

Python3.5


import os

foo = os.listdir()

If you want to see the contents of a particular directory, give that path.

Bash


foo=$(ls bar)

Python3.5


import os

foo = os.listdir("bar")

To use pathlib.Path, use the method ʻiterdir (). Since it is a generator that is returned, either use a for loop or list it with list ()`.

Python3.5+pathlib


from pathlib import Path

foo = Path().iterdir() # foo=$(ls)
bar = Path("baz").iterdir() # bar=$(ls baz)

# ls qux
for path in Path("qux").iterdir():
  print(path)

If you want to use pattern matching, use glob.glob or glob.iglob instead. The former returns a list and the latter returns an iterator, so the latter should be used to pass to a for statement, map, filter, etc.

Bash


audio_files=$(ls *.wav)

ls *.tmp | xargs rm -f

Python3.5


import glob
import os

audio_files = glob.glob("*.wav")

for tmp_file in glob.iglob("*.tmp"):
    os.remove(tmp_file)

If you use pathlib.Path, use the method glob.

Python3.5+pathlib


from pathlib import Path

for tmp_file in Path().glob("*.tmp"):
    tmp_file.unlink()

Creating a directory

If you want to create a single directory, use ʻos.mkdir`.

Bash


mkdir test_dir

Python3.5


import os

os.mkdir("test_dir")

For pathlib.Path, use the method mkdir.

Python3.5+pathlib


from pathlib import Path

Path("test_dir").mkdir()

If you want to limit permissions beyond the default values (777 & umask values), use mode.

Bash


mkdir -m 700 secret_dir

Python3.5


import os

os.mkdir("secret_dir", mode=0o700)

Note that if you try to create a directory with this function but it already exists, the exception FileExitsError will be thrown. This argument can also be used with the pathlib.Path.mkdir method.

If you want to create directories recursively, use ʻos.makedirs. If you don't mind the directory already exists, use the ʻexist_ok option. You can also use the mode option as well as ʻos.mkdir`.

Bash


mkdir -p /usr/local/my_software

Python3.5


import os

os.makedirs("/usr/local/my_software", exist_ok=True)

If you do not specify ʻexist_ok = True and the directory already exists, the exception ʻOSError will be thrown. For pathlib, use the parents option of the mkdir method.

Python3.5+pathlib


from pathlib import Path

Path("/usr/local/my_software").mkdir(parents=True, exist_ok=True)

Search inside the directory

Bash


find . -type d -exec chmod 700 {} \;
find . -type f -exec chmod 600 {} \;

Python


import os

for root, dirs, files in os.walk("."):
    for dir in dirs:
        os.chmod(0o700, os.path.join(root,dir))
    for file in files:
        os.chmod(0o600, os.path.join(root, file))

If you pass an instance of the Path class as an argument, you can pass it as it is if it is Python 3.6 or later. In the case of 3.5, use str to convert it to a character string once.

Permissions / ownership / UID / GID

Change permissions

Use ʻos.chmod`.

Bash


chmod 700 directory

Python3.5


import os

os.chmod("directory", 0o700)

The Path class also has a method with the same name.

Python3.5+pathlib


from pathlib import Path

Path("directory").chmod(0o700)

Also, when specifying in the original permission + α format, such as when assigning execution bits, use it together with ʻos.stat` ・ bit operation.

Bash


chmod +x script.sh

Python3.5


import os

os.chmod("script.sh", os.stat("script.sh").st_mode | 0o111)

For the Path class, use the stat method instead of ʻos.stat`.

Python3.5+pathlib


from pathlib import Path

script_path = Path("script.sh")
script_path.chmod(script_path.stat().st_mode | 0o111)

Obtaining UID / GID

Use ʻos.getuid () and ʻos.getgid () to get the UID and GID of the current user, respectively.

Python3.5


import os, sys

if os.getuid() != 0:
    print("Run this script as root", file=sys.stderr)
    exit(1)

Also, use pwd.getpwnam and grp.getgrnam to get the UID and group GID of the user with the specified name, respectively. The pw_uid and gr_gid attributes of each return value are the UID and GID, respectively. If there is no user group with the specified name, the exception KeyError will be returned, so catch it as appropriate.

Python3.5


import grp
import pwd

assert pwd.getpwnam("root").pw_uid == 0
assert grp.getgrnam("root").gr_gid == 0

Change of ownership

Use the shutil.chown command.

Bash


chown root:www /var/www/html

Python3.5


import shutil

shutil.chown("/var/www/html", "root", "www")

Note: These functions are not implemented on Windows. Also, the pwd and grp modules are not available from Windows.

Copy / move / delete files / directories

copy

Use shutil.copy to copy the file normally. Use it in the same way as the cp command.

import shutil

shutil.copy("old.txt", "new.txt")
shutil.copy("old.txt", "directory")

If you also want to copy the metadata, use shutil.copy2. Usage is the same as copy. If you want to copy between file objects, use shutil.copyfileobj.

import shutil

with open("old.txt", "rb") as old:
    with open("new.txt", "wb") as new:
        shutil.copyfileobj(old, new)

If you want to copy the directory, use shutil.copytree.

Move

Use shutil.move. There is also ʻos.rename`, but it is for files only.

Delete

Use ʻos.remove` to remove the file.

Bash


rm file

Python3.5


import os
from pathlib import Path

os.remove("file") # or Path("file").unlink()

If you want to remove an empty directory, use the ʻos.rmdir or the method ʻunlink of the Path class. Use shutil.rmtree to recursively delete the entire non-empty directory.

Bash


rm -rf /

Python3.5


import shutil

shutil.rmtree("/")

Creating a symbolic link

It can be created with the symlink_to method of the ʻos.symlink or Path` class.

Bash


ln -s foo bar

Python3.5


import os

os.symlink("foo", "bar") # or Path("bar").symlink_to("foo")

File download

You may want to use the requests package. If you want to get the contents of the file and parse it directly in Python later, write as follows.

import requests

req = requests.get("http://server/page.html")
html = req.text

If you want to download a large file, you can write:

import requests
import posixpath
import os
import shutil

req = requests.get("http://server/largefile.deb", stream=True)

with open(os.path.join("~/download",posixpath.basename(req.url)), "wb") as f:
    shutil.copyfileobj(req.raw, f)

If you cannot use all the packages, use the ʻurllib.request` module (https://docs.python.jp/3/library/urllib.request.html) instead.

Alternative to the test command

Path existence check

Bash


[[ -e "file.txt" ]] && echo "Exist" || echo "Not exist"

Python3.5


import os

print("Exist" if os.path.exists("file.txt") else "Not exist")

Python3.5+pathlib


from pathlib import Path

print("Exist" if Path("file.txt").exists() else "Not exist")

File / directory check

Bash


[[ -f "file.txt" ]] && echo "File" || echo "Not file"
[[ -d "directory" ]] && echo "Directory" || echo "Not directory"

Python3.5


import os

print("File" if os.path.isfile("file.txt") else "Not file")
print("Directory" if os.path.isdir("directory") else "Not directory")

Python3.5+pathlib


from pathlib import Path

print("File" if Path("file.txt").is_file() else "Not file")
print("Directory" if Path("directory").is_dir() else "Not directory")

Permission check

Use ʻos.access` to check permissions for the current user.

Bash


[[ -r 'file.txt' ]] && echo 'True' || echo 'False'
[[ -r 'file.txt' -a -w 'file.txt' ]] && echo 'True' || echo 'False'
[[ -r 'directory' -a -x 'directory' ]] && echo 'True' || echo 'False'

Python3.5


import os

print(os.access("file.txt", os.R_OK))
print(os.access("file2.txt", os.R_OK | os.W_OK))
print(os.access("directory", os.R_OK | os.X_OK))

Check update date

The modification date timestamp can be obtained with the st_mtime attribute of the named tuple, which can be obtained with the ʻos.stat or the method statof thePath` class.

File size check

The file size can also be obtained with the st_size attribute of the return value of the method stat of the ʻos.stator thePath` class. Note that in the case of a directory, it is not the total size of the files contained in it.

Archive compression / decompression

Use shutil

Use shutil.unpack_archive to decompress. This is the easiest for various tarballs and zips.

Bash


tar xf foo.tar.xz

Python


import shutil

shutil.unpack_archive("foo.tar.xz")

You can specify the decompression destination with the second argument of this function.

Bash


tar xf foo.tar.gz -C bar

Python


import shutil

shutil.unpack_archive("foo.tar.gz", "bar")

Use shutil.make_archive if you want to compress an entire directory. For example, if you want to compress a directory named bar to create an archive named foo.tar.xz, write:

Bash


tar cf foo.tar.xz bar

Python


import shutil

shutil.make_archive("foo", "xztar", base_dir="bar")

gz ・ bz2 ・ xz

These files can be read and written like ordinary files by using the gzip, bz2, and lzma modules, respectively.

import gzip
import bz2
import lzma
import shutil

with gzip.open("file.gz", "rb") as gz:
    with bz2.open("file.bz2", "wb") as bzip2:
        shutil.copyfileobj(gz, bzip2)
    with lzma.open("file.xz", "wb") as xz:
        shutil.copyfileobj(gz, xz)

tar Use the tarfile module. Use ʻextractall ()` for decompression. For example, use it as follows.

import tarfile

with tarfile.open("tarfile.tar.xz") as f:
    f.extractall()

If you specify a directory as the first argument, it will be expanded there. Get the file list of the tar file with members (). (The list will be returned)

Use ʻadd () when creating a tar file, but be careful about the file mode specified by ʻopen (). Only w is considered uncompressed. For example, if you want to create an archive in tar.xz format, write as follows.

import tarfile

with tarfile.open("archive.tar.xz", "w:xz") as f:
    f.add("directory")

Add : (compressed format name) after w like this.

zip Use zipfile.ZipFile. (http://docs.python.jp/3.6/library/zipfile.html#zipfile-objects) For example, if you want to extract all the contents of the ZIP file in the current directory, write as follows.

import zipfile

with zipfile.ZipFile("foo.zip") as ar:
    ar.extractall()

Since there is a file name encoding problem in the zip file, it may be good to call the OS zip / ʻunzip command with subprocess.run`. (Windows / Ubuntu Japanese Remix)

Temporary files directory

Temporary files and directories use the tempfile module.

temporary file

If you do not want to use the file externally, use tempfile.TemporaryFile.

import tempfile

with tempfile.TemporaryFile() as temp_file_handler:
  temp_file_handler.write(some_byte_array)
  # ...
  some_bytes = temp_file_handler.read(length)
  # ...

For external use, use tempfile.NamedTemporaryFile. It has a name attribute, which allows you to get the filename.

import tempfile
import subprocess

with tempfile.NamedTemporaryFile() as temp_file:
  temp_file.write(some_byte_array)
  # ...
  subprocess.run(("external-process", temp_file.name), check=True)
  # ...

If you want to specify the extension / prefix, use suffix / prefix. For example, the following code

import tempfile

with tempfile.NamedTemporaryFile(prefix="tempfile-", suffix=".deb") as temp_file:
  print(temp_file.name)

Prints a string like /tmp/tempfile-hz1u73cr.deb and exits.

In both cases, the file will be deleted when the with statement is exited. Also, the default open mode for files is w + b, or binary mode.

Temporary directory

Temporary directories are created with tempfile.TemporaryDirectory. Normally, you would use the with statement as follows:

import tempfile
import os
import subprocess

with tempfile.TemporaryDirectory() as temp_dir_name:
  temp_conf_name = os.path.join(temp_dir_name, "hoge.conf")
  with open(temp_conf_name, "w") as temp_conf:
    print("hoge=hogehoge", file=temp_conf)
  subprocess.run(("sudo", "hoge", temp_conf_name), check=True)

In this case, if you exit the with statement, the temporary directory and the files in it will be deleted. Also, in this example, the type of temp_dir_name is str.

Environment variable

You can get and set it in a dictionary (like) named ʻos.environ`.

Python3.5


import os

print("256 colors" if "TERM" in os.environ and "256color" in os.environ["TERM"] else "16 colors")

Python3.5


import os
import pip

if "CUDA_ROOT" not in os.environ:
    os.environ["CUDA_ROOT"] = "/usr/local/cuda"
pip.main(["install", "-U", "chainer"])

If you want to embed environment variables in a string, use ʻos.path.expandvars`.

Python3.5


import os

WORKING_DIR = os.path.join(COMMON_ROOT, os.path.expandvars("work-$USER"))

String manipulation of the path by the method of the Path class

Path concatenation

Python3.5+pathlib


from pathlib import Path

#All Path("/bin/bash")return it
Path("/bin","bash")
Path("/bin") / "bash"
Path("/bin") / Path("bash")
Path("/bin").joinpath("bash")
Path("/bin").joinpath(Path("bash"))

Path split

Python3.5+pathlib


from pathlib import Path

p = Path("dir/dir2/file.tar.gz")
print(p.name) # file.tar.gz
print(p.parent) # dir/dir2
print([str(dir) for dir in p.parents]) # ['dir/dir2', 'dir', '.']

print(p.stem) # file.tar
print(p.suffix) # .gz
print(p.suffixes) # ['.tar', '.gz']

Change the name and extension while keeping the parent directory

Python3.5+pathlib


from pathlib import Path

ico = Path("img/favicon.ico")

gitignore = ico.with_name(".gitignore") # img/.gitignore
png = ico.with_suffix(".png ") # img/favicon.png

String manipulation

Path manipulation (deprecated; use of Path class recommended)

Various delimiters can be obtained with ʻos.sep (directory), ʻos.extsep (extension), and ʻos.pathsep` (environment variable PATH).

Python3.5


import os

print(os.sep, os.extsep, os.pathsep)
# Windows: \ . ;
#other than that: / . :

Use ʻos.path.join` to join paths.

Python3.5


import os

os.makedirs(os.path.join(PROJECT_ROOT, "build"), exist_ok=True)

Use ʻos.path.split, ʻos.path.basename, ʻos.path.dirname` to split the path into basename and dirname.

Python3.5


import os

dirname, basename = os.path.split(FILE_PATH)
assert dirname == os.path.dirname(FILE_PATH)
assert basename == os.path.basename(FILE_PATH)

Use ʻos.path.splitext` to extract the extension / non-extension part of the path.

Python3.5


import os

root, ext = os.path.splitext("/etc/nginx/nginx.conf")
print(root, ext)
# /etc/nginx/nginx .conf

Use ʻos.path.expanduserto handle paths that use~` to represent your home directory.

Bash


echo "Your home directory is: $HOME"

Python3.5


import os

print("Your home directory is:", os.path.expanduser("~"))

If you are conscious of compatibility with Windows, only use it to get such a home directory, and use ʻos.path.join` together.

Also, use ʻos.path.expandvars` to work with paths that include environment variables.

Python3.5


import os

os.makedirs(os.path.join(PROJECT_ROOT, "build", os.path.expandvars("python-$PYTHON_VERSION")), exist_ok=True)

There is also a way to use pathlib, but I will not explain it at this time. https://docs.python.jp/3/library/pathlib.html

Other

Other than that, I plan to write something equivalent to sed, ʻawk, and grep`.

reference

http://qiita.com/supersaiakujin/items/12451cd2b8315fe7d054

Recommended Posts

Write a batch script with Python3.5 ~
I want to write to a file with Python
POST json with Python3 script
Make a fortune with Python
Write to csv with Python
Create a directory with python
[Python] A memo to write CSV vertically with Pandas
A program to write Lattice Hinge with Rhinoceros with Python
[Piyopiyokai # 1] Let's play with Lambda: Creating a Python script
[Python] What is a with statement?
Write a binary search in Python
I replaced the Windows PowerShell cookbook with a python script.
How to write a Python class
Solve ABC163 A ~ C with Python
A python graphing manual with Matplotlib.
[Python] Write to csv file with Python
Let's make a GUI with python.
Let's create a script that registers with Ideone.com in Python.
Create a virtual environment with Python!
Write A * (A-star) algorithm in Python
I made a fortune with Python.
Write a Residual Network with TFLearn
Building a virtual environment with Python 3
Solve ABC168 A ~ C with Python
Make a recommender system with python
Write a pie chart in Python
[Python] Generate a password with Slackbot
Solve ABC162 A ~ C with Python
Solve ABC167 A ~ C with Python
Solve ABC158 A ~ C with Python
Let's make a graph with python! !!
[Linux] Write a deployment tool using rsync with a shell script
Launch a Python script as a service
[Python] Inherit a class with class variables
I made a daemon with Python
How to batch start a python program created with Jupyter notebook
How to pass arguments to a Python script in SPSS Modeler Batch
Batch download images from a specific URL with python Modified version
Write a script to calculate the distance with Elasticsearch 5 system painless
[Pyenv] Building a python environment with ubuntu 16.04
Spiral book in Python! Python with a spiral book! (Chapter 14 ~)
Create a Python function decorator with Class
Creating a simple PowerPoint file with Python
Building a Python3 environment with Amazon Linux2
Install Python as a Framework with pyenv
Build a blockchain with Python ① Create a class
Add a Python data source with Redash
Execute Python script with cron of TS-220
Create a dummy image with Python + PIL.
Write the test in a python docstring
I made a character counter with Python
[Python] Drawing a swirl pattern with turtle
I drew a heatmap with seaborn [Python]
[Python] Create a virtual environment with Anaconda
Let's create a free group with Python
A memo with Python2.7 and Python3 on CentOS
Building a Python 3.6 environment with Windows + PowerShell
Map rent information on a map with python
Search the maze with the python A * algorithm
[Python] Create a Batch environment using AWS-CDK
Write a short property definition in Python