I was soberly addicted to calling awscli from a Python 2.7 script registered in crontab

Summary

--When calling from python, enter the command ʻaws cli with an absolute path. --/ usr / local / bin / aws s3 cp s3: //hoge/fuga/foo.txt bar.txt`.

I didn't use boto because I felt that I got an error because I only had access to the bottom of the bucket.

Thing you want to do

$ aws s3 cp s3://hoge/fuga/foo_20160711.txt .

I want to bring the file from S3. The file name changes every day depending on the date. If the download is successful, process the import, or something. I'm going to call it with python for the time being.

I tried it

fetch.py

import datetime
import os
import stat
import subprocess
import tempfile


def main():
	fd, tmp = tempfile.mkstemp()
    os.close(fd)
    os.chmod(tmp, stat.S_IRWXU | stat.S_IROTH)
    filename = 'foo_{}.txt'.format(datetie.datetime.now().strftime('%Y%m%d'))
    subprocess.check_call([
	    'aws',
     	's3',
	    'cp',
	    's3://hoge/fuga/{}'.format(filename),
		tmp,
    	'--quiet',
    ])
	do_something(tmp)

if __name__ == '__main__':
	main()

Make a code like this. I ran it from the command line and confirmed that it worked, so I threw it into crontab.

0 1 * * * /usr/local/bin/python /home/vagrant/fetch.py

It was no good.

Reasons not good

When I look at the execution history, I usually get an error. ʻIt is said that the aws command does not exist. By the way, I heard somewhere that the config file of ʻawscli is in the home directory, so the config cannot be read unless the home directory is specified exactly in the environment variable. However, since it is said that there is no ʻawscommand in the first place, we have not reached the problem of environment variables. Maybe there is noPATH. I think it can be solved by adding PATH` or specifying the location of the command directly.

Fix

--Add home directory to crontab --Pass environment variables to subprocess.check_call () --Search for the aws command with which and refer to it directly

HOME=/home/vagrant
0 1 * * * /usr/local/bin/python /home/vagrant/fetch.py

fetch.py

import datetime
import os
import stat
import subprocess
import tempfile


def do_something(name):
	return


def main():
	fd, tmp = tempfile.mkstemp()
    os.close(fd)
    os.chmod(tmp, stat.S_IRWXU | stat.S_IROTH)
    filename = 'foo_{}.txt'.format(datetie.datetime.now().strftime('%Y%m%d'))
    subprocess.check_call([
	    '/usr/local/bin/aws',
     	's3',
	    'cp',
	    's3://hoge/fuga/{}'.format(filename),
		tmp,
    	'--quiet',
    ], env=os.environ.copy())
	do_something(tmp)

if __name__ == '__main__':
	main()

It almost worked.