Periodically execute python script with cron in venv environment

To register a python job in cron and execute it periodically in a venv environment

Conclusion

You can use python under the venv environment you want to use.

$ crontab -e
* * * * * cd [Absolute path]; venv/bin/python foo.py

Concrete example

I want to write the date and time to a file in jsonline format once a minute.

OS environment

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.14.6
BuildVersion:	18G87

Create a directory, create a venv virtual environment, and install the package jsonlines used this time.

$ mkdir [place]
$ cd [place]
$ python -V
Python 3.7.1
$ python -m venv venv
$ source venv/bin/activate
(venv) $ pip install jsonlines

Write a python script. The date and time of processing is written as jsonline to out.jsonl. Also print for confirmation.

dt2jsonl.py

import datetime
import jsonlines

dt = datetime.datetime.now()
dict_now = {'date': str(dt.date()), 'time': str(dt.time())}

with jsonlines.open('out.jsonl', mode='a') as writer:
    writer.write(dict_now)

print('updated: '+ str(dt))

When processed, the output looks like this.

(venv) $ python dt2jsonl.py
updated: 2020-01-26 17:39:23.435616
(venv) $ cat out.jsonl
{"date": "2020-01-26", "time": "17:39:23.435616"}

Try registering with cron as python dt2jsonl.py. Since it is * * * * *, it should be processed once a minute. Write stderr and stdout to /tmp/cron.log with >> /tmp/cron.log 2> & 1 to check the error and print result.

(venv) $ crontab -e
* * * * * python dt2jsonl.py >> /tmp/cron.log 2>&1

When waiting, there is an error in /tmp/cron.log

(venv) $ tail -f /tmp/cron.log
python: can't open file 'dt2jsonl.py': [Errno 2] No such file or directory

It seems that dt2jsonl.py cannot be found, so check where cron is.

(venv) $ crontab -e
* * * * * pwd >> /tmp/cron.log 2>&1

(venv) $ tail -f /tmp/cron.log
/Users/[User]

I'm in the user directory, so I'll tell you the absolute path of the script.

(venv) $ crontab -e
* * * * * python /Users/[User]/[place]/dt2jsonl.py >> /tmp/cron.log 2>&1

(venv) $ tail -f /tmp/cron.log
Traceback (most recent call last):
  File "/Users/[User]/[place]/dt2jsonl.py", line 2, in <module>
    import jsonlines
ImportError: No module named jsonlines

This time I got to a python script, but without jsonlines. Since it is not running in the venv environment, check which python cron is using.

(venv) $ crontab -e
* * * * * which python >> /tmp/cron.log 2>&1; python -V >> /tmp/cron.log 2>&1

(venv) $ tail -f /tmp/cron.log
/usr/bin/python
Python 2.7.10

I'm using the system python2.7, so I should do something about it. Check which python you should use.

(venv) $ which python
/Users/[User]/[place]/venv/bin/python

Rewrite to python under venv.

(venv) $ crontab -e
* * * * * /Users/[User]/[place]/venv/bin/python /Users/[User]/[place]/dt2jsonl.py >> /tmp/cron.log 2>&1

After a short wait, I got the print result in /tmp/cron.log.

(venv) $ tail -f /tmp/cron.log
updated: 2020-01-26 17:48:00.924120

(venv) $ cat out.jsonl
{"date": "2020-01-26", "time": "17:39:23.435616"}

I thought that it was also written as jsonline in out.jsonl, but it was executed manually at the beginning, not the result of executing it with cron. It was in the user directory because it was run in the user directory.

(venv) $ cat ~/out.jsonl
{"date": "2020-01-26", "time": "17:48:00.924120"}
{"date": "2020-01-26", "time": "17:49:01.102146"}
{"date": "2020-01-26", "time": "17:50:00.278025"}

The location of the output is different, but cron handled it in the venv environment. It is good to specify the output of the python script with an absolute path, This time, change the execution directory of cron.

(venv) $ crontab -e
* * * * * cd /Users/[User]/[place]; venv/bin/python dt2jsonl.py >> /tmp/cron.log 2>&1

This is what I expected.

(venv) $ tail -f /tmp/cron.log
updated: 2020-01-26 17:55:00.837256

(venv) $ cat out.jsonl
{"date": "2020-01-26", "time": "17:39:23.435616"}
{"date": "2020-01-26", "time": "17:55:00.837256"}

Wait a minute

(venv) $ cat out.jsonl
{"date": "2020-01-26", "time": "17:39:23.435616"}
{"date": "2020-01-26", "time": "17:55:00.837256"}
{"date": "2020-01-26", "time": "17:56:00.986909"}
{"date": "2020-01-26", "time": "17:57:01.167183"}
{"date": "2020-01-26", "time": "17:58:00.273073"}