Create a simple scheduled batch using Docker's Python Image and parse-crontab

This article is the 14th day of Docker Advent Calendar 2015.

Summary

Main subject

This story is from PyCon JP 2015's Talk Session "[Baseball Hack! ~ Data analysis and visualization using Python](http://www.slideshare.net/shinyorke/hackpython-pyconjp" Baseball Hack! ~ Data using Python. This is an excerpt and detailed explanation version of the story "I made a batch to acquire baseball data with Docker and parse-crontab!" That was shown in the second half of "Analysis and Visualization") ".

Basically, there are many parts that I write for my convenience and thoughts, so I am waiting for opinions and suggestions m (_ _) m

Also, in making this material,

** Implementing scheduled execution processing in Python (GAUJIN.JP/Gojin) **

I referred to the above entry.

Who are you

Self-introduction

Dockar history

background

[Announcement of XP Festival 2015] before PyCon JP 2015 ,,, [http://www.slideshare.net/shinyorke/agile-baseball-science-52692504 "Agile Baseball Science --Agile Baseball that works well for the brain Baseball data was needed for Hanashi "), and development began.

Task

Docker Hub Python image does not include crontab

turning point! ~ Can crontab work with Python alone?

I was just making an announcement for PyCon JP 2015, so I asked on Twitter to "reprint it!" → Thank you for your kind consent.

sample & commentary

The baseball code is a little complicated (& I don't want to explain it too much), so I made a sample.

python-crontab-docker-example(GitHub)

The recommended environment is Python 3.4.x or higher.

By the way, it works fine with the latest version Python 3.5.1 at the moment (2015/12/14)!

Job setting & execution

Define JobController to execute batch and JobSettings to manage the next execution time & interval until execution.

The argument of JobController.run () is the familiar crontab setting (* * * * *).

scheduler/job.py


#!/usr/bin/env python
# -*- coding: utf-8 -*-

import time
import functools
import logging
from crontab import CronTab
from datetime import datetime, timedelta
import math

__author__ = 'Shinichi Nakagawa'


class JobController(object):
    """
Job execution controller
    """

    @classmethod
    def run(cls, crontab):
        """
Processing execution
        :param crontab: job schedule
        """
        def receive_func(job):
            @functools.wraps(job)
            def wrapper():

                job_settings = JobSettings(CronTab(crontab))
                logging.info("->- Process Start")
                while True:
                    try:
                        logging.info(
                            "-?- next running\tschedule:%s" %
                            job_settings.schedule().strftime("%Y-%m-%d %H:%M:%S")
                        )
                        time.sleep(job_settings.interval())
                        logging.info("->- Job Start")
                        job()
                        logging.info("-<- Job Done")
                    except KeyboardInterrupt:
                        break
                logging.info("-<- Process Done.")
            return wrapper
        return receive_func


class JobSettings(object):
    """
Output settings
    """

    def __init__(self, crontab):
        """
        :param crontab: crontab.CronTab
        """
        self._crontab = crontab

    def schedule(self):
        """
Next run
        :return: datetime
        """
        crontab = self._crontab
        return datetime.now() + timedelta(seconds=math.ceil(crontab.next()))

    def interval(self):
        """
Time to next execution
        :return: seconds
        """
        crontab = self._crontab
        return math.ceil(crontab.next())

batch body

Import JobController, Pool the process to be executed, and execute it in parallel.

The sample is made like teaching baseball every Friday night at Tamori Club and every day at 18:00.

As a point to note

That's about it.

batch.py


#!/usr/bin/env python
# -*- coding: utf-8 -*-

import logging
from multiprocessing import Pool
from scheduler.job import JobController

__author__ = 'Shinichi Nakagawa'


#Note that the Docker Image Timezone is UTC!
@JobController.run("20 15 * * 5")
def notice_tmr_club():
    """
It's time for the Tamori Club(Tokyo)
    :return: None
    """
    logging.info("The Tamori Club will begin! !! !!")


#Note that the Docker Image Timezone is UTC!(I said it twice because it's important)
@JobController.run("00 9 * * *")
def notice_baseball():
    """
Teach the time of Yakiu
    :return: None
    """
    logging.info("It's time to go! !! !! !!")


def main():
    """
method to run crontab
    :return: None
    """
    #Log settings(Info level, format, timestamp)
    logging.basicConfig(
        level=logging.INFO,
        format="time:%(asctime)s.%(msecs)03d\tprocess:%(process)d" + "\tmessage:%(message)s",
        datefmt="%Y-%m-%d %H:%M:%S"
    )

    #Register the job you want to run with crontab
    jobs = [notice_tmr_club, notice_baseball]

    # multi process running
    p = Pool(len(jobs))
    try:
        for job in jobs:
            p.apply_async(job)
        p.close()
        p.join()
    except KeyboardInterrupt:
        logging.info("exit")


if __name__ == '__main__':
    main()

Dockerfile

This is simple.

Include the GitHub code, launch and exit.

Dockerfile


# Python crontab sample

FROM python:3.5.1

MAINTAINER Shinichi Nakagawa <[email protected]>

# add to application
RUN mkdir /app
WORKDIR /app
ADD requirements.txt /app/
RUN pip install -r requirements.txt
ADD ./scheduler /app/scheduler/
ADD *.py /app/

docker-compose

Write the docker startup settings in docker-compose.yml.

However, move batch.py and finish.

docker-compose.yml


batch:
  build: .
  dockerfile: ./Dockerfile
  command: python batch.py
  container_name: python_crontab_example

Start-up! !! !!

If you docker-compose up (or docker run) and move like this, it's OK.

$ docker-compose up
Creating python_crontab_example
Attaching to python_crontab_example
python_crontab_example | time:2015-12-13 13:45:09.463   process:9       message:->- Process Start
python_crontab_example | time:2015-12-13 13:45:09.464   process:8       message:->- Process Start
python_crontab_example | time:2015-12-13 13:45:09.465   process:9       message:-?- next running        schedule:2015-12-18 15:20:00
python_crontab_example | time:2015-12-13 13:45:09.465   process:8       message:-?- next running        schedule:2015-12-14 09:00:00

Summary & to the next person

Summary

To the next

Thank you @tamai.

Recommended Posts

Create a simple scheduled batch using Docker's Python Image and parse-crontab
[Python] Create a Batch environment using AWS-CDK
Create a web map using Python and GDAL
Create a Mac app using py2app and Python3! !!
Create a python GUI using tkinter
Create a simple Python development environment with VS Code and Docker
Create a dummy image with Python + PIL.
Create a simple GUI app in Python
[Python] Create a linebot to write a name and age on an image
I tried to create a sample to access Salesforce using Python and Bottle
Create a simple momentum investment model in Python
Create a MIDI file in Python using pretty_midi
Create a Python image in Django without a dummy image file and test the image upload
Create a simple reception system with the Python serverless framework Chalice and Twilio
Create a data collection bot in Python using Selenium
Create a color sensor using a Raspberry Pi and a camera
Try creating a compressed file using Python and zlib
Create a Python module
[CRUD] [Django] Create a CRUD site using the Python framework Django ~ 1 ~
Create an image file using PIL (Python Imaging Library).
[Python] Create a ValueObject with a complete constructor using dataclasses
Let's make a simple game with Python 3 and iPhone
Create a High Dynamic Range Image (HDR) with OpenCV and Python (Mertens, Robertson, Debevec)
Create a batch of images and inflate with ImageDataGenerator
Implementing a generator using Python> link> yield and next ()> yield
[CRUD] [Django] Create a CRUD site using the Python framework Django ~ 2 ~
[Python] Mask the image into a circle using Pillow
Create a Python environment
Create a decent shell and python environment on Windows
Create a company name extractor with python using JCLdic
Create a simple API just to input and output JSON files ~ Python / Flask edition ~
[CRUD] [Django] Create a CRUD site using the Python framework Django ~ 3 ~
[CRUD] [Django] Create a CRUD site using the Python framework Django ~ 4 ~
[CRUD] [Django] Create a CRUD site using the Python framework Django ~ 5 ~
[Python] Start a batch file from Python and pass variables.
I want to create a karaoke sound source by separating instruments and vocals using Python
Building a Python environment on a Mac and using Jupyter lab
[Python] Create a date and time list for a specified period
[Python] Chapter 01-03 About Python (Write and execute a program using PyCharm)
Let's create a PRML diagram with Python, Numpy and matplotlib.
[Python Kivy] How to create a simple pop up window
Create code that outputs "A and pretending B" in python
Create a simple video analysis tool with python wxpython + openCV
Create a simple Python development environment with VSCode & Docker Desktop
Python: Create a dictionary from a list of keys and values
Create a GCE instance from a GCR Docker image using terraform
Create a gadget-like transparent background image type window using wxpython
Shoot time-lapse from a PC camera using Python and OpenCV
Create a simple CRUD app using Django's generic class view
[Python] Accessing and cropping image pixels using OpenCV (for beginners)
I made a Chatbot using LINE Messaging API and Python
Create a Wox plugin (Python)
Create a function in Python
Create a dictionary in Python
Create a (simple) REST server
# 1 [python3] Simple calculation using variables
Create JIRA tickets using Python
Create a python numpy array
Create a simple textlint server
Create a directory with python
I made a simple circuit with Python (AND, OR, NOR, etc.)