Automatic acquisition of stock price data with docker-compose

Introduction

When selecting a stock, the tool prepared on the securities company's site is used, but chart display is enough, but when you want to do a slightly different screening or when backtesting system trade etc. You want to have your own stock price data, right?

There is a site called Inexhaustible that publishes daily stock price data of Japanese stocks. In order to have the stock price data by yourself, I will try to get the stock price list of the day at the fixed time every day from here.

Python script to get stock price data from inexhaustible sites

For example, if the url of the data acquisition of the inexhaustible site is the stock price data of 2020/11/04,

http://mujinzou.com/k_data/2020/20_11/T201104.zip

Example script to get the stock price data of the day and unzip it:

 -*- coding: utf-8 -*-
import urllib.request
import zipfile
import datetime
import os
import logging
import sys

def download_csv_zip(output_dir, year, month, day):
    base_url = 'http://mujinzou.com/k_data/'+str(year)+'/'
    subdir = '{0:02d}_{1:02d}/'.format(year-2000, month)
    filename = 'T20{0:02d}{1:02d}.zip'.format(month, day)
    url = base_url + subdir + filename
    now = datetime.datetime.now()
    try:
        urllib.request.urlretrieve(url, filename)
        with zipfile.ZipFile(filename, 'r')as zf:
            zf.extractall(output_dir)
        os.remove(filename)
        logging.info(str(now)+' success to get file: '+filename)
    except urllib.error.HTTPError:
        logging.error(str(now)+' error')
                

def main():
    args = sys.argv
    if len(args)==2:
        output_dir = args[1]
        if output_dir[-1]!='/':
            output_dir = output_dir + '/'
    else:
        output_dir = './'
    logging.basicConfig(level=logging.INFO)
    today = datetime.date.today()
    download_csv_zip(output_dir, today.year, today.month, today.day)

if __name__ == '__main__':
    main()

Example of usage:

$ python get_todays_csv_zip_from_muzinzo.py ~/tmp

You can specify the download destination directory as an argument. If not specified, the download destination will be the current directory.

Please note that the stock price data of the day will be placed on the site after the evening. In other words, the stock price data for the day can be obtained with this script from the evening of the day to midnight.

Run python script with docker-compose

As a stepping stone for scheduled execution of python scripts, run python scripts with docker-compose.

docker-compose.yml and Dockerfile are borrowed from the following.

https://qiita.com/reflet/items/4b3f91661a54ec70a7dc Let's easily create a python3 environment with docker

Directory structure

python3\
    Dockerfile
    docker-compose.yml
    opt\
        get_todays_csv_zip_from_muzinzo.py
        exec_cmd.sh

FROM python:3
USER root

RUN apt-get update
RUN apt-get -y install locales && \
    localedef -f UTF-8 -i ja_JP ja_JP.UTF-8
ENV LANG ja_JP.UTF-8
ENV LANGUAGE ja_JP:ja
ENV LC_ALL ja_JP.UTF-8
ENV TZ JST-9
ENV TERM xterm

RUN apt-get install -y vim less cron
RUN pip install --upgrade pip
RUN pip install --upgrade setuptools
RUN echo '0 20 * * 1-5 root /root/opt/exec_cmd.sh' >> /etc/crontab
CMD ["cron", "-f"]

version: '3'
services:
  python3:
    restart: always
    build: .
    container_name: 'python3'
    working_dir: '/root/opt'
    tty: true
    volumes:
      - ./opt:/root/opt

Start-up

$ docker-compose up -d --build

Run python script

$ docker-compose exec python3 python /root/opt/get_todays_csv_zip_from_muzinzo.py

This will generate the stock price data "Tyymmdd.csv" for the day in the opt directory.

Run python scripts on time every day

Comment out the last two lines of the previous Dockerfile. This is the setting for cron to execute the script exec_cmd.sh from Monday to Friday at 20:00.

  ...
RUN echo '0 20 * * 1-5 root /root/opt/exec_cmd.sh' >> /etc/crontab
CMD ["cron", "-f"]

!/bin/bash
/usr/local/bin/python /root/opt/get_todays_csv_zip_from_muzinzo.py /root/opt >> /root/opt/exec_cmd.log 2>&1

Don't forget this

$ chmod a+x exec_cmd.sh

After preparing up to this point, start it on the always-on server.

$ docker-compose up -d --build

If you not only acquire stock price data but also store it in the database together, you can use the latest stock price data at any time, which is very convenient.

The execution environment of python and the scheduled execution by cron are put together in a container and finished neatly.