Automatic acquisition of stock price data with docker-compose

Introduction

When selecting a stock, the tool prepared on the securities company's site is used, but chart display is enough, but when you want to do a slightly different screening or when backtesting system trade etc. You want to have your own stock price data, right?

There is a site called Inexhaustible that publishes daily stock price data of Japanese stocks. In order to have the stock price data by yourself, I will try to get the stock price list of the day at the fixed time every day from here.

Python script to get stock price data from inexhaustible sites

For example, if the url of the data acquisition of the inexhaustible site is the stock price data of 2020/11/04,

http://mujinzou.com/k_data/2020/20_11/T201104.zip

Example script to get the stock price data of the day and unzip it:

 -*- coding: utf-8 -*-
import urllib.request
import zipfile
import datetime
import os
import logging
import sys

def download_csv_zip(output_dir, year, month, day):
    base_url = 'http://mujinzou.com/k_data/'+str(year)+'/'
    subdir = '{0:02d}_{1:02d}/'.format(year-2000, month)
    filename = 'T20{0:02d}{1:02d}.zip'.format(month, day)
    url = base_url + subdir + filename
    now = datetime.datetime.now()
    try:
        urllib.request.urlretrieve(url, filename)
        with zipfile.ZipFile(filename, 'r')as zf:
            zf.extractall(output_dir)
        os.remove(filename)
        logging.info(str(now)+' success to get file: '+filename)
    except urllib.error.HTTPError:
        logging.error(str(now)+' error')
                

def main():
    args = sys.argv
    if len(args)==2:
        output_dir = args[1]
        if output_dir[-1]!='/':
            output_dir = output_dir + '/'
    else:
        output_dir = './'
    logging.basicConfig(level=logging.INFO)
    today = datetime.date.today()
    download_csv_zip(output_dir, today.year, today.month, today.day)

if __name__ == '__main__':
    main()

Example of usage:

$ python get_todays_csv_zip_from_muzinzo.py ~/tmp

You can specify the download destination directory as an argument. If not specified, the download destination will be the current directory.

Please note that the stock price data of the day will be placed on the site after the evening. In other words, the stock price data for the day can be obtained with this script from the evening of the day to midnight.

Run python script with docker-compose

As a stepping stone for scheduled execution of python scripts, run python scripts with docker-compose.

docker-compose.yml and Dockerfile are borrowed from the following.

https://qiita.com/reflet/items/4b3f91661a54ec70a7dc Let's easily create a python3 environment with docker

Directory structure

python3\
    Dockerfile
    docker-compose.yml
    opt\
        get_todays_csv_zip_from_muzinzo.py
        exec_cmd.sh
FROM python:3
USER root

RUN apt-get update
RUN apt-get -y install locales && \
    localedef -f UTF-8 -i ja_JP ja_JP.UTF-8
ENV LANG ja_JP.UTF-8
ENV LANGUAGE ja_JP:ja
ENV LC_ALL ja_JP.UTF-8
ENV TZ JST-9
ENV TERM xterm

RUN apt-get install -y vim less cron
RUN pip install --upgrade pip
RUN pip install --upgrade setuptools
RUN echo '0 20 * * 1-5 root /root/opt/exec_cmd.sh' >> /etc/crontab
CMD ["cron", "-f"]
version: '3'
services:
  python3:
    restart: always
    build: .
    container_name: 'python3'
    working_dir: '/root/opt'
    tty: true
    volumes:
      - ./opt:/root/opt

Start-up

$ docker-compose up -d --build

Run python script

$ docker-compose exec python3 python /root/opt/get_todays_csv_zip_from_muzinzo.py

This will generate the stock price data "Tyymmdd.csv" for the day in the opt directory.

Run python scripts on time every day

Comment out the last two lines of the previous Dockerfile. This is the setting for cron to execute the script exec_cmd.sh from Monday to Friday at 20:00.

  ...
RUN echo '0 20 * * 1-5 root /root/opt/exec_cmd.sh' >> /etc/crontab
CMD ["cron", "-f"]
!/bin/bash
/usr/local/bin/python /root/opt/get_todays_csv_zip_from_muzinzo.py /root/opt >> /root/opt/exec_cmd.log 2>&1

Don't forget this

$ chmod a+x exec_cmd.sh

After preparing up to this point, start it on the always-on server.

$ docker-compose up -d --build

If you not only acquire stock price data but also store it in the database together, you can use the latest stock price data at any time, which is very convenient.

The execution environment of python and the scheduled execution by cron are put together in a container and finished neatly.

Recommended Posts

Automatic acquisition of stock price data with docker-compose
Download Japanese stock price data with python
Acquisition of time series data (daily) of stock prices
Get stock price data with Quandl API [Python]
Stock price forecast using deep learning [Data acquisition]
Data is missing when getting stock price data with Pandas-datareader
Get stock price with Python
[Stock price analysis] Learn pandas with Nikkei 225 (004: Change read data to Nikkei 225)
Automatic acquisition of gene expression level data by python and R
[Stock price analysis] Learning pandas with fictitious data (001: environment preparation-file reading)
[Stock price analysis] Learning pandas with fictitious data (003: Type organization-candlestick chart)
Stock price acquisition code by scraping (Selenium)
[Spotify API] Looking back on 2020 with playlists --Part.1 Acquisition of playlist data
Acquisition of plant growth data Acquisition of data from sensors
Stock Price Forecast with TensorFlow (LSTM) ~ Stock Forecast Part 1 ~
Recommendation of Altair! Data visualization with Python
Example of efficient data processing with PANDAS
Get the stock price of a Japanese company with Python and make a graph
Convert data with shape (number of data, 1) to (number of data,) with numpy.
Automatic operation of Chrome with Python + Selenium + pandas
Automatic version sync of chrome driver binary with chrome
CSV output of pulse data with Raspberry Pi (CSV output)
View details of time series data with Remotte
Let's do web scraping with Python (stock price)
Python / Automatic low wrench unfitting of experimental data
"Getting stock price time series data from k-db.com with Python" Program environment creation memo
Stock Price Forecast with TensorFlow (Multilayer Perceptron: MLP) ~ Stock Forecast Part 2 ~
Overview and tips of seaborn with statistical data visualization
Challenge principal component analysis of text data with Python
[Basics of data science] Collecting data from RSS with python
Extract the band information of raster data with python