background

Due to the spread of COVID-19 infection, Jakuso was closed and we lost our place to play mahjong set in Tenhou's private room.
At first, I manually copied and pasted the logs to manage my grades and saved them in Google Docs, but as I played mahjong every day, it became more and more troublesome.
Therefore, I decided to create an application in Python that automatically downloads logs (scrape), aggregates them, and (and displays graphs) automatically.
Furthermore, we made this tool a LINE bot so that all participants can see the information at any time.

Over View

App overview chart

The app mainly consists of the following two functions.

Automatically download and save Tenhou's log (purple arrow in the figure)
Reply aggregate information in response to user (LINE) request (green arrow in the figure)

This time, I will explain the first application (program).

Complete image

In response to the wake word (here, "shukei" or "gurafu"), the request menu (here, "balance", "arrival order", "tip", "today]) is displayed.
If you select "Balance", the total score, the number of games, and the total number of chips, which is the sum of all the past logs, will be displayed.
From the "Gurafu" wake word, you can display a graph of past scores and chip transitions.
I haven't implemented it this time, but the balance of direct games and the average score may be interesting.

Languages, tools, services, modules, etc.

Language used

Python 3.7.3
Anaconda is used in the local development environment, but Anaconda is irrelevant because the final application runs on Heroku.

Tools and services used

This time, I created an application using the following services that can be used for free.

Heroku

Heroku Home
Service for running programs on the web
This time, we will use it as two execution environments, the LINE bot execution application and the log scraping script.

LINE bot (LINE Messaging API)

[LINE Messaging API official website](https://developers.line.biz/ja/docs/messaging-api/overview/#messaging-api mechanism)
You can operate the created LINE bot with Python
I chose LINE as the UI this time.

Dropbox API

I used DropBox to store the logs.
Using API, you can operate files and folders on Dropbox with Python
Initially, I created an app using the API of Google Drive (and Google Docs) that I originally used, but the Google Docs file (.gdocs) does not support the API (I just could not find it) ?) So I stopped. Also, when I edit a text file (.txt) with Google Docs, I'm quite frustrated by the mystery specification that changes to .gdocs, so I don't think I'll use it again.
In that respect, Dropbox's API is very simple and easy to understand, and even I, a beginner, could use it immediately. recommendation! !! !!

AWS S3 storage

Amazon cloud service
Temporarily save the graph written in Python in response to the request from LINE, and display the image on LINE from the link of the file.
You may be able to go with Dropbox without using S3 (I used S3 before I started using Dropbox, so I confused multiple services). I'm sorry it's hard to understand (sweat

Module used

Flask==0.12.2
line-bot-sdk==1.8.0
boto3==1.9.4
pandas==0.24.2
matplotlib==3.0.3
numpy==1.16.2
dropbox==10.1.2

Get log of Tenhou private room

This time, we will get the log of Tenhou private room by scraping. I will post more and more.

Synopsis

The method of downloading the Tenhou log was as described in Tenhou Official / Log.
In particular, "https://tenhou.net/sc/raw/dat/sca{yyyymmdd}.log.gz" Download the .gz file (scrape.gz) from the URL ({yyyymmdd} is the date of the log to be acquired) using urllib, expand the contents, and make it a .txt file (scrape.txt) once. __ The log of the private room is sca__.
Next, we want to be able to access the logs from Heroku's server in the future, so we will be able to manage the logs with Dropbox.
I will download the log file saved in Dropbox, add the log of the day, and then upload it to Dropbox again.
Since it is not good to manually execute the script every day, I will automatically get the log for the previous day at 0:30 AM every day. Periodic script execution can be set on Heroku. For the method of periodic execution, I referred to Regular execution of weather Python on Heroku -Qiita.

Code example

All modules should be included by default in Anaconda environment.
If there is nothing, install it with pip or conda.
download4 is a self-made module used for exchanging files with Dropbox (details will be explained later).

# scrape_log.py
# coding *-utf-8-*

import os
import pprint
import time
import urllib.error
import urllib.request
import gzip
import shutil
import datetime
from datetime import date,timedelta
import download4

def download_file(url, dst_path):
    try:
        with urllib.request.urlopen(url) as web_file:
            data = web_file.read()
            with open(dst_path, mode='wb') as local_file:
                local_file.write(data)
    except urllib.error.URLError as e:
        print(e)

Main part
I would like to get the log of the day before executing the script, but since the server time of __Heroku (execution environment) is UTC (World Standard Time), it is necessary to consider the effect of the time difference of 9 hours. There is !! __. 0:30 Japan time is 15:30 on the day before __ in UTC, so you should log the day when the script is executed. (By the way, in the local test environment, the log is fetched one day before the execution date, so in that case, it is shifted by one day using `` `timedelta``` as commented out.
Since the rule is set to scrape the Tenhou log in the `.gz``` file, after scraping as the .gz file, the `` `.txt file It has been rewritten to.
Use the `download``` method defined on the self-made module download4 to download the .txt file (here `logvol1.txt) that stores the past logs from Dropbox and scrape it. After adding the log for the previous day, re-upload with the upload``` method.
We use if to write out only the logs for our private room (example: C1234).

if __name__ == "__main__":
    dt_now = datetime.datetime.now()
    yyyymmdd = dt_now.strftime('%Y%m%d')
    # dt_1day_past = dt_now - timedelta(days=1) #Find the log 1 hour ago
    # yyyymmdd = dt_1day_past.strftime('%Y%m%d')
    print(yyyymmdd)

    fname = 'sca{}.log.gz'.format(yyyymmdd)
    URL = "https://tenhou.net/sc/raw/dat/"+fname
    
    dst_path = 'scrape.gz'
    download_file(URL, dst_path)

    with gzip.open('scrape.gz', mode='r') as f_in:
        with open('scrape.txt', 'wb') as f_out:
            shutil.copyfileobj(f_in, f_out)
    with open('scrape.txt') as f:
        lines = f.readlines()


    download4.download("/logvol1.txt","temp.txt")
    with open("temp.txt",'a') as f:
        f.write("{}\n".format(yyyymmdd))
        for line in lines:
            roomid = line.split()[0]
            if roomid == "C1234":　#Private room ID
                f.write("{}".format(line)) 
                print(line)

    download4.upload("temp.txt","/logvol1.txt")

Contents of self-made module download4
It's just a function of downloading and uploading with Dropbox.
Refer to the site below for API usage registration of Dropbox, creation of application folders, etc.
When you create an application, a folder named "Apps" will be created in the Dropbox home directory, and it seems that the working folder will be directly under it.
When downloading the Dropbox API (`download_file (self, file_from, file_to)`), it seems that you cannot download without `` `file_to```, so you need to prepare a file in advance. did.
Also, `file_from``` had to be prefixed with / , like ``` download ('/ logvol1.txt', "log.txt") `. .. Now you can refer to the working folder (directly under) of the application folder.
REFERENCES
[Upload files from python script to Dropbox](Upload files from https://www.it-swarm.dev/ja/python/python-script to dropbox / 1047611873 /)
Manipulate Dropbox folders using Dropbox API (under investigation)

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import dropbox

class TransferData:
    def __init__(self, access_token):
        self.access_token = access_token

    def upload_file(self, file_from, file_to):
        """upload a file to Dropbox using API v2
        """
        dbx = dropbox.Dropbox(self.access_token)

        with open(file_from, 'rb') as f:
            dbx.files_upload(f.read(), file_to,mode=dropbox.files.WriteMode.overwrite)
    
    def download_file(self,file_from,file_to):
        """download a file to Dropbox using API v2
        """
        dbx = dropbox.Dropbox(self.access_token)

        with open(file_to, 'rb') as f:
            dbx.files_download_to_file(file_to, file_from)
    


def upload(file_from,file_to):
    access_token = "Hogehoge"　#Access token to the application folder
    transferData = TransferData(access_token)
    # API v2
    transferData.upload_file(file_from, file_to)

def download(file_from,file_to):
    access_token = "Hogehoge"　#Access token to the application folder
    transferData = TransferData(access_token)
    # API v2
    transferData.download_file(file_from, file_to)

if __name__ == '__main__':
    # upload()
    download('/logvol1.txt',"log.txt")

After that, if you push these files to Heroku and execute them automatically at regular intervals, the log file (`` `logvol1.txt```) on Dropbox will be updated every day.
Refer to here for the usage and settings of Heroku.
[For beginners] Easy LINE Bot development in Heroku environment using Python](https://miyabi-lab.space/blog/21)
Regular execution of weather Python on Heroku -Qiita

in conclusion

This time it was up to the scrape of the Tenhou log.
Thank you to our ancestors for your reference.
Next time, I will write about the response part of LINE bot.
I myself am not a main business of programming, so if you have any questions about writing styles or techniques such as "I should do this more!" Or "This is not good security!", Please comment! !! !! !!

p.s.

This is my first post, but it is difficult to write. .. .. (I'll fix it little by little lol)
Mahjong is fun!
Of course, Jongso is fun, but the environment where you can play mahjong with your scattered friends at any time is too great!
Let's survive COVID-19 in Tenhou!

Create an automatic grade management app for Tenhou private room with LINE bot and Python Part 1