A story about automating online mahjong (Mahjong Soul) with OpenCV and machine learning

Overview

By using OpenCV template matching and GUI operation module from python The mahjong tiles on the web browser could be recognized and clicked by the BOT, and the play could be automated. In addition, machine learning was used for the logic part of which mahjong tile to click.

If you replace the template match search image, you can use it not only in Mahjong Soul but also in other mahjong games in general, and if you change the machine learning part, you can repeatedly ask for rational choice under specific conditions. It can be applied.

• In-game images of Mahjong Soul are used throughout the article to help understand the content, but they are strongly blurred from the viewpoint of copyright protection.

Target audience

People who like machine learning (like mahjong) People who are interested in automating Windows and GUI operations but wonder what OpenCV is

A person who likes Mahjong Soul but has a broken heart because it is too difficult to run a trial event I wrote in an image processing laboratory at a university in the past ~~ Killing time ~~ People who are interested in scripts

My ~~ impure ~~ motivation

The other day, I was addicted to Mahjong Soul. I'm tired of it lately.

If you play against several stations on a daily basis or participate in an event, you will be able to reach the gacha reward, so continue playing the game ~~ and pick up a cute girl ~~

But I can't get along with a stranger on the net who even knows the rules ~~ like work ~~ mahjong. (Especially the event matches with other players regardless of ability, and it is too hard to play seriously because the ranking and reward after playing do not match)

However, if you just move the simple Tsumo-kiri BOT, you will be judged to be away from your desk and the worst account will be banned. (I was surprised at this. Do not do it.)

So

--Browser operation by simulating mouse click operation ――Choose a tile and throw it away & you can aggress

I decided to make a BOT.

Component

Acquisition of hand tile + Tsumo tile information

Overview

I used a mechanism called template matching in OpenCV.

OpenCV (Open Source Computer Vision Library) ... An open source computer vision library developed and published by Intel. A dream-like library that can realize various functions required for processing images and videos on a computer.

Template matching ... A dream-like process that searches the input image for the most similar part to the template image (partial image).

I could have made it as a Windows application in C #, but ~~ It's faster to charge Mahjong Soul directly than to spend so much effort ~~ This time I adopted python which can access OpenCV quickly.

After installing python

$ pip install opencv-python

You can use OpenCV just by using it. I used to have a hard time doing tutorials ... I think you should refer to here for environment construction

Install OpenCV with pip https://qiita.com/fiftystorm36/items/1a285b5fbf99f8ac82eb

All pattern tile images are prepared in advance, 13 tiles are cut out from the actual play screen capture, and tile information is obtained by searching for matching tile images.

Attention of screen capture

Please note that browser size = app screen size does not hold.

Mahjong Soul is a web application designed to match the aspect ratio at all resolutions, as you can see by transforming the browser. [^ 1] In other words, the drawing area that is deformed by the browser and does not match the original aspect ratio of the game is displayed as a jet black area.

Here, template match is vulnerable to detection targets whose size has expanded or contracted, so it is important to acquire the screen (capture) size at the beginning of the process and then add adjustment processing to match the size of the prepared tile template image. is. (If you play the game with a fixed display environment + browser size every time, this process is unnecessary because the size of the captured image and the template image match each time)

• Various modules for python that capture the browser are also distributed, but in my method, after simply taking a full screen capture, the position of the edge of the application screen is detected by template matching, so that the pure application screen can be displayed. I decided to cut it out.

Image elements located at two points as far as possible are extracted in advance as a template image so that there is less error when calculating the scale. It is a good idea to select an image that does not change depending on the situation and does not have similar features on the screen (≈ characteristic that is easy to identify).

Also, at this time, it is necessary to prepare 3-5 types of template images of different sizes (or to scale them during processing) and adopt the template image with the highest degree of matching and its scale. .. [^ 2](Because the size of the image prepared as a template and the actual captured image are different, the entire process becomes unstable if the detection of the screen area fails)

By measuring the size of the image extracted as a template image and the distance between the two points at that time in advance and holding it as data, it is possible to obtain the relative position of the place where the Tehai are lined up even if the display environment changes. Become. By cutting out the hand tiles one by one (cv2.crop) based on either of the templates, the image could be extracted as shown below. (Since I collected the hand-painted image data by manual capture, it is a little rattling. It was okay because it did not have a big effect on the recognition accuracy)

Dora 牌 always has a shiny on-screen effect, but in terms of image processing, I have a very bad feeling (laughs).

Tehai recognition

By the way, in order to recognize the tile itself, template match each of the extracted tile images from the "tile list image" as shown below.

(* The entered Kuman image matches a part of the tile list image as a template)

You can create data in which tiles are lined up in a horizontal row, and the x-coordinates of the detected image will tell you the true identity of the tiles. (It was hard to make this all mahjong tile side by side data ...) The process seems to be fine, but it took 2.7 seconds so far. .. .. Mahjong Soul has 5 seconds, so you can spend up to 5 seconds per process. To put it a little more, it cannot be said that the start of loop processing = the moment when it hits, so at least the monitoring loop at intervals of 2.5 seconds (turn start → processing start (after Max 2.5s has passed) → processing completion (2.5s) → It's necessary to finish the turn), so it feels awkward (;'∀') The machine learning part also takes processing time ... Well, let's refactor it later![^ 3]

"What to cut" problem

Overview and precedents

13 I searched for a program that outputs tiles that should be discarded by inputting hand tiles and tiles. There isn't. Rather, the mahjong game + AI area was in the midst of competing for this area. I'm sorry seniors.

Furthermore, my purpose is not "the strongest Majan AI that I thought of", but a BOT that keeps playing so wet that Mahjong Soul management can not be determined as a robot, so I will make this area appropriately.

I tried to divide it into the famous Amazon named github with the keyword "mahjong"

"How many cuts" judgment part from the mahjong application "Computer Mahjong" that runs on HTML5 + JavaScript https://github.com/kobalab/Majiang [Operation demo] http://kobalab.net/majiang/dapai.html

This looked pretty good. The author has left a log of the hardships related to the "cut what" judgment logic on his blog, so I think that it will be quite helpful for those who aim for the strongest AI.

However, since this is javascript, it is a little troublesome to connect data input / output from python. .. .. I was looking for something that could be used as it is in python.

Adoption of machine learning

How many mahjong AIs by deep learning https://github.com/hogeki/dlmahjong

For those who have never touched machine learning, readme is a little difficult to get along with, so the procedure after git clone to work is described below.

First, we will introduce TensorFlow [^ 4] with pip, but here it will be happier if the versions are matched exactly. seriously. You should listen to the story of the human pillar (true face)

pip tensorflow==1.70 (2020.8.5 Follow the contents of the current git repository)

Thankfully, Tenhou's score data (.txt) is also included, so you can learn it as it is and save it as a parameter.

python mahjong_ai.py --train --save

The training result parameters should now be saved as external data (trained model / trained model). In the future, by using this model as an argument, it will be possible to perform judgment processing to cut how many.

Since the function of "passing 13 + 1 tile data as information and telling what to cut" is not implemented in the corresponding git script (only continuous tests and its performance evaluation described later) You need to make the part yourself. The code is below.

If you enter the information of Tehai + Tsumo tile as an argument with the character string of Tenhou specification, it will tell you what to cut. (The above training model is the default argument)

Implementation script

mahjong_ai.py(Additional part)

#calc_Since the dahai function is supposed to be called from an external file
#Mahjong the accompanying file_ai.Refer to the file attached to py with import
import mahjong_common as mjc
import mahjong_loader as mjl

#Addition of import due to function addition
import unicodedata
import re

def init_sess():

    #calc_Since the dahai function is supposed to be called from an external file
    #Make the part that sets sess a function and save it globally.
    # (Mahjong about what sess is_common.Can be read from py)
    global sess
    
    make_model()
    saver = tf.train.Saver()
    sess = tf.Session()
    sess.run(tf.global_variables_initializer())
    sess.run(tf.local_variables_initializer())
    saver.restore(sess, "ckpt/my_model")

def calc_dahai(hands_str,tsumo_str):

    #1st argument input example
    #hands_str="6p6p7p8p3s4s5s5s7s8s East East East"

    #2nd argument input example
    #tsumo_str="6s"

    #Init before execution_sess()If you don't, it won't work
    global sess

    #Converts a Tehai character string to an array of character strings for each Tehai
    tehai_tmp=[]

    #Since 1-byte characters and 2-byte characters are mixed and difficult to process, convert from 2-byte characters to 1-byte characters
    #(It's a shit-like specification, but it can't be helped because the input data from Tenhou is that way.)
    hands_str = hands_str.replace('east', 'HI')
    hands_str = hands_str.replace('South', 'MI')
    hands_str = hands_str.replace('West', 'NI')
    hands_str = hands_str.replace('North', 'KI')
    hands_str = hands_str.replace('White', 'SI')
    hands_str = hands_str.replace('Departure', 'HA')
    hands_str = hands_str.replace('During ~', 'NA')
        
    #Store in an array every two characters
    tehai_str_list_by_2char = re.split('(..)',hands_str)[1::2]

    #Undo characters
    tehai_str_list_by_2char = [s.replace('HI', 'east') for s in tehai_str_list_by_2char]
    tehai_str_list_by_2char = [s.replace('MI', 'South') for s in tehai_str_list_by_2char]
    tehai_str_list_by_2char = [s.replace('NI', 'West') for s in tehai_str_list_by_2char]
    tehai_str_list_by_2char = [s.replace('KI', 'North') for s in tehai_str_list_by_2char]
    tehai_str_list_by_2char = [s.replace('SI', 'White') for s in tehai_str_list_by_2char]
    tehai_str_list_by_2char = [s.replace('HA', 'Departure') for s in tehai_str_list_by_2char]
    tehai_str_list_by_2char = [s.replace('NA', 'During ~') for s in tehai_str_list_by_2char]

    #Mahjong array of handicraft strings_common.Convert to an array of numbers determined by py
    for tehai_one in tehai_str_list_by_2char:#tehai_tmp:
        h = mjc.get_hai_number(tehai_one)
        tehai_num_array[h] += 1
    
    print("--------Mahjong AI judgment result--------")

    tstr = mjc.get_string_from_tehai(tehai_num_array)
    print("Tehai:" + tstr)

    tsumo = mjc.get_hai_number(tsumo_str)
    print("Self-reliance:" + mjc.get_hai_string(tsumo))

    tehai_num_array[tsumo] += 1

    #Since there is also a function that determines whether or not it has been turned
    #It is also possible to collect grades or have them process something individually
    #if mjc.is_agari(tehai_num_array):
    #    print("Won")
    
    #Judgment processing by machine learning
    ai_outs = sess.run(out, feed_dict={x:[tehai_num_array]})

    dahai = get_ai_dahai(tehai_num_array, ai_outs[0])
    #print("Hit:" + mjc.get_hai_string(dahai))

    return(mjc.get_hai_string(dahai))

Also, as an aside from the main purpose of this article, as a function that is the original purpose of this git repository, using the model that was learned and completed, a random mahjong tile simulator is looped and the number of trials is sufficient. It makes it possible to test the tempai rate etc. in.

python mahjong_ai.py --run

[Result of simulating 1000 times]

If you want to improve the functionality of AI, you can collect more score data for learning, or train professionals and good people. It's interesting because changing the training data will change the test results.

I won't go into the description of machine learning here, but I've asked you to briefly touch on the basics of machine learning, so please play around with it. ~~ And if you have a good AI, please give it to me ~~ By the way, it seems a little difficult to collect a large amount of Tenhou's score data at this point. If you have a good script that can collect data, please share it in the comments, and all human beings ~~ and mainly I ~~ will be saved.

Mouse simulation

I tried various things, but the tutorial below was easy to understand. After piping the required modules, copy and paste will work.

Let Python do iterative work with PyAutoGui https://qiita.com/hirohiro77/items/78e26a59c2e45a0fe4e3

The calculator app will start up in front of you and you can drag the mouse without permission. At first glance, I even applauded "Oh".

If you copy and paste only the part necessary for mouse click of this and click the coordinate position calculated by the above operation, it is OK

Knead and mix the three elements so far to complete. Code description is no longer omitted. If there is demand, I will raise it to git.

State of operation (video)

Click the thumbnail to jump to the link to youtube

It's a medium-smelling playing, but it looks like you have a choice of tiles towards Agari.

Mechanko It is a strength of the machine that both simple tile selection and tile selection that is a little confusing for humans can be processed in the same time.

Observing, the current issues are as follows.

(1) I managed to make 18 shots so that I couldn't use up my 20 seconds, but depending on the situation, I might exceed my time.

There is room for refactoring in this part, but I don't care because it's okay to lose due to time out.

After all, I don't expect to pop or chee, so it seems that processing will be more stable if you always select the "No squeal" option when playing a game.

(2) Since the data at the time of learning does not include information on the abandoned tiles and dora of another family, the tiles will not shake even if the other family reaches as in the video, and the dora tiles will be discarded without worrying about it ( Lol)

If you are interested in machine learning and want to enhance this feature, the next step is to learn Dora information and self-wind / cold information. It is an important step to extend the functionality to train data of different dimensions (that is, the meaning of the data) as one input parameter. (Mahjong_common.py side needs to be repaired)

(However, if river information and reach information of other families are included in the learning, the input parameters will become enormous, and if you do not devise it properly, stable operation itself will be difficult.)

③ It does not support Damaten, Tenpai, and Dark Can.

From the concept itself, it is a machine that aims at Tsumoagari in a straight line.

afterwards

It was banned (laughs)

That's a joke, so I refactored the number and frequency of template matching and the processing procedure so as not to run out of time, and tried using it in league games as well.

You can win, lose, or lose, but after all, the machine doesn't get tired, so it keeps playing all the time. (Currently progressive, only while my computer is running ...)

I would like to measure the ability of the model that simply trained the Tenhou data included in git introduced at the beginning to finally stabilize in which league.

I would like to add it again at a later date, so please look forward to it.

◆ About fear of infringement of rights to Mahjong Soul in the article

This time, I posted a video of Mahjong Soul in the service and in the article of BOT that the management side of Mahjong Soul did not intend, but it is used within the range that does not conflict with the official guidelines, and the copyright etc. We have determined that we have not infringed. However, if you do the same thing with reference to this article, please do so at your own risk. For reference, we will present the terms of use of Mahjong Soul that may conflict with the following.

Guidelines for live game distribution and video posting (excerpts from prohibited items)

・ Use for purposes other than showing the game live, such as politics, religion, ** promotion of specific beliefs ** ~~ (applicable w) ~~

Mahjong Soul Terms of Use, Article 11 Prohibitions

(12) The act of illegally acquiring various contents provided by the Company through this service, or the act of promoting this. ~~ (Getting gacha rewards through BOT operation, etc.) ~~ (16) Acts that interfere with or may interfere with the operation and use of this service. (21) Acts similar to the preceding items and other acts that the Company deems inappropriate.

[^ 1]: Later verification revealed that the aspect ratio may have fluctuated slightly, but it seems to be within the margin of error. In my script, I decided to measure and correct the vertical and horizontal scales separately.

[^ 2]: I think there are various theories about this content, but it's just my best practice. It's a problem that always occurs when you adopt a technique called template matching, but do you know any other good method (in less time)?

[^ 3]: I haven't tried refactoring afterwards in the projects I've entered so far. Mainly due to lack of resources or motivation. .. ..

[^ 4]: TensorFlow ... A software library developed by Google and released as open source for use in machine learning.