Automate sushi making with Python

Hello. Suddenly everyone Sushi hitting Are you doing? Most people who have practiced typing should have played this game. Sushi hitting is a browser game. So, I thought that it would be easy to automate if I got HTML with Python, so I decided to make it.

I was trying to get the html Actually, sushi making uses a technology called OpenGL, It seems that the characters are displayed as an image. Therefore, it is a policy change. Let's recognize the image.

First, press the start button and the recommended button.

import pyautogui
import time
x,y=pyautogui.locateCenterOnScreen("susida.png ")
print(x)
print(y)
pyautogui.click(x, y+100)
print("I pressed the start button.")
time.sleep(1)
pyautogui.click(x, y+100)
print("I pressed the recommendation.")
time.sleep(1)
pyautogui.typewrite(" ")
print("start.")
time.sleep(2)

If you specify it in absolute coordinates, it will change depending on the window size etc. I decided to use the relative coordinates from susida.png. susida.png←susida.png Then click on that coordinate with pyautogui. Installation of pyautogui

pip install pyautogui

is.

Next is the image recognition part.

import pyocr
import pyocr.builders
from PIL import Image
import sys
import cv2

i=1
result=0
tools=pyocr.get_available_tools()

if len(tools) == 0:
    print("The OCR tool was not found.")
    sys.exit(1)

tool=tools[0]

while(i<201):
    sc=pyautogui.screenshot(region=(x-110, y+85, 210, 25))

    sc.save("original.png ")

    original=cv2.imread("original.png ",0)
    threshold=100

    ret, img_thresh = cv2.threshold(original, threshold, 255, cv2.THRESH_BINARY)
    cv2.imwrite("gray.png ", img_thresh)

    hanten = cv2.imread("gray.png ")
    hanten2=cv2.bitwise_not(hanten)
    cv2.imwrite("sushida_sc.png ", hanten2)


    img_org=Image.open("sushida_sc.png ")

    builder = pyocr.builders.TextBuilder()
    tmp=result
    result=tool.image_to_string(img_org, lang="eng", builder=builder)
    if(tmp != result):
        print(i,"Dish:",result)
        pyautogui.typewrite(result)
    i=i+1
    time.sleep(0.3)

Pyocr is used for image recognition. Please see here for the introduction method ↓ https://gammasoft.jp/blog/ocr-by-python/ First of all, I usually take a screenshot of the Roman alphabet. However, this makes the background transparent and reduces the recognition accuracy. So, it will be binarized. Then invert it to make the text black. If you type in what you recognize with pyautogui, the automation of sushi making is complete. I stopped at 200 because if I eat too much sushi, I get an error. The full text is below.

import pyautogui
import time
import pyocr
import pyocr.builders
from PIL import Image
import sys
import cv2

i=1
result=0
tools=pyocr.get_available_tools()

if len(tools) == 0:
    print("The OCR tool was not found.")
    sys.exit(1)

tool=tools[0]

x,y=pyautogui.locateCenterOnScreen("susida.png ")
print(x)
print(y)
pyautogui.click(x, y+100)
print("I pressed the start button.")
time.sleep(1)
pyautogui.click(x, y+100)
print("I pressed the recommendation.")
time.sleep(1)
pyautogui.typewrite(" ")
print("start.")
time.sleep(2)
while(i<201):
    sc=pyautogui.screenshot(region=(x-110, y+85, 210, 25))

    sc.save("original.png ")

    original=cv2.imread("original.png ",0)
    threshold=100

    ret, img_thresh = cv2.threshold(original, threshold, 255, cv2.THRESH_BINARY)
    cv2.imwrite("gray.png ", img_thresh)

    hanten = cv2.imread("gray.png ")
    hanten2=cv2.bitwise_not(hanten)
    cv2.imwrite("sushida_sc.png ", hanten2)


    img_org=Image.open("sushida_sc.png ")

    builder = pyocr.builders.TextBuilder()
    tmp=result
    result=tool.image_to_string(img_org, lang="eng", builder=builder)
    if(tmp != result):
        print(i,"Dish:",result)
        pyautogui.typewrite(result)
    i=i+1
    time.sleep(0.3)

YouTube:https://www.youtube.com/watch?v=kZ0MKTQXgj0

Recommended Posts

Automate sushi making with Python
I tried to automate sushi making with python
Automate python testing with CircleCI
FizzBuzz with Python3
[Python] Automate Pelican builds with Travis CI
Scraping with Python
Scraping with Python
Python with Go
Twilio with Python
Integrate with Python
Play with 2016-Python
AES256 with python
Tested with Python
python starts with ()
with syntax (Python)
Bingo with python
Zundokokiyoshi with python
Excel with Python
Microcomputer with Python
Cast with python
Serial communication with Python
Zip, unzip with python
Django 1.11 started with Python3.6
Python with eclipse + PyDev.
Socket communication with Python
Data analysis with python 2
Scraping with Python (preparation)
Try scraping with Python.
Learning Python with ChemTHEATER 03
Sequential search with Python
"Object-oriented" learning with python
Handling yaml with python
Solve AtCoder 167 with python
Serial communication with python
[Python] Use JSON with Python
Learning Python with ChemTHEATER 05-1
Learn Python with ChemTHEATER
Run prepDE.py with python3
1.1 Getting Started with Python
Collecting tweets with Python
Binarization with OpenCV / Python
3. 3. AI programming with Python
Kernel Method with Python
Scraping with Python + PhantomJS
Drive WebDriver with python
[Python] Redirect with CGIHTTPServer
Voice analysis with python
Think yaml with python
Operate Kinesis with Python
Use DynamoDB with Python
Zundko getter with python
Handle Excel with python
Ohm's Law with Python
Run Blender with python
Python starting with Windows 7
Automate AtCoder submission (Python)
Heatmap with Python + matplotlib
Multi-process asynchronously with python
Python programming with Atom
Learning Python with ChemTHEATER 02
Use Python 3.8 with Anaconda