[Pyto] I tried to use a smartphone as a flick keyboard for PC

image2.jpg

Introduction

It seems that many people are unable to type while it is said that young people are away from PCs. However, young people are accustomed to handling smartphones instead of being able to use PCs, and the flick input of current high school girls is as fast as a feature phone gal.

In that case, it seems that it has been said for some time that PC input should be done by flicking, but there was no software that could be easily tried.

So, I made a flick keyboard using an app called Pyto that allows Python programming on iOS.

Existing product

I also found an image where I was trying to create a flick keyboard using the touch panel.

environment

[IOS side] iPhone SE 2020 (iOS13.4.1) Pyto (12.0.3)

[PC side] Windows 10 Home 64bit (1909) Python 3.7.4

Library pyautogui (0.9.50) pyperclip (1.8.0)

specification

Sends the input data of the iOS device to the PC using socket communication.

Functions are character input, Enter, and Backspace. It is possible to add functions such as the cross key and Delete, but this time we limited it to character input.

Kanji conversion is performed on the iOS side, and after the conversion is confirmed, press Confirm again to send the characters to the PC.

Source code and commentary

The entire source code is located on GitHub. There are 2 files on the iOS side and 1 file on the PC side.

iOS side

text_client.py

# text_client.py
import socket

class Sender():
    def __init__(self, port, ipaddr, timeout=5.0):
        self.port = port
        self.ipaddr = ipaddr
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.sock.settimeout(timeout)
    
    def connect(self):
        self.sock.connect((self.ipaddr, self.port))            
    
    def send(self, text, enc="utf-8"):
        """
Send data
        
        Parameters
        ----------
        text : str
Text to send
        enc : str
Encoding type. None with no encoding
        """
        if(enc is not None):
            text_ = text.encode(enc)
        else:
            text_ = text
        self.sock.sendall(text_)
    
    def close(self):
        self.sock.close()
 

Text transmission class. Send data using socket communication. We will use this in the second file.

flick_kb.py

# flick_kb.py
import sys
import socket
import pyto_ui as ui
import text_client as myclient

class MySender(myclient.Sender):
    def __init__(self, port, ipaddr, timeout=5.0):
        print(port, ipaddr)
        super().__init__(port, ipaddr, timeout)
        self.editflag = False
    
    def connect(self):
        try:
            super().connect()
            self.editflag = True
            print("connection")
        except socket.timeout:
            print("timed out")
            super().close()
            sys.exit()
    
    def send_text(self, text):
        self.send(text)
    
    def send_end_flag(self):
        #Send a flag to inform the PC of the end of the application
        if(self.editflag):
            self.send(b"\x00", enc=None)
            self.editflag = False
            
    def close(self):
        self.send_end_flag()
        super().close()
    
    def did_end_editing(self, sender):
        if(self.editflag):
            if sender.text == " ":
                self.send(b"\x0a", enc=None)  # enter
                sender.text = " "
            elif sender.text == "":
                self.send(b"\x08", enc=None)  # backspace
            else:
                self.send_text(sender.text[1:])  #Send text without the previous space
                sender.text = " "
    
        sender.superview["text_field1"].become_first_responder()  #Keep the text box out of focus
    
    def did_change_text(self, sender):
        if sender.text == "":
            sender.text = " "  #For backspace detection
            self.send(b"\x08", enc=None)  # backspace       

def main():
    args = sys.argv
    if(len(args)<=2):
        print("Input arguments. [PORT] [IP Address]")
        sys.exit()
    else:
        port = int(args[1])
        ipaddr = args[2]
    
    #Transmitter
    print("start connection...")
    mysender = MySender(port=port, ipaddr=ipaddr, timeout=5.0)
    mysender.connect()
    
    #GUI part
    view = ui.View()
    view.background_color = ui.COLOR_SYSTEM_BACKGROUND
    
    text_field = ui.TextField(placeholder="Back Space")
    text_field.name = "text_field1"
    text_field.text = " "
    text_field.become_first_responder()
    text_field.action = mysender.did_change_text
    text_field.did_end_editing = mysender.did_end_editing
    text_field.return_key_type = ui.RETURN_KEY_TYPE_DONE
    text_field.width = 300
    text_field.center = (view.width/2, view.height/2)
    text_field.flex = [
        ui.FLEXIBLE_BOTTOM_MARGIN,
        ui.FLEXIBLE_TOP_MARGIN,
        ui.FLEXIBLE_LEFT_MARGIN,
        ui.FLEXIBLE_RIGHT_MARGIN
    ]
    view.add_subview(text_field)
    
    ui.show_view(view, ui.PRESENTATION_MODE_SHEET)
    
    #End processing
    mysender.close()
    print("end")

if __name__=="__main__":
    main()

It is a flick keyboard body. I'm using pyto's pyto_ui.TextField. The GUI part is almost the same as the sample code of Pyto.

This is TextField, but when you press Enter, it automatically loses focus. Since continuous input is not possible with that, the method of calling the function become_first_responder () and displaying it again is taken. Therefore, the keyboard disappears for a moment after pressing the confirm key.

Also, since the mysender.did_change_text method set in text_field.action of the GUI part is called even before the Japanese characters are confirmed, it seems that it cannot be sent at the moment when the conversion is confirmed. .. Therefore, the text is sent by pressing the confirm button twice, "Confirm conversion → Confirm". You need to press the confirm key three times to insert a line break.

Backspace does not respond in the absence of any text. Therefore, I tried to put a space in the text box in advance. It is determined that Backspace was pressed when this space was erased. After it is erased, the process sender.text =" " `is used to immediately insert a new space.

I used UTF-8 LF (0x0a), Backspace (0x08), and NULL (0x00) for Enter, Backspace, and exit code, respectively, but they are only used as flags, so they have no original meaning. Therefore, the line feed code reflects the settings of the PC OS and editor.

PC side

flick_kb_receiver.py

# flick_kb_receiver.py
import sys
import time
import socket
import pyautogui
import pyperclip
import threading

def type_text(text):
    #Enter the given characters (copy and paste to clipboard)
    pyperclip.copy(text)
    pyautogui.hotkey("ctrl", "v")
    return True

def type_backspace():
    pyautogui.typewrite(["backspace"])
    return True

def type_enter():
    pyautogui.typewrite(["enter"])
    return True

class Receiver():
    def __init__(self, port, ipaddr=None, set_daemon=True):
        """
Receiver

        Parameters
        ----------
        port : int
Port number to use
        ipaddr : None or str
IP address of the receiving PC. Automatically acquired with None.
        set_daemon : bool
Do you want to daemonize threads? Stops the main thread without waiting for the receiver thread to end.
        """
        if(ipaddr is None):
            host = socket.gethostname()
            ipaddr = socket.gethostbyname(host)
        self.ipaddr = ipaddr
        self.port = port
        self.set_daemon = set_daemon
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.loopflag = False
        print("ip:{0} port:{1}".format(self.ipaddr, self.port))
    
    def loop(self):
        self.sock.settimeout(0.5)
        self.sock.bind((self.ipaddr, self.port))
        self.sock.listen(1)
        print("start listening...")
        while(self.loopflag):
            try:
                conn, addr = self.sock.accept()
            except socket.timeout:
                continue
            print("accepted")
            with conn:
                while(self.loopflag):
                    # print("waiting...")
                    data = conn.recv(1024)
                    if(not data):
                        break
                    if(data[:1]==b"\x08"):  #If you press backspace in succession, multiple messages will be sent at the same time (example: b)."\x08\x08\x08), so check only the first 8 bytes
                        print("> BS")
                        type_backspace()
                    elif(data==b"\x0a"):
                        print("> Enter")
                        type_enter()
                    elif(data==b"\x00"):
                        print("STOP CLIENT")
                        break
                    else:
                        text = data.decode("utf-8")
                        print(">", text)
                        type_text(text)

    def start_loop(self):
        self.loopflag = True
        self.thread = threading.Thread(target=self.loop)
        if(self.set_daemon):
            self.thread.setDaemon(True)
        self.thread.start()
        print("start_thread")
    
    def stop_loop(self):
        print("stop loop")
        self.loopflag = False
        time.sleep(0.6)  #Wait until socket times out
        if(not self.set_daemon):
            print("waiting to stop client...")  #Wait for the sender to stop
            self.thread.join()
        print("stop_thread")
    
    def close_sock(self):
        self.sock.close()
        print("socket closed")

def main():
    #Command line arguments
    ipaddr = None
    args = sys.argv
    if(len(args)<=1):
        print("Usage: flick_kb_receiver [PORT] [IP (optional)]")
        sys.exit()
    elif(len(args)==2):
        port = int(args[1])
    else:
        port = int(args[1])
        ipaddr = args[2]

    #Main processing
    receiver = Receiver(port=port, ipaddr=ipaddr)
    receiver.start_loop()
    while True:
        stopper = input()
        if(stopper=="s"):
            receiver.stop_loop()
            break
    receiver.close_sock()

if __name__=="__main__":
    main()

It is the PC side. I used pyautogui for character input. pyautogui is a library that allows general PC operations such as key operations and mouse operations. If the received binary data is not Enter, Backspace, or exit code, it is decoded and sent to the text input process.

However, it seems that Japanese input is not possible with pyautogui, so the method is to save the text to the clipboard with the type_text method and then paste it (Reference. / 2017/10/18/231 750)).

Regarding Backspace, if you delete characters continuously on the smartphone side, multiple Backspace codes \ x08 may be sent at the same time like \ x08 \ x08 \ x08. Therefore, I write ʻif (data [: 1] == b" \ x08 "):to see only the first byte. By the way, if it isdata [0], 8` (value converted to decimal number) will be returned.

The communication part runs in a separate thread, and the main thread waits for the program end command. While running, you can exit the program by typing s in the console and pressing Enter. However, since the communication section thread does not stop when communication is connected, the communication section is daemonized and forcibly stopped when the main thread ends. I don't know if this is a good way.

motion

File placement

[PC]

flick_kb_receiver.py

[iOS]

flick_keyboard (arbitrary folder)
├── flick_kb.py (executable file)
├── text_client.py

Execution procedure

Connection / input

  1. Execute the port number as a command line argument on the PC side Example: python flick_kb_receiver.py 50001
  2. Open flick_kb.py in the Pyto app on the iOS side and press the gear mark at the bottom right. Execute after entering the port number and IP address in Arguments in it Example: 50001 192.168.1.2
  3. Place the cursor on any text editor or search window on your PC.
  4. After confirming the input in the text box on the iOS side, press Confirm again.
  5. Text is entered on the PC side
12C402D0-A9C4-41A1-9B3A-AF6F0A083CF8.jpeg Screen to set Arguments in Pyto

Disconnect / end

  1. Stop the program on the iOS side
  2. Type "s" on the console on the PC side and press Enter to stop the program on the PC side.

If an error occurs in the import section on the iOS side, it means that folder access is not permitted, so allow it from the padlock mark at the bottom right of the Pyto editor screen.

The firewall access permission is given when the PC is started for the first time, but it cannot be connected unless public access is permitted.

State of operation

E8D546BF-13AA-411C-8FE0-097E427CB509.gif

The GIF above is a later combination of PC and iPhone screens. From connection to inputting characters to VS Code and disconnecting. Almost no delay was felt.

in conclusion

I think flick input is a very good method as a keyboard for inputting Japanese on a small touch panel. This time, I actually tried flick input on a PC, and I felt that the threshold to the PC would be lowered by increasing the options for flick input for those who cannot use the PC keyboard. There was little movement of the line of sight between the PC and the smartphone, and I was able to input without any discomfort.

However, I can type faster with a PC keyboard, so there is little benefit to using flick input. Is it possible to hit it with one hand or use it while sleeping? After all, the keyboard input of the PC can make use of all the fingers, and the symbols are easy to type, so it is good to use it.

Recommended Posts

[Pyto] I tried to use a smartphone as a flick keyboard for PC
I tried to explain what a Python generator is for as easily as possible.
I tried to create a bot for PES event notification
I tried to make a strange quote for Jojo with LSTM
I tried to create a linebot (preparation)
I tried to make a Web API
I made a Docker container to use JUMAN ++, KNP, python (for pyKNP).
I tried to make a face diagnosis AI for a female professional golfer ①
I tried to make a face diagnosis AI for a female professional golfer ②
I tried to make a skill that Alexa will return as cold
I think it's a loss not to use profiler for performance tuning
I tried to build a super-resolution method / ESPCN
I tried to use lightGBM, xgboost with Boruta
I tried to build a super-resolution method / SRCNN ①
I tried to make AI for Smash Bros.
I tried to generate a random character string
I tried to build a super-resolution method / SRCNN ③
I tried to create a button for Slack with Raspberry Pi + Tact Switch
I tried to build a super-resolution method / SRCNN ②
[Python] I tried to get the type name as a string from the type function
I tried to make a ○ ✕ game using TensorFlow
[Python] I want to use only index when looping a list with a for statement
I tried to create a reinforcement learning environment for Othello with Open AI gym
When I searched for "as a Service" from AaaS to ZaaS, I saw various services.
Create a dataset of images to use for learning
I tried to make a "fucking big literary converter"
I tried to create a table only with Django
I tried to summarize how to use matplotlib of python
I tried to draw a route map with Python
I tried to implement a pseudo pachislot in Python
I tried to implement a recommendation system (content-based filtering)
[Go + Gin] I tried to build a Docker environment
Convenient to use matplotlib subplots in a for statement
I tried to summarize how to use pandas in python
I tried a simple RPA for login with selenium
I tried to draw a configuration diagram using Diagrams
I tried to debug.
I tried to paste
I tried using a library (common thread) that makes Python's threading package easier to use
[Git] I tried to make it easier to understand how to use git stash using a concrete example
[For those who want to use TPU] I tried using the Tensorflow Object Detection API 2
I tried to implement a volume moving average with Quantx
I tried to implement a basic Recurrent Neural Network model
How to use Fujifilm X-T3 as a webcam on Ubuntu 20.04
I tried porting the code written for TensorFlow to Theano
I tried to automatically create a report with Markov chain
[Markov chain] I tried to read a quote into Python.
I tried using Tensorboard, a visualization tool for machine learning
I tried to solve a combination optimization problem with Qiskit
I tried "How to get a method decorated in Python"
How to use cuML SVC as a Gridsearch CV classifier
I want to use a virtual environment with jupyter notebook!
I didn't know how to use the [python] for statement
I tried to automate [a certain task] using Raspberry Pi
I stumbled when I tried to install Basemap, so a memorandum
I tried to sort a random FizzBuzz column with bubble sort.
I tried to make a stopwatch using tkinter in python
How to use a file other than .fabricrc as a configuration file
I tried to divide with a deep learning language model
I tried to make a simple text editor using PyQt
Use a scripting language for a comfortable C ++ life-OpenCV-Port Python to C ++-