[Pyto] I tried to use a smartphone as a flick keyboard for PC
Introduction
It seems that many people are unable to type while it is said that young people are away from PCs. However, young people are accustomed to handling smartphones instead of being able to use PCs, and the flick input of current high school girls is as fast as a feature phone gal.
In that case, it seems that it has been said for some time that PC input should be done by flicking, but there was no software that could be easily tried.
So, I made a flick keyboard using an app called Pyto that allows Python programming on iOS.
Existing product
- FlickTyper
When you connect your smartphone and PC with this device, your smartphone becomes a flick keyboard. There is also a FlickTyper BT that allows you to connect wirelessly using Bluetooth. It seems easy and good that there is no need to install a dedicated driver on the PC side. -Google Japanese Input Physical Flick Version This is the April Fool's Day story released by Google in 2016. -Physical Flick Keyboard It is a keyboard assembly kit that makes the above Google story practical. It seems that it was made by an individual. It was sold on consignment at Switch Science.
I also found an image where I was trying to create a flick keyboard using the touch panel.
environment
[IOS side] iPhone SE 2020 (iOS13.4.1) Pyto (12.0.3)
[PC side] Windows 10 Home 64bit (1909) Python 3.7.4
Library pyautogui (0.9.50) pyperclip (1.8.0)
specification
Sends the input data of the iOS device to the PC using socket communication.
Functions are character input, Enter, and Backspace. It is possible to add functions such as the cross key and Delete, but this time we limited it to character input.
Kanji conversion is performed on the iOS side, and after the conversion is confirmed, press Confirm again to send the characters to the PC.
Source code and commentary
The entire source code is located on GitHub. There are 2 files on the iOS side and 1 file on the PC side.
iOS side
text_client.py
# text_client.py
import socket
class Sender():
def __init__(self, port, ipaddr, timeout=5.0):
self.port = port
self.ipaddr = ipaddr
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.settimeout(timeout)
def connect(self):
self.sock.connect((self.ipaddr, self.port))
def send(self, text, enc="utf-8"):
"""
Send data
Parameters
----------
text : str
Text to send
enc : str
Encoding type. None with no encoding
"""
if(enc is not None):
text_ = text.encode(enc)
else:
text_ = text
self.sock.sendall(text_)
def close(self):
self.sock.close()
Text transmission class. Send data using socket communication. We will use this in the second file.
flick_kb.py
# flick_kb.py
import sys
import socket
import pyto_ui as ui
import text_client as myclient
class MySender(myclient.Sender):
def __init__(self, port, ipaddr, timeout=5.0):
print(port, ipaddr)
super().__init__(port, ipaddr, timeout)
self.editflag = False
def connect(self):
try:
super().connect()
self.editflag = True
print("connection")
except socket.timeout:
print("timed out")
super().close()
sys.exit()
def send_text(self, text):
self.send(text)
def send_end_flag(self):
#Send a flag to inform the PC of the end of the application
if(self.editflag):
self.send(b"\x00", enc=None)
self.editflag = False
def close(self):
self.send_end_flag()
super().close()
def did_end_editing(self, sender):
if(self.editflag):
if sender.text == " ":
self.send(b"\x0a", enc=None) # enter
sender.text = " "
elif sender.text == "":
self.send(b"\x08", enc=None) # backspace
else:
self.send_text(sender.text[1:]) #Send text without the previous space
sender.text = " "
sender.superview["text_field1"].become_first_responder() #Keep the text box out of focus
def did_change_text(self, sender):
if sender.text == "":
sender.text = " " #For backspace detection
self.send(b"\x08", enc=None) # backspace
def main():
args = sys.argv
if(len(args)<=2):
print("Input arguments. [PORT] [IP Address]")
sys.exit()
else:
port = int(args[1])
ipaddr = args[2]
#Transmitter
print("start connection...")
mysender = MySender(port=port, ipaddr=ipaddr, timeout=5.0)
mysender.connect()
#GUI part
view = ui.View()
view.background_color = ui.COLOR_SYSTEM_BACKGROUND
text_field = ui.TextField(placeholder="Back Space")
text_field.name = "text_field1"
text_field.text = " "
text_field.become_first_responder()
text_field.action = mysender.did_change_text
text_field.did_end_editing = mysender.did_end_editing
text_field.return_key_type = ui.RETURN_KEY_TYPE_DONE
text_field.width = 300
text_field.center = (view.width/2, view.height/2)
text_field.flex = [
ui.FLEXIBLE_BOTTOM_MARGIN,
ui.FLEXIBLE_TOP_MARGIN,
ui.FLEXIBLE_LEFT_MARGIN,
ui.FLEXIBLE_RIGHT_MARGIN
]
view.add_subview(text_field)
ui.show_view(view, ui.PRESENTATION_MODE_SHEET)
#End processing
mysender.close()
print("end")
if __name__=="__main__":
main()
It is a flick keyboard body. I'm using pyto's pyto_ui.TextField. The GUI part is almost the same as the sample code of Pyto.
This is TextField, but when you press Enter, it automatically loses focus. Since continuous input is not possible with that, the method of calling the function become_first_responder () and displaying it again is taken. Therefore, the keyboard disappears for a moment after pressing the confirm key.
Also, since the mysender.did_change_text method set in text_field.action of the GUI part is called even before the Japanese characters are confirmed, it seems that it cannot be sent at the moment when the conversion is confirmed. .. Therefore, the text is sent by pressing the confirm button twice, "Confirm conversion → Confirm". You need to press the confirm key three times to insert a line break.
Backspace does not respond in the absence of any text. Therefore, I tried to put a space in the text box in advance. It is determined that Backspace was pressed when this space was erased. After it is erased, the process sender.text =" " `is used to immediately insert a new space.
I used UTF-8 LF (0x0a), Backspace (0x08), and NULL (0x00) for Enter, Backspace, and exit code, respectively, but they are only used as flags, so they have no original meaning. Therefore, the line feed code reflects the settings of the PC OS and editor.
PC side
flick_kb_receiver.py
# flick_kb_receiver.py
import sys
import time
import socket
import pyautogui
import pyperclip
import threading
def type_text(text):
#Enter the given characters (copy and paste to clipboard)
pyperclip.copy(text)
pyautogui.hotkey("ctrl", "v")
return True
def type_backspace():
pyautogui.typewrite(["backspace"])
return True
def type_enter():
pyautogui.typewrite(["enter"])
return True
class Receiver():
def __init__(self, port, ipaddr=None, set_daemon=True):
"""
Receiver
Parameters
----------
port : int
Port number to use
ipaddr : None or str
IP address of the receiving PC. Automatically acquired with None.
set_daemon : bool
Do you want to daemonize threads? Stops the main thread without waiting for the receiver thread to end.
"""
if(ipaddr is None):
host = socket.gethostname()
ipaddr = socket.gethostbyname(host)
self.ipaddr = ipaddr
self.port = port
self.set_daemon = set_daemon
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.loopflag = False
print("ip:{0} port:{1}".format(self.ipaddr, self.port))
def loop(self):
self.sock.settimeout(0.5)
self.sock.bind((self.ipaddr, self.port))
self.sock.listen(1)
print("start listening...")
while(self.loopflag):
try:
conn, addr = self.sock.accept()
except socket.timeout:
continue
print("accepted")
with conn:
while(self.loopflag):
# print("waiting...")
data = conn.recv(1024)
if(not data):
break
if(data[:1]==b"\x08"): #If you press backspace in succession, multiple messages will be sent at the same time (example: b)."\x08\x08\x08), so check only the first 8 bytes
print("> BS")
type_backspace()
elif(data==b"\x0a"):
print("> Enter")
type_enter()
elif(data==b"\x00"):
print("STOP CLIENT")
break
else:
text = data.decode("utf-8")
print(">", text)
type_text(text)
def start_loop(self):
self.loopflag = True
self.thread = threading.Thread(target=self.loop)
if(self.set_daemon):
self.thread.setDaemon(True)
self.thread.start()
print("start_thread")
def stop_loop(self):
print("stop loop")
self.loopflag = False
time.sleep(0.6) #Wait until socket times out
if(not self.set_daemon):
print("waiting to stop client...") #Wait for the sender to stop
self.thread.join()
print("stop_thread")
def close_sock(self):
self.sock.close()
print("socket closed")
def main():
#Command line arguments
ipaddr = None
args = sys.argv
if(len(args)<=1):
print("Usage: flick_kb_receiver [PORT] [IP (optional)]")
sys.exit()
elif(len(args)==2):
port = int(args[1])
else:
port = int(args[1])
ipaddr = args[2]
#Main processing
receiver = Receiver(port=port, ipaddr=ipaddr)
receiver.start_loop()
while True:
stopper = input()
if(stopper=="s"):
receiver.stop_loop()
break
receiver.close_sock()
if __name__=="__main__":
main()
It is the PC side. I used pyautogui for character input. pyautogui is a library that allows general PC operations such as key operations and mouse operations. If the received binary data is not Enter, Backspace, or exit code, it is decoded and sent to the text input process.
However, it seems that Japanese input is not possible with pyautogui, so the method is to save the text to the clipboard with the type_text method and then paste it (Reference. / 2017/10/18/231 750)).
Regarding Backspace, if you delete characters continuously on the smartphone side, multiple Backspace codes \ x08 may be sent at the same time like \ x08 \ x08 \ x08. Therefore, I write ʻif (data [: 1] == b" \ x08 "):to see only the first byte. By the way, if it isdata [0], 8` (value converted to decimal number) will be returned.
The communication part runs in a separate thread, and the main thread waits for the program end command. While running, you can exit the program by typing s in the console and pressing Enter. However, since the communication section thread does not stop when communication is connected, the communication section is daemonized and forcibly stopped when the main thread ends. I don't know if this is a good way.
motion
File placement
[PC]
flick_kb_receiver.py
[iOS]
flick_keyboard (arbitrary folder)
├── flick_kb.py (executable file)
├── text_client.py
Execution procedure
Connection / input
- Execute the port number as a command line argument on the PC side
Example:
python flick_kb_receiver.py 50001 - Open
flick_kb.pyin the Pyto app on the iOS side and press the gear mark at the bottom right. Execute after entering the port number and IP address in Arguments in it Example:50001 192.168.1.2 - Place the cursor on any text editor or search window on your PC.
- After confirming the input in the text box on the iOS side, press Confirm again.
- Text is entered on the PC side
Screen to set Arguments in Pyto
Disconnect / end
- Stop the program on the iOS side
- Type "s" on the console on the PC side and press Enter to stop the program on the PC side.
If an error occurs in the import section on the iOS side, it means that folder access is not permitted, so allow it from the padlock mark at the bottom right of the Pyto editor screen.
The firewall access permission is given when the PC is started for the first time, but it cannot be connected unless public access is permitted.
State of operation
The GIF above is a later combination of PC and iPhone screens. From connection to inputting characters to VS Code and disconnecting. Almost no delay was felt.
in conclusion
I think flick input is a very good method as a keyboard for inputting Japanese on a small touch panel. This time, I actually tried flick input on a PC, and I felt that the threshold to the PC would be lowered by increasing the options for flick input for those who cannot use the PC keyboard. There was little movement of the line of sight between the PC and the smartphone, and I was able to input without any discomfort.
However, I can type faster with a PC keyboard, so there is little benefit to using flick input. Is it possible to hit it with one hand or use it while sleeping? After all, the keyboard input of the PC can make use of all the fingers, and the symbols are easy to type, so it is good to use it.
Recommended Posts