About keyboard events on Mac Catalyst [Xcode & Swift]

Introduction

In this article, for example, when creating a ** calculator app **, place a display ** label **, number keys and a calculation (+/-/×/÷/=) key ** button **, The scene of operating with the button ** tap ** or the mouse ** click ** is normal, but here is the story when you want to operate by pressing the ** key ** of the physical keyboard. (If the app has a text input field, there is nothing wrong with it.) Also, it is a story when you want to support all of iOS (including iPad OS) and Mac OS (Mac Catalyst) with the same source code.

The environment at the time of writing the article is as follows.

Development environment version

• MacOS ： 11.1 (20C69)
• Xcode ： 12.3 (12C33)
• Swift ： 5.3.2 (swiftlang-1200.0.45 clang-1200.0.32.28)

Target environment version

• iOS/iPadOS ： 14.0
• MacOS ： 11.0

1. How to pick up keyboard events

For MacOS apps with APPKit, keyDown (with :) and keyUp (with::: ) You can implement the method, but this method does not exist in the UIView class. In UIKit, pressesBegan (: with :) and pressesEnded (: with :) of the UIView class I will implement the pressesended) method, but it works fine on UIKit native iOS/iPad OS, but on Mac (Catalyst) some keys work but most of the keys are "Warning: insertText reached" on the console. Is output and the above method is not called. A phenomenon similar to this article [^ 0], but not resolved.

Another option is to use the GCKeyboard class by Game Controller supported by iOS14. The outline of how to use it is as follows.

if let keyboard = GCKeyboard.coalesced?.keyboardInput {
    // bind to any key-down/up
    keyboard.keyChangedHandler = {
        (keyboard, key, keyCode, pressed) in
        // compare button to GCKeyCode
・ ・ ・
    }

    // bind to specific key-down/up
    keyboard.button(forKeyCode: .spacebar)?.valueChangedHandler = {
        (key, value, pressed) in
        // SpaceBar was pressed or released
・ ・ ・
    }
}

As far as I've tried, using the GCKeyboard class worked on all iOS/iPad OS and Mac (Catalyst). However, for Mac, I encountered a phenomenon that a beep sound is produced by pressing some keys. This is the same phenomenon that occurred when implementing the keyDown (with :) and keyUp (with :) methods of the NSView class in a MacOS app with APPKit, but the workaround in the MacApp [^ 1] ] [^ 2] [^ 3] cannot be applied on Mac (Catalyst) that does not use NSView.

2. Beep sound avoidance

The workaround for the beep sound when using the GC Keyboard on Mac (Catalyst) is listed here [^ 4]. It is a dummy implementation of pressesBegan`` pressesEnded to prevent the message from being transmitted to the application level. I'm worried, so I also implemented a dummy of pressesCancelled.

override func pressesBegan(_ presses: Set<UIPress>, with event: UIPressesEvent?) { }
override func pressesEnded(_ presses: Set<UIPress>, with event: UIPressesEvent?) { }
override func pressesCancelled(_ presses: Set<UIPress>, with event: UIPressesEvent?) { }

3. Handling of GC Key Code

The handling of keyboard events by the GCKeyboard class is a very low level handling. The GC Key Code type tells you which key was pressed/released, but this type does not always match the marking on the key top of the keyboard, so be careful.

** Here are some examples. ** **

・ Equal symbol (=)

On a JIS keyboard, the shift key and- (hyphen) key are used, so the notification is divided into two events, leftShift (rightShift) and hyphen. Since it is the = (equal) key on the US keyboard, it is notified by one event of equalSign.

In other words, even if the characters are the same, the code notified by the JIS/US keyboard may be different. The left and right shift keys are distinguished and notified independently.

・ Number zero (0)

Normally, the number keys are lined up at the top of the keyboard, but with a full keyboard, the number keys are also placed on the right side as a numeric keypad, and the same 0 (zero number) has a different code. The number 0 at the top of the keyboard is zero. The number 0 on the numeric keypad is keypad0. As an aside, I personally wanted it to be key0, just like the alphabet, instead of zero.

In other words, even with the same key top character, the code notified when the key location is different is different.

・ Handling of a and A (lowercase/uppercase)

Alphabetic characters are usually entered in lowercase unless they are Caps Locked. To enter in uppercase, press the shift key at the same time. On the other hand, if CapsLock is set, it will be entered in uppercase, and if you press it at the same time as the shift key, it will be in lowercase. In the case of shift key ** + ** A, it depends on whether it is interpreted as uppercase or lowercase, and the state of CapsLock, but it is unknown whether this can be known. This handling can be difficult for apps that want to be case sensitive. (Pressing the Caps Lock key after starting the application is passed through the event, but if it is already in the Caps Lock state before starting the application, it has the opposite meaning)

If it is UIKey type, you can know the CapsLock status by modifierFlags, so is it possible to judge by using the pressesBegan event together? ~~ (I haven't tried it, so I'm not sure if it can be used together) ~~ (Added 2021.1.7) We have verified that the pressesBegan event can be used together, but it still requires some ingenuity.

・ Notes on using the pressesBegan event together

Behavior is different between iOS and Mac (Catalyst). Since there is a possibility of an OS bug, specify the version at the time of verification.

• MacOS 11.1(20C69)
• iOS 14.2 (18B92)

Method invocation (in order)

• Mac Catalyst

As explained in Section 1, pressesBegan / pressesEnded is not called in the A key.

• iOS

The order in which keyChangedHandler andpressesBegan (/ Ended)are called is reversed from that of Mac (Catalys).

Handling of Caps Lock keys

Since the alphaShift of UIPress.key.modifierFlags passed to PressesBegan indicates the CapsLock state, the CapsLock state after starting the application can be correctly recognized on both Mac/iOS. However, if the Caps Lock state is set before the application is started, neither Mac/iOS can correctly recognize the Caps Lock state. The reason is that on Mac, the A key does not call PressesBegan. This is because in the case of iOS, alphaShift of UIPress.key.modifierFlags is set in the opposite direction. iOS may be a bug.

・ Identification of the keyboard itself

There is a big difference in the keyboard layout between JIS and US keyboards, but it has not been investigated how to distinguish whether the keys on the JIS keyboard were pressed or the keys on the US keyboard were pressed. ~~ I can't tell this apart. (Added 2021.1.8) GCPhysicalInputProfile can be used to determine the number of keys, but there is no information that can clearly identify the keyboard type. In the first place, when you press the key ** ^ ** (hat) to the right of the hyphen on the JIS keyboard, a equalSign is returned. This is the code for the US keyboard layout.

Since the keys used are limited if it is a calculator application, it can be realized by paying attention only to the above points, but developing a screen editor like vi requires considerable effort to handle keyboard events. Seem.

4. Mac keyboard example

The keyboard layout of the genuine Apple keyboard is as follows.

JIS keyboard

US keyboard

Finally

"I can't pick up keyboard events on Mac (Catalyst). When you hit the key, you will hear a beep. This survey took almost a day, so I've summarized it here. ~~ Uninvestigated/unverified items will be added later when necessary. ~~ (Added 2021.1.8) Since Game Controller is a framework for game apps in the first place, it may be impossible to develop a screen editor like vi with this. that's all

bonus

An extension that visualizes (characterizes) UIKeyModifierFlags and GCKeyCode is included.

extension UIKeyModifierFlags {
    var toString: String {
        var result = "["
        let keys: [UIKeyModifierFlags] = [.alphaShift, .shift, .control, .alternate, .command, .numericPad]
        let strs = ["alphaShift", "shift", "control", "alternate", "command", "numericPad"]
        for n in keys.indices {
            if self.contains(keys[n]) {
                if result.count == 1 {
                    result += strs[n]
                } else {
                    result += " ," + strs[n]
                }
            }
        }
        result += "]"
        return result
    }
}

extension GCKeyCode {
    var toString: String {
        let str: String
        switch self {
        case .F1: str = "F1"
        case .F10: str = "F10"
        case .F11: str = "F11"
        case .F12: str = "F12"
        case .F2: str = "F2"
        case .F3: str = "F3"
        case .F4: str = "F4"
        case .F5: str = "F5"
        case .F6: str = "F6"
        case .F7: str = "F7"
        case .F8: str = "F8"
        case .F9: str = "F9"
        case .LANG1: str = "LANG1"
        case .LANG2: str = "LANG2"
        case .LANG3: str = "LANG3"
        case .LANG4: str = "LANG4"
        case .LANG5: str = "LANG5"
        case .LANG6: str = "LANG6"
        case .LANG7: str = "LANG7"
        case .LANG8: str = "LANG8"
        case .LANG9: str = "LANG9"
        case .application: str = "application"
        case .backslash: str = "backslash"
        case .capsLock: str = "capsLock"
        case .closeBracket: str = "closeBracket"
        case .comma: str = "comma"
        case .deleteForward: str = "deleteForward"
        case .deleteOrBackspace: str = "deleteOrBackspace"
        case .downArrow: str = "downArrow"
        case .eight: str = "eight"
        case .end: str = "end"
        case .equalSign: str = "equalSign"
        case .escape: str = "escape"
        case .five: str = "five"
        case .four: str = "four"
        case .graveAccentAndTilde: str = "graveAccentAndTilde"
        case .home: str = "home"
        case .hyphen: str = "hyphen"
        case .insert: str = "insert"
        case .international1: str = "international1"
        case .international2: str = "international2"
        case .international3: str = "international3"
        case .international4: str = "international4"
        case .international5: str = "international5"
        case .international6: str = "international6"
        case .international7: str = "international7"
        case .international8: str = "international8"
        case .international9: str = "international9"
        case .keyA: str = "keyA"
        case .keyB: str = "keyB"
        case .keyC: str = "keyC"
        case .keyD: str = "keyD"
        case .keyE: str = "keyE"
        case .keyF: str = "keyF"
        case .keyG: str = "keyG"
        case .keyH: str = "keyH"
        case .keyI: str = "keyI"
        case .keyJ: str = "keyJ"
        case .keyK: str = "keyK"
        case .keyL: str = "keyL"
        case .keyM: str = "keyM"
        case .keyN: str = "keyN"
        case .keyO: str = "keyO"
        case .keyP: str = "keyP"
        case .keyQ: str = "keyQ"
        case .keyR: str = "keyR"
        case .keyS: str = "keyS"
        case .keyT: str = "keyT"
        case .keyU: str = "keyU"
        case .keyV: str = "keyV"
        case .keyW: str = "keyW"
        case .keyX: str = "keyX"
        case .keyY: str = "keyY"
        case .keyZ: str = "keyZ"
        case .keypad0: str = "keypad0"
        case .keypad1: str = "keypad1"
        case .keypad2: str = "keypad2"
        case .keypad3: str = "keypad3"
        case .keypad4: str = "keypad4"
        case .keypad5: str = "keypad5"
        case .keypad6: str = "keypad6"
        case .keypad7: str = "keypad7"
        case .keypad8: str = "keypad8"
        case .keypad9: str = "keypad9"
        case .keypadAsterisk: str = "keypadAsterisk"
        case .keypadEnter: str = "keypadEnter"
        case .keypadEqualSign: str = "keypadEqualSign"
        case .keypadHyphen: str = "keypadHyphen"
        case .keypadNumLock: str = "keypadNumLock"
        case .keypadPeriod: str = "keypadPeriod"
        case .keypadPlus: str = "keypadPlus"
        case .keypadSlash: str = "keypadSlash"
        case .leftAlt: str = "leftAlt"
        case .leftArrow: str = "leftArrow"
        case .leftControl: str = "leftControl"
        case .leftGUI: str = "leftGUI"
        case .leftShift: str = "leftShift"
        case .nine: str = "nine"
        case .nonUSBackslash: str = "nonUSBackslash"
        case .nonUSPound: str = "nonUSPound"
        case .one: str = "one"
        case .openBracket: str = "openBracket"
        case .pageDown: str = "pageDown"
        case .pageUp: str = "pageUp"
        case .pause: str = "pause"
        case .period: str = "period"
        case .power: str = "power"
        case .printScreen: str = "printScreen"
        case .quote: str = "quote"
        case .returnOrEnter: str = "returnOrEnter"
        case .rightAlt: str = "rightAlt"
        case .rightArrow: str = "rightArrow"
        case .rightControl: str = "rightControl"
        case .rightGUI: str = "rightGUI"
        case .rightShift: str = "rightShift"
        case .scrollLock: str = "scrollLock"
        case .semicolon: str = "semicolon"
        case .seven: str = "seven"
        case .six: str = "six"
        case .slash: str = "slash"
        case .spacebar: str = "spacebar"
        case .tab: str = "tab"
        case .three: str = "three"
        case .two: str = "two"
        case .upArrow: str = "upArrow"
        case .zero: str = "zero"
        default: str = "???"
        }
        return str
    }
}