UI Automation Part 2 in Python

Continuation of UI Automation in Python This time I will click the button.

ControlPattern Controls that support UI Automation provide an operable interface in the form of Control Pattern. In the case of buttons, InvokePattern is supported, so if you get this and call Invoke (), which is a method of InvokePattern, you can realize button click. You can find out if an AutomationElement supports InvokePattern by looking at the element's IsInvokePatternAvailable.

def click_button(element):
    """IUIAutomationInvokePattern for the specified element.Invoke()Click with

If the IsInvokePatternAvailable property of the specified element is False, do nothing
    """
    isClickable = element.GetCurrentPropertyValue(UIA_IsInvokePatternAvailablePropertyId)
    if isClickable == True:
        ptn = element.GetCurrentPattern(UIA_InvokePatternId)
        ptn.QueryInterface(IUIAutomationInvokePattern).Invoke()

Details are below. UI Automation Control Patterns Overview http://msdn.microsoft.com/en-us/library/ee671194(v=vs.85).aspx#controlpatterninterfaces

The .NET version has a Japanese translation (although it is a machine translation). UI Automation Control Pattern Overview http://msdn.microsoft.com/ja-jp/library/ms752362(v=vs.110).aspx

Preparation

Before actually clicking the button, put it together, including the previous ones.

uiauti.py


# -*- coding: utf-8 -*-
"""
uiautil : UI Automation utility

It only works on Windows.
"""
import comtypes
from comtypes import CoCreateInstance
import comtypes.client
from comtypes.gen.UIAutomationClient import *

__uia = None
__root_element = None

def __init():
    global __uia, __root_element
    __uia = CoCreateInstance(CUIAutomation._reg_clsid_,
                             interface=IUIAutomation,
                             clsctx=comtypes.CLSCTX_INPROC_SERVER)
    __root_element = __uia.GetRootElement()
    
def get_window_element(title):
    """Gets the AutomationElement of the window specified in title
    
If the title is duplicated on the desktop, return the first one found
    """
    win_element = __root_element.FindFirst(TreeScope_Children,
                                           __uia.CreatePropertyCondition(
                                           UIA_NamePropertyId, title))
    return win_element

def find_control(base_element, ctltype):
    """Has the specified control type ID from the specified base element subtree
Returns a sequence of elements
    """
    condition = __uia.CreatePropertyCondition(UIA_ControlTypePropertyId, ctltype)
    ctl_elements = base_element.FindAll(TreeScope_Subtree, condition)
    return [ ctl_elements.GetElement(i) for i in range(ctl_elements.Length) ]
   
def lookup_by_name(elements, name):
    """Elements with the specified Name property from the specified sequence of elements
Returns the first of them.
Returns None if there is no hit

   """
    for element in elements:
        if element.CurrentName == name:
            return element

    return None

def lookup_by_automationid(elements, id):
    """Has the specified AutomationId property from the specified sequence of elements
Returns the first of the elements.
Returns None if there is no hit

   """
    for element in elements:
        if element.CurrentAutomationId == id:
            return element
    return None
    
def click_button(element):
    """IUIAutomationInvokePattern for the specified element.Invoke()Click with

If the IsInvokePatternAvailable property of the specified element is False, do nothing
    """
    isClickable = element.GetCurrentPropertyValue(UIA_IsInvokePatternAvailablePropertyId)
    if isClickable == True:
        ptn = element.GetCurrentPattern(UIA_InvokePatternId)
        ptn.QueryInterface(IUIAutomationInvokePattern).Invoke()


if __name__ == '__main__':
    __init()

Try to calculate automatically with a calculator

Test by running uiautil.py on IDLE. Start the calculator and decide the property that identifies each button. This time I will use AutomationId.

>>> 
>>> win = get_window_element('calculator')
>>> btns = find_control(win, UIA_ButtonControlTypeId)
>>> for b in btns: print btns.index(b), b.CurrentName, b.CurrentAutomationId

0 Clear memory 122
1 backspace 83
2 7 137
3 4 134
4 1 131
5 0 130
<Omission>
21 subtraction 94
22 Addition 93
23 Memory subtraction 126
24 square root 110
25 percent 118
26 Reciprocal 114
27 equal sign 121
28 Minimize
29 Maximize
30 close
>>> def one_plus_one():
	btn_1 = lookup_by_automationid(btns, "131")
	btn_plus = lookup_by_automationid(btns, "93")
	btn_equal = lookup_by_automationid(btns, "121")
	click_button(btn_1)
	click_button(btn_plus)
	click_button(btn_1)
	click_button(btn_equal)

	
>>> one_plus_one()
>>> 

2 was displayed on the calculator!

This time I am running on Windows 8.1 Pro (x64), but as shown below, it seems that AutomaionId is not guaranteed between builds and releases, so AutomaionId may be different on other OS such as Windows 7. Also, there are cases where AutomationId is not assigned as in minimization and maximization.

From the Description of UIA_AutomationIdPropertyId in http://msdn.microsoft.com/en-us/library/ee684017(v=vs.85).aspx.

Although support for AutomationId is always recommended for better automated testing support, this property is not mandatory. Where it is supported, AutomationId is useful for creating a test automation script that runs regardless of the UI language. Clients should make no assumptions regarding the AutomationId values exposed by other applications. AutomationId is not guaranteed to be stable across different releases or builds of an application.

How to check Automation Element

It seems that the basis for specifying AutomationElement depends on the creation of the application to be operated. Therefore, you will want to find out in advance what properties the Automation Elements of the operation target application have. As in the example above, the method of actually hitting AutomationElement one by one is troublesome, so I searched for a tool that could be easily investigated. there were.

It's also in the Windows SDK from the beginning.

You can use Inspect.exe and Visual UI Automation Verify included in the Windows SDK to check the properties of each element in the AutomationElement tree under the desktop.

Windows Software Development Kit (SDK) for Windows 8.1 http://msdn.microsoft.com/en-us/windows/desktop/bg162891.aspx

If you install the Windows SDK, you will see the following (xxx is x86, x64, arm. The version may differ depending on the version of the SDK you have installed). Visual Studio is not required.

%ProgramFiles%\Windows Kits\8.1\bin\xxx

For details on how to use the tool, see below. Inspect http://msdn.microsoft.com/en-us/library/windows/desktop/dd318521(v=vs.85).aspx

Visual UI Automation Verify http://msdn.microsoft.com/en-us/library/windows/desktop/jj160544(v=vs.85).aspx

Recommended Posts

UI Automation Part 2 in Python
UI Automation in Python
Transpose CSV files in Python Part 1
[Automation with python! ] Part 1: Setting file
[Automation with python! ] Part 2: File operation
Quadtree in Python --2
Basic Linear Algebra Learned in Python (Part 1)
QGIS + Python Part 2
Python in optimization
CURL in python
Metaprogramming in Python
Python 3.3 in Anaconda
Geocoding in python
SendKeys in Python
GUI creation in python using tkinter part 1
AM modulation and demodulation in Python Part 2
Meta-analysis in Python
Unittest in python
QGIS + Python Part 1
Epoch in Python
Discord in Python
Draw a heart in Python Part 2 (SymPy)
Sudoku in Python
DCI in Python
quicksort in python
N-Gram in Python
Programming in python
Plink in Python
Constant in python
Lifegame in Python.
FizzBuzz in Python
Sqlite in python
Python: Scraping Part 1
StepAIC in Python
N-gram in python
LINE-Bot [0] in Python
Csv in python
Disassemble in Python
Reflection in Python
Constant in python
nCr in Python.
format in python
Scons in Python3
Puyo Puyo in python
python in virtualenv
PPAP in Python
Quad-tree in Python
Reflection in Python
Chemistry in Python
Hashable in python
DirectLiNGAM in Python
LiNGAM in Python
Flatten in python
Python3 Beginning Part 1
Python: Scraping Part 2
flatten in python
[Automation] Extract the table in PDF with Python
[Python] Implemented automation in excel file copying work
Use Python in your environment from Win Automation
Getting rid of DICOM images in Python Part 2
Transpose CSV file in Python Part 2: Performance measurement