How to automatically operate a screen created in Java on Windows
Preface
Most Windows screens are automatically manipulated using the methods described below.
** Automation of "Akanchan Kawaiitta" by RPA Nine People ** https://qiita.com/mima_ita/items/d4655de865f30bb51c65
Actually, there is a troublesome case, which is the case of making a screen in Java. This time, let's consider whether automatic operation is possible using a screen made in Java as an example.
Experiment environment Windows10 Home Java 8 Visual Studio 2019 PowerShell 5.1 UiPath 2019.10.0-beta 111
How to create a Java screen
Swing may be used as the main method for creating screens in Java, or JavaFx may be used.
Screen created by Swing
Create a simple Swing screen by referring to the page below.
ToDoListPane.java
package SwingSample;
import java.awt.BorderLayout;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import javax.swing.DefaultListModel;
import javax.swing.JButton;
import javax.swing.JList;
import javax.swing.JPanel;
import javax.swing.JScrollPane;
import javax.swing.JTextField;
/**
*ToDo list
*Reference below
* https://www.atmarkit.co.jp/ait/articles/0609/23/news027.html
*/
public class ToDoListPane extends JPanel {
private JList<String> toDoList;
private DefaultListModel<String> toDoListModel;
private JTextField toDoInputField;
private JButton addButton;
public ToDoListPane() {
super(new BorderLayout());
//Generate list
toDoListModel = new DefaultListModel<String>();
toDoList = new JList<String>(toDoListModel);
JScrollPane listScrollPane = new JScrollPane(toDoList);
//Generate text field for adding tasks
toDoInputField = new JTextField();
//Generation of each button
JPanel buttonPanel = new JPanel();
addButton = new JButton("add to");
//Set listener on button
addButton.addActionListener(new AddActionHandler());
buttonPanel.add(addButton);
add(listScrollPane, BorderLayout.NORTH);
add(toDoInputField, BorderLayout.CENTER);
add(buttonPanel, BorderLayout.SOUTH);
}
/**
*Handler for additional button actions
*/
private class AddActionHandler implements ActionListener {
public void actionPerformed(ActionEvent e) {
//Add text field content to list model
toDoListModel.addElement
(toDoInputField.getText());
}
}
}
All the code is below. https://github.com/mima3/testjavagui/tree/master/java/Swing001
Screen created with JavaFx
JavaFx also creates a simple screen.
Main.fxml
<?xml version="1.0" encoding="UTF-8"?>
<?import java.lang.*?>
<?import javafx.scene.control.*?>
<?import javafx.scene.layout.*?>
<?import javafx.scene.layout.AnchorPane?>
<AnchorPane xmlns:fx="http://javafx.com/fxml/1" xmlns="http://javafx.com/javafx/2.2" fx:controller="ctrl.Controller">
<!-- TODO Add Nodes -->
<children>
<Pane layoutX="0.0" layoutY="-14.0" prefHeight="297.0" prefWidth="345.0">
<children>
<Label layoutX="14.0" layoutY="14.0" text="list" />
<ListView id="" fx:id="list" layoutX="14.0" layoutY="30.0" prefHeight="198.0" prefWidth="317.0" />
<Button id="" fx:id="btnAdd" layoutX="14.0" layoutY="262.0" mnemonicParsing="false" onAction="#onAddButtonClicked" text="add to" />
<TextField id="" fx:id="textBox" layoutX="14.0" layoutY="228.0" prefHeight="15.9609375" prefWidth="317.0" />
</children>
</Pane>
</children>
</AnchorPane>
Controler.java
package ctrl;
import java.net.URL;
import java.util.ResourceBundle;
import javafx.event.ActionEvent;
import javafx.fxml.FXML;
import javafx.fxml.Initializable;
import javafx.scene.control.Button;
import javafx.scene.control.ListView;
import javafx.scene.control.TextField;
public class Controller implements Initializable {
@FXML
private TextField textBox;
@FXML
private Button btnAdd;
@FXML
private ListView<String> list;
@Override
public void initialize(URL location, ResourceBundle resources) {
//TODO auto-generated method stub
textBox.setText("please fill in the value.");
}
@FXML
public void onAddButtonClicked(ActionEvent event) {
//Set a character string in the text box
list.getItems().add(textBox.getText());
textBox.setText("");
}
}
All the code is below. https://github.com/mima3/testjavagui/tree/master/java/Java8Fx001
Precautions when creating a JavaFx screen with Java 11
JavaFX has been separated from the Oracle JDK since JDK 11. Therefore, when creating a JavaFx screen, the following procedure is required.
(1) Download JavaFX. https://gluonhq.com/products/javafx/
(2) Add the jar in lib in the downloaded folder to the reference library of the project.
(3) At runtime ** When running from the command line **
C:\pleiades201904\java\11\bin\java --module-path=C:\tool\lib\javafx-sdk-11.0.2\lib\ --add-modules=javafx.controls --add-modules=javafx.swing --add-modules=javafx.base --add-modules=javafx.fxml --add-modules=javafx.media --add-modules=javafx.web -jar Java11Fx.jar
** Execution configuration when running in Eclipse **
Automatic operation of UI Automation
Check if the created Java screen can be operated via UI Automation using inspect.exe ..
For Swing:
You can see that UI Automation has not retrieved the control information. In other words, ** Applications created with Swing cannot be operated via UI Automation **.
For JavaFx:
You can see that the elements of UIAutomation have been acquired and the ControlType has been set appropriately. Let's actually use PowerShell to perform automatic operations.
Add-Type -AssemblyName UIAutomationClient
Add-Type -AssemblyName UIAutomationTypes
Add-type -AssemblyName System.Windows.Forms
$source = @"
using System;
using System.Windows.Automation;
using System.Runtime.InteropServices;
using System.Windows.Forms;
using System.Drawing;
public class AutomationHelper
{
// https://culage.hatenablog.com/entry/20130611/1370876400
[DllImport("user32.dll")]
extern static uint SendInput(uint nInputs, INPUT[] pInputs, int cbSize);
[StructLayout(LayoutKind.Sequential)]
struct INPUT
{
public int type;
public MOUSEINPUT mi;
}
[StructLayout(LayoutKind.Sequential)]
struct MOUSEINPUT
{
public int dx;
public int dy;
public int mouseData;
public int dwFlags;
public int time;
public IntPtr dwExtraInfo;
}
const int MOUSEEVENTF_LEFTDOWN = 0x0002;
const int MOUSEEVENTF_LEFTUP = 0x0004;
static public void Click()
{
//Declaration of struct array
INPUT[] input = new INPUT[2];
//Left button Down
input[0].mi.dwFlags = MOUSEEVENTF_LEFTDOWN;
//Left button Up
input[1].mi.dwFlags = MOUSEEVENTF_LEFTUP;
//Batch generation of events
SendInput(2, input, Marshal.SizeOf(input[0]));
}
static public void MouseMove(int x, int y)
{
var pt = new System.Drawing.Point(x, y);
System.Windows.Forms.Cursor.Position = pt;
}
static public void SendKeys(string key)
{
System.Windows.Forms.SendKeys.SendWait(key);
}
public static AutomationElement RootElement
{
get
{
return AutomationElement.RootElement;
}
}
public static AutomationElement GetMainWindowByTitle(string title) {
PropertyCondition cond = new PropertyCondition(AutomationElement.NameProperty, title);
return RootElement.FindFirst(TreeScope.Children, cond);
}
public static AutomationElement ChildWindowByTitle(AutomationElement parent , string title) {
try {
PropertyCondition cond = new PropertyCondition(AutomationElement.NameProperty, title);
return parent.FindFirst(TreeScope.Children, cond);
} catch {
return null;
}
}
public static AutomationElement WaitChildWindowByTitle(AutomationElement parent, string title, int timeout = 10) {
DateTime start = DateTime.Now;
while (true) {
AutomationElement ret = ChildWindowByTitle(parent, title);
if (ret != null) {
return ret;
}
TimeSpan ts = DateTime.Now - start;
if (ts.TotalSeconds > timeout) {
return null;
}
System.Threading.Thread.Sleep(100);
}
}
}
"@
Add-Type -TypeDefinition $source -ReferencedAssemblies("UIAutomationClient", "UIAutomationTypes", "System.Windows.Forms", "System.Drawing")
# 5.If it is 0 or later, it is easier to describe it with using.
$autoElem = [System.Windows.Automation.AutomationElement]
#List all controls that meet the specified conditions below the window
function findAllElements($form, $condProp, $condValue) {
$cond = New-Object -TypeName System.Windows.Automation.PropertyCondition($condProp, $condValue)
return $form.FindAll([System.Windows.Automation.TreeScope]::Element -bor [System.Windows.Automation.TreeScope]::Descendants, $cond)
}
#Get one control that meets the specified conditions below the window
function findFirstElement($form, $condProp, $condValue) {
$cond = New-Object -TypeName System.Windows.Automation.PropertyCondition($condProp, $condValue)
return $form.FindFirst([System.Windows.Automation.TreeScope]::Element -bor [System.Windows.Automation.TreeScope]::Descendants, $cond)
}
#Convert elements to Value Pattern
function convertValuePattern($elem) {
return $elem.GetCurrentPattern([System.Windows.Automation.ValuePattern]::Pattern) -as [System.Windows.Automation.ValuePattern]
}
function convertSelectionItemPattern($elem) {
return $elem.GetCurrentPattern([System.Windows.Automation.SelectionItemPattern]::Pattern) -as [System.Windows.Automation.SelectionItemPattern]
}
#Enter text in the element
#TxtValuePtn for Java 8.Alternative for SetValue not working properly
function sendTextValue($textCtrl, $message) {
[AutomationHelper]::MouseMove($textCtrl.Current.BoundingRectangle.X + 5, $textCtrl.Current.BoundingRectangle.Y + 5)
[AutomationHelper]::Click()
[AutomationHelper]::SendKeys("^(a)")
[AutomationHelper]::SendKeys("{DEL}")
[AutomationHelper]::SendKeys($message)
Start-Sleep 1
}
#Main processing
$mainForm = [AutomationHelper]::GetMainWindowByTitle("TODO list")
if ($mainForm -eq $null) {
Write-Error "Launch the JavaFx screen"
exit 1
}
$mainForm.SetFocus()
$editType = [System.Windows.Automation.ControlType]::Edit
$textCtrl = findFirstElement $mainForm $autoElem::ControlTypeProperty $editType
#In case of Java8, an error occurs in SetValue of ValuePattern
#$txtValuePtn = convertValuePattern $textCtrl
#$txtGetValue = $txtValuePtn.Current.Value
#Write-Host "Change before:$txtGetValue"
#$txtValuePtn.SetValue("Wafuru");
sendTextValue $textCtrl "Waffle"
$btnCtrl = findFirstElement $mainForm $autoElem::NameProperty "add to"
$btnInvoke = $btnCtrl.GetCurrentPattern([System.Windows.Automation.InvokePattern]::Pattern) -as [System.Windows.Automation.InvokePattern]
$btnInvoke.Invoke()
#2nd character
sendTextValue $textCtrl "Cat"
$btnInvoke.Invoke()
#3rd character
sendTextValue $textCtrl "dog"
$btnInvoke.Invoke()
#List selection
$listitemType = [System.Windows.Automation.ControlType]::ListItem
$listitems = findAllElements $mainForm $autoElem::ControlTypeProperty $listitemType
$listPtn = convertSelectionItemPattern $listitems[1]
$listPtn.Select()
Execution result
When this is executed, the screen using JavaFx of Java11 will be completed normally, but the screen using JavaFx of Java8 will output the following error.
Error when operating JavaFx with UI Automation
When setting a value with Value Pattern of Ui Automation for JavaFx created with Java8, the following error appears.
** PowerShell side **
"1"Specifying the number of arguments"SetValue"Exception occurred while calling: ""
Occurrence location C:\dev\testjavagui\out\javafx_auto_err.ps1:146 characters:1
+ $txtValuePtn.SetValue("Wafuru");
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : COMException
** Java side **
Exception in thread "JavaFX Application Thread" java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer
at javafx.scene.control.TextInputControl.executeAccessibleAction(TextInputControl.java:1590)
at javafx.scene.Node$19.executeAction(Node.java:9649)
at com.sun.glass.ui.Accessible$ExecuteAction.run(Accessible.java:177)
at com.sun.glass.ui.Accessible$ExecuteAction.run(Accessible.java:173)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.glass.ui.Accessible.lambda$executeAction$5(Accessible.java:190)
at com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:389)
at com.sun.glass.ui.Accessible.executeAction(Accessible.java:187)
at com.sun.glass.ui.win.WinAccessible.SetValueString(WinAccessible.java:1262)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$null$152(WinApplication.java:177)
at java.lang.Thread.run(Thread.java:748)
This error does not occur for JavaFx created with Java 11.
Automatic operation using Java Access Bridge
Use the Java Access Bridge (https://docs.oracle.com/javase/10/access/java-access-bridge-architecture.htm#JSACC-GUID-FAFC50E6-DEFD-4808-9E04-65AD717F33D6) Windows will be able to operate the Java GUI.
Be aware of whether the process that uses Java or Java Access Bridge is 32bit or 64bit.
Exploring GUI elements using Java Access Bridge
First, I will explain how to use Access Bridge Explorer, which is a GUI element exploration tool using Java Access Bridge.
(1) Enable Java Access Bridge
%JRE_HOME%\bin\jabswitch -enable
(2) Confirm that WindowsAccessBridge-64.dll exists in% JRE_HOME% \ jre \ bin, and add% JRE_HOME% \ jre \ bin to the environment variable PATH. If it is an older version, you need to download it from the following. https://www.oracle.com/technetwork/java/javase/tech/index-jsp-136191.html
(3) Download Access Bridge Explorer from the following. https://github.com/google/access-bridge-explorer/releases
(4) Start Access Bridge Explorer.
- With Java Access Bridge, you can only operate screens created with Swing. It is impossible to operate the screen created by JavaFx.
Working with Java Access Bridge from .NET
A sample of operating Java Access Bridge from .NET was published below. https://github.com/jdog3/JavaAccessBridge.Net-Sample
Based on the above, the following is a sample that enables click operation and operation from the console application. https://github.com/mima3/testjavagui/tree/master/cs
using JabApiLib.JavaAccessBridge;
using System;
using System.Collections.Generic;
using System.Text;
namespace JabApiCsharpSample
{
class Program
{
static void Main(string[] args)
{
//JabApi.Windows_run();
JabHelpers.Init();
int vmID = 0;
JabHelpers.AccessibleTreeItem javaTree = null;
javaTree = JabHelpers.GetComponentTreeByTitle("ToDo list", out vmID);
//Text settings
JabHelpers.AccessibleTreeItem txt = javaTree.children[0].children[1].children[0].children[0].children[1];
JabApi.setTextContents(vmID, txt.acPtr, "Warosuwarosu");
JabHelpers.AccessibleTreeItem button = javaTree.children[0].children[1].children[0].children[0].children[2].children[0];
List<string> actionList = JabHelpers.GetAccessibleActionsList(vmID, button.acPtr);
Console.WriteLine("Operable actions-------------");
foreach (string a in actionList)
{
Console.WriteLine(a);
}
//Click execute
JabHelpers.DoAccessibleActions(vmID, button.acPtr, "click");
//
JabApi.setTextContents(vmID, txt.acPtr, "Irohanihohe");
JabHelpers.DoAccessibleActions(vmID, button.acPtr, "click");
//
JabApi.setTextContents(vmID, txt.acPtr, "Dust slimy");
JabHelpers.DoAccessibleActions(vmID, button.acPtr, "click");
//List contents
Console.WriteLine("List list-------------");
javaTree = JabHelpers.GetComponentTreeByTitle("ToDo list", out vmID);
JabHelpers.AccessibleTreeItem list = javaTree.children[0].children[1].children[0].children[0].children[0].children[0].children[0];
foreach (JabHelpers.AccessibleTreeItem listitem in list.children)
{
Console.WriteLine(listitem.name );
}
JabHelpers.DoAccessibleActions(vmID, list.children[1].acPtr, "click");
Console.ReadLine();
}
}
}
The operations that can be performed with DoAccessibleActions are different for each control, and you can find out what you can do with GetAccessibleActions. In JabApi, API of Java Access Bridge The functions that call are collectively implemented. This time it is assumed that it is running on 64-bit, so change the following line in JabApi.cs if necessary.
public static class JabApi
{
public const String WinAccessBridgeDll = @"WindowsAccessBridge-64.dll";
In addition, Windows_run, which is the initial processing of Java Access Bridge, requires a message pump, and if the message is not processed, the subsequent processing will not operate normally. That's why the original .NET to Java Access Bridge Operation Sample says that Windows_run must be included when FormLoad. .. This time, DoEvents is executed after Windows_run as follows so that it works on the console.
// Windows_run needs a message pump
// https://stackoverflow.com/questions/50582769/windowsaccessbridge-for-java-automation-using-c-sharp
public static void Init()
{
JabApi.Windows_run();
DoEvents();
}
Execution result
Example in PowerShell
You can write a script that performs the same operation in PowerShell based on C #. The JabApi.dll you are using should be Download or Source Code. / testjavagui / tree / master / cs ) Please compile. The DLL listed on GitHub is 64bit + .NET 2.0, so it cannot be used depending on the environment.
#64bit premise
$dllPath = Split-Path $MyInvocation.MyCommand.Path
Set-Item Env:Path "$Env:Path;$dllPath"
Add-Type -Path "$dllPath\JabApi.dll"
[JabApiLib.JavaAccessBridge.JabHelpers]::init()
$vmID = 0
$javaTree = [JabApiLib.JavaAccessBridge.JabHelpers]::GetComponentTreeByTitle("ToDo list",[ref]$vmID)
$txt = $javaTree.children[0].children[1].children[0].children[0].children[1]
[JabApiLib.JavaAccessBridge.JabApi]::setTextContents($vmID, $txt.acPtr, "Warosuwarosu")
#click
$button = $javaTree.children[0].children[1].children[0].children[0].children[2].children[0]
[JabApiLib.JavaAccessBridge.JabHelpers]::DoAccessibleActions($vmID, $button.acPtr, "click")
#
[JabApiLib.JavaAccessBridge.JabApi]::setTextContents($vmID, $txt.acPtr, "Ahhhh")
[JabApiLib.JavaAccessBridge.JabHelpers]::DoAccessibleActions($vmID, $button.acPtr, "click")
#
[JabApiLib.JavaAccessBridge.JabApi]::setTextContents($vmID, $txt.acPtr, "Good")
[JabApiLib.JavaAccessBridge.JabHelpers]::DoAccessibleActions($vmID, $button.acPtr, "click")
#Check for updates
$javaTree = [JabApiLib.JavaAccessBridge.JabHelpers]::GetComponentTreeByTitle("ToDo list",[ref]$vmID)
$list = $javaTree.children[0].children[1].children[0].children[0].children[0].children[0].children[0]
foreach($item in $list.children) {
Write-Host $item.name
}
[JabApiLib.JavaAccessBridge.JabHelpers]::DoAccessibleActions($vmID, $list.children[1].acPtr, "click")
For UIPath
You can operate Java GUI by installing Java extension from the tool.
When the extension is installed, UiPathJavaBridgeV8_x64.dll is stored in "% JRE_HOME% \ bin ".
After installing the extension, you will be able to create screens as usual.
Execution result
Other options
It may be possible to perform automatic operations using the GUI test framework. Since it was different from the purpose of this time, I have not investigated it in detail.
Automation Automation is a framework that makes it easy to test Swing and JavaFx GUIs.
It can be written in Java, but it can also be written in a Groovy script like the one below.
clickOn 'text:Some Button'
doubleClickOn 'username-input'
type 'my-username'
clickOn 'text:Login'
TestFX A simple and clean testing framework for JavaFX. https://github.com/TestFX/TestFX
AssertJ Swing AssertJ Swing seems to be able to test Swing's GUI. Now it's a fork of Fest Swing.
Java Swing UI test driver replacement for Fest [closed] https://stackoverflow.com/questions/31168990/java-swing-ui-test-driver-replacement-for-fest
reference
- Javafx Listview Add and edit element
- How do I use JavaFX 11 in Eclipse? -Run JavaFX with OpenJFX + OpenJDK -Preparing the environment for java11 and javaFx with Ubuntu 18.4 -[For super beginners] JavaFX super introduction
- Java Access Bridge Architecture
- Java Swing UI test driver replacement for Fest [closed]
- Automation
- Two simple tools to let you test your Swing/JavaFX applications easily
- TestFx
- WindowsAccessBridge for Java Automation using C#
Recommended Posts