It became possible to take screenshots by starting the smartphone (Android terminal) application with Python using Appium on Windows 10 and automating the screen transition operation.

Introduction

The starting point for me to do this is that our development members work from 5 am on weekdays and the rate of work on Saturdays and Sundays is increasing, which has become a problem. When asked why, the main thing was "** Check the operation of the smartphone app in real time ** in the comprehensive test (on the network that can not be accessed unless you go to work **". .. .. So, I wondered if it would be possible to automatically operate this smartphone app on an in-house PC at a certain time, and to take a screenshot of the screen ** without going to work. , I searched and implemented it.

What I was able to do

・ I was able to automatically start and operate the smartphone (actual device / Android device) app from my computer. -Take a screenshot and store it on a file server ・ Perform this regularly by specifying a date and time such as Saturday, Sunday, or morning.

[Constraints] ・ Keep your computer running ・ Set the smartphone so that it does not go to sleep and keep it running.

With the configuration of ↓, I was able to do well.

・ The OS of the personal computer is Windows 10. ・ Smartphone is an Android OS terminal ・ USB data communication cable is required (* It is different from charging. Please note) (* This time, the smartphone was connected to the company's comprehensive environment for the purpose, but of course, automatic operation is possible with the same configuration even when connecting to the Internet (WiFi, etc.).)

The tools and programming languages ​​used on the PC are as follows. You need to install these on your PC.

・ Python ・ Appium (GUI) ・ Other items required for Appium to operate -Task scheduler (* installed on Windows 10 from the beginning)

Environmental preparation

Install the Appium GUI version on your Windows 10 PC. The reason I recommend the GUI version is that it makes programming much easier. When operating, it may be okay to use the CUI version.

For environment preparation, I referred to the following site. https://qiita.com/emurin/items/fe640327f847c474f1e8

① Java installation -Set the environment variable "JAVA_HOME" (Example: Set "C: \ Program Files \ Java \ jdk-15. **") -Set to the environment variable "path" (example: set "% JAVA_HOME% \ bin") -To be on the safe side, check that the command can be executed with "java -versioin" etc. at the command prompt.

② Install Python ・ Download the installer -When you start the installer, you can check something like "Add Python 3.x to PATH" to automatically add it to the environment variable "path". -Execute "pip install Appium-Python-Client" at the command prompt (* This installs the library to be used later.)

③ Install Android Studio -When you start it and make the initial settings, the installation will run, and a directory called "Android" will be created in "C: \ Users \ **** \ AppData \ Local" and installed. -Set to the environment variable "ANDROID_HOME" (Example: "C: \ Users \ **** \ AppData \ Local \ Android \ Sdk") -Set to the environment variable "path" (Example: Set "% ANDROID_HOME% \ platform-tools") ・ Here, just in case, type "adb devices" etc. and confirm that the response is returned.

④ Install Appium GUI for Windows OK if you start as follows

Try connecting with a smartphone

Connect your PC and smartphone with a USB data communication cable.

The screen looks like this.

From Settings, open the System menu and open Developer Options.

If this "Developer option" is not displayed, tap the "Build number" part of "Terminal information" 5 or 6 times to temporarily display the "Developer option". Should be.

Open "Developer Options", check "ON", Check "Do not put into sleep mode" and "USB debugging".

At the command prompt, type "adb devices" and the following will be displayed. Now you are ready.

Connect from Appium GUI client

From here is the production. First, start the GUI client of Appium and try to access the screen of the smartphone.

First, start the Appium GUI client on your PC and click "Start Server v *****".

Then, the Appium server will be started.

Here, press the magnifying glass button on the upper right.

It will take some time to start, but you should see a screen like ↓. Therefore, I will describe where I want to connect at the bottom left of the screen, but this time there is only one Android terminal, so for the time being, a feeling like ↓ is enough. ↓ Please use for copy and paste

{
  "platformName": "Android"
}

It is recommended to save it as it will be used repeatedly.

Then press the "Start Session" button at the bottom right.

The screen of the smartphone is displayed like this. But it is amazing!

Programming in Python with Appium

In order to operate the smartphone app automatically, the ID or location (xpath) of each element is required. This is of course difficult to know unless you are a developer, but with the ** Appium GUI client, you can know the ID and location (xpath) even if you are not the developer. ** **

The PC and smartphone are connected, and in the Appium GUI client, click the icon "Start Recording" like the ↓ eye. Then, it will be as if you are recording like ↓. In this state, if you select the "Calculator" app icon on the smartphone screen on the left side of the screen, that element will be displayed in the lower right window, so press the "tap" button.

Then, the "Calculator" application will start on the smartphone on the PC screen, but

At this time, the smartphone (actual machine) also moves in conjunction with this.

And at the same time, can you see that the program code is being output? .. The pull-down that says "Python" in the upper right corner of the screen can be changed to "Java" or "Ruby".

In the same way, this time, select the "1" button and click "tap" in the element window on the right, and it will look like ↓.

The rest is a repetition of this.

Press the "+" button and press the "tap" button Press the "2" button and then the "tap" button Press the "=" button and then the "tap" button

With this kind of feeling, the button operation is generated as a program code. Then, click the icon button with the "arrow in the square" next to the pull-down that selects "Python" in the upper right, and the entire program code including the import statement will be generated.

↓ Here is the generated code.

# This sample code uses the Appium python client
# pip install Appium-Python-Client
# Then you can paste this into a file and simply run with Python

from appium import webdriver

caps = {}
caps["platformName"] = "android"
caps["ensureWebviewsHavePages"] = True

driver = webdriver.Remote("http://localhost:4723/wd/hub", caps)

el1 = driver.find_element_by_accessibility_id("calculator")
el1.click()
el2 = driver.find_element_by_id("com.google.android.calculator:id/digit_1")
el2.click()
el3 = driver.find_element_by_accessibility_id("Add")
el3.click()
el4 = driver.find_element_by_id("com.google.android.calculator:id/digit_2")
el4.click()
el5 = driver.find_element_by_accessibility_id("equal")
el5.click()

driver.quit()

Copy and paste this into a text file, and save it with an appropriate file name with a ".py" extension. If you save it with the file name "test01.py", you can execute it with the following command from the command prompt.

> python [path]/test01.py

Write in the program code

Add processing code such as taking screenshots, waiting, or assertions to the "test01.py (provisional)" created earlier as needed.

screenshot

You can take a screenshot below. Insert it where you need it.

driver.save_screenshot('[path]/screenshoto1.png')

Wait

If you want to temporarily stop the process due to screen transition or data load time, you can sleep it. The following is an example of waiting for 2 seconds. Insert it where you need it.

sleep(2)

Wait until an element is found

In the case of sleep processing, it is inefficient because it forces you to wait for that time. So you can also write a method of waiting until an element is found.

There are several ways to write it, but an example is ↓. The specification in xpath is taken as an example.

def find_element_by_xpath_ex(xpath):
    """
0 until element found.Repeatedly inspect with a 5-second sleep, and return the element as soon as it is found.
After repeating, if it is not found even after sleeping for 10 seconds or more, None is returned.
    """
    tmp = 0
    el = None
    while True:
        try:
            el = driver.find_element_by_xpath(xpath)
            if el is not None:
                break
        except NoSuchElementException:
            if tmp > 10:
                break
            sleep(0.5)
            tmp += 0.5
    return el

The app exit does not work well, so put the app in the background

In order to execute it repeatedly on a regular basis, it is necessary to return to the initial state (in this case, the state where the "Telephone District" icon can be tapped). For that, I want to quit the app, but it didn't return no matter what I did, such as "driver.quit ()" or "driver.close_app ()" (´; ω; `) That's why I made it to the initial state by moving the ↓ application to the background.

driver.background_app(-1)

Regular execution

For periodic execution, create a bat file in which the python command "> python [path] /test01.py" is written, and register the time etc. with the task scheduler installed as standard in Windows. This makes it possible to execute it regularly.

Regarding re-execution when an error occurred, the task scheduler did not work, so I ended up programming in the bat file. This is described in the previous article ↓. https://qiita.com/jun_higuche/items/523e11502b5e0847571d

Other app operations

This time, I explained it with the "Calculator" application, so I only selected the element (button) and tapped it, but there are other "swipe" and "tap without specifying the element".

"Swipe" is possible from here. It will be implemented by specifying two points, the start position and the end position.

"Tap without specifying an element (tap by specifying a location on the X-axis and Y-axis on the screen)" is possible from here. Of course, it depends on the actual machine, so I think you should not use it much.

In addition, there is a tab called Actions, from which you can perform various operations.

There is a possibility that the operation on the Appium client is not all, so let's think about how to check it in the API document below and write it manually. http://appium.io/docs/en/about-appium/api/ It's very easy to understand because it also describes how to write in each programming language.

Operation that is unlikely to be possible

Pinch-in / pinch-out (the action of using two fingers to zoom in and out) is missing and may not be possible. Other than that, it seems unsuitable for operations such as tapping a small object and dragging it. It seems that the ideal creation is that an object is assigned an ID, that can be specified, and a numerical value can be given by parameters.

Finally

For the time being, I wanted to perform automatic operation on an Android device, so I connected it to a Windows 10 PC to enable automatic operation. I've been spinning it for about two weeks now and I've got screenshots without any problems! Really wonderful and stable operation! That's because I think about various patterns (when an error occurs at startup, when it starts from an unexpected screen after the error, etc.), and something like "In the case of such a screen, do this kind of operation." I'm making a lot of it.

In the future, there is a demand for automatic operation tests on iPhone as well, so I will try it on iMac + iPhone. Probably almost the same, but ... Again, if it works, I'll write an article on Qiita again.

See you again. I hope it helps someone m (_ _) m