Seeking a unified way to wait and get for state changes in Selenium for Python elements

Introduction

Selenium provides ʻexpected_conditions` to examine the state change of an element. But,

--There is no provision to check the state change of ʻenabled / disabled --Only the type that passeslocator is prepared as the function to check the state change of clickable`. --When you get an element with XPath, you don't know what to do when you want to check the state change of an element represented by the relative path of an element.

There are many aspects that are difficult to use. The purpose of this article is to try to get rid of these.

Assumed reader

It is intended for those who have used selenium for the time being. However, it is not for advanced users. For those who know what XPATH or find_element () is.

environment

Python 3.8.3 selenium 3.141.0 geckodriver v0.26.0 Firefox 77.0.1 (64-bit)

Result source

For the time being, only the source of the result is displayed without explanation. (Although it is Python, it seems that a stone can be thrown because it is camelCase instead of snake_case) I haven't completely tested it because there are some conditional branches that I haven't used.

Also, in this article, we will proceed on the assumption that the module shown in the following source is ʻimport`.

python


import logging

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import (UnexpectedAlertPresentException, NoAlertPresentException,
                                        ElementNotVisibleException, TimeoutException, NoSuchElementException)

logger = logging.getLogger(__name__)
logger.addHandler(logging.NullHandler())

class CheckState():

  def __init__(self, locator=(), element=None, state="enabled"):
    self.locator = locator
    self.element = element
    self.state = state

  def __call__(self, driver):
    try:
      if self.element is not None and self.locator == ():
        element = self.element
      elif self.element is not None and self.locator != ():
        element = self.element.find_element(*self.locator)
      elif self.locator != ():
        element = driver.find_element(*self.locator)
      else:
        return False

      if self.state == "enabled":
        return element if element.is_enabled() == True else False
      elif self.state == "disabled":
        return element if element.is_enabled() == False else False
      elif self.state == "selected":
        return element if element.is_selected() == True else False
      elif self.state == "unselected":
        return element if element.is_selected() == False else False
      elif self.state == "displayed":
        return element if element.is_displayed() == True else False
      elif self.state == "undisplayed":
        return element if element.is_displayed() == False else False
      elif self.state == "clickable":
        if element.is_enabled() == False:
          return False
        return element if element.is_displayed() == True else False
      else:
        return False
    except Exception as e:
      logger.debug(f"CheckState: {type(e)}, {e}, {self.locator}, {self.element}, {self.state}")
      return False


def findElement(driver, locator=(), element=None, state="enabled", must=True, wait=30, interval=0.5, ignore=False):
  try:
    if element is None and locator == ():
      raise ValueError

    driverWait = WebDriverWait(driver, wait, interval)
    return driverWait.until(CheckState(locator=locator, element=element, state=state))

  except TimeoutException:
    if must == True and ignore == False:
      logger.error(f"findElement: {locator}, {element}, {state}, {must}, {wait}, {interval}, {ignore}")
      raise ValueError
    return None
  except Exception as e:
    if ignore == True:
      return None
    logger.error(f"findElement: {type(e)}, {e}")
    raise e


def isDriver(driver):
  if isinstance(driver, webdriver.remote.webdriver.WebDriver):
    return True
  return False

A function that checks the current state of an element

For example element = driver.find_element(by, value) If you get the element by

--ʻElement.is_enabled () : Whether it is ʻenabled --ʻElement.is_displayed () : Whether it is displayed on the screen --ʻElement.is_selected () : Whether it is selected --ʻElement.get_attribute (name) : Get attribute or property --ʻElement.get_property (name) : Get property --ʻElement.value_of_css_property (property_name) `: Get the value of CSS property

You can check the status with. In this article, I would like to consider the simplest ʻelement.is_enabled (), ʻelement.is_displayed (), ʻelement.is_selected ()`.

ʻExpected_conditions` function to check state changes

In ʻexpected_conditions`,

The following functions are provided to check the state change of.

--ʻEC.element_to_be_selected (element) : Whether ʻelement is ʻis_selected () == True --ʻEC.element_located_to_be_selected (locator): Whether the element indicated by locator is ʻis_selected () == True --ʻEC.element_selection_state_to_be (element, is_selected): Whether ʻelement is ʻis_selected () == is_selected --ʻEC.element_located_selection_state_to_be (locator, is_selected) : ʻis_selected () == is_selected --ʻEC.visibility_of (element) : Whether ʻelement is ʻis_displayed () == True --ʻEC.visibility_of_element_located (locator): Whether the element indicated by locator is ʻis_displayed () == True --Whether ʻEC.invisibility_of_element (element): ʻelement is ʻis_displayed () == False --ʻEC.invisibility_of_element_located (locator) : Whether the element indicated by locator is ʻis_displayed () == False

The argument here is

It will be. On the other hand

The function that checks only is not prepared and can be substituted.

It will be. However, clickable often does not serve the purpose because it looks at ʻelement.is_enabled () and element.is_displayed ()`.

How to use

In general, the function of ʻexpected_conditions` is

python


driver = webdriver.Firefox(executable_path=path, options=options, service_log_path="nul")
driver.get(url)

locator = (By.XPATH, "//div[@id='cat']")
element = WebDriverWait(driver, 30, 1).until(EC.visibility_of_element_located(locator))

Use in combination with WebDriverWait (). Until () etc. Let's take a closer look at this.

Expected conditions

The first is ʻEC.visibility_of_element_located () `. This can be done as follows

python


func = EC.visibility_of_element_located(locator)
element = func(driver)

func returns a function that takes one argument (assuming driver). Then, if you pass driver to func and execute it, if the element indicated by locator is displayed, that element will be returned, and if it is not displayed, False will be returned. In other words, func is a function likefind_element (locator)that does not throw an exception if it fails.

Also, find_element () can be changed as follows:

python


relativeLocator = (By.XPATH, "./div[@class='meow']") #Relative path
child = element.find_element(*relativeLocator)

You can also get the element (child) under ʻelement. When I try to do something similar with ʻEC.element_to_be_clickable, I get:

python


child = EC.visibility_of_element_located(relativeLocator)(element)

It seems that other ʻexpected_conditionsfunctions can get elements of relative paths as well. However, when I read the (found) description, it seems to assume an absolute path fromdriver`. Looking at the source on GitHub seems to be fine, but I'm a little worried. Therefore, I would like to prepare some other means when dealing with relative paths.

WebDriverWait

Let's go back a little and look at WebDriverWait (). Until ().

WebDriverWait () takes the following arguments (one omitted):

-- driver: Assuming driver --timeout: Maximum wait time --poll_frequency: Trial interval

And wait.until () takes one argument as described below.

--method: A function that takes one driver as an argument. This function returns False on failure and non-False` on success

If you write as follows,

python


timeout = 30
poll_frequency = 1
wait = WebDriverWait(driver, timeout, poll_frequency)
element = wait.until(method)

The behavior of wait.until (method) is

  1. The contents of the driver variable (usually assumed to be a driver instance) are passed to method.
  2. Continue running for up to 30 seconds at 1 second intervals until method (driver) succeeds (returns something other than False).
  3. If method (driver) succeeds, its return value is returned.
  4. If it does not succeed after 30 seconds, an exception will be thrown.

It will be.

combination

From the above explanation, if you write as follows,

python


locator = (By.XPATH, "//div[@id='cat']")
element = WebDriverWait(driver, 30).until(EC.visibility_of_element_located(locator))

If the element indicated by locator exists and is displayed, that element is assigned to ʻelement. If the element is not found or becomes displayed after 30 seconds, an exception will be thrown. As you may have noticed, you can get the relative element from ʻelement by doing the following.

python


relativeLocator = (By.XPATH, "./div[@class='meow']") #Relative path
child = WebDriverWait(element, 30).until(EC.visibility_of_element_located(relativeLocator))

Correspond to ʻenabled / disabled`

ʻExpected_conditions does not have a function corresponding to ʻenabled (disabled), but it is easy to create one. Assuming it is called from WebDriverWait (). Until ()

--A function that takes one driver as an argument. This function returns False on failure and non-False` on success

You can see that you should create the function. However, if it is a function, only driver can be passed unless you use a global variable, so you will create Class. The simplest way to make it is as follows.

python


class IsEnabled():

  def __init__(self, locator=(), state=True):
    self.locator = locator
    self.state = state

  def __call__(self, driver):
    try:
      if self.locator == ():
        return False
      element = driver.find_element(*self.locator)
      return element if element.is_enabled() == self.state else False

    except Exception as e:
      return False

It can be used as follows:

python


locator = (By.XPATH, "//div[@id='cat']")
element = WebDriverWait(driver, 30, 1).until(IsEnabled(locator))

There are two types of ʻexpected_conditions, one that takes locator and the other that takes ʻelement. However, the one you created must specify locator. And it does not support relative paths. I will try to improve it so that it can handle these.

python


class IsEnabled():

  def __init__(self, locator=(), element=None, state=True):
    self.locator = locator
    self.element = element
    self.state = state

  def __call__(self, driver):
    try:
      if self.element is not None and self.locator == ():
        element = self.element
      elif self.element is not None and self.locator != ():
        element = self.element.find_element(*self.locator)
      elif self.locator != ():
        element = driver.find_element(*self.locator)
      else:
        return False

      return element if element.is_enabled() == state else False

    except Exception as e:
      return False

By doing this, you can use it as follows.

python


#Get the element indicated by locator when enabled
element = WebDriverWait(driver, 30, 1).until(IsEnabled(locator=locator))

#Returns element when element is enabled
element = WebDriverWait(driver, 30, 1).until(IsEnabled(element=element))

#Get when the relative element from element becomes enable
child = WebDriverWait(driver, 30, 1).until(IsEnabled(element=element, locator=relativeLocator))

The CheckState shown at the beginning corresponds to yet another state.

Prepare a function that changes to find_element ()

Now you can get the element with the same description, like find_element (). Besides, you can get this while watching the change of state. However, it is troublesome to write the following every time, and it can be confusing when used in combination with find_element ().

python


element = WebDriverWait(driver, 30, 1).until(CheckState(element=element, state="clickable"))

So we define a wrapper function that changes find_element (). Personally, it doesn't seem like I'm getting the element from the name of the function.

I think it's also a problem that you make too many wrapper functions and you don't know what you're using.

python


def findElement(driver, locator=(), element=None, state="enabled", must=True, wait=30, interval=0.5, ignore=False):
  try:
    if element is None and locator == ():
      raise ValueError

    driverWait = WebDriverWait(driver, wait, interval)
    return driverWait.until(CheckState(locator=locator, element=element, state=state))

  except TimeoutException:
    if must == True and ignore == False:
      logger.error(f"findElement: {locator}, {element}, {state}, {must}, {wait}, {interval}, {ignore}")
      raise ValueError
    return None
  except Exception as e:
    if ignore == True:
      return None
    logger.error(f"findElement: {type(e)}, {e}")
    raise e

It is used as follows.

python


locator = (By.XPATH, "//div[@id='cat']")
element = findElement(driver, locator=locator):

Finally

Now, I think you can write it neatly. However, depending on the behavior of the site, it is difficult to make various adjustments. Even if it has worked so far, it is often the case that the site is slow and stumbling in unexpected places. After all, I can't let go of time.sleep () There is a story that I am not good at it (´ ・ ω ・ `)

Recommended Posts

Seeking a unified way to wait and get for state changes in Selenium for Python elements
A simple way to avoid multiple for loops in Python
A standard way to develop and distribute packages in Python
Do you want to wait for general purpose in Python Selenium?
How to get a stacktrace in python
Get a token for conoha in python
Wait for another window to open in Selenium
A clever way to time processing in Python
Searching for an efficient way to write a Dockerfile in Python with poetry
[Mac] A super-easy way to execute system commands in Python and output the results
How to swap elements in an array in Python, and how to reverse an array.
Get the number of specific elements in a python list
Developed a library to get Kindle collection list in Python
How to define multiple variables in a python for statement
Tips for coding short and easy to read in Python
Recursively get the Excel list in a specific folder with python and write it to Excel.
I tried "How to get a method decorated in Python"
Useful tricks related to list and for statements in Python
How to get the last (last) value in a list in Python
How to get a list of built-in exceptions in python
I also tried to imitate the function monad and State monad with a generator in Python
Introducing a good way to manage DB connections in Python
Just try to receive a webhook in ngrok and python
An easy way to view the time taken in Python and a smarter way to improve it
How to determine the existence of a selenium element in Python
How to compare lists and retrieve common elements in a list
What to do if you get a minus zero in Python
How to get a string from a command line argument in python
[Introduction to Python] How to use the in operator in a for statement?
[For beginners] How to register a library created in Python in PyPI
I tried to make a periodical process with Selenium and Python
Create a CGH for branching a laser in Python (laser and SLM required)
Tips for using Selenium and Headless Chrome in a CUI environment
How to display bytes in the same way in Java and Python
Assignments and changes in Python objects
Selenium and python to open google
[Python] How to create a dictionary type list, add / change / delete elements, and extract with a for statement
Get a capture of the entire web page in Selenium Python VBA
How to get a specific column name and index name in pandas DataFrame
Click the Selenium links in order to get the elements of individual pages
How to get a value from a parameter store in lambda (using python)
How to put a half-width space before letters and numbers in Python.
Connect to postgreSQL from Python and use stored procedures in a loop.
Implementation of particle filters in Python and application to state space models
What is the fastest way to create a reverse dictionary in python?
How to stop a program in python until a specific date and time
Build a lightweight server in Python and listen for Scratch 2 HTTP extensions
selenium: wait for element with AND / OR
Easy way to use Wikipedia in Python
A way to understand Python duck typing
How to use is and == in Python
I tried to make a periodical process with CentOS7, Selenium, Python and Chrome
Steps to put dlib in Python Tools for Visual Studio and have fun
Tips for those who are wondering how to use is and == in Python
How to count the number of elements in Django and output to a template
Use libsixel to output Sixel in Python and output a Matplotlib graph to the terminal.
For those who want to learn Excel VBA and get started with Python
[Cloudian # 10] Try to generate a signed URL for object publishing in Python (boto3)
A complete guidebook to using pyenv, pip and python in an offline environment
I searched for the skills needed to become a web engineer in Python
Python script to get a list of input examples for the AtCoder contest