Scraping with Selenium in Python

Introduction

I summarized Selenium when scraping with Python. Scraping is pretty fun. I have used a wide range of other uses, such as shopping automation and data collection.

environment

Windows10 Python Selenium Dedicated browser driver you want to use (eg Chrome Driver)

Install Selenium

If you haven't built a Python environment yet, click here. How to build an environment for Python is posted. (Windows version)

If you have already built a Python environment, please install Selenium.

pip install selenium

Download browser driver

You will need a driver that matches the browser you are using. (Example: Chrome Driver)

Open a browser using Selenium in Python

Open your browser using Selenium.

from selenium import webdriver

driver = webdriver.Chrome() #()Driver in_describe path
driver.get(URL)
driver.close()
driver.quit()

First, open the browser on the first line Specify the URL you want to open in the second line The third line closes the page Line 4 closes the browser

driver_path is ok without description when the driver is in the same directory.

Get element

Next is the acquisition of the elements required to operate the page. I often use these two.

id get_id = driver.find_element_by_id ("write id in this ") It is recommended because the id will surely be obtained.

xpath get_xpath = driver.find_element_by_xpath ("write xpath in this ") Since it often changes when the screen changes, I rewrite the xpath each time.

And if you can't get it because it's not on HTML, but you want that value, it's recommended to get it from js. price = driver.execute_script ("describe js processing") By writing as above, you can get the value that is not on the screen but is processed on the js side.

There are times when js doesn't have it either, so I have no choice but to give up at that time: sweat:

Click the button

When you want to click a button on a form, you can click it with click ().

driver.find_elemnt_by_id ('write id in this'). Click () You can click the button element in the above process.

Let's use sleep processing

Often, there are times when an error occurs because the element cannot be removed. In that case, use sleep () and wait for the screen to appear.

in conclusion

There are many other ways to use selenium, but I wrote this article to let you know what you can do. If you want to know more details, please see articles written by other people and increase your knowledge. And let's automate everything!

reference

https://kurozumi.github.io/selenium-python/installation.html https://qiita.com/nadechin/items/0a34e2182132cc1a821b

Recommended Posts

Scraping with selenium in Python
Scraping with Selenium in Python
Scraping with Selenium in Python (Basic)
Scraping with Selenium [Python]
Scraping with Selenium + Python Part 1
Scraping with Tor in Python
Scraping with Selenium + Python Part 2
Scraping with selenium
Scraping with Python
Scraping with Python
Scraping with Selenium
I was addicted to scraping with Selenium (+ Python) in 2020
Successful scraping with Selenium
Scraping with Python (preparation)
Try scraping with Python.
Scraping with Python + PhantomJS
ScreenShot with Selenium (Python)
Python web scraping selenium
Practice web scraping with Python and Selenium
Scraping with Python + PyQuery
Scraping RSS with Python
Log in to Yahoo Business with Selenium Python
Achieve scraping with Python & CSS selector in 1 minute
I tried scraping with Python
Web scraping with python + JupyterLab
Working with LibreOffice in Python
Web scraping notes in python3
Festive scraping with Python, scrapy
Debugging with pdb in Python
Python: Working with Firefox with selenium
Working with sounds in Python
Tweet with image in Python
Combined with permutations in Python
Web scraping using Selenium (Python)
Scraping weather forecast with python
[Python + Selenium] Tips for scraping
I tried scraping with python
Web scraping beginner with python
I-town page scraping with selenium
[Scraping] Python scraping
Number recognition in images with Python
Try scraping with Python + Beautiful Soup
Testing with random numbers in Python
Scraping with Node, Ruby and Python
GOTO in Python with Sublime Text 3
Working with LibreOffice in Python: import
Web scraping with Python ① (Scraping prior knowledge)
CSS parsing with cssutils in Python
Web scraping with Python First step
I tried web scraping with python.
Scraping with Python and Beautiful Soup
Numer0n with items made in Python
Scraping with Beautiful Soup in 10 minutes
Use rospy with virtualenv in Python3
Write selenium test code in python
Let's do image scraping with Python
Use Python in pyenv with NeoVim
Heatmap with Dendrogram in Python + matplotlib
Get Qiita trends with Python scraping
Read files in parallel with Python
Password generation in texto with python