Automated testing method combining Beautiful Soup and Selenium (Python)

1. 1. Overview

We sometimes embed tags to collect specific data in the HTML of web pages, and we used automated tests to test whether the embedded tags were correct.

How to search HTML data using Beautiful Soup

However, if it is a static page, I could test it using Beautiful soup, but I could not get HTML data for a screen with strong security such as an SSL-enabled screen.

Therefore, if you cannot get the HTML data with Beautiful soup, we decided to use Selenium to move to the target screen and get the HTML page.

2. program

Below is a program that acquires HTML data using Beautiful soup and Selenium.

test.py


import time
from selenium import webdriver
from bs4 import BeautifulSoup
import re

#From the screen that can be transitioned to the screen that could not be acquired by Beautiful soup
driver.get("test.html")
driver.find_element_by_css_selector("test").click()

#If you can transition to the target screen
source = driver.page_source
soup = BeautifulSoup(source,'html.parser')
elems = soup.find_all("script",text=re.compile("test"))

#Move to the next screen
driver.find_element_by_css_selector("test").click()
…

For parsing HTML data, you can use Beautiful Soup as it is.

source = driver.page_source
soup = BeautifulSoup(source,'html.parser')

There is no problem if you use Selenium's ** ".page_source" ** to get the HTML data.

3. 3. Summary

Create the above program for each required screen and you're done. Also, in the case of the above program, Chrome will start when you start it, so it may be better to start it with Headless. (I don't use Headless so much because Selenium often stops with an error ...)

Reference: I tried using Headless Chrome from Selenium

Recommended Posts

Automated testing method combining Beautiful Soup and Selenium (Python)
My Beautiful Soup (Python)
I tried various things with Python: scraping (Beautiful Soup + Selenium + PhantomJS) and morphological analysis.
Challenge Python3 and Selenium Webdriver
Try scraping with Python + Beautiful Soup
Scraping with Python, Selenium and Chromedriver
[Python] A memorandum of beautiful soup4
Selenium and python to open google
[Python] Difference between function and method
Settings when using Python 3 requests and Beautiful Soup with crostini on Chromebook
Sort anime faces by scraping anime character pages with Beautiful Soup and Selenium
[Python3] Understand the basics of Beautiful Soup
Python + Selenium Frequently used operation method summary
[Python] Difference between class method and static method
[Python] Scraping a table using Beautiful Soup
Practice web scraping with Python and Selenium
Beautiful Soup
Try out the touch of data-driven testing with Selenium Python Bindings and py.test
Instant method grammar for Python and Ruby (studying)
I tried web scraping using python and selenium
Try running Google Chrome with Python and Selenium