How to make Selenium as light as possible

Requirements

-Although it's late, Raspberry Pi also tried scraping the card company. -Since I tried scraping with Selenium using Raspeye Zero, many timeouts occurred. -We have made it possible to implement scraping as lightly as possible. -Manage passwords at your own risk. ..

code

from selenium import webdriver 
from selenium.webdriver.chrome.options import Options 
from selenium.webdriver.common.by import By 
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.support.ui import WebDriverWait 
import time
from bs4 import BeautifulSoup as BS 
import re
import requests

 options = Options()
 options.add_argument("no-sandbox")
 options.add_argument("--disable-extensions")
 options.add_argument("--headless")
 options.add_argument('--disable-gpu')  
 options.add_argument('--ignore-certificate-errors')
 options.add_argument('--allow-running-insecure-content')
 options.add_argument('--disable-web-security')
 options.add_argument('--disable-desktop-notifications')
 options.add_argument("--disable-extensions")
 options.add_argument('--lang=ja')
 options.add_argument('--blink-settings=imagesEnabled=false')
 options.add_argument('--disable-dev-shm-usage')
 options.add_argument('--proxy-server="direct://"')
 options.add_argument('--proxy-bypass-list=*')    
 options.add_argument('--start-maximized')
 driver = webdriver.Chrome(options=options)

 driver.get (http:///~~~)

Also, in order to reduce the reading range, it is possible to read up to the Class immediately before scraping. Described with maximum timeout

 wait = WebDriverWait(driver, 300);
 element=wait.until(EC.presence_of_element_located((By.CLASS_NAME,"fluid")));

After that, read and process with BeautifulSoup

 res = driver.page_source.encode('utf-8')
 print("loading")
 soup=BS(res,"html.parser")

Now I've managed to avoid the timeout.

Recommended Posts

How to make Selenium as light as possible
How to debug selenium
How to collect tweets from tweetid as soon as possible (72000 tweets / hour)
How to make a Japanese-English translation
How to make a slack bot
How to make a crawler --Advanced
How to make a recursive function
[EC2] How to run selenium webdriver
How to make a deadman's switch
[Blender] How to make a Blender plugin
[Blender] How to make Blender scripts multilingual
How to make a crawler --Basic
How to make AWS Lambda Layers when running selenium × chrome on AWS Lambda
Try to make capture software with as high accuracy as possible with python (1)
How to make Word Cloud characters monochromatic
[Python] How to make a class iterable
How to install CatBoost [as of January 2020]
[Cocos2d-x] How to make Script Binding (Part 2)
How to make multi-boot USB (Windows 10 compatible)
How to make a Backtrader custom indicator
How to make a Pelican site map
[Cocos2d-x] How to make Script Binding (Part 1)
Try to make a capture software with as high accuracy as possible with python (2)
Dedicated to beginners! How to learn programming without spending as much money as possible
How to make a dialogue system dedicated to beginners
How to disguise a ZIP file as a PNG file
How to make an embedded Linux device driver (11)
How to make WTForms TextArea correspond to file drop
How to make an embedded Linux device driver (8)
How to make Spigot plugin (for Java beginners)
How to make an embedded Linux device driver (1)
How to make an embedded Linux device driver (4)
How to make multiple kernels selectable on Jupyter
How to make a dictionary with a hierarchical structure.
How to make an embedded Linux device driver (7)
How to make an embedded Linux device driver (2)
How to make scrapy JSON output into Japanese
How to make an embedded Linux device driver (3)
How to deal with SessionNotCreatedException when using Selenium
How to make a QGIS plugin (package generation)
I read "How to make a hacking lab"
[Blender x Python] How to make an animation
How to make an embedded Linux device driver (6)
How to make Substance Painter Python plugin (Introduction)
How to output "Ketsumaimo" as standard output in Python
[Blender x Python] How to make vertex animation
How to make an embedded Linux device driver (5)
How to make an embedded Linux device driver (10)
How to make Python faster for beginners [numpy]
How to make Python Interpreter changes in Pycharm
How to make Linux compatible with Japanese keyboard
How to make an embedded Linux device driver (9)
Inspired by "How to make pure functional JavaScript"
How to make AWS rekognition recognize local image files
[Continued] Inspired by "How to make pure functional JavaScript"
How to make Yubico Yubikey recognized in Manjaro Linux
Explain in detail how to make sounds with python
How to make a shooting game with toio (Part 1)
How to make unit tests Part.2 Class design for tests
How to export Cybozu Live My Calendar as iCalendar
How to make an interactive CLI tool in Golang