I tried using Selenium with Headless chrome

This time I will try to operate Headless Chrome using Selenium.

The code is summarized in GitHub.

Postscript

Execution environment

What is Headless Chrome?

A mode that operates without displaying Chrome introduced from Google Chrome 59. As a result, it can be used in a server environment without automatic testing and UI.

Preparation

  1. Install the latest version Chrome
  2. Download Python 3.6 (recommended to use virtualenv)
  3. Install dependent libraries such as Selenium

Python dependent library

This time I will use the following library

** Installed drivers are saved under ~ / .wdm / by default

Dependent library installation

Install the latest version of pre-release, selene. The above libraries are included at the same time when selene is installed. Also, six is dependent for some reason, but I didn't enter it together, so I will install it separately.

pip install selene --pre pip install six

Sample code

It's a simple Google search and screenshots of the results.

sample_selene.py


#!/usr/bin/env python
# -*- coding: utf-8 -*-

from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from selene.driver import SeleneDriver
from webdriver_manager.chrome import ChromeDriverManager

# run chrome headless
options = Options()
options.add_argument('--headless')

# install chromedriver if not found and start chrome
driver = SeleneDriver.wrap(webdriver.Chrome(executable_path=ChromeDriverManager().install(), chrome_options=options))

# search 'python' in google
driver.get('https://www.google.co.jp/')
input = driver.find_element_by_name('q')
input.send_keys('Python')
input.send_keys(Keys.RETURN)

# save screen shot
driver.save_screenshot('result.png')

driver.quit()

The execution result is as follows.

result.png

Parallelization of sample code

sample_selene_parallel.py


#!/usr/bin/env python
# -*- coding: utf-8 -*-

from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from selene.driver import SeleneDriver
from webdriver_manager.chrome import ChromeDriverManager
from joblib import Parallel, delayed

# search 'keyword' in google
def google(url, keyword):
    # run chrome headless
    options = Options()
    options.add_argument('--headless')
    driver = SeleneDriver.wrap(webdriver.Chrome(executable_path=ChromeDriverManager().install(), chrome_options=options))
    driver.get(url)
    input = driver.find_element_by_name('q')
    input.send_keys(keyword)
    input.send_keys(Keys.RETURN)

    # save screen shot
    driver.save_screenshot(keyword + '.png')

    driver.quit()

url = 'https://www.google.co.jp/'
keywords = ['Python', 'Google', 'Selenium']

# n_jobs=-1 means use all of the resources you can`
Parallel(n_jobs=-1)(delayed(google)(url,keyword) for keyword in keywords)

The execution result is as follows. Python results are omitted. Google.png

Selenium.png

Reference link

Recommended Posts

I tried using Selenium with Headless chrome
I tried using Headless Chrome from Selenium
I tried using parameterized
I tried using argparse
I tried using mimesis
I tried using anytree
I tried using aiomysql
I tried using coturn
I tried using Pipenv
I tried using matplotlib
I tried using "Anvil".
I tried using Hubot
I tried using ESPCN
I tried using openpyxl
I tried using Ipython
I tried using PyCaret
I tried using cron
I tried using ngrok
I tried using face_recognition
I tried using Jupyter
I tried using PyCaret
I tried using Heapq
I tried using doctest
I tried using folium
I tried using jinja2
I tried using folium
I tried using time-window
I tried web scraping using python and selenium
I tried using mecab with python2.7, ruby2.3, php7
I tried DBM with Pylearn 2 using artificial data
I tried using a database (sqlite3) with kivy
I tried fp-growth with python
I tried scraping with Python
I tried to make a periodical process with CentOS7, Selenium, Python and Chrome
I tried Learning-to-Rank with Elasticsearch!
[I tried using Pythonista 3] Introduction
I tried using easydict (memo).
I tried face recognition using Face ++
I tried using Random Forest
I tried clustering with PyCaret
I tried using BigQuery ML
I tried using Amazon Glacier
I tried using git inspector
[Python] I tried using OpenPose
I tried using magenta / TensorFlow
I tried gRPC with Python
I tried scraping with python
I tried a simple RPA for login with selenium
I tried to create an environment to check regularly using Selenium with AWS Fargate
I tried using AWS Chalice
I tried using Slack emojinator
I tried using the Python library from Ruby with PyCall
I tried using the DS18B20 temperature sensor with Raspberry Pi
[Python scraping] I tried google search top10 using Beautifulsoup & selenium
I tried handwriting recognition of runes with CNN using Keras
I tried using Rotrics Dex Arm # 2
I tried trimming efficiently with OpenCV
I tried summarizing sentences with summpy
I tried machine learning with liblinear
I tried web scraping with python.
I tried moving food with SinGAN