Execution environment

maxOS Sierra ver. 10.12.6
Python 3.6.1
Google Chrome 59.0

What is Headless Chrome?

A mode that operates without displaying Chrome introduced from Google Chrome 59. As a result, it can be used in a server environment without automatic testing and UI.

Windows only seems to be supported from Chrome 60

Preparation

Install the latest version Chrome
Download Python 3.6 (recommended to use virtualenv)
Install dependent libraries such as Selenium

Python dependent library

This time I will use the following library

Selene-Selenide implemented in Python makes Selenium easier to handle *
python webdriver manager --Automatically installs and manages browser drivers **
Selenium
Joblib --Used for parallel execution

API is subject to change as it is still in alpha version

** Installed drivers are saved under ~ / .wdm / by default

Dependent library installation

Install the latest version of pre-release, selene. The above libraries are included at the same time when selene is installed. Also, six is dependent for some reason, but I didn't enter it together, so I will install it separately.

pip install selene --pre pip install six

Sample code

It's a simple Google search and screenshots of the results.

`sample_selene.py`


#!/usr/bin/env python
# -*- coding: utf-8 -*-

from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from selene.driver import SeleneDriver
from webdriver_manager.chrome import ChromeDriverManager

# run chrome headless
options = Options()
options.add_argument('--headless')

# install chromedriver if not found and start chrome
driver = SeleneDriver.wrap(webdriver.Chrome(executable_path=ChromeDriverManager().install(), chrome_options=options))

# search 'python' in google
driver.get('https://www.google.co.jp/')
input = driver.find_element_by_name('q')
input.send_keys('Python')
input.send_keys(Keys.RETURN)

# save screen shot
driver.save_screenshot('result.png')

driver.quit()

The execution result is as follows.

Parallelization of sample code

`sample_selene_parallel.py`


#!/usr/bin/env python
# -*- coding: utf-8 -*-

from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from selene.driver import SeleneDriver
from webdriver_manager.chrome import ChromeDriverManager
from joblib import Parallel, delayed

# search 'keyword' in google
def google(url, keyword):
    # run chrome headless
    options = Options()
    options.add_argument('--headless')
    driver = SeleneDriver.wrap(webdriver.Chrome(executable_path=ChromeDriverManager().install(), chrome_options=options))
    driver.get(url)
    input = driver.find_element_by_name('q')
    input.send_keys(keyword)
    input.send_keys(Keys.RETURN)

    # save screen shot
    driver.save_screenshot(keyword + '.png')

    driver.quit()

url = 'https://www.google.co.jp/'
keywords = ['Python', 'Google', 'Selenium']

# n_jobs=-1 means use all of the resources you can`
Parallel(n_jobs=-1)(delayed(google)(url,keyword) for keyword in keywords)

The execution result is as follows. Python results are omitted.

Reference link

https://developers.google.com/web/updates/2017/04/headless-chrome
https://github.com/yashaka/selene
https://github.com/SergeyPirogov/webdriver_manager
http://qiita.com/orangain/items/db4594113c04e8801aad

I tried using Selenium with Headless chrome

Postscript