Try Selenium Grid with Docker

Try installing Chrome on the Anaconda image, but it still takes a long time to build, so cut out only the Chrome part as a separate image. I was wondering if there was one, but I found out that there is a mechanism called Selenium Grid that can operate WebDriver via REST API, so I tried it.

Start Selenium Grid

A Docker image that can use Selenium Grid is officially released, so use this.

SeleniumHQ/docker-selenium: Docker images for Selenium Grid Server (Standalone, Hub, and Nodes).

If you want to use multiple browsers, you need to launch Hub and Node of each browser respectively, but this time I just want to try it with Chrome, so I used the image of Standalone.

docker-compose.yml


version: "3"

services:

  chrome:
    image: selenium/standalone-chrome
    ports:
      - 4444:4444
    volumes:
      - /dev/shm:/dev/shm

The REST API is exposed on port 4444. The API for operating WebDriver seems to be / wd / hub.

$ docker-compose up

Use Selenium Grid from Python

If you want to use Selenium Grid from Python, use selenium.webdriver.Remote.

$ pip3 install selenium

main.py


import sys
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities


def search(driver, query):
    driver.get('https://google.com/')

    q = driver.find_element_by_name('q')
    q.send_keys(query)
    q.submit()

    r = driver.find_element_by_class_name('g').find_element_by_class_name('r')
    title = r.find_element_by_tag_name('h3').text
    url = r.find_element_by_tag_name('a').get_attribute('href')

    return title, url


if __name__ == '__main__':
    query = ' '.join(sys.argv[1:])
    options = {
        'command_executor': 'http://localhost:4444/wd/hub',
        'desired_capabilities': DesiredCapabilities.CHROME,
    }
    with webdriver.Remote(**options) as driver:
        title, url = search(driver, query)
        print(f'{title}\n{url}')

I did a Google search and wrote a script to display the title and URL of the top results.

$ python3 main.py qiita
Qiita
https://qiita.com/

Recommended Posts

Try Selenium Grid with Docker
Try building JupyterHub with Docker
Scraping with selenium
Scraping with selenium ~ 2 ~
Let's try gRPC with Go and Docker
Try using Selenium
Scraping with Selenium
Beginning with Selenium
Try running Google Chrome with Python and Selenium
[Python3] [Ubuntu16] [Docker] Try face recognition with OpenFace
Tftp server with Docker
Successful scraping with Selenium
Try scraping with Python.
ScreenShot with Selenium (Python)
Use python with docker
Try SNN with BindsNET
Proxy server with Docker
Hello, World with Docker
Scraping with Selenium [Python]
Try regression with TensorFlow
Install selenium on Mac and try it with python
Try to factorial with recursion
Try function optimization with Optuna
Try deep learning with TensorFlow
Try using PythonTex with Texpad.
Scraping with selenium in Python
Scraping with Selenium + Python Part 1
Try edge detection with OpenCV
Try implementing RBM with chainer.
Try Google Mock with C
Try using matplotlib with PyCharm
Try programming with a shell!
Try GUI programming with Hy
Try an autoencoder with Pytorch
Try matrix operation with NumPy
Python: Working with Firefox with selenium
Try implementing XOR with PyTorch
Try running CNN with ChainerRL
Scraping with Selenium in Python
Try Deep Learning with FPGA
Implement PyTorch + GPU with Docker
Prepare python3 environment with Docker
Try tweeting automatically using Selenium.
Try running Python with Try Jupyter
Memories of fighting with Selenium
Try implementing perfume with Go
Scraping with Selenium + Python Part 2
Try face recognition with Python
Try OpenCV with Google Colaboratory
Try machine learning with Kaggle
Try TensorFlow MNIST with RNN
I-town page scraping with selenium
Rails application building with Docker
Try using folium with anaconda
Try Deep Learning with FPGA-Select Cucumbers
Try scraping with Python + Beautiful Soup
Reinforcement learning 13 Try Mountain_car with ChainerRL.
Try running tensorflow on Docker + anaconda
Japaneseize Matplotlib with Alpine using Docker
Until you start Jupyter with Docker
Easy Slackbot with Docker and Errbot