[CENTOS] Scraping with Selenium [Python]

Scraping is done using Selenium in Python.

0. Environment

MacOSX10.11.1

For CentOS

Xvfb and more needed Reference: CentOS6.4 + Selenium environment creation

1. Install Selenium with pip


$ sudo pip install selenium

For CentOS

You need to install the library separately.


$ sudo pip install pyvirtualdisplay

http://stackoverflow.com/questions/26070834/how-to-fix-selenium-webdriverexception-the-browser-appears-to-have-exited-befor

2. Run

2.1. Import Selenium

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

#When using pyvirtualdisplay
from pyvirtualdisplay import Display
display = Display(visible=0, size=(1024, 768))
display.start()

2.2. Open a browser (FireFox)


driver = webdriver.Firefox()

Later, when running the script on Linux (CentOS) FireFox is more convenient (easier), so I use FireFox.

2.3. Open the URL

url = "http://google.com"
driver.get(url)

2.4. Close your browser

driver.close()

#When using pyvirtualdisplay
display.stop()

Recommended Posts

Scraping with Selenium [Python]
Scraping with selenium in Python
Scraping with Selenium + Python Part 1
Scraping with Selenium in Python
Scraping with Selenium + Python Part 2
Scraping with selenium
Scraping with selenium ~ 2 ~
Scraping with Python
Scraping with Python
Scraping with Selenium
Scraping with Selenium in Python (Basic)
Scraping with Python, Selenium and Chromedriver
Scraping with Python (preparation)
Scraping with Python + PhantomJS
ScreenShot with Selenium (Python)
Python web scraping selenium
Scraping with Python + PyQuery
Scraping RSS with Python
Practice web scraping with Python and Selenium
I tried scraping with Python
Web scraping with python + JupyterLab
Scraping with chromedriver in python
Festive scraping with Python, scrapy
Python: Working with Firefox with selenium
Scraping with Tor in Python
[Scraping] Python scraping
Scraping weather forecast with python
[Python + Selenium] Tips for scraping
I tried scraping with python
Web scraping beginner with python
I-town page scraping with selenium
I was addicted to scraping with Selenium (+ Python) in 2020
Scraping with Node, Ruby and Python
Web scraping with Python ① (Scraping prior knowledge)
Web scraping with Python First step
I tried web scraping with python.
Scraping with Python and Beautiful Soup
Let's do image scraping with Python
Get Qiita trends with Python scraping
"Scraping & machine learning with Python" Learning memo
Get weather information with Python & scraping
Python scraping notes
[Python / Selenium] XPath
FizzBuzz with Python3
Python Scraping get_ranker_categories
Statistics with python
Python with Go
Twilio with Python
Integrate with Python
Play with 2016-Python
AES256 with python
Tested with Python
Beginning with Selenium
python starts with ()
with syntax (Python)
Python: Scraping Part 1
Zundokokiyoshi with python
Scraping using Python
Excel with Python
Microcomputer with Python
Python: Scraping Part 2