Ruby Scraping-Move Selenium Headless with VPS.

Development environment

ruby2.7.1 centos7

procedure

① Let's add Gem

gem install gem install selenium-webdriver

② Install Google Chrome

install yum install google-chrome-stable Check the version google-chrome -version

③ Install Google Chrome

Download the same version of Chrome Driver as google-chrome from http://chromedriver.chromium.org/downloads

Below is a command from Google Chrome version 83.0.4103.39 wget https://chromedriver.storage.googleapis.com/index.html?path=83.0.4103.39/

Deployment unzip chromedriver_linux64.zip Move sudo mv chromedriver ~/usr/local/bin/ Change permissions sudo chmod 755 /usr/local/bin/chromedriver

Ready

④ Sample code using Selenium

scraping.rb


def scraping
  url = 'https://google.com/' #URL you want to open
  options = Selenium::WebDriver::Chrome::Options.new #New option
  options.add_argument('--headless') #Headless option added
  options.add_argument('--disable-gpu') #Disable GPU
  options.add_argument('--window-size=4000,4000') #Maximize screen
  driver = Selenium::WebDriver.for :chrome, options: options #New driver to reflect options
#Scraping code as you like
.
.
.
.
end

--headless and --disable-gpu added for headless processing for Linux The addition of the option --window-size = 4000,4000 is written because scraping can be done only in the part displayed on the screen.

* For those who create automatic Bitcoin trading tools (because there are many these days)

Selenium is great for automating sites that don't have an API. You don't need Selenium for automatic Bitcoin trading. I think it is better to make it using Bitflyer's API as shown below. http://benzenetarou.hatenablog.com/entry/bitcoin/automatic_trade/1

Recommended Posts

Ruby Scraping-Move Selenium Headless with VPS.
Creating a browser automation tool with Ruby + Selenium
Getting Started with Ruby
Evolve Eevee with Ruby
Ruby version switching with rbenv
I tried DI with Ruby
GraphQL Client starting with Ruby
Ruby: Send email with Starttls
Format Ruby with VS Code
Integer check method with ruby
[Ruby] problem with if statement
Studying with CodeWar (ruby) ⑤ proc
Use Ruby with Google Colab
Getting Started with Ruby Modules
[ruby] Method call with argument
Install Ruby on Sakura's VPS
[Linux: Ruby] Use the driver with selenium without specifying the browser driver path