Automat WEB process with Heroku + Selenium + Chrome

Automat WEB process with Heroku + Selenium + Chrome

-[Automate web process with Heroku + Selenium + Chrome](# heroku--selenium--chrome% e3% 81% a7web% e3% 83% 97% e3% 83% ad% e3% 82% bb% e3% 82% b9% e3% 82% 92% e8% 87% aa% e5% 8b% 95% e5% 8c% 96% e3% 81% 99% e3% 82% 8b) -[Introduction](#% e3% 81% af% e3% 81% 98% e3% 82% 81% e3% 81% ab) -[Purpose](#% e7% 9b% ae% e7% 9a% 84) -[Related Articles](#% e9% 96% a2% e9% 80% a3% e3% 81% 99% e3% 82% 8b% e8% a8% 98% e4% ba% 8b) -[Execution environment](#% e5% ae% 9f% e8% a1% 8c% e7% 92% b0% e5% a2% 83) -[Source Code](#% e3% 82% bd% e3% 83% bc% e3% 82% b9% e3% 82% b3% e3% 83% bc% e3% 83% 89) -[Scenario and Prerequisites](#% e3% 82% b7% e3% 83% 8a% e3% 83% aa% e3% 82% aa% e3% 81% a8% e5% 89% 8d% e6% 8f% 90% e6% 9d% a1% e4% bb% b6) -[Create Automatic Process](#% e8% 87% aa% e5% 8b% 95% e3% 83% 97% e3% 83% ad% e3% 82% bb% e3% 82% b9% e3% 81% ae% e4% bd% 9c% e6% 88% 90) -[Python API configuration](# python-api% e6% a7% 8b% e6% 88% 90) -[API Mainframe](#api% e3% 83% a1% e3% 82% a4% e3% 83% b3% e3% 83% 95% e3% 83% ac% e3% 83% bc% e3% 83% a0) -[Get FX Rate](# fx% e3% 83% ac% e3% 83% bc% e3% 83% 88% e5% 8f% 96% e5% be% 97) -[Heroku Settings](# heroku% e3% 81% ae% e8% a8% ad% e5% ae% 9a) -[Chrome and Driver Settings](# chrome% e3% 81% a8driver% e3% 81% ae% e8% a8% ad% e5% ae% 9a) -[Driver Version Settings](#driver% e3% 83% 90% e3% 83% bc% e3% 82% b8% e3% 83% a7% e3% 83% b3% e3% 81% ae% e8% a8 % ad% e5% ae% 9a)

Introduction

Although it is an article on Mac environment, the procedure is the same for Windows environment. Please read and try the environment-dependent part.

Purpose

After reading this article to the end, you will be able to:

--Automate web processes using Selenium and Chrome Driver --Set Chrome and Driver on Heroku

Web process automation

Obtain and display ** USD / JPY ** from the FX chart rate of Yahoo! Finance.

View manually from browser Display automatically acquired data
Screenshot 2019-11-17 16.21.49.png Screenshot 2019-11-17 0.33.50.png

Related articles

Execution environment

environment Ver.
macOS Mojave 10.14.6
Python 3.7.3
Flask 1.1.1
selenium 3.141.0
chromedrive 78.0.3904.70
google-chrome 78.0.3904.97

Source code

I think that understanding will deepen if you read while actually following the implementation contents and source code. Please use it by all means.

GitHub

Scenarios and prerequisites

  1. Obtain and display ** USD / JPY ** from the FX chart rate of Yahoo! Finance.
  2. It is supposed to be saved in MySQL or PostgreSQL, but DB related is out of scope for the purpose of explaining automation.
  3. The API framework adopts Flask.

Creating an automated process

Python API configuration

tree.sh


python-Selenium
  ├── fx_rate
  │   ├── __init__.py
  │   └── utility.py
  └── main.py

API mainframe

main.py


import datetime
import logging
import os

from flask import Flask

from fx_rate.utility import get_fx_rate

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

app = Flask(__name__)


@app.route('/fx-rate', methods=['GET'])
def get():
    usd_jpy = get_fx_rate()
    res = 'timestamp={}, USDJPY={}'.format(
        datetime.datetime.utcnow() + datetime.timedelta(hours=9), usd_jpy)
    logger.info(res)
    return res, 200


if __name__ == '__main__':
    host = os.getenv('HOST', '0.0.0.0')
    port = int(os.getenv('PORT', 5000))
    app.run(host=host, port=port, debug=True)

Forex rate acquisition

utility.py


import datetime

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By


def get_fx_rate():
    options = Options()
    options.add_argument('--headless')
    driver = webdriver.Chrome(options=options)
    driver.get('https://info.finance.yahoo.co.jp/fx/')
    usd_jpy = driver.find_element(By.ID, 'USDJPY_top_bid').text
    driver.quit()
    return usd_jpy


if __name__ == '__main__':
    usd_jpy = get_fx_rate()
    print('timestamp={}, USDJPY={}'.format(datetime.datetime.utcnow() +
                                           datetime.timedelta(hours=9), usd_jpy))

Heroku settings

Chrome and Driver settings

Add the following from ** Add buildpack ** in the Settings> Buildpacks section.

スクリーンショット 2019-11-16 21.12.36.png
Buildpack URL
chromedrive https://github.com/heroku/heroku-buildpack-chromedriver.git
google-chrome https://github.com/heroku/heroku-buildpack-google-chrome.git

`If you deploy to Heroku, it will be installed automatically. Please register in advance. If you have already deployed or want to redeploy without changing the source code, try the empty commit below. ``

allow_empty.sh


~$ git commit --allow-empty -m "allow empty commit"
~$ git push heroku master

Driver version setting

Normally, no setting is required. If the version of ** chromedrive ** and ** google-chrome ** are different, such as when Chrome is upgraded, you need to specify the version of ** chromedrive **.

  1. Click ** Reveal Config Vars ** in the Settings> Config Vars section.
  2. Enter ** KEY ** and ** VALUE ** and click ** Add **.
スクリーンショット 2019-11-16 21.12.25.png
KEY VALUE
CHROMEDRIVER_VERSION Supported version(Example: 78.0.3904.70)

Recommended Posts

Automat WEB process with Heroku + Selenium + Chrome
Easy web app with Python + Flask + Heroku
Practice web scraping with Python and Selenium
I tried to make a periodical process with CentOS7, Selenium, Python and Chrome
Scraping with selenium
Scraping with selenium ~ 2 ~
Download PDF with selenium chromedriver (Chrome 57) + (Other notes)
Automatic operation of Chrome with Python + Selenium + pandas
Try running Google Chrome with Python and Selenium
Scraping with Selenium
Beginning with Selenium
Automatic login to ServiceNow with Selenium Web Driver
Automate Chrome with Python and Selenium on your Chromebook
How to switch to smartphone mode with Python + Selenium + Chrome
Deploy a web app created with Streamlit to Heroku
Successful scraping with Selenium
ScreenShot with Selenium (Python)
Scraping with Selenium [Python]
Python web scraping selenium
Gaussian process with pymc3
How to deploy a web app made with Flask to Heroku
(Failure) Deploy a web app made with Flask on heroku