-[Automate web process with Heroku + Selenium + Chrome](# heroku--selenium--chrome% e3% 81% a7web% e3% 83% 97% e3% 83% ad% e3% 82% bb% e3% 82% b9% e3% 82% 92% e8% 87% aa% e5% 8b% 95% e5% 8c% 96% e3% 81% 99% e3% 82% 8b) -[Introduction](#% e3% 81% af% e3% 81% 98% e3% 82% 81% e3% 81% ab) -[Purpose](#% e7% 9b% ae% e7% 9a% 84) -[Related Articles](#% e9% 96% a2% e9% 80% a3% e3% 81% 99% e3% 82% 8b% e8% a8% 98% e4% ba% 8b) -[Execution environment](#% e5% ae% 9f% e8% a1% 8c% e7% 92% b0% e5% a2% 83) -[Source Code](#% e3% 82% bd% e3% 83% bc% e3% 82% b9% e3% 82% b3% e3% 83% bc% e3% 83% 89) -[Scenario and Prerequisites](#% e3% 82% b7% e3% 83% 8a% e3% 83% aa% e3% 82% aa% e3% 81% a8% e5% 89% 8d% e6% 8f% 90% e6% 9d% a1% e4% bb% b6) -[Create Automatic Process](#% e8% 87% aa% e5% 8b% 95% e3% 83% 97% e3% 83% ad% e3% 82% bb% e3% 82% b9% e3% 81% ae% e4% bd% 9c% e6% 88% 90) -[Python API configuration](# python-api% e6% a7% 8b% e6% 88% 90) -[API Mainframe](#api% e3% 83% a1% e3% 82% a4% e3% 83% b3% e3% 83% 95% e3% 83% ac% e3% 83% bc% e3% 83% a0) -[Get FX Rate](# fx% e3% 83% ac% e3% 83% bc% e3% 83% 88% e5% 8f% 96% e5% be% 97) -[Heroku Settings](# heroku% e3% 81% ae% e8% a8% ad% e5% ae% 9a) -[Chrome and Driver Settings](# chrome% e3% 81% a8driver% e3% 81% ae% e8% a8% ad% e5% ae% 9a) -[Driver Version Settings](#driver% e3% 83% 90% e3% 83% bc% e3% 82% b8% e3% 83% a7% e3% 83% b3% e3% 81% ae% e8% a8 % ad% e5% ae% 9a)
Although it is an article on Mac environment, the procedure is the same for Windows environment. Please read and try the environment-dependent part.
After reading this article to the end, you will be able to:
--Automate web processes using Selenium and Chrome Driver --Set Chrome and Driver on Heroku
Web process automation
Obtain and display ** USD / JPY ** from the FX chart rate of Yahoo! Finance.
View manually from browser | Display automatically acquired data |
---|---|
environment | Ver. |
---|---|
macOS Mojave | 10.14.6 |
Python | 3.7.3 |
Flask | 1.1.1 |
selenium | 3.141.0 |
chromedrive | 78.0.3904.70 |
google-chrome | 78.0.3904.97 |
I think that understanding will deepen if you read while actually following the implementation contents and source code. Please use it by all means.
tree.sh
python-Selenium
├── fx_rate
│ ├── __init__.py
│ └── utility.py
└── main.py
main.py
import datetime
import logging
import os
from flask import Flask
from fx_rate.utility import get_fx_rate
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
app = Flask(__name__)
@app.route('/fx-rate', methods=['GET'])
def get():
usd_jpy = get_fx_rate()
res = 'timestamp={}, USDJPY={}'.format(
datetime.datetime.utcnow() + datetime.timedelta(hours=9), usd_jpy)
logger.info(res)
return res, 200
if __name__ == '__main__':
host = os.getenv('HOST', '0.0.0.0')
port = int(os.getenv('PORT', 5000))
app.run(host=host, port=port, debug=True)
utility.py
import datetime
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
def get_fx_rate():
options = Options()
options.add_argument('--headless')
driver = webdriver.Chrome(options=options)
driver.get('https://info.finance.yahoo.co.jp/fx/')
usd_jpy = driver.find_element(By.ID, 'USDJPY_top_bid').text
driver.quit()
return usd_jpy
if __name__ == '__main__':
usd_jpy = get_fx_rate()
print('timestamp={}, USDJPY={}'.format(datetime.datetime.utcnow() +
datetime.timedelta(hours=9), usd_jpy))
Add the following from ** Add buildpack ** in the Settings> Buildpacks section.
Buildpack | URL |
---|---|
chromedrive | https://github.com/heroku/heroku-buildpack-chromedriver.git |
google-chrome | https://github.com/heroku/heroku-buildpack-google-chrome.git |
`If you deploy to Heroku, it will be installed automatically. Please register in advance. If you have already deployed or want to redeploy without changing the source code, try the empty commit below. ``
allow_empty.sh
~$ git commit --allow-empty -m "allow empty commit"
~$ git push heroku master
Normally, no setting is required. If the version of ** chromedrive ** and ** google-chrome ** are different, such as when Chrome is upgraded, you need to specify the version of ** chromedrive **.
KEY | VALUE |
---|---|
CHROMEDRIVER_VERSION | Supported version(Example: 78.0.3904.70) |
Recommended Posts