This article is a relay article of 2021 New Year Advent Calendar Tech Connect! of Link Information Systems. Tech Connect! Is relayed by group members of engineer.hanzomon as an advent calendar to start on its own.
This article is for people who missed the sale during this time.
It is a common one that only integrates the scraped result with Slack, but in order to differentiate it from other articles, Google Cloud Platform (hereinafter GCP), Microsoft Azure (hereinafter Azure), Amazon Web Services (hereinafter AWS) Let's deploy each one.
Go to the My Nintendo Store Oniken page once a day and link to Slack if it's on sale.
Check https://store-jp.nintendo.com/robots.txt. There is no problem because "/ list/software /" was not disallowed.
Change the name to something that is easy to understand.
The one who just has to move for the time being.
import json
import os
import requests
from bs4 import BeautifulSoup
res = requests.get(os.environ['TARGET_URL'])
soup = BeautifulSoup(res.text, 'html.parser')
div = soup.select_one('div.productDetail--detail__pricePrice')
if div is not None:
item = {
"text": div.text.strip()
}
json_data = json.dumps(item)
res = requests.post(os.environ['WEBHOOK_URL'], json_data)
Get the following information (price) with div = soup.select_one ('div.productDetail--detail__pricePrice')
.
If you don't get this, it's not on sale and will not work with Slack.
The part of res = requests.post (os.environ ['WEBHOOK_URL'], json_data)
.
Just post the json data to the webhook URL.
This time it's a simple message, but if you want to make something complicated, I think you should try various things with Block Kit Builder.
requirements.txt
beautifulsoup4==4.9.1
requests==2.24.0
Cloud Functions Responsible for scraping and linking with Slack.
"Basic" as you like. Set the "trigger" as shown in the image and "save" it. ** Make a note of the URL as you will use it later **
Name | Value |
---|---|
TARGET_URL | URL of Oniken page |
WEBHOOK_URL | Slack's Webhook URL that you noted earlier |
After setting, click "Next".
Please note that it is not a copy and paste.
After copying and pasting, "deploy".
This will take some time.
Cloud Scheduler Use it to run Cloud Functions once a day. ** You can create up to 3 jobs with a free frame **
There is a region selection screen after this screen, but I forgot to take a screenshot. Please choose your favorite place.
Name, description, etc. as you like.
Item | Value |
---|---|
Target | HTTP |
URL | Cloud Functions trigger URL created earlier |
HTTP method | GET |
After setting, "Create".
This is the end of deploying to GCP.
With this configuration, you can't write code on the Azure portal, so you'll need to do a lot of preparation. I will omit the details because it seems to be long.
As you like except the red frame. Others are "confirm and create" by default.
"Move to resource" when the deployment is complete.
Set the following two and "Save".
Name | Value |
---|---|
TARGET_URL | URL of Oniken page |
WEBHOOK_URL | Slack's Webhook URL that you noted earlier |
It is an operation with Visual Studio Code.
This time, we will work in the azure_sample folder created on Desktop.
Select "Timer trigger".
It is specified to run at 01:00 every day.
After waiting for a while, a file called __init__.py
will be opened, so rewrite it with the contents prepared in advance.
There is requirements.txt in the project folder (azure_sample this time), so edit it.
It will take some time.
It is deployed properly.
This is the end of deploying to Azure. Thanks to the Timer trigger, it was possible to realize it by the function application alone.
Lambda
You can name the function as you like. After selecting options and runtime, "Create Function".
python library from bs4 to urllib3.
lambda_function.py
is a script prepared in advance for lambda.
oniken.zip
├ bs4
├ certifi
├ chardet
├ idna
├ requests
├ soupsieve
├ urllib3
└ lambda_function.py
lambda_function.py
import json
import os
import requests
from bs4 import BeautifulSoup
def lambda_handler(event, context):
res = requests.get(os.environ['TARGET_URL'])
soup = BeautifulSoup(res.text, 'html.parser')
div = soup.select_one('div.productDetail--detail__pricePrice')
if div is not None:
item = {
"text": div.text.strip()
}
json_data = json.dumps(item)
res = requests.post(os.environ['WEBHOOK_URL'], json_data)
return "success"
Set the following two and "Save".
Name | Value |
---|---|
TARGET_URL | URL of Oniken page |
WEBHOOK_URL | Slack's Webhook URL that you noted earlier |
It is specified to run at 02:00 every day.
This is the end of deploying to AWS. It was troublesome because I had to upload the library myself. (GCP, Azure just write requirements.txt)
Notifications were scheduled to arrive at 00:00 (GCP), 01:00 (Azure), and 02:00 (AWS), but Azure and AWS are off by about 9 hours. GCP set the time zone in Cloud Scheduler, but by the way, I feel that there was no such setting in Azure and AWS.
Azure Looking at the log, it was running at 01:03 (UTC). Is there a 3 minute gap?
AWS Looking at the log, it was also running at 02:00 (UTC).
This is my homework because it has moved.
Azure was overwhelmingly easy if the environment could be built. Timer trigger is great.
type | Easy | Usually | Difficult |
---|---|---|---|
Environment is not built | GCP | AWS | Azure |
Environment built | Azure | GCP | AWS |
--Create a function in Azure using Visual Studio Code and Python --Configure function app settings in Azure | Microsoft Docs --Python AWS Lambda Deployment Package --AWS Lambda
** What is that? ** ** Quoted from Microsoft Store Oniken Page.
Inspired by the harsh gameplay of 8-bit action/platform in the post-inventory future controlled by sinister companies, the demon sword is against enemies, traps, and challenging boss gauntlets. Striking the agility and sword.
As you can see from a little reading, Japanese is strange.
** Find the translation source ** Quoted from Microsoft Store Oniken Page. The same text on the publisher's product page.
In a post-apocalyptic future controlled by a sinister corporation, ninja mercenary Zaku is the last hope for humanity. Inspired by the demanding gameplay of 8-bit action/platformers, Oniken pits your agility and swordplay against a gauntlet of enemies, traps, and challenging bosses.
The "table of contents" was the "apocalyptic".
** Get translated ** As far as the original text (Japanese page) is seen, it seems that some kind of translation will be done, so I will ask you to translate the text of the English page on some sites.
In the post-apocalyptic future dominated by sinister companies, Ninja Mercenary Zaku is the last hope for humanity. Inspired by the demanding gameplay of 8-bit action/platformers, Oniken defeats agility and swordsmanship against enemies, traps and challenging boss gauntlets.
In a post-apocalyptic future dominated by evil companies, the ninja mercenary Zaku was humanity's last hope. Inspired by the rigorous gameplay of 8-bit action/platformers, "Oniken" uses agility and swordsmanship to break through obstacles such as enemies, traps, and bosses.
In a post-apocalyptic future dominated by sinister companies, Ninja Mercenary Zaku is the last hope of mankind. Inspired by the tough gameplay of 8-bit action/platforms, Oniken pits your agility and sword play against enemies, traps, and challenging boss gauntlets.
Ninja mercenary Zaku is the last hope for humanity in the backward movement after being dominated by an ominous enterprise. Inspired by the tough game of 8-bit action/small plates, the demon sword swiftly pours swords on enemies, traps, and challenging bosses, and swords on its abdomen.
** Google Translate results are close to the original ** Except for DeepL Translator, the three results are almost similar, and Google Translate is quite close to the original text except for the Ninja Mercenary Zaku part. Let's see the difference by arranging the original text and the result of Google Translate side by side.
--Original
Inspired by the ** harsh ** gameplay of 8-bit ** action/** platform ** in the future ** controlled ** by the sinister company ** after inventory ** Being done, the demon sword will ** hit ** your ** agility and ** sword ** against ** enemies, traps, ** and ** challenging bosses ** gauntlets **. **.
--Google Translate (excluding the Ninja Mercenary Zaku part)
In the future ** dominated ** dominated by sinister companies ** after the end , Oniken was inspired by ** demanding ** gameplay of 8-bit action/ platformers ** ** Defeats ** agility and ** swordsmanship ** against ** gauntlets ** of enemies, traps, and challenging bosses **.
It seems that there is only a level difference that the translation result has changed with the update of Google Translate.
** Deepening mystery ** I've come to the point that the text on the Japanese page is probably based on the results of Google Translate, but there remains a mystery that I can't understand.
--Tablebook I have no idea what and how the apocalypse becomes a table of contents. English → (Translation) → Japanese → (Speech to Text) → (Speech to Text) → If you use Japanese, there is a possibility that you will make a mistake in hearing at Speech to Text, but you have to do something like this. It doesn't make sense.
--The disappeared ninja mercenary Zaku Zaku is the main character of Oniken. When translating
in conclusion Please let me know if anyone can get closer to the answer.
Recommended Posts