Create a Twitter BOT service with GAE / P + Tweepy + RIOT API! (Part 2)

Synopsis

Continuing from Part 1, in the second part, I would like to actually refer to various APIs and operate the BOT service. The source code is operated at GitBucket, and the finished product is operated at here.

Sequential processing in Task Queue

Since there are API usage restrictions to monitor each user's battle record and post it to Twitter if there is an update, it is necessary to control the process to be executed at intervals of seconds. Since Cron, which will be described later, can only be called at a minimum interval of 1 minute, each user's processing uses a mechanism called Task Queue provided by GAE. As an image of the Task Queue, the tasks are thrown into a container called Bucket, and the tasks are processed according to the settings in the order in which they were put first. If you increase the Bucket size, it will be parallel processing. The configuration file is queue.yaml.

queue.yaml


queue:
- name: tweet #Anything is fine
  rate: 0.8/s #0 per second.Process at a pace of 8 times
  bucket_size: 1 #Number to process in parallel
  retry_parameters:
    task_retry_limit: 0 #Number of retries when processing fails

Now, use add ()` `` to actually throw a task from Python into the Bucket set in` `` queue.yaml. Specify `name``` set in queue.yaml``` for `` `` queue_name appropriately. Specify the address of the process to be called in ```url appropriately`. You can also pass parameters.

launcher.py


from google.appengine.api.taskqueue import add

import webapp2

class modelTask(db.Model): #Queued tasks
    resion = db.StringProperty()
    summoner_name = db.StringProperty()
    summoner_id = db.IntegerProperty()
    latest_game = db.IntegerProperty()
    access_key = db.StringProperty()
    access_secret = db.StringProperty()
    date_success = db.DateTimeProperty(auto_now_add=True)
    date = db.DateTimeProperty(auto_now_add=True)

class mainHandler(webapp2.RequestHandler):
    def get(self):
        qs = modelTask.all().order('-date_success')
        for q in qs: #Add all tasks to Queue
            add(queue_name='tweet', url='/tweet', params={'qid': q.key().id()})

app = webapp2.WSGIApplication([ ('/launcher', mainHandler) ])

Next, implement the process specified in ```url appropriately` that is called when the turn comes around in the Task Queue. Called by the POST method from the Task Queue.

tweet.py


#! -*- coding: utf-8 -*-
from google.appengine.ext import db
from google.appengine.api.urlfetch import fetch
from django.utils.simplejson import loads

import webapp2, tweepy
from datetime import datetime

from laucher import modelTask

CONSUMER_KEY = '********************'
CONSUMER_SECRET = '**************************************'
RIOT_KEY = '***********************************'

class mainHandler(webapp2.RequestHandler):
    def post(self):
        getGame(long(self.request.get('qid')))

def getGame(qid):
    q = modelTask().get_by_id(qid, parent=None)
    #Call the RIOT API
    result = fetch('https://prod.api.pvp.net/api/lol/'+q.resion+'/v1.3/game/by-summoner/'+str(q.summoner_id)+'/recent?api_key='+RIOT_KEY)
    if result.status_code == 200:
        #Set various values obtained from API
        j = loads(result.content)['games'][0]
        if j['stats']['win'] == True:
            win = 'victory'
        else:
            win = 'defeat'
        try:
            kill = str(j['stats']['championsKilled'])
        except:
            kill = '0'
        try:
            death = str(j['stats']['numDeaths'])
        except:
            death = '0'
        try:
            assist = str(j['stats']['assists'])
        except:
            assist = '0'

        game_type = j['subType']

        #Post on Twitter if there is an update in the final battle time
        if j['createDate'] > q.latest_game:
            q.latest_game = j['createDate']
            q.put()
            if tweet(q.summoner_name+'Mr. latest'+game_type+'The battle record is'+kill+'kill'+death+'death'+assist+'Assist'+win+'is . http://tol.orfx.jp #Tweet_of_Legends', q.access_key, q.access_secret):
                q.date_success = datetime.now()
                q.put()

#Twitter post processing
def tweet(message, access_key, access_secret):
    try:
        auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
        auth.set_access_token(access_key, access_secret)
        api = tweepy.API(auth_handler=auth, api_root='/1.1', secure=True)
        api.update_status(status=message)
        
        return True
    except:
        return False

app = webapp2.WSGIApplication([ ('/tweet', mainHandler) ])

Periodic processing with Cron

I use Cron because I want to periodically call the process of adding the above task to the queue. The configuration file is cron.yaml. This time, I'll try to run it every 10 minutes from 1 am to 6 pm, and every 5 minutes otherwise.

cron.yaml


cron:
- description: tweet job
  url: /launcher
  schedule: every 10 minutes from 1:00 to 17:59
  timezone: Asia/Tokyo #Don't forget the time zone when specifying the time
  
- description: tweet job
  url: /launcher
  schedule: every 5 minutes from 18:00 to 0:59
  timezone: Asia/Tokyo

Resource distribution with Backend

By the way, if you run it with this Cron (every 5 to 10 minutes) setting, the instance startup time is calculated in 15-minute increments, so Frontend instance startup time will definitely consume 24 hours or more. In addition to that, registration page processing etc. will be added, so I am worried about the 28 hours of the free frame (as of March 2014). So, let's take advantage of Backend. Backend is originally a mechanism for doing behind-the-scenes work such as batch processing and asynchronous processing, but this time it is simply used to earn instance startup time (9 hours). The configuration file is backend.yaml.

backend.yaml


backends:
- name: tweet #anything is fine
  class: B1 #Minimize processing resources
  options: dynamic #Make the instance non-resident

Specify `` `target``` to make the process Backend in Cron. This time, I will process it with Backend from 6 pm to 1 am.

cron:
- description: tweet job
  url: /launcher
  schedule: every 10 minutes from 1:00 to 17:59
  timezone: Asia/Tokyo
  
- description: tweet job
  url: /launcher
  schedule: every 5 minutes from 18:00 to 0:59
  timezone: Asia/Tokyo
  target: tweet #backend.yaml name

TaskQueue can also be processed by Backend by specifying add (target =) `` `. Use `` `get_backend ()` `` to see if you are running with Backend. Also, when Backend is started, `` `/ _ah / start is forcibly called. Since it is called first, you can describe the process at startup, but this time we will describe the empty process.

launcher.py


from google.appengine.api.taskqueue import add
from google.appengine.api.backends import get_backend

import webapp2

class modelTask(db.Model):
    resion = db.StringProperty()
    summoner_name = db.StringProperty()
    summoner_id = db.IntegerProperty()
    latest_game = db.IntegerProperty()
    access_key = db.StringProperty()
    access_secret = db.StringProperty()
    date_success = db.DateTimeProperty(auto_now_add=True)
    date = db.DateTimeProperty(auto_now_add=True)

class mainHandler(webapp2.RequestHandler):
    def get(self):
        qs = modelTask.all().order('-date_success')
        target = get_backend()
        if target is None: #If it is started by Backend, TaskQueue will be the same.
            for q in qs:
                add(queue_name='tweet', url='/tweet', params={'qid': q.key().id()})
        else:
            for q in qs:
                add(queue_name='tweet', url='/tweet', params={'qid': q.key().id()}, target='tweet')

class startHandler(webapp2.RequestHandler): #Forced to be called when Backend starts
    def get(self):
        return

app = webapp2.WSGIApplication([ ('/launcher', mainHandler), ('/_ah/start', startHandler) ])

Reduce resource consumption with Memcache

Memcache can be used to significantly reduce resource consumption for processes that frequently browse the data store but rarely update it. In RIOT API game-v1.3, Champion is returned by ID, so it is necessary to convert the ID to a name based on the information of champion-v1.1. Since you can't hit the API frequently, the information acquired by champion-v1.1 is stored once in the data store. Then copy the information stored in Memcache. Use `memcache.add (key, value)` to add data to Memcache.

champion.py


from google.appengine.api import memcache
from google.appengine.ext import db
from google.appengine.api.urlfetch import fetch
from django.utils.simplejson import loads

import webapp2

RIOT_KEY = '***********************************'

class modelChampion(db.Model): #Champion information storage model
    name = db.StringProperty()
    date = db.DateTimeProperty(auto_now_add=True)

class mainHandler(webapp2.RequestHandler):
    def get(self):
        #Champion information acquisition
        result = fetch('https://prod.api.pvp.net/api/lol/na/v1.1/champion?api_key='+RIOT_KEY)
        if result.status_code == 200:
            js = loads(result.content)['champions']
            for j in js:
                #Champion information storage
                modelchampion = modelChampion().get_or_insert(str(j['id']))
                modelchampion.name = j['name']
                modelchampion.put()

                #Copy champion information to Memcache
                memcache.add("champion_"+str(j['id']), j['name'], 86399)

app = webapp2.WSGIApplication([ ('/champion', mainHandler) ], debug=True)

Use `memcache.get (key)` to reference the data added to Memcache. Since the data added to Memcache may be lost, it is necessary to describe the processing at that time as well. Now, let's add the champion name to the posted content in tweet.py.

tweet.py


from google.appengine.api import memcache
from champion import modelChampion

def getGame(qid):
    q = modelQueue().get_by_id(qid, parent=None)
    result = fetch('https://prod.api.pvp.net/api/lol/'+q.resion+'/v1.3/game/by-summoner/'+str(q.summoner_id)+'/recent?api_key='+RIOT_KEY)
    if result.status_code == 200:
        j = loads(result.content)['games'][0]
        if j['stats']['win'] == True:
            win = 'victory'
        else:
            win = 'defeat'
        try:
            kill = str(j['stats']['championsKilled'])
        except:
            kill = '0'
        try:
            death = str(j['stats']['numDeaths'])
        except:
            death = '0'
        try:
            assist = str(j['stats']['assists'])
        except:
            assist = '0'

        #Get champion information from Memcache
        champion = memcache.get("champion_"+str(j['championId']))
        if champion is None: #If you can't get champion information from Memcache, from the data store
            champion = modelChampion.get_by_key_name(str(j['championId'])).name

        game_type = j['subType']
        
        if j['createDate'] > q.latest_game:
            q.latest_game = j['createDate']
            q.put()
            if tweet(q.summoner_name+'Mr. latest'+game_type+'The battle record is'+champion+'so'+kill+'kill'+death+'death'+assist+'Assist'+win+'soす 。 http://tol.orfx.jp #Tweet_of_Legends', q.access_key, q.access_secret):
                q.date_success = datetime.now()
                q.put()

Summary

This is the first memorandum of app created with GAE. Once you get the hang of it, it's very easy to create an app from the next time, so it's recommended!

Recommended Posts

Create a Twitter BOT service with GAE / P + Tweepy + RIOT API! (Part 1)
Create a Twitter BOT service with GAE / P + Tweepy + RIOT API! (Part 2)
Steps to create a Twitter bot with python
I made a Twitter BOT with GAE (python) (with a reference)
Create a real-time auto-reply bot using the Twitter Streaming API
Create a Twitter BOT with the GoogleAppEngine SDK for Python
I made a Twitter Bot with Go x Qiita API x Lambda
[LINE Messaging API] Create a BOT that connects with someone with Python
Create a bot that boosts Twitter trends
Create a web service with Docker + Flask
Let's make a Twitter Bot with Python!
Tweet the weather forecast with a bot Part 2
Make a Twitter trend bot with heroku + Python
Create a LINE BOT with Minette for Python
[Life hack] Women's Anna support bot with Twitter API
Create a tweet heatmap with the Google Maps API
[LINE Messaging API] Create parrot return BOT with Python
Create a slack bot
I wrote a script to create a Twitter Bot development environment quickly with AWS Lambda + Python 2.7
Create a list in Python with all followers on twitter
Create a machine learning app with ABEJA Platform + LINE Bot
Create a Mastodon bot with a function to automatically reply with Python
Tornado-Let's create a Web API that easily returns JSON with JSON
Create a web API that can deliver images with Django
Create a social integration API for smartphone apps with Django
Create an API with Django
Use Twitter API with Python
Create a homepage with django
Create a heatmap with pyqtgraph
Support yourself with Twitter API
Create a directory with python
Successful update_with_media with twitter API
[AWS Hands-on] Let's create a celebrity identification service with a serverless architecture!
Create a word cloud with only positive / negative words on Twitter
Try to create a Qiita article with REST API [Environmental preparation]
Create a REST API to operate dynamodb with the Django REST Framework
How to create a serverless machine learning API with AWS Lambda
LINE BOT (Messaging API) development with API Gateway and Lambda (Python) [Part 2]
Beginners will make a Bitcoin automatic trading bot aiming for a lot of money! Part 2 [Transaction with API]