Disconnect and reconnect

Perhaps because it's a Rotor human, the Stream API maintains an HTTP connection, but it's unreliable, or ** it disconnects even if nothing happens. Rather, I am wary of being disconnected ** because there is no problem. Even if you look at the previous reference book (pocket reference), it is written in such a way that you need to assume reconnection because it will be disconnected if something happens. …… But, even if you look at the page of a well-known library or the example of implementing it by yourself, there is no page that describes reconnection after disconnection within the range that you can see. * ...... Can you do that with the API? If so, it's never been so easy ...

(Note for those who know: I'm writing in chronological order in the brain while coding and remembering, so please wait for a while)

Library selection

I have a lot of things to think about, but I decided to consider a Twitter-connected library with a lot of Japanese materials from the perspective of referring to the wonderful code of my predecessors.

• Of course, it needs to support Stream API
• Ideally, you should get a quick answer by google. Based on that, I decided to use Tweepy .

When I searched for a library from the same viewpoint on the database side,

• Anyway, the amount of coding description is small
• You can usually find the answer by google.
• It looks like MongoDB official or something close to that That's why I decided to use PyMongo .

An iron plate that tends to be apt? Well, if Shiroto had to do something about it, he would have to use the standard one ...

Development environment.

After deciding what to use, let's prepare a development environment and a test environment. I'm a Gatchigachi Win shop who came in from VB and came in C → VC ++ → C #. Of course, the environment that can be used for development is also Windows, so it goes without saying that it is ideal to be able to develop on Win until just before release = implementation. Or rather, if you don't have an IDE (Integrated Development Environment), you will die instantly. Even more so, if you ask me anything on Linux, I have no choice but to sit down.

Fortunately, since it is a script language, there is not much dependence on the environment, and nowadays the installation of libraries etc. is automated, so the trouble should be much less than in the past.

• Use Anaconda as a way to install Python and its peripheral libraries in bulk. There is no problem with installation, leaving everything up to environment variables.
• I thought about MongoDB for a moment, but Windows version exists , so that's okay. To say.
  At first, I thought about setting it on Docker for Windows that can also use Hyper-V , but When I actually try it, I stumbled on data persistence and the solution also fails, so it's annoying ~~ ~~ Win version is fine.

I think this is fine for infrastructure, but I want to use IDE, etc. ~~ I licked the world ~~ When I was googled in a familiar environment, Python Tools for Visual Studio is something like that. ** Moreover, you can easily debug on the spot by calling Win version Python such as Anaconda **. Because there is only this anymore

• Anaconda (Python 3.5.2 :: Anaconda 4.1.1 (64-bit) )
• MongoDB for Win(MongoDB shell version: 3.2.10)
• VisualStudio2013 + Python Tools For Visual Studio(Ver.2.2.2)
• Tweepy + PyMongo (latest at the time of acquisition)

Decided to develop and test with the configuration. As a concern,

1. Is there any problem with execution in the execution environment? Is there any part that depends on the environment?
2. Can you do enough testing in the development environment? Will there be any unforeseen problems?
3. The version around the infrastructure may change depending on the development environment and execution environment.

Around, but ... Well, in any case, the actual machine test is necessary at the end, and if you do not write it well, no problem will occur, so leave it for the time being. (If this is a job, I'm afraid I have to pack it all around ...)

Program requirements

The program I'm making this time carries the demon's rule that once you start running, you can keep running anything for 3 months and stop it. Therefore, we will implement it by focusing on the essential functions, and do something else by other means **.

Functions to be implemented with the highest priority

• Receive Public Stream from Twitter
• Storage of received data in MongoDB
• Ability to reconnect in the event of an unexpected disconnect

Function to be implemented as a second priority

• A function to record when an event such as connection, disconnection, or reconnection occurs.
• In addition to the above, a function to notify the occurrence of an event with Twitter's Direct Message (parental desire to contact)

Features to implement if possible

• Notification of daily flow rate, remaining storage capacity, etc.

Function to consider if there is spare capacity

• Automatically ends when a specific date and time arrives

Functions not included in the required specifications

• Blacklisted specific account repelling feature (as a feature within the program).
• Processing of tweet data assuming real-time analysis (as a function in the program)

Well, it looks like this. Let's start with the elements with the highest priority and gradually improve the degree of perfection.

Twitter reception program for the time being

With the vow of "** I have O'Reilly's Python tutorial " and " I google if I don't understand **", I started creating a demonstration program for the time being. From Creating a New Project in Visual Studio, specify "Python Application" and you're ready to write Python code in your familiar editor. This is convenient. After typing the tutorial code, execute it with [F5]. You can try the execution with the same procedure as the C language console application, so there is really no stress. I'm sorry I can't step.

Well, if you can't talk to Twitter first, it's out of the question, so let's start by making that area. You need to install Tweepy first. If you think that you can type the "pip" command from Anaconda Prompt, this can also be executed from Visual Studio.

1. Select [View]-> [Other Windows]-> [Python Environments] from the menu bar to display.
2. Select pip from the center dropdown in the Python Environments window.
3. Enter "tweepy" in the "Search PyPl and installed packages" text box.
4. Click "" pip install tweepy "from PyPl". Then the installation is completed.

It's as fun as NuGet.

Stream based on some pages found by google for the time being and tutorial of the original Tweepy Write a program to get tweets with API.

tweetCheck.py

#!/usr/bin/env python
# -*- coding:utf-8 -*-

import tweepy
#Prepare the variables required to execute the Twitter API by yourself.
CK = ''   # Consumer Key
CS = ''   # Consumer Secret
AT = ''   # Access Token
AS = ''   # Accesss Token Secert

class Listener(tweepy.StreamListener):
    def on_status(self, status):
        print(status.text.encode('shift_jis', 'ignore'))
        return True

    def on_error(self, status_code):
        print('Error occurred: ' + str(status_code))
        return True

#Main processing from here
auth = tweepy.OAuthHandler(CK, CS)
auth.set_access_token(AT, AS)     #Obtaining an access token

listener = Listener()                       #Instance of Listener class
stream = tweepy.Stream(auth, listener)      #Reception starts from here.

#Select one and uncomment it.
#stream.filter(track=['#xxxxxx'])  #Filter by specified search word
stream.sample()                    #1 from all tweets on Twitter%pick up
#stream.userstream()               #User's own TL

…… Eh, 31 lines (blank lines, including comments)? What can I do with this? ?? Execute while thinking. I can't read it (because it's UTF-8), but I can receive it. Forcibly terminate with Ctrl + C.

Notes

It feels like it's moving quickly, but it's actually clogged in two places.

• Only one line is displayed, but it is forcibly terminated immediately.
  → The character code of the py file created by Visual Studio was "Shift-JIS".
  Select UTF-8 from [Menu] → [File] → [Detailed settings for saved file] and save.
• If you execute it after fixing the above, it will be displayed for a moment, but it will be forcibly terminated. The timing is random.
  → The print under "def on_status (self, status):" was initially "print (status.text)", but
  <a href="http://lab.hde.co" .jp / 2008/08 / pythonunicodeencodeerror.html "> I heard that I die trying to display characters that cannot be displayed at the command prompt.
  When I tried to convert the encoding, it came to be displayed although it was garbled.

The former is okay if you fix it the first time. The latter is okay because it doesn't need to be displayed if it's always moving. If it's the first time Python can manage to this extent in a few days, it may be unexpectedly in time for the end of October.

Next time, I will flesh out this sauce. (Continue)