Program for Twitter Trend Analysis (Personal Note)
Hello. I am currently studying Twitter trends ** as the title suggests in my second year student at Komazawa University ** GMS Faculty **. ** This article introduces research, code, and potential references.
Originally, I was interested in Twitter trends, and since I was in the first grade, I have been using Python, Twitter API, and MeCab, but it was ** a primitive thing that morphologically analyzes and aggregates word by word **. Besides, I was playing with language and location information and appearing kanji () ↓ Then, in the same way as N-gram, for example, we recorded every 2-12 vocabulary and aggregated all ** simple trend analysis **. As a side note, I made a lot of manual rules about where the particles don't come and what the auxiliary verbs are, as in the Twitter trend. This is the beginning of the second grade ↓ After that, when it came to what to study, I had the idea of defining and modeling a stationary trend that fluctuates throughout the day, but I was wondering what it would be like. Then, a teacher at a certain university pointed out that the points and rules were in the hands of others. It's about the summer of the second grade ↓ I was wandering a little, but https://pj.ninjal.ac.jp/corpus_center/goihyo.html ** I arrived at the National Institute for Japanese Language and Language Unidic, Classification Vocabulary, Semantic Classification **, and tried to do semantic normalization and clustering of words. It was quite good, but the correspondence table of multiple dictionaries? It was supposed to be a homonym when reverse lookup, but the meaning was referred to by an extreme person, so some people got involved, but the problem is I don't think so. Also, there is an unknown word in Unidic, and there is a problem that the trend is not taken into consideration. Is it about October in the second grade? ↓ When I was able to normalize the meaning, I used CaboCha because I had been thinking from the previous stage that I wanted to use the dependency to create a trend with the dependency of the meaning. ** This was a hassle and I couldn't do it on the seminar server or AWS, so I had a hard time installing it on my Windows and binding it with subprocess (since Python has only 32bit). Since the position to separate Unidic and Cabocha (Can Cabocha also be specified in the dictionary?) Was different, we matched them in the intersection in the way of the least common multiple. ↓ As a result, we were able to create a semantic trend based on the dependency of meaning. ** However, since it says "It should be a homonym, but the meaning is referred to by an extreme person", I made a correction, and that is "Hit in the campaign! There is a method called "follow and tweet", and in "simple trend analysis", there was a method to delete using duplication of expressed trends, but I decided to use that as well. ↓ So far, we have been able to analyze "meaning trends based on the dependency of meaning" and "simple expression trends", and now I think that meaning trends are difficult to separate, unlike expressions. I mean, it's a deep psychology or something big, and I think it's important to make it concrete in this research. Independent? Aggregating by semantic trend accompanying the expression trend, ** estimating the expression trend from the meaning trend, how much both can be used properly, how to combine ** are future issues. Now in November of my second year, I am doing my best for my graduation research. I wrote it because I had a paragraph. (Maybe I should have written it earlier)
Nowadays, the expression trend is a list of characters, and the meaning trend is a list of meanings in which characters are replaced with meanings.
** If you want to use it, please let me know even on Twitter. I was worried about that. Please assume that you own the copyright. ** ** I also wanted to write it in a backup sense. The environment is Windows 10-64bit Python 3 (Miscellaneous) https://twitter.com/kenkensz9
First, it is a program for estimating the expression trend. custam_freq_sentece.txt is the full text obtained for parsing custam_freq_tue.txt is a trend candidate. custam_freq.txt is a trend. custam_freq_new.txt is the program that puts out the longest trend by excluding duplicates from the trend. In addition, trends are changed every hour. The part of freshtime = int (time.time () * 1000)-200000. This should be changed according to the speed of tweet acquisition. Also, there is a word list badword that will not be processed if this word is included in the tweet in the first place, and this may not be the version of the program, but management is poor, so please feel free to use it.
hyousyutu_trend.py
# coding: utf-8
import tweepy
import datetime
import re
import itertools
import collections
from pytz import timezone
import time
import MeCab
#import threading
#from multiprocessing import Pool
import os
#import multiprocessing
import concurrent.futures
import urllib.parse
#import
import pdb; pdb.set_trace()
import gc
import sys
import emoji
consumer_key = ""
consumer_secret = ""
access_token = ""
access_token_secret = ""
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
authapp = tweepy.AppAuthHandler(consumer_key,consumer_secret)
apiapp = tweepy.API(authapp)
#Authentication
m_owaka = MeCab.Tagger("-Owakati")
m_ocha = MeCab.Tagger("-Ochasen")
#mecab morpheme decomposition definition
lang_dict="{'en': 'English', 'und': 'unknown', 'is': 'Icelandic', 'ay': 'Aymara', 'ga': 'Irish', 'az': 'Azerbaigen', 'as': 'Assamese', 'aa': 'Afar', 'ab': 'Aphazian', 'af': 'Afrikaans', 'am': 'Amharic', 'ar': 'Arabic', 'sq': 'Albanian', 'hy': 'Armenian', 'it': 'Italian', 'yi': 'Yiddish', 'iu': 'Inuktitut', 'ik': 'Inupia', 'ia': 'Interlingua', 'ie': 'Interlingue', 'in': 'Indonesian', 'ug': 'Uyghur', 'cy': 'Welsh', 'vo': 'Volapuk', 'wo': 'Wolof', 'uk': 'Ukrainian', 'uz': 'Uzbek', 'ur': 'Urdu', 'et': 'Estonian', 'eo': 'Esperanto', 'or': 'Orian', 'oc': 'Okitan', 'nl': 'Dutch', 'om': 'Oromo', 'kk': 'Kazakh', 'ks': 'Kashmir', 'ca': 'Catalan', 'gl': 'Galician', 'ko': 'Korean', 'kn': 'Kannada', 'km': 'Cambodian', 'rw': 'Kyawanda', 'el': 'Greek language', 'ky': 'Kyrgyz', 'rn': 'Kirundi', 'gn': 'Guarani', 'qu': 'Quetua', 'gu': 'Gujarati', 'kl': 'Greenlandic', 'ku': 'Kurdish', 'ckb': '中央Kurdish', 'hr': 'Croatian', 'gd': 'Gaelic', 'gv': 'Gaelic', 'xh': 'Xhosa', 'co': 'Corsican', 'sm': 'Samoan', 'sg': 'Sangho', 'sa': 'Sanskrit', 'ss': 'Swati', 'jv': 'Javanese', 'ka': 'Georgian', 'sn': 'Shona', 'sd': 'Sindhi', 'si': 'Sinhala', 'sv': 'Swedish', 'su': 'Sudanese', 'zu': 'Zulu', 'es': 'Spanish', 'sk': 'Slovak', 'sl': 'Slovenian', 'sw': 'Swahili', 'tn': 'Setswana', 'st': 'Seto', 'sr': 'Serbian', 'sh': 'セルボCroatian', 'so': 'Somali', 'th': 'Thai', 'tl': 'Tagalog', 'tg': 'Tajik', 'tt': 'Tatar', 'ta': 'Tamil', 'cs': 'Czech language', 'ti': 'Tigrinya', 'bo': 'Tibetan', 'zh': 'Chinese', 'ts': 'Zonga', 'te': 'Telugu', 'da': 'Danish', 'de': 'German', 'tw': 'Twi', 'tk': 'Turkmen', 'tr': 'Turkish', 'to': 'Tongan', 'na': 'Nauruan', 'ja': 'Japanese', 'ne': 'Nepali', 'no': 'Norwegian', 'ht': 'Haitian', 'ha': 'Hausa', 'be': 'White Russian', 'ba': 'Bashkir', 'ps': 'Pasito', 'eu': 'Basque', 'hu': 'Hungarian', 'pa': 'Punjabi', 'bi': 'Bislama', 'bh': 'Bihari', 'my': 'Burmese', 'hi': 'Hindi', 'fj': 'Fijian', 'fi': 'Finnish', 'dz': 'Bhutanese', 'fo': 'Faroese', 'fr': 'French', 'fy': 'Frisian', 'bg': 'Bulgarian', 'br': 'Breton', 'vi': 'Vietnamese', 'iw': 'Hebrew', 'fa': 'Persian', 'bn': 'Bengali', 'pl': 'Polish language', 'pt': 'Portuguese', 'mi': 'Maori', 'mk': 'Macedonian', 'mg': 'Malagasy', 'mr': 'Malata', 'ml': 'Malayalam', 'mt': 'Maltese', 'ms': 'Malay', 'mo': 'Moldavian', 'mn': 'Mongolian', 'yo': 'Yoruba', 'lo': 'Laota', 'la': 'Latin', 'lv': 'Latvian', 'lt': 'Lithuanian', 'ln': 'Lingala', 'li': 'Limburgish', 'ro': 'Romanian', 'rm': 'Rate romance', 'ru': 'Russian'}"
lang_dict=eval(lang_dict)
lang_dict_inv = {v:k for k, v in lang_dict.items()}
#Language dictionary
all=[]
#List initialization
if os.path.exists('custam_freq_tue.txt'):
alll=open("custam_freq_tue.txt","r",encoding="utf-8-sig")
alll=alll.read()
all=eval(alll)
del alll
#all=[]
#Ready to export
#freq_write=open("custam_freq.txt","w",encoding="utf-8-sig")
sent_write=open("custam_freq_sentece.txt","a",encoding="utf-8-sig", errors='ignore')
#Ready to export
use_lang=["Japanese"]
use_type=["tweet"]
#config
uselang=""
for k in use_lang:
k_key=lang_dict_inv[k]
uselang=uselang+" lang:"+k_key
#config preparation
def inita(f,k):
suball=[]
small=[]
for s in k:
if not int(f)==int(s[1]):
#print("------",f)
suball.append(small)
small=[]
#print(s[0],s[1])
small.append(s)
f=s[1]
suball.append(small)
#If 2 is included
return suball
def notwo(al):
micro=[]
final=[]
kaburilist=[]
for fg in al:
kaburilist=[]
if len(fg)>1:
for v in itertools.combinations(fg, 2):
micro=[]
for s in v:
micro.append(s[0])
micro=sorted(micro,key=len,reverse=False)
kaburi=len(set(micro[0]) & set(micro[1]))
per=kaburi*100//len(micro[1])
#print(s[1],per,kaburi,len(micro[0]),len(micro[1]),"m",micro)
if per>50:
kaburilist.append(micro[0])
kaburilist.append(micro[1])
else:
final.append([micro[0],s[1]])
#print("fin1",micro[0],s[1])
if micro[0] in micro[1]:
pass
#print(micro[0],micro[1])
#print("included"*5)
#if micro[0] in kaburilist:
# kaburilist.remove(micro[0])
else:
pass
#print(fg[0][1],fg[0][0])
final.append([fg[0][0],fg[0][1]])
#print("fin3",fg[0][0],fg[0][1])
#if kaburilist:
#longword=max(kaburilist,key=len)
#final.append([longword,s[1]])
##print("fin2",longword,s[1])
#kaburilist.remove(longword)
#kaburilist=list(set(kaburilist))
#for k in kaburilist:
# if k in final:
# final.remove(k)
# #print("finremove1",k)
return final
def siage(fin):
fin=list(map(list, set(map(tuple, fin))))
finallen = sorted(fin, key=lambda x: len(x[0]))
finallendic=dict(finallen)
finalword=[]
for f in finallen:
finalword.append(f[0])
#print("f1",finalword)
notwo=[]
for v in itertools.combinations(finalword, 2):
#print(v)
if v[0] in v[1]:
#print("in")
if v[0] in finalword:
finalword.remove(v[0])
#print("f2",finalword)
finall=[]
for f in finalword:
finall.append([f,finallendic[f]])
finall = sorted(finall, key=lambda x: int(x[1]), reverse=True)
#print("final",finall)
kk=open("custam_freq_new.txt", 'w', errors='ignore')
kk.write(str(finall))
kk.close()
def eval_pattern(use_t):
tw=0
rp=0
rt=0
if "tweet" in use_t:
tw=1
if "retweet" in use_t:
rt=1
if "reply" in use_t:
rp=1
sword=""
if tw==1:
sword="filter:safe OR -filter:safe"
if rp==0:
sword=sword+" exclude:replies"
if rt==0:
sword=sword+" exclude:retweets"
elif tw==0:
if rp==1 and rt ==1:
sword="filter:reply OR filter:retweets"
elif rp==0 and rt ==0:
print("NO")
sys.exit()
elif rt==1:
sword="filter:retweets"
elif rp==1:
sword="filter:replies"
return sword
pat=eval_pattern(use_type)+" "+uselang
#config read function and execution
def a(n):
return n+1
def f(k):
k = list(map(a, k))
return k
def g(n,m):
b=[]
for _ in range(n):
m=f(m)
b.append(m)
return b
#Serial number list generation
def validate(text):
if re.search(r'(.)\1{1,}', text):
return False
elif re.search(r'(..)\1{1,}', text):
return False
elif re.search(r'(...)\1{1,}', text):
return False
elif re.search(r'(...)\1{1,}', text):
return False
elif re.search(r'(....)\1{1,}', text):
return False
else:
return True
#Function to check for duplicates
def eval_what_nosp(c,i):
no_term=[]
no_start=[]
no_in=[]
koyu_meisi=[]
if re.findall(r"[「」、。)(『』&@_;【/＜＞,！】＼／@]", c[0]):
no_term.append(i)
no_start.append(i)
no_in.append(i)
if len(c) == 4:
if "suffix" in c[3]:
no_start.append(i)
if "Proper noun" in c[3]:
koyu_meisi.append(i)
if c[3]=="noun-Non-independent-General":
no_term.append(i)
no_start.append(i)
no_in.append(i)
if "Particle" in c[3]:
no_term.append(i)
no_start.append(i)
#no_in.append(i)
if c[3]=="Particle-Attributive":
no_start.append(i)
if c[3]=="Particle":
no_start.append(i)
if "O" in c[2]:
if c[3]=="noun-Change connection":
no_term.append(i)
no_start.append(i)
no_in.append(i)
if len(c) == 6:
if c[4]=="Sahen Suru":
no_start.append(i)
if c[3]=="verb-Non-independent":
no_start.append(i)
if "suffix" in c[3]:
no_start.append(i)
if c[3]=="Auxiliary verb":
if c[2]=="Ta":
no_start.append(i)
no_in.append(i)
if c[3]=="Auxiliary verb":
if c[2]=="Absent":
no_start.append(i)
if c[3]=="Auxiliary verb":
if "Continuous use" in c[5]:
no_term.append(i)
no_start.append(i)
if c[2]=="To do":
if c[3]=="verb-Independence":
if c[5]=="Continuous form":
no_start.append(i)
no_in.append(i)
if c[2]=="Become":
if c[3]=="verb-Independence":
no_start.append(i)
no_in.append(i)
if c[2]=="Teru":
if c[3]=="verb-Non-independent":
no_start.append(i)
no_in.append(i)
if c[2]=="is":
if c[3]=="Auxiliary verb":
no_start.append(i)
no_in.append(i)
if c[2]=="Chau":
if c[3]=="verb-Non-independent":
no_start.append(i)
no_in.append(i)
if c[2]=="is there":
if c[3]=="verb-Independence":
no_term.append(i)
no_start.append(i)
no_in.append(i)
if c[2]=="Auxiliary verb":
if c[3]=="Special da":
no_term.append(i)
no_start.append(i)
no_in.append(i)
if c[2]=="Trout":
if c[3]=="Auxiliary verb":
no_term.append(i)
no_start.append(i)
no_in.append(i)
if "Continuous use" in c[5]:
no_term.append(i)
if c[5]=="Word connection":
no_start.append(i)
if c[2]=="Give me":
if c[3]=="verb-Non-independent":
no_start.append(i)
no_in.append(i)
x=""
y=""
z=""
koyu=""
if no_term:
x=no_term[0]
if no_start:
y=no_start[0]
if no_in:
z=no_in[0]
if koyu_meisi:
koyu=koyu_meisi[0]
#print("koyu",koyu)
koyu=int(koyu)
return x,y,z,koyu
small=[]
nodouble=[]
seq=""
def process(ty,tw,un,tagg):
global all
global seq
global small
global nodouble
tw=tw.replace("\n"," ")
sent_write.write(str(tw))
sent_write.write("\n")
parselist=m_owaka.parse(tw)
parsesplit=parselist.split()
parseocha=m_ocha.parse(tw)
l = [x.strip() for x in parseocha[0:len(parseocha)-5].split('\n')]
nodouble=[]
no_term=[]
no_start=[]
no_in=[]
km_l=[]
for i, block in enumerate(l):
c=block.split('\t')
#sent_write.write("\n")
#sent_write.write(str(c))
#sent_write.write("\n")
#print(str(c))
ha,hi,hu,km=eval_what_nosp(c,i)
no_term.append(ha)
no_start.append(hi)
no_in.append(hu)
km_l.append(km)
#Completed writing
if km_l[0]:
for r in km_l:
strin=parsesplit[r]
if not strin in nodouble:
all.append([strin,un])
nodouble.append(strin)
for s in range(2,8):
#A chain of 2 to 8.
#Important because you can improve accuracy instead of making it heavier
num=g(len(parsesplit)-s+1,range(-1,s-1))
for nr in num:
#2 for one sentence-All streets of 8 chains
#print(no_term)
if not len(set(nr) & set(no_in)):
if not nr[-1] in no_term:
if not nr[0] in no_start:
small=[]
#print(str(parsesplit))
for nr2 in nr:
#print(nr2,parsesplit[nr2])
#Add word to small at the position indexed by the array inside
small.append(parsesplit[nr2])
seq="".join(small)
judge_whole=0
bad_direct_word=["Like","\'mat","I\'mat"]
#if "" in seq:
# judge_whole=1
#if "" in seq:
# judge_whole=1
for bd in bad_direct_word:
if seq==bd:
judge_whole=1
break
parselist=m_owaka.parse(seq)
l = [x.strip() for x in parseocha[0:len(parseocha)-5].split('\n')]
for n in range(len(l)):
if len(l[n].split("\t"))==6:
if l[n].split("\t")[3]=="verb-Independence":
if len(l[n+1].split("\t"))==6:
if l[n+1].split("\t")[3]:
judge_whole=1
break
if judge_whole==0:
if validate(seq) and len(seq) > 3 and not re.findall(r'[「」、。『』/\＼／@]', seq):
if not seq in nodouble:
#Continuous avoidance
all.append([seq,un])
nodouble.append(seq)
#print("Added successfully",seq)
#Do not aggregate the same word twice
else:
#print("Already included",seq)
pass
else:
#print("Exclusion",seq)
pass
else:
#print("The beginning is no_is start",seq)
pass
else:
#print("The end is no_term",seq)
pass
#print("\n")
#print(parsesplit)
#print(l)
if tagg:
print("tagg",tagg)
for sta in tagg:
all.append(["#"+str(sta),un])
#Include tag
N=1
#Number of tweets acquired
def print_varsize():
import types
print("{}{: >15}{}{: >10}{}".format('|','Variable Name','|',' Size','|'))
print(" -------------------------- ")
for k, v in globals().items():
if hasattr(v, 'size') and not k.startswith('_') and not isinstance(v,types.ModuleType):
print("{}{: >15}{}{: >10}{}".format('|',k,'|',str(v.size),'|'))
elif hasattr(v, '__len__') and not k.startswith('_') and not isinstance(v,types.ModuleType):
print("{}{: >15}{}{: >10}{}".format('|',k,'|',str(len(v)),'|'))
def collect_count():
global all
global deadline
hh=[]
tueall=[]
#print("alllll",all)
freshtime=int(time.time()*1000)-200000
deadline=-1
#import pdb; pdb.set_trace()
#print(N_time)
print(len(N_time))
for b in N_time:
if int(b[1]) < freshtime:
deadline=b[0]
print("dead",deadline)
dellist=[]
if not deadline ==-1:
for b in N_time:
print("b",b)
if int(b[0]) < int(deadline):
dellist.append(b)
for d in dellist:
N_time.remove(d)
#print(N_time)
#import pdb; pdb.set_trace()
#time.sleep(2)
#import pdb; pdb.set_trace()
for a in all:
if int(a[1]) > freshtime:
#Number of tweets you want to get/45*Subtract the value of 1000. Now 5000/45*1000=112000
tueall.append(a[0])
#print("tuealllappend"*10)
#print(tueall)
else:
all.remove(a)
#print("allremove",a)
#import pdb; pdb.set_trace()
c = collections.Counter(tueall)
c=c.most_common()
#print("c",c)
#print(c)
for r in c:
if r and r[1]>1:
hh.append([str(r[0]),str(r[1])])
k=str(hh).replace("[]","")
freq_write=open("custam_freq.txt","w",encoding="utf-8-sig", errors='ignore')
freq_write.write(str(k))
#import pdb; pdb.set_trace()
oldunix=N_time[0][1]
newunix=N_time[-1][1]
dato=str(datetime.datetime.fromtimestamp(oldunix/1000)).replace(":","-")
datn=str(datetime.datetime.fromtimestamp(newunix/1000)).replace(":","-")
dato=dato.replace(" ","_")
datn=datn.replace(" ","_")
#print(dato,datn)
#import pdb; pdb.set_trace()
freq_writea=open("trenddata/custam_freq-"+dato+"-"+datn+"--"+str(len(N_time))+".txt","w",encoding="utf-8-sig", errors='ignore')
freq_writea.write(str(k))
#import pdb; pdb.set_trace()
freq_write_tue=open("custam_freq_tue.txt","w",encoding="utf-8-sig", errors='ignore')
freq_write_tue.write(str(all))
#print(c)
def remove_emoji(src_str):
return ''.join(c for c in src_str if c not in emoji.UNICODE_EMOJI)
def deEmojify(inputString):
return inputString.encode('ascii', 'ignore').decode('ascii')
def get_tag(tw,text_content):
taglist=[]
entities=eval(str(tw.entities))["hashtags"]
for e in entities:
text=e["text"]
taglist.append(text)
for _ in range(len(taglist)+2):
for s in taglist:
text_content=re.sub(s,"",text_content)
#text_content=re.sub(r"#(.+?)+ ","",text_content)
return taglist,text_content
def get_time(id):
two_raw=format(int(id),'016b').zfill(64)
unixtime = int(two_raw[:-22],2) + 1288834974657
unixtime_th = datetime.datetime.fromtimestamp(unixtime/1000)
tim = str(unixtime_th).replace(" ","_")[:-3]
return tim,unixtime
non_bmp_map = dict.fromkeys(range(0x10000, sys.maxunicode + 1), '')
N_time=[]
def gather(tweet,type,tweet_type,removed_text):
global N
global N_all
global lagtime
global all_time
global all
global auth
global N_time
if get_time(tweet.id):
tim,unix=get_time(tweet.id)
else:
exit
#Get detailed tweet time
#original_text=tweet.text
nowtime=time.time()
tweet_pertime=str(round(N/(nowtime-all_time),1))
lag=str(round(nowtime-unix/1000,1))
#Calculate lag
lang=lang_dict[tweet.lang]
print(N_all,N,tweet_pertime,"/s","+"+lag,tim,type,tweet_type,lang)
#Information display.(All tweets, processed tweets, processing speed, lag, real time, acquisition route, tweet type, language)
print(removed_text.replace("\n"," "))
taglist,tag_removed_text=get_tag(tweet,removed_text)
#import pdb; pdb.set_trace()
#print(type(tweet))
#import pdb; pdb.set_trace()
#Exclude tags
noemoji=remove_emoji(tag_removed_text)
try:
process(tweet_type,tag_removed_text,unix,taglist)
N_time.append([N,unix])
print("trt",tag_removed_text)
except Exception as pe:
print("process error")
print(pe)
#import pdb; pdb.set_trace()
#Send to actual processing
surplus=N%1000
if surplus==0:
#sumprocess()
try:
collect_count()
except Exception as eeee:
print(eeee)
#exit
#Let's count
cft_read=open("custam_freq.txt","r",encoding="utf-8-sig")
cft_read=cft_read.read()
cft_read=eval(cft_read)
max_freq=cft_read[0][1]
#Maximum value
allen=inita(max_freq,cft_read)
#Make a list of trends with the same frequency.
finf=notwo(allen)
#Find and remove duplicate strings and trends
siage(finf)
#New it_Write as freq
print_varsize()
#Display memory information
N=N+1
#streaming body
def judge_tweet_type(tweet):
text = re.sub("https?://[\w/:%#\$&\?\(\)~\.=\+\-]+","",tweet.text)
if tweet.in_reply_to_status_id_str :
text=re.sub(r"@[a-zA-Z0-9_]* ","",text)
text=re.sub(r"@[a-zA-Z0-9_]","",text)
return "reply",text
else:
head= str(tweet.text).split(":")
if len(head) >= 2 and "RT" in head[0]:
text=re.sub(r"RT @[a-zA-Z0-9_]*: ","",text)
return "retwe",text
else:
return "tweet",text
badword=["Question box","Let's throw marshmallows","I get a question","Participation in the war","Delivery","@","Follow","Application","Smartphone RPG","Gacha","S4live","campaign","Drift spirits","Present","Cooperative live","We are accepting consultations completely free of charge","Omikuji","Chance to win","GET","get","shindanmaker","Hit","lottery"]
N_all=0
def gather_pre(tweet,type):
global N_all
N_all=N_all+1
#Count all tweets passing through here
go=0
for b in badword:
if b in tweet.text:
go=1
break
#Judge whether there is a bad word in the text, GO judgment because it is not included in 0
if go == 0:
if tweet.lang=="ja":
tweet_type,removed_text=judge_tweet_type(tweet)
#Determine tweet type
if tweet_type=="tweet":
try:
gather(tweet,type,tweet_type,removed_text)
#print(type(tweet))
#Send to gather processing.
except Exception as eee:
#gather("Ah","Ah","Ah","Ah")
#import pdb; pdb.set_trace()
pass
lagtime=0
def search(last_id):
#print(pat)
global pat
time_search =time.time()
for status in apiapp.search(q=pat,count="100",result_type="recent",since_id=last_id):
#Get newer tweets than the last tweet you got with search
gather_pre(status,"search")
#search body
interval = 2.16
#search call interval
#min2
trysearch=0
#search number of calls
class StreamingListener(tweepy.StreamListener):
def on_status(self, status):
global time_search
global trysearch
gather_pre(status,"stream")
time_stream=time.time()
time_stream-time_search % interval
if time_stream-time_search-interval>interval*trysearch:
#For a certain period of time(interbal)Execute search every time.
last_id=status.id
#executor = concurrent.futures.ThreadPoolExecutor(max_workers=8)
#executor.submit(search(last_id))
#When trying parallel processing
search(last_id)
trysearch=trysearch+1
#streaming body
def carry():
listener = StreamingListener()
streaming = tweepy.Stream(auth, listener)
streaming.sample()
#stream call function
time_search =time.time()
#The time when the search was last executed, but defined before the stream
executor = concurrent.futures.ThreadPoolExecutor(max_workers=8)
#Parallel definition
all_time=time.time()
#Execution start time definition
try:
carry()
except Exception as e:
import pdb; pdb.set_trace()
print(e)
#import pdb; pdb.set_trace()
pass
#except Exception as ee:
#print(ee)
#import pdb; pdb.set_trace()
#carry body and error handling
Below is a program of meaning trends, but I can recommend it with confidence because it is being highly acclaimed (). The part I bring from the thesaurus is not something I wrote, but I will leave it.
imi_trend.py
from bs4 import BeautifulSoup
import collections
import concurrent.futures
import datetime
import emoji
import itertools
import MeCab
from nltk import Tree
import os
from pathlib import Path
from pytz import timezone
import re
import spacy
import subprocess
import sys
import time
import tweepy
import unidic2ud
import unidic2ud.cabocha as CaboCha
from urllib.error import HTTPError, URLError
from urllib.parse import quote_plus
from urllib.request import urlopen
m=MeCab.Tagger("-d ./unidic-cwj-2.3.0")
os.remove("bunrui01.csv")
os.remove("all_tweet_text.txt")
os.remove("all_kakari_imi.txt")
bunrui01open=open("bunrui01.csv","a",encoding="utf-8")
textopen=open("all_tweet_text.txt","a",encoding="utf-8")
akiopen=open("all_kakari_imi.txt","a",encoding="utf-8")
catedic={}
with open('categori.txt') as f:
a=f.read()
aa=a.split("\n")
b=[]
bunrui01open.write(",,,")
for i, j in enumerate(aa):
catedic[j]=i
bunrui01open.write(str(j))
bunrui01open.write(",")
bunrui01open.write("\n")
print(catedic)
with open('./BunruiNo_LemmaID_ansi_user.csv') as f:
a=f.read()
aa=a.split(",\n")
b=[]
for bb in aa:
if len(bb.split(","))==2:
b.append(bb.split(","))
word_origin_num_to_cate=dict(b)
with open('./cate_rank2.csv') as f:
a=f.read()
aa=a.split("\n")
b=[]
for bb in aa:
if len(bb.split(","))==2:
b.append(bb.split(","))
cate_rank=dict(b)
class Synonym:
def getSy(self, word, target_url, css_selector):
try:
#Encoded because the URL to access contains Japanese
self.__url = target_url + quote_plus(word, encoding='utf-8')
#Access and parse
self.__html = urlopen(self.__url)
self.__soup = BeautifulSoup(self.__html, "lxml")
result = self.__soup.select_one(css_selector).text
return result
except HTTPError as e:
print(e.reason)
except URLError as e:
print(e.reason)
sy = Synonym()
alist = ["Selection"]
#Use "Japanese Thesaurus Associative Thesaurus" to search
target = "https://renso-ruigo.com/word/"
selector = "#content > div.word_t_field > div"
#for item in alist:
# print(sy.getSy(item, target, selector))
consumer_key = ""
consumer_secret = ""
access_token = ""
access_token_secret = ""
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)
authapp = tweepy.AppAuthHandler(consumer_key,consumer_secret)
apiapp = tweepy.API(authapp)
#Authentication(Here api)
def remove_emoji(src_str):
return ''.join(c for c in src_str if c not in emoji.UNICODE_EMOJI)
def get_tag(tw,text_content):
taglist=[]
entities=eval(str(tw.entities))["hashtags"]
for e in entities:
text=e["text"]
taglist.append(text)
for _ in range(len(taglist)+2):
for s in taglist:
text_content=re.sub(s,"",text_content)
#text_content=re.sub(r"#(.+?)+ ","",text_content)
return taglist,text_content
def get_swap_dict(d):
return {v: k for k, v in d.items()}
def xcut(asub,a):
asub.append(a[0])
a=a[1:len(a)]
return asub,a
def ycut(asub,a):
asub.append(a[0])
a=a[1:len(a)]
return asub,a
def bunruikugiri(lastx,lasty):
hoge=[]
#import pdb; pdb.set_trace()
editx=[]
edity=[]
for _ in range(500):
edity,lasty=ycut(edity,lasty)
#target=sum(edity)
for _ in range(500):
target=sum(edity)
#rint("sum",sum(editx),"target",target)
if sum(editx)<target:
editx,lastx=xcut(editx,lastx)
elif sum(editx)>target:
edity,lasty=ycut(edity,lasty)
else:
hoge.append(editx)
editx=[]
edity=[]
if lastx==[] and lasty==[]:
return hoge
break
all_appear_cate=[]
all_unfound_word=[]
all_kumiawase=[]
nn=1
all_kakari_imi=[]
def process(tw,ty):
global nn
wordnum_toword={}
catenum_wordnum={}
word_origin_num=[]
mozisu=[]
try:
tw=re.sub("https?://[\w/:%#\$&\?\(\)~\.=\+\-]+","",tw)
tw=tw.replace("#","")
tw=tw.replace(",","")
tw=tw.replace("\u3000","") #Important for matching the number of characters
tw=re.sub(re.compile("[!-/:-@[-`{-~]"), '', tw)
parseocha=m.parse(tw)
print(tw)
l = [x.strip() for x in parseocha[0:len(parseocha)-5].split('\n')]
bunrui_miti_sentence=[]
for i, block in enumerate(l):
if len(block.split('\t')) > 1:
c=block.split('\t')
d=c[1].split(",")
#Word processing process
print(d,len(d))
if len(d)>9:
if d[10] in ["To do"]:
word_origin_num.append(d[10])
bunrui_miti_sentence.append(d[8])
mozisu.append(len(d[8]))
elif d[-1] in word_origin_num_to_cate:
word_origin_num.append(int(d[-1]))
wordnum_toword[int(d[-1])]=d[8]
bunrui_miti_sentence.append(word_origin_num_to_cate[str(d[-1])])
mozisu.append(len(d[8]))
else:
#print("nai",d[8])
#Display of unknown words
all_unfound_word.append(d[10])
bunrui_miti_sentence.append(d[8])
mozisu.append(len(c[0]))
else:
mozisu.append(len(c[0]))
all_unfound_word.append(c[0])
bunrui_miti_sentence.append(c[0])
#else:
# mozisu.append(l[])
#print("kouho",word_origin_num,"\n")
#Words to original numbers
#print(tw)
#If you look at sentences made with semantic classification and unknown words
for s in bunrui_miti_sentence:
print(s," ",end="")
print("\n")
stn=0
cmd = "echo "+str(tw)+" | cabocha -f1"
cmdtree="echo "+str(tw)+" | cabocha "
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE,shell=True)
proctree = subprocess.Popen(cmdtree, stdout=subprocess.PIPE, stderr=subprocess.PIPE,shell=True)
proc=proc.communicate()[0].decode('cp932')
proctree=proctree.communicate()[0].decode('cp932')
print(proctree)
proclist=proc.split("\n")
#print(proc)
#f1 information
#print(proclist)
#Listing information
procnumlist=[]
wordlis=[]
eachword=""
num=0
for p in proclist:
if p[0]=="*":
f=p.split(" ")[1]
t=p.split(" ")[2].replace("D","")
procnumlist.append([f,t])
if eachword:
wordlis.append([num,eachword])
num=num+1
eachword=""
elif p=="EOS\r":
wordlis.append([num,eachword])
num=num+1
eachword=""
break
else:
#print("aaaaa",p.split("\t")[0])
eachword=eachword+p.split("\t")[0]
tunagari_num_dict=dict(procnumlist)
print(tunagari_num_dict)
bunsetu_num_word=dict(wordlis)
#print(bunsetu_num_word)
bunsetu_mozisu=[]
for v in bunsetu_num_word.values():
bunsetu_mozisu.append(len(v))
if sum(bunsetu_mozisu) != sum(mozisu):
return
#print("mozisu",mozisu)
#print("bunsetumozi",bunsetu_mozisu)
res=bunruikugiri(mozisu,bunsetu_mozisu)
#print("res",res)
nnn=0
small_cateandcharlist=[]
big_cateandcharlist=[]
for gc in res:
for _ in range(len(gc)):
print(bunrui_miti_sentence[nnn],end=" ")
if bunrui_miti_sentence[nnn] in list(catedic.keys()):
small_cateandcharlist.append(bunrui_miti_sentence[nnn])
nnn=nnn+1
#Unknown words and particles are considered to be the same, so the mecabne gold dictionary can be used.
if small_cateandcharlist==[]:
big_cateandcharlist.append(["null"])
else:
big_cateandcharlist.append(small_cateandcharlist)
small_cateandcharlist=[]
print("\n")
#print("bcacl",big_cateandcharlist)
twewtnai_kakari_imi=[]
if len(big_cateandcharlist)>1 and len(big_cateandcharlist)==len(bunsetu_num_word):
#Dependencies and morphological analysis delimiters do not match
for kk, vv in tunagari_num_dict.items():
if vv != "-1":
for aaw in big_cateandcharlist[int(kk)]:
for bbw in big_cateandcharlist[int(vv)]:
twewtnai_kakari_imi.append([aaw,bbw])
if not "Rank symbol" in str([aaw,bbw]):
if not "null" in str([aaw,bbw]):
if not "Number sign" in str([aaw,bbw]):
if not "Things" in str([aaw,bbw]):
all_kakari_imi.append(str([aaw,bbw]))
akiopen.write(str([aaw,bbw]))
else:
break
else:
return
akiopen.write("\n")
akiopen.write(str(bunrui_miti_sentence))
akiopen.write("\n")
akiopen.write(str(tw))
akiopen.write("\n")
print("tki",twewtnai_kakari_imi)
tweetnai_cate=[]
word_cate_num=[]
for k in word_origin_num:
if str(k) in word_origin_num_to_cate:
ram=word_origin_num_to_cate[str(k)]
print(ram,cate_rank[ram],end="")
tweetnai_cate.append(ram)
all_appear_cate.append(ram)
word_cate_num.append(catedic[ram])
catenum_wordnum[catedic[ram]]=int(k)
stn=stn+1
else:
if k in ["To do"]:
all_appear_cate.append(k)
tweetnai_cate.append(k)
print("\n")
#print(tweetnai_cate)
#import pdb; pdb.set_trace()
for k in tweetnai_cate:
if k in catedic:
aac=catedic[k]
#print("gyaku",word_cate_num)
#print("wt",wordnum_toword)
#print("cw",catenum_wordnum)
bunrui01open.write(str(tw))
bunrui01open.write(",")
bunrui01open.write(str(tim))
bunrui01open.write(",")
bunrui01open.write(str(unix))
bunrui01open.write(",")
ps=0
for tt in list(range(544)):
if int(tt) in word_cate_num:
a=catenum_wordnum[tt]
#Word number from the sword
bunrui01open.write(str(wordnum_toword[a]))
#Word from word number
bunrui01open.write(",")
ps=ps+1
else:
bunrui01open.write("0,")
bunrui01open.write("end")
bunrui01open.write("\n")
textopen.write(str(nn))
textopen.write(" ")
textopen.write(tw)
textopen.write("\n")
nn=nn+1
#Put all the streets
for k in list(itertools.combinations(tweetnai_cate,2)):
all_kumiawase.append(k)
except Exception as ee:
print(ee)
import pdb; pdb.set_trace()
pass
def judge_tweet_type(tweet):
if tweet.in_reply_to_status_id_str:
return "reply"
else:
head= str(tweet.text).split(":")
if len(head) >= 2 and "RT" in head[0]:
return "retwe"
else:
return "tweet"
#Judging whether it is a rip, retweet, or tweet
def get_time(id):
two_raw=format(int(id),'016b').zfill(64)
unixtime = int(two_raw[:-22],2) + 1288834974657
unixtime_th = datetime.datetime.fromtimestamp(unixtime/1000)
tim = str(unixtime_th).replace(" ","_")[:-3]
return tim,unixtime
#Tweet time from id
N=1
def gather(tweet,type,tweet_typea):
global all_appear_cate
global N
global all_time
global tim
global unix
tim,unix=get_time(tweet.id)
original_text=tweet.text.replace("\n","")
taglist,original_text=get_tag(tweet,original_text)
nowtime=time.time()
tweet_pertime=str(round(N/(nowtime-all_time),1))
lag=str(round(nowtime-unix/1000,1))
#lang=lang_dict[tweet.lang]
try:
process(remove_emoji(original_text),tweet_typea,)
except Exception as e:
print(e)
#import pdb; pdb.set_trace()
pass
print(N,tweet_pertime,"/s","+"+lag,tim,type,tweet_typea)
N=N+1
if N%500==0:
ccdd=collections.Counter(all_appear_cate).most_common()
for a in ccdd:
print(a)
#ccdd=collections.Counter(all_unfound_word).most_common()
#for a in ccdd:
# print("Absent",a)
ccdd=collections.Counter(all_kumiawase).most_common(300)
for a in ccdd:
print(a)
ccdd=collections.Counter(all_kakari_imi).most_common(300)
for a in ccdd:
print("all_kakari_imi",a)
#import pdb; pdb.set_trace()
#All tweets of stream and search are collected
def pre_gather(tw,ty):
#print(ty)
# if "http://utabami.com/TodaysTwitterLife" in tw.text:
print(tw.text)
if ty=="stream":
tweet_type=judge_tweet_type(tw)
if tw.lang=="ja" and tweet_type=="tweet":
gather(tw,ty,tweet_type)
elif ty=="search":
gather(tw,ty,"tweet")
def search(last_id):
time_search =time.time()
for status in apiapp.search(q="filter:safe OR -filter:safe -filter:retweets -filter:replies lang:ja",count="100",result_type="recent",since_id=last_id):
pre_gather(status,"search")
#search body
class StreamingListener(tweepy.StreamListener):
def on_status(self, status):
global time_search
global trysearch
pre_gather(status,"stream")
time_stream=time.time()
time_stream-time_search % interval
if time_stream-time_search-interval>interval*trysearch:
last_id=status.id
#executor = concurrent.futures.ThreadPoolExecutor(max_workers=2)
#executor.submit(search(last_id))
search(last_id)
trysearch=trysearch+1
#streaming body
def carry():
listener = StreamingListener()
streaming = tweepy.Stream(auth, listener)
streaming.sample()
interval = 2.1
trysearch=0
time_search =time.time()
#executor = concurrent.futures.ThreadPoolExecutor(max_workers=2)
all_time=time.time()
try:
#executor.submit(carry)
carry()
except Exception as er:
print(er)
import pdb; pdb.set_trace()
pass
For every 500 tweets Number of occurrences of simple meaning Number of occurrences of 2-gram meaning The number of occurrences of a meaningful continuous? Also for all Tweet text, Unidic analysis information, CaboCha dependency, replacement with semantic classification, etc.
bunrui01.csv-The horizontal axis is the meaning classification of 544, the vertical axis is the tweet, 0 does not exist, 1 is the csv to write so that it is the corresponding word all_tweet_text-Processed tweets and what number they are all_kakari_imi-Dependent meaning pair, meaning classification replacement, text categori.txt-A txt that describes the semantic classification of 544 and creates a dictionary with index at runtime. For more information on BunruiNo_LemmaID_ansi_user.csv, see https://pj.ninjal.ac.jp/corpus_center/goihyo.html As you can see, it is a correspondence table of word original numbers and meaning classifications. cate_rank2.csv-A dictionary of the order of appearance of semantic classifications created at one time.
I will explain the other variables later,
It's for my own memo, and people who understand it will do their best to understand it, so I will do this.
Recommended Posts