A story that visualizes the present of Qiita with Qiita API + Elasticsearch + Kibana

How to get basic post data

$ curl -XGET 'https://qiita.com/api/v2/items?page=3&per_page=20'

If you do, I think that the result will be returned for the time being. The meaning of this command is to get 20 pieces of data on the 3rd page. In other words, note that the number of data that can be obtained is 20, not 3 * 20.

You can decide page and per_page yourself, but there are restrictions. It is summarized in the table below.

If you get 100 post data from each page, you can get up to 10000 post data.

Python code that sends post data (json format) to Elasticsearch as it is

forward_json.py

# coding: utf-8
import json
import requests
from elasticsearch import Elasticsearch

#Address of the server on which Elasticsearch is installed
server_address = "xxx.xxx.xxx.xxx"
#If installed and standard settings, the port is 9200
port = str(9200)
#Create an instance of Elasticsearch
es = Elasticsearch("%s:%s" % (server_address, port))
#end point
endpoint = 'https://qiita.com/api/v2/items'
for p in range(1, 11): #Perform the following processing from page 1 to page 10 in the same way.
    payload = {'page': p, 'per_page': '100'} #Get 100 data per page
    r =  requests.get(endpoint, params=payload).json() #Receive the result in json format
    '''
    #For your reference
    print type(r)
    # => <type 'list'>
    print r[0].keys() 
    # => [u'body', u'group', u'rendered_body', u'url', u'created_at', u'tags', u'updated_at', u'private', u'coediting', u'user', u'title', u'id']
    '''
    for it in r: #Loop through the list of results
        #Insert all the data! !!
        #This time I tried to name index qiita
        es.index(index='qiita', doc_type='qiita', id=it['id'], body=it)

In server_address, write the address of the server where Elasticsearch and Kibana are installed. When you run this code, Elasticsearch should store a total of 1000 post data.

Visualization with Kibana

Go to http //xxx.xxx.xxx.xxx:5601. I fetched 1000 post data from Qiita API and stored it in Elasticsearch. This is a screenshot with numbers and graph descriptions added in red. For the time being, the user name was hidden.

There are times when the number of posts is extremely high, so click on it.

Click the green circle in the image above to change the page.

It seems that the number of posts during that time period has increased due to the angry posts by enthusiastic users.

Summary of results

• You can see who posted how much in 1000 post data
• You may get various information by following people who are currently posting with enthusiasm.
• The number of tags is 1076, but the Japanese tag was not obtained well, and the tag "learning" was regarded as two tags, "learning" and "learning". Future tasks
• The table below summarizes the 1000 most popular tags.

The result was! After all there are many programming tags! Actually, I wanted to know the stock number ranking of tags, but I stopped because it was difficult to obtain the stock number from the posted data without authentication. I would like to challenge when I have time. This time, I found that there are many posts with python tags, so I would like to continue posting so that I can add python tags.