I scraped the Organization member team and made a ranking

SDT's Qiita Organization is getting hotter and hotter! So I tried to make a ranking somehow ~ (What does that mean !!)

Source code

nnsnodnb/sdt-qiita-ranking | GitHub

Public page sample

スクリーンショット 2016-07-23 午前1.22.55.png

It's almost like this. Approximately ... I can only see the ones in SDT yet, but in the future, I'm thinking or not thinking about creating an API so that ranking can be taken in any organization when I have free time !!

Scraping

package

requirements.txt


beautifulsoup4==4.5.0
bottle==0.12.9
Jinja2==2.8
MarkupSafe==0.23

Don't ask if you used Jinja2 or something :; (∩´﹏`∩) ;:

Import the package ♪

from urllib.request import urlopen
from bs4 import BeautifulSoup

Put together the parts to put together

get_soup()


url = 'http://qiita.com/organizations/smartdt/members'
html = urlopen(url)
return = BeautifulSoup(html, 'html.parser')

Member list and number of contributions

get_contribution()


soup = get_soup()

contribution_array = {}
# organizationMemberList_Get all li tags of item class
members = soup.findAll('li', class_='organizationMemberList_item')

for member in members:
    #Get the first a tag for each member
    a = member.find('a')
    #Organization Member List for each member_Get the first div tag of the memberProfile class
    profiles = member.find('div', 'organizationMemberList_memberProfile')
    #OrganizationMemberList from the div tags obtained above_Get all div tags of memberStats class and specify the second one
    contribute = profiles.findAll('div', 'organizationMemberList_memberStats')[1]

    # a.attrs['href'] -> nnsnodnb
    name = a.attrs['href'].split('/')[1]
    # contribute.text -> 50 Contribution
    number = int(contribute.text.split(' Con')[0])
    
    contribution_array.update({name: number})

Member icon

get_membericon()


soup = get_soup()

icons = {}
# organizationMemberList_Get all img tags of icon class
member_icons = soup.findAll('img', 'organizationMemberList_icon')
# organizationMemberList_Get all li tags of item class
members = soup.findAll('li', class_='organizationMemberList_item')

for index in range(len(members)):
    a = members[index].find('a')
    name = a.attrs['href'].split('/')[1]
    #Extract the src attribute from each member of the member icon list
    icon = member_icons[index].attrs['src']

    icons.update({name: icon})

This time, name is set as the key in order to relate the ** member list and the number of contributions ** and the ** member icon **.

Sort by ranking

I'm using a dictionary because I can't have a relationship between key and value in a list. However, since the dictionary is just a dictionary, it is output out of order. Want to cry So it takes the form of taking out in order and typing in the list. Also, don't forget reverse () because the data is added in ascending order when you hit it.

ranking()


contribution = get_contribution()

names, numbers = [], []
for key, value in sorted(contribution.items(), key=lambda x:x[1]):
    names.append(key)
    numbers.append(value)
    
user_icons = get_membericon()

names.reverse()
numbers.reverse()

Pour into template

This time I used Bottle as the web framework. There are many ways to use it if you google! Should be!

from bottle import template, get, run

Basically this is fine

Flow to template


return template('hoge')
return template('foo', name = name)

If you do it like this, it will be manageable ~ For hoge and foo, create a views folder in the application directory and create hoge.tpl and foo.tpl below! Same as HTML

If you want to run Python code in hoge.tpl, you can run it with% . Also, if you like template ('foo', name = name), you can replace it with {{name}} in hoge.tpl. ~~ Mustache It's the same. ~~

Also, if you want to end the loop etc. using for or ʻif`, you can do as follows

for sample


% for i in range(10):
% #Output to terminal
% print(i)
% end

if sample


% if name == 'nnsnodnb':
% <p>Yaho ~!</p>
% else:
% <p>Who are you wwwww</p>
% end

After all it feels bad without indentation! !! Of course, you can also write hoge.tpl with indentation.

If you code loosely like this, a page like a public page sample will be created.

Bottle execution

main()


run(host = '0.0.0.0')
$ python hoge.py

Summary

Just by the existence of such a thing, everyone's fighting power can be visualized and the target has been visualized loosely!

If you have any questions, please leave a comment or edit request!

Recommended Posts

I scraped the Organization member team and made a ranking
[Python] I analyzed the diary of a first-year member of society and made a positive / negative judgment on the life of a member of society.
I made a tool to automatically back up the metadata of the Salesforce organization
〇✕ I made a game
I made AI patroll the net and created a gadget ranking Web service that is updated once a week
I made a LINE BOT with Python and Heroku
I made a command to markdown the table clipboard
I made a small donation to the non-profit organization "Open Source Robot Foundation" OSRF
I made a function to check the model of DCGAN
I made a dot picture of the image of Irasutoya. (part1)
I made a VGG16 model using TensorFlow (on the way)
I made a python text
I made a discord bot
I made a dot picture of the image of Irasutoya. (part2)
I made a Chatbot using LINE Messaging API and Python
Hannari Python At the LT meeting in December, I made a presentation on "Python and Bayesian statistics".
I made a Line bot that guesses the gender and age of a person from an image
I made a program to solve (hint) Saizeriya's spot the difference
I made a simple circuit with Python (AND, OR, NOR, etc.)
I want to record the execution time and keep a log.
[Python3] I made a decorator that declares undefined functions and methods.
I made a simple network camera by combining ESP32-CAM and RTSP.
I made a game called Battle Ship using pygame and tkinter
I made a program that solves the spot the difference in seconds
I made a Nyanko tweet form with Python, Flask and Heroku
I made RNN learn a sine wave and made a prediction: Hyperparameter adjustment
I relaxed the conditions a little and let optuna solve Sudoku
I made a slack bot that notifies me of the temperature
I made a scaffolding tool for the Python web framework Bottle
I made a Discord chat reading bot by imitating the appearance
I made a command to display a colorful calendar in the terminal
I made a Chatbot using LINE Messaging API and Python (2) ~ Server ~
I made a program that automatically calculates the zodiac with tkinter
I made a chatbot with Tensor2Tensor and this time it worked
[Kaggle] I made a collection of questions using the Titanic tutorial
I made a C ++ learning site
I made a Line-bot using Python!
I made a CUI-based translation script (2)
I made a wikipedia gacha bot
I made a fortune with Python.
I made a CUI-based translation script
I made a daemon with Python
I made a POST script to create an issue on Github and register it in the Project
I made a calendar that automatically updates the distribution schedule of Vtuber
[Django] I made a field to enter the date with 4 digit numbers
I made a kitchen timer to be displayed on the status bar!
I made a network to convert black and white images to color images (pix2pix)
I made a simple timer that can be started from the terminal
I made a program to input what I ate and display calories and sugar
I made a GAN with Keras, so I made a video of the learning process.
I made a library konoha that switches the tokenizer to a nice feeling
I made a program to check the size of a file in Python
I made a mistake in fetching the hierarchy with MultiIndex of pandas
I made a function to see the movement of a two-dimensional array (Python)
[Python] I made a web scraping code that automatically acquires the news title and URL of Nikkei Inc.
I made a new AWS S3 bucket
I made a dash docset for Holoviews
I made a payroll program in Python!
I touched "Orator" so I made a note
I made a character counter with Python
Beginner: I made a launcher using dictionary