Python --Explanation and usage summary of the top 24 packages

With pip install, we extracted the top 20 packages from the familiar PyPi by total and weekly downloads. This is a summary of how to use the 24 popular packages. I was surprised at the result of 16 overlaps in total for the week. I didn't know how to use only the pyasn1 package that handles ASN.1, so please let me know.

PyPi Downloads Ranking

1.simplejson A library that encodes and decodes json. It boasts more than 100 million downloads in the package that ranks first overall and weekly. For speed only, ujson is faster.

simplejson_1.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
import simplejson

# dict to json
d = {i: i**3 for i in xrange(10)}
json_str = simplejson.dumps(d)
print(type(json_str), json_str)

# json to dict
json_dict = simplejson.loads(json_str)
print(type(json_dict), json_dict)

Execution result

>>>python simplejson_1.py 
(<type 'str'>, '{"0": 0, "1": 1, "2": 8, "3": 27, "4": 64, "5": 125, "6": 216, "7": 343, "8": 512, "9": 729}')
(<type 'dict'>, {'1': 1, '0': 0, '3': 27, '2': 8, '5': 125, '4': 64, '7': 343, '6': 216, '9': 729, '8': 512})

2.setuptools Package management software. easy_install is also one of the features of setuptools. It's often the first to install pip, which also manages packages. Installing pip will install setuptools. distribute was merged into setuptools in 2013. Starting with Python 3.4, pip is included in Python itself. See PEP453 for details.

3.requests Requests is an HTTP library designed to be easy for people to use. You can write HTTP GET, POST, basic authentication, and OAuth 2.0 authentication concisely.

requests_3.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
import requests
import simplejson

#Code to HTTP GET to API that responds with json type
host = "api-sandbox.oanda.com"
url = "https://{}/v1/candles?instrument=EUR_USD&count=100&candleFormat=midpoint&" \
      "granularity=D&dailyAlignment=0&alignmentTimezone=America%2FNew_York".format(host)
response = requests.get(url)
#HTTP STATUS is normal
assert response.status_code == 200
data = simplejson.loads(response.text)
print(type(data), data)

Execution result

>>>python ./requests_3.py 
(<type 'dict'>, {u'instrument': u'EUR_USD', u'candles': [{u'complete': True, u'closeMid': 1.09102, u'highMid': 1.09347, u'lowMid': 1.084825, u'volume': 31402, u'openMid': 1.0863, u'time': u'2015-08-05T04:00:00.000000Z'}, {u'complete': True, u'closeMid': 1.09141, u'highMid': 1.0944, u'lowMid': 1.08735, u'volume': 22130, u'openMid': 1.09104, u'time': 
...

4.virtualenv You can easily build multiple Python environments on one device. I switch between Python2.7 + Django1.5 environment, PyPy environment, and Python3.5 environment.

Installation, environment construction, usage

# install
sudo pip install virtualenv
sudo pip install virtualenvwrapper

#hoge environment construction
mkvirtualenv --python=/usr/local/bin/python3 hoge

#Environment list
workon

#Environment switching
workon hoge

#Get out of the environment
deactivate

5.distribute Package management software. Installing pip installs setuptools and distribute was merged into setuptools in 2013. Therefore, there is no problem if package management is done with pip.

6.six six is a compatibility library for Python 2 and 3 series. Provides utility functions for writing single code that works in both 2 and 3 systems.

six_6.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
import six

#Run in python2 environment
print (six.PY2, six.PY3)


#Use the meta class of 3 series in python2 series environment
class ABCMeta(type):
    pass


@six.add_metaclass(ABCMeta)
class MyABC(object):
    pass

abc = MyABC()
print abc

Execution result

>>>python six_6.py 
(True, False)
<__main__.MyABC object at 0x101d02550>

7.pip Python package management software. There are other package management software such as setuptools and distribute, but if you get lost, I think the correct answer is to manage packages using pip in principle. It was developed because easy_install was unpopular, and since Python 3.4 it has been included in Python itself. Details PEP453

Get to install pip itself-pip.Easy to use py.

curl -kL https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python

8.certifi certifi is a library that precisely verifies the authenticity of SSL certificates using a root certificate authority (CA). Rather than using it alone, it is often installed at the same time because it depends on other libraries.

certifi_8.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
import certifi
#Route Certificate Authority(CA)Output the file installation location of
print(certifi.where())

Execution result

>>>python ./certifi_8.py 
/xxxxx/lib/python2.7/site-packages/certifi/cacert.pem

9.boto An interface for operating Amazon Web Services from Python. In my own experience, I used it to pull the server list of the deploy destination from the tag set in the EC2 instance at the time of deploy, or to access S3 and get the file list.

access s3 bucket with boto

import boto
from boto.s3.key import Key
import boto.s3.connection

AWS_ACCESS_KEY_ID = '<access key>'
AWS_SECRET_ACCESS_KEY = '<my secret key>'
Bucketname = 'Bucket-name' 


conn = boto.s3.connect_to_region('ap-southeast-1',
       aws_access_key_id=AWS_ACCESS_KEY_ID,
       aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
       is_secure=True,               # uncommmnt if you are not using ssl
       calling_format = boto.s3.connection.OrdinaryCallingFormat(),
       )
bucket = conn.get_bucket(Bucketname)

10.pbr A library for consistently managing setuptools. You can write package information in Setup.cfg in Metadata2.0 format without writing package information in setup.py. Default values are entered in setup.cfg to unify the format of each library.

setup.py

import setuptools
from pbr import util

setuptools.setup(
    **util.cfg_to_args())

setup.cfg

[metadata]
name = pbr
author = OpenStack
author-email = [email protected]
summary = Python Build Reasonableness
...

11.wincertstore The wincert store provides an interface for accessing Windows CA certificates and CRLs (Certificate Revocation Lists). Rather than using it alone, I think that it is often installed along with other packages.

12.python-dateutil Extends the Python standard datetime module. You can calculate the end of the month, calculate the "next ◯ day of the week", and calculate the "last ◯ day of the month". Since I have never used it, I implemented it by referring to How to use python-dateutil.

python-dateutil.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
from dateutil.relativedelta import relativedelta, FR
from datetime import date
import calendar

#Calculate the end of the month and return November 30th properly
print "Month-end calculation:", date(2015, 11, 26) + relativedelta(day=31)

#Return next Tuesday
print "Next tuesday:", date(2015, 11, 28) + relativedelta(weekday=calendar.TUESDAY)

#Last Friday of the month
first_day = date(2015, 11, 1)
rdl = relativedelta(day=31, weekday=FR(-1))
print "Last Friday of the month:", first_day + rdl

Execution result

>>>python ./python-dateutil.py 
Month-end calculation: 2015-11-30
Next tuesday: 2015-12-01
Last Friday of the month: 2015-11-27

13.Jinja2 A Python template engine. It can be used with Google App Engine. As the document says, "It's a template engine that is conscious of Django Template," I think it's easy for people who use Django. In a simple PJ, I think that it will be an option when it is "Fat (fat) with Django".

14.nose A unit test framework that makes it easy to write unit tests. The assert function is sufficient for simple use, but when generating multiple test files and testing them all at once, it is convenient to use the nosetest command to test all py files whose file names start with "test".

test_nose_14.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
from unittest import TestCase
from nose.tools import ok_, eq_


def sum(a, b):
    return a + b


def is_even(n):
    return (n % 2 == 0)


class HogeTestCase(TestCase):
    def setUp(self):
        print 'before test'

    def tearDown(self):
        print 'after test'

    def test_sum(self):
        eq_(sum(1, 2), 3)
        eq_(sum(5, 11), 16)
        eq_(sum(0, 0), 0)
        #error
        eq_(sum(0, 0), 10000)

    def test_is_even(self):
        ok_(is_even(2))
        ok_(not is_even(3))
        #If you get an error with assert, it will not work properly
        assert is_even(15)

Execution result

>>>nosetests
FF
======================================================================
FAIL: test_is_even (test_nose_14.HogeTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/xxxx/qiita/pypi_ranking/test_nose_14.py", line 33, in test_is_even
    assert is_even(15)
AssertionError: 
-------------------- >> begin captured stdout << ---------------------
before test

--------------------- >> end captured stdout << ----------------------

======================================================================
FAIL: test_sum (test_nose_14.HogeTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/xxxx/qiita/pypi_ranking/test_nose_14.py", line 27, in test_sum
    eq_(sum(0, 0), 10000)
AssertionError: 0 != 10000
-------------------- >> begin captured stdout << ---------------------
before test

--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 2 tests in 0.005s

FAILED (failures=2)

15.lxml A library for parsing XML. It is often used for more specialized libraries by wrapping lxml. Beutiful Soup may be a typical example.

lxml_15.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
import requests
from lxml import html

# HTTP GET
url = "http://google.co.jp/"
response = requests.get(url)

#Parse with lxml
root = html.fromstring(response.text)

#A tag output
anchors = root.xpath('//a')
for anchor in anchors:
    print(anchor.attrib['href'])

Execution result

>>>python lxml_15.py 
http://www.google.co.jp/imghp?hl=ja&tab=wi
http://www.google.co.jp/maps?hl=ja&tab=wl
https://play.google.com/?hl=ja&tab=w8
...

16.docutils If you define the document in reStructuredText format, you can output HTML, XML, Latex files with one source.

17.MarkupSafe You can output safe strings escaped in XML / HTML / XHTML format.

mks_17.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals

from markupsafe import Markup, escape

#escape
print "Escape output:", escape("<script>alert(document.cookie);</script>")

#Markup
tmpl = Markup("<em>%s</em>")
print "Markup output:", tmpl % "Peter > Lustig"

Execution result

>>>python ./mks_17.py 
Escape output: <script>alert(document.cookie);</script>
Markup output: <em>Peter &gt; Lustig</em>

18.pyasn1 ASN.1 is a language for "defining the structure of information". The information whose structure is defined by ASN.1 is called an "ASN.1 object". pyasn1 is a library for defining ASN.1 in python. I wrote it with reference to PyASN1 programmer's manual. If used well, I think you can implement serialization and deserialization processing. If anyone knows how to use it, please let me know.

pyasn1_18.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
from pyasn1.type import univ
"""
Definition
;; values specification
age-of-universe INTEGER ::= 13750000000
mean-martian-surface-temperature INTEGER ::= -63
"""

ageOfUniverse = univ.Integer(13750000000)
print ageOfUniverse

meanMartianSurfaceTemperature = univ.Integer(-63)
print meanMartianSurfaceTemperature

Execution result

>>>python ./pyasn1_18.py 
13750000000
-63

19.PyYAML YAML is a data serialization format designed for human readability and interaction with scripting languages. PyYAML is a YAML parser for Python. The API protocol is defined as yaml and the same model is shared between Python and other languages, and it is used when mapping when excel data is converted to json.

pyyaml_19.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
import yaml

yaml_str = """
fields:
- column: id
  name: ID
  type: int
  validate:
    unique: true

- column: campaign_id
  name:Campaign ID
  type: int

- column: description
  name:Detailed explanation of the campaign
  type: char
"""

y = yaml.load(yaml_str)
print type(y), y

Execution result

>>>python ./pyyaml_19.py 
<type 'dict'> {'fields': [{'column': 'id', 'validate': {'unique': True}, 'type': 'int', 'name': 'ID'}, {'column': 'campaign_id', 'type': 'int', 'name': u'\u30ad\u30e3\u30f3\u30da\u30fc\u30f3ID'}, {'column': 'description', 'type': 'char', 'name': u'\u30ad\u30e3\u30f3\u30da\u30fc\u30f3\u8a73\u7d30\u8aac\u660e'}]}

20.pytz Global timezone definition for Python. For example, you usually manage the time with utc + 0, and you can concisely write the process to output in Japan time of utc + 9 only when it is displayed.

pytz_20.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
from datetime import datetime
import pytz

#standard
print datetime.now()

#Japan time
jst = datetime.now(tz=pytz.timezone('Asia/Tokyo'))
print jst

#New York time
nyt = datetime.now(tz=pytz.timezone('US/Eastern'))
print nyt

#Time difference
print "diff:", nyt - jst

Execution result

>>>python ./pytz_20.py 
2015-11-20 13:56:54.633965
2015-11-20 13:56:54.661743+09:00
2015-11-19 23:56:54.665424-05:00
diff: 0:00:00.003681

10th place a week. Pycrypto

Python encryption toolkit. We offer a variety of cryptographic algorithms and protocols.

Perform AES encryption and decryption

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
from Crypto.Cipher import AES
import base64
import os

#Block size of Cipher object; 16, 24, or 32 for AES
BLOCK_SIZE = 32

#If it is less than the block length, fill it with Padding
PADDING = '{'

#Function to fill the part less than the block length with Paddng
pad = lambda s: s + (BLOCK_SIZE - len(s) % BLOCK_SIZE) * PADDING

#Base64 encryption and decryption functions
EncodeAES = lambda c, s: base64.b64encode(c.encrypt(pad(s)))
DecodeAES = lambda c, e: c.decrypt(base64.b64decode(e)).rstrip(PADDING)

#Randomly generate secret key
secret = os.urandom(BLOCK_SIZE)

#Create Cipher object with secret key
cipher = AES.new(secret)

#Encrypt string
encoded = EncodeAES(cipher, 'password')
print 'Encrypted string:', encoded

#Decrypt a string
decoded = DecodeAES(cipher, encoded)
print 'Decrypted string:', decoded

Execution result

>>>python ./aes_w10.py 
Encrypted string: P2vLpfpZgQyh1DPiY7a9TQfjGIiw3HcCh1qBxBwvtBk=
Decrypted string: password

16th place a week. Cffi

An external function interface for calling C code.

cffi_w16.py

from cffi import FFI

ffi = FFI()
ffi.cdef("""
int printf(const char *format, ...);   // copy-pasted from the man page
""")
C = ffi.dlopen(None)  # loads the entire C namespace
arg = ffi.new("char[]", "world")  # equivalent to C code: char arg[] = "world";
C.printf("hi there, %s.\n", arg)  # call printf

Execution result

>>>python ./cffi_w16.py 
hi there, world.

17th place a week. Colorama

You can color the Text on the console.

colorama_w17

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
from colorama import Fore, Back, Style
print(Fore.RED + 'Akairo!')
print(Back.GREEN + 'Red word+Background')
print(Style.DIM + '+Hazy letters')
print(Style.RESET_ALL)
print('Text color after reset')

■ Execution result

20th place a week. Rsa

RSA implementation. It supports encryption and decryption, signature and signature verification, and key generation. ..

rsa_w20.py

# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
import rsa

#Key generation
(bob_pub, bob_priv) = rsa.newkeys(512)

#encryption
message = 'hello Bob!'.encode('utf8')
crypto = rsa.encrypt(message, bob_pub)
print crypto

#Composite
message = rsa.decrypt(crypto, bob_priv)
print(message.decode('utf8'))

Execution result

>>>python ./rsa_w20.py 
i??u)"?}%??4?????4????:?n?Z????A?6?>?XF[/?XA%usc?ؙ?
hello Bob!

reference

PyPI Ranking Python package management technology summary How to use python-dateutil

When the sample code was written on the PyPi top page of the package, it was quite easy to get an overview, so I decided to imitate it. The 21st to 50th libraries in the weekly ranking were lined up with interesting libraries such as sqlalchemy and supervisord, so I'd like to use them someday.