Compress python data and write to sqlite

compress.py


#!/usr/bin/python
# -*- encoding: utf-8 -*-
import sys

import zlib
from base64 import binascii

#-----------------------------------------------------------
# COMPRESS
#-----------------------------------------------------------
def z_compress(data):

    #return buffer(zlib.compress(data))
    return zlib.compress(data)

#-----------------------------------------------------------
# DECOMPRESS
#-----------------------------------------------------------
def z_decompress(data):

    return zlib.decompress(data)

#-----------------------------------------------------------
# B2A
#   Binary >> Base64 Ascii
#-----------------------------------------------------------
def b2a(data):

    return binascii.b2a_base64(data)

#-----------------------------------------------------------
# A2B
#   Base64 Ascii >> Binary
#-----------------------------------------------------------
def a2b(data):

    return binascii.a2b_base64(data)

#-----------------------------------------------------------
# TEST
#-----------------------------------------------------------
def test():

    data = '1234567890123456789012345678901234567890123456789012345678901234567890'

    print 'ROW DATA:   %3d bytes' % sys.getsizeof(data)

    data = z_compress(data)
    print 'COMPRESS:   %3d bytes' % sys.getsizeof(data)

    data = b2a(data)
    print 'B2A:        %3d bytes' % sys.getsizeof(data)

    data_lists = sqlite_test(data)

    for data in data_lists:
        print 'ROW_DATA:   %3d bytes' % sys.getsizeof(data)

        data = a2b(data)
        print 'A2B:        %3d bytes' % sys.getsizeof(data)

        data = z_decompress(data)
        print 'DECOMPRESS: %3d bytes' % sys.getsizeof(data)

def sqlite_test(data):

    import sqlite3

    #---Connect to sqlite
    con = sqlite3.connect(":memory:")
    cur = con.cursor()

    #---Check if there is a table
    sql = 'SELECT COUNT(*) FROM sqlite_master WHERE type="table" AND name=?'
    cur.execute(sql, ('test',))

    #---Create when there is no table
    if not cur.fetchone()[0]:
        sql = 'CREATE TABLE test(data TEXT)'
        con.execute(sql)

    #---Data plunge
    sql  = 'INSERT INTO test (data) VALUES (?)'
    con.execute(sql, (data,))
    con.commit()

    #---Data acquisition
    sql = 'SELECT * from test'
    cur.execute(sql)
    row_lists = []
    for row in cur:
        row_lists.append(row[0])

    #---Returns data
    return row_lists

if __name__ == '__main__' : test()

result

ROW DATA:   110 bytes
COMPRESS:    61 bytes
B2A:         69 bytes
ROW_DATA:   168 bytes
A2B:         61 bytes
DECOMPRESS: 110 bytes

Recommended Posts

Compress python data and write to sqlite
Dump SQLite3 data and migrate to MySQL
[python] Compress and decompress
[Python] How to read data from CIFAR-10 and CIFAR-100
Data retrieval from MacNote3 and migration to Write
Write CSV data to AWS-S3 with AWS-Lambda + Python
It's not easy to write Python, it's easy to write numpy and scipy
Write tests in Python to profile and check coverage
Write data to KINTONE using the Python requests module
Connect to sqlite from python
Write to csv with Python
Build a Python environment and transfer data to the server
[Introduction to Data Scientists] Basics of Python ♬ Functions and classes
[Python] Use this to read and write wav files [wavio]
[Introduction to Python] Combine Nikkei 225 and NY Dow csv data
Get additional data to LDAP with python (Writer and Reader)
[Python] How to FFT mp3 data
[Introduction to Python3 Day 1] Programming and Python
[Python] Write to csv file with Python
How to use SQLite in Python
Python logging and dump to json
Selenium and python to open google
[Python / Ruby] Understanding with code How to get data from online and write it to CSV
Solving AOJ's Algorithm and Introduction to Data Structures in Python -Part1-
processing to use notMNIST data in Python (and tried to classify it)
Solving AOJ's Algorithm and Introduction to Data Structures in Python -Part2-
Solving AOJ's Algorithm and Introduction to Data Structures in Python -Part4-
Difference in how to write if statement between ruby ​​and python
Move data to LDAP with python Change / Delete (Writer and Reader)
Solving AOJ's Algorithm and Introduction to Data Structures in Python -Part3-
[Introduction to Data Scientists] Basics of Python ♬ Conditional branching and loops
[Introduction to Data Scientists] Basics of Python ♬ Functions and anonymous functions, etc.
Data pipeline construction with Python and Luigi
How to package and distribute Python scripts
From Python to using MeCab (and CaboCha)
[Introduction to Python3, Day 17] Chapter 8 Data Destinations (8.1-8.2.5)
Bind methods to Python classes and instances
[Introduction to Python3, Day 17] Chapter 8 Data Destinations (8.3-8.3.6.1)
How to write Python document comments (Docstrings)
Bind to class to read and write YAML
Convert Excel data to JSON with python
Fractal to make and play with Python
Write O_SYNC file in C and Python
[Introduction to Python3 Day 19] Chapter 8 Data Destinations (8.4-8.5)
Convert FX 1-minute data to 5-minute data with Python
Python data structure and operation (Python learning memo ③)
Trying to handle SQLite3 with Python [Note]
Read Python csv and export to txt
Read and write JSON files in Python
Easily graph data in shell and Python
python: How to use locals () and globals ()
How to use "deque" for Python data
[Python] How to calculate MAE and RMSE
How to use Python zip and enumerate
How to use is and == in Python
How to write Ruby to_s in Python
[Introduction to Data Scientists] Basics of Python ♬
Exchange encrypted data between Python and C #
How to write pydoc and multi-line comments
How to write the correct shebang in Perl, Python and Ruby scripts
Try to write python code to generate go code --Try porting JSON-to-Go and so on