Gzip the data by streaming

What you want to do

I wanted to dynamically create and return Gzip data in Flask Streaming Contents.

code

GzipStream class (self-made).

#!/usr/bin/env python
# encoding: utf-8

from gzip import GzipFile
from StringIO import StringIO


class GzipStream(object):
    
    def __init__(self):
        self._io = StringIO()
        self._gf = GzipFile(fileobj=self._io, mode='wb')
        self._pos = 0
    
    def push(self, content):
        '''
        push a part of content.
        '''
        assert not self._gf.closed
        self._gf.write(content)
        self._gf.flush()
    
    def pop(self):
        '''
        :return: A part of gzipped content, or "None" that means EOF.
        '''
        current_pos = self._io.len
        
        if self._gf.closed and current_pos == self._pos:
            return None
        
        part = self._io.getvalue()[self._pos:current_pos]
        self._pos = current_pos
        
        return part
    
    def close(self):
        '''
        you must call this.
        '''
        if not self._gf.closed:
            self._gf.close()
    
    def __enter__(self):
        '''You don't have to'''
        return self
    
    def __exit__(self, exc_type, exc_value, traceback):
        '''You don't have to'''
        self.close()

Try using it as follows.

>>> from streamgzip import GzipStream

>>> gs = GzipStream()

>>> gs.push('hello ')
>>> gs.pop()
'\x1f\x8b\x08\x00\xfd\xcf\x0cX\x02\xff\xcaH\xcd\xc9\xc9W\x00\x00\x00\x00\xff\xff'

>>> gs.push('world ')
>>> gs.push('!!')
>>> gs.pop()
'*\xcf/\xcaIQ\x00\x00\x00\x00\xff\xffRT\x04\x00\x00\x00\xff\xff'

>>> gs.close()
>>> gs.pop()
'\x03\x00\xb7\xc8\xd4%\x0e\x00\x00\x00'

>>> print gs.pop()
None

Try to verify that the data is correct.

>>> from gzip import GzipFile
>>> from StringIO import StringIO

>>> io = StringIO()
>>> io.write('\x1f\x8b\x08\x00\xfd\xcf\x0cX\x02\xff\xcaH\xcd\xc9\xc9W\x00\x00\x00\x00\xff\xff*\xcf/\xcaIQ\x00\x00\x00\x00\xff\xffRT\x04\x00\x00\x00\xff\xff\x03\x00\xb7\xc8\xd4%\x0e\x00\x00\x00')
>>> io.seek(0)
>>> gzfile = GzipFile(fileobj=io, mode='rb')
>>> gzfile.read()
'hello world !!'

Recommended Posts

Gzip the data by streaming
List the AMIs used by AWS Data Pipeline
Split data by threshold
Training data by CNN
Correlation by data preprocessing
Let's visualize the rainfall data released by Shimane Prefecture
Classify data by k-means method
Visualization of data by prefecture
Data acquired by Django releted
What I saw by analyzing the data of the engineer market
Let's visualize the river water level data released by Shimane Prefecture
Export the contents acquired by Twitter Streaming API in JSON format
[pepper] Pass the JSON data obtained by python request to the tablet.
The first time a programming beginner tried simple data analysis by programming
Prepare a high-speed analysis environment by hitting mysql from the data analysis environment
First satellite data analysis by Tellus
Start data science on the cloud
Shift the data for 3 months Shift the data for n months
Understanding the Tensor (3): Real World Data
Get the complete bitflyer tick data
Install the data files with setup.py
Check the data summary in CASTable
10 selections of data extraction by pandas.DataFrame.query
Animation of geographic data by geopandas
Save the audio data acquired by the browser in wav format on the server
Let's play with the corporate analysis data set "CoARiJ" created by TIS ①
One-click data prediction for the field realized by fully automatic machine learning
Extract and plot the latest population data from the PDF data provided by the city
Let's play with the corporate analysis data set "CoARiJ" created by TIS ②