Generate error correction code to restore data corruption with zfec library

Using the zfec library, I tried to generate an error correction code that can be restored even if some data is damaged.

Image to restore data corruption

The zfec library uses the RAID-5 algorithm to restore data corruption. The Raid-5 algorithm can be easily understood by reading here. It's a simple mechanism using XOR, so you can understand it immediately.

スクリーンショット 2016-02-25 16.33.13.png

スクリーンショット 2016-02-25 16.33.19.png

スクリーンショット 2016-02-25 16.33.36.png

Features of zfec library

·early ・ There are APIs for python, c, Haskel ・ Can handle char arrays

https://pypi.python.org/pypi/zfec

Installation

pip install zfec

Generate error correction code from 2 blocks and verify

スクリーンショット 2016-02-25 16.33.13.png

2DataBlock


# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
import zfec

#Error detection correction
data_block1 = str('aiueo')
data_block2 = str('kakik')

#Error detection code from data blocks 1 and 2:Generate 3
_, _, data_block3 = zfec.Encoder(2, 3).encode([data_block1, data_block2])
print 'data block 3 is... {}'.format(data_block3)
# >>> data block 3 is... uyI}g

#Data block 1 and error detection code:Restore 3 to 2
_, repaired_data_block2 = zfec.Decoder(2, 3).decode([data_block1, data_block3, ], [0, 2])
assert data_block2 == repaired_data_block2

#Data block 2 and error detection code:Restore 3 to 1
repaired_data_block1, _ = zfec.Decoder(2, 3).decode([data_block2, data_block3, ], [1, 2])
assert data_block1 == repaired_data_block1


Generate error correction code from 5 blocks and verify

5DataBlock


# -*- coding: utf-8 -*-
from __future__ import absolute_import, unicode_literals
import zfec

#Error detection correction
data_block1 = str('aiueo')
data_block2 = str('kakik')
data_block3 = str('ukeko')
data_block4 = str('sasis')
data_block5 = str('useso')

#Error detection code from data blocks 1 and 2:Generate 3
blocks = zfec.Encoder(5, 6).encode([data_block1,
                                    data_block2,
                                    data_block3,
                                    data_block4,
                                    data_block5])

ecc_block = blocks[-1]
print str('ecc block is... {}').format(ecc_block)
# >>> ecc block is... J?\j

#Data block 1-Restore 5 from 4 and ECC blocks
repaired_blocks = zfec.Decoder(5, 6).decode([data_block1,
                                             data_block2,
                                             data_block3,
                                             data_block4,
                                             ecc_block], [0, 1, 2, 3, 5])
assert data_block5 == repaired_blocks[4]

#Data block 1,2,4,Restore 3 from 5 and ECC blocks
repaired_blocks = zfec.Decoder(5, 6).decode([data_block1,
                                             data_block2,
                                             data_block4,
                                             data_block5,
                                             ecc_block], [0, 1, 3, 4, 5])
assert data_block3 == repaired_blocks[2]

Recommended Posts

Generate error correction code to restore data corruption with zfec library
Library comparison summary to generate PDF with Python
Settings to debug the contents of the library with VS Code
Try to process Titanic data with preprocessing library DataLiner (Append)
Try to process Titanic data with preprocessing library DataLiner (Encoding)
Try to process Titanic data with preprocessing library DataLiner (conversion)
Command to generate QR code
Use with Cabocha to automatically generate "IOB2 tag corpus" learning data
Addicted to character code by inserting and extracting data with SQLAlchemy
Try to process Titanic data with preprocessing library DataLiner (Drop edition)
How to deal with imbalanced data
Code to randomly generate a score
How to deal with imbalanced data
How to Data Augmentation with PyTorch
Generate fake table data with GAN
I installed the library with Visual Studio Code, but Unable to import