The rough difference between Unicode and UTF-8 (and their friends)
What is Unicode
A character set. Manage by adding an integer value called code point to each character. There is.
What are UTF-8, UTF-16, UTF-32?
[Character encoding](https://ja.wikipedia.org/wiki/%E6%96%87%E5%AD%97%E7%AC%A6%E5%8F%B7%E5%8C%96% E6% 96% B9% E5% BC% 8F). Converts the code point integer value to a byte string for computer use.
What is endian?
The order in which bytes are arranged when recording data consisting of multiple bytes in memory or when sending and receiving over a network. Sometimes called ** byte order **. For big endian, arrange from the upper side, and for little endian, arrange from the lower side.
Verification code
This is the code when I verified it with Python when writing this article.
In [1]: import unicodedata
In [2]: import binascii
In [3]: unicodedata.name('Depression') #Look up the name
Out[3]: 'CJK UNIFIED IDEOGRAPH-9B31'
In [4]: ord('Depression') #Examine the code point
Out[4]: 39729
In [5]: binascii.hexlify('Depression'.encode('UTF-8')) # UTF-Encode to byte string at 8 and then convert to hexadecimal representation
Out[5]: b'e9acb1'
In [6]: binascii.hexlify('Depression'.encode('UTF-16'))
Out[6]: b'fffe319b'
In [7]: binascii.hexlify('Depression'.encode('UTF-16LE'))
Out[7]: b'319b'
In [8]: binascii.hexlify('Depression'.encode('UTF-16BE'))
Out[8]: b'9b31'
In [9]: binascii.hexlify('Depression'.encode('UTF-32'))
Out[9]: b'fffe0000319b0000'
In [10]: binascii.hexlify('Depression'.encode('UTF-32LE'))
Out[10]: b'319b0000'
In [11]: binascii.hexlify('Depression'.encode('UTF-32BE'))
Out[11]: b'00009b31'
In [12]: binascii.hexlify('Pleasure'.encode('UTF-16'))
Out[12]: b'fffeeb5f1f61'
In [13]: binascii.hexlify('Pleasure'.encode('UTF-16LE'))
Out[13]: b'eb5f1f61'
In [14]: binascii.hexlify('Pleasure'.encode('UTF-16BE'))
Out[14]: b'5feb611f'
Recommended Posts