--Environment --Windows10 Pro version 1909 - Python 3.8.5 - Pandas 1.0.5
Traceback (most recent call last):
File "C:/path/to/my_code.py", line 258, in <module>
csv = read_files(target_dir)
File "C:/path/to/my_code.py", line 74, in read_files
data = pd.read_csv(file, encoding="shift_jis")
File "C:\path\to\venv\lib\site-packages\pandas\io\parsers.py", line 676, in parser_f
return _read(filepath_or_buffer, kwds)
File "C:\path\to\venv\lib\site-packages\pandas\io\parsers.py", line 448, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "C:\path\to\venv\lib\site-packages\pandas\io\parsers.py", line 880, in __init__
self._make_engine(self.engine)
File "C:\path\to\venv\lib\site-packages\pandas\io\parsers.py", line 1114, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "C:\path\to\venv\lib\site-packages\pandas\io\parsers.py", line 1891, in __init__
self._reader = parsers.TextReader(src, **kwds)
File "pandas\_libs\parsers.pyx", line 529, in pandas._libs.parsers.TextReader.__cinit__
File "pandas\_libs\parsers.pyx", line 720, in pandas._libs.parsers.TextReader._get_header
File "pandas\_libs\parsers.pyx", line 916, in pandas._libs.parsers.TextReader._tokenize_rows
File "pandas\_libs\parsers.pyx", line 2063, in pandas._libs.parsers.raise_parser_error
UnicodeDecodeError: 'shift_jis' codec can't decode byte 0xee in position 4225: illegal multibyte sequence
my_code.py
data = pd.read_csv(file, encoding="shift_jis")
This is in test2.csv, ・ Hashigodaka "Taka" ・ Tachisaki "Saki" It is caused by the mixture of windows extension strings such as>. Points to note when letting pandas read csv of excel output --Qiita
I'm crazy so I tried the combination. *. I tried using "Takasaki" as the extended character.
read_csv Character code |
file of Character code |
file of With extended characters |
file of No extended characters |
---|---|---|---|
shift_jis | shift_jis | error | OK |
shift_jis | cp392 | error | OK |
cp932 | shift_jis | error editor(Sublime Text)so If you put extended characters and save 保存はsoきるけど警告が出る。 You should notice that it is strange ... |
OK |
cp932 | cp932 | OK | OK |
shift with cp932_Error when reading a file with extended characters with jis
# ...abridgement...
UnicodeDecodeError: 'cp932' codec can't decode byte 0x86 in position 5: illegal multibyte sequence
If you use a file with Japanese on Windows, it is better to set it to cp932
.
I also studied CP932 and SJIS.
my_code.py
data = pd.read_csv(file, encoding="cp932")
Recommended Posts