A real way for people using python 3.8.0-2 from windows to work with multibyte characters

First conclusion

--Start with python -X utf8 --If that doesn't work, set the environment variable and set the variable name PYTHONIOENCODING and the value ʻutf-8`.

at first

"This article is written for people who use python3.8.0-python3.8.2 from windows. I don't know about python3.7 or earlier or newer versions than python3.8.2. " I dare to declare it like this.

Until you get to this article

  1. Trying to run http.server or something, for example
  2. Being bullied by ʻUnicodeDecodeError`
  3. Execute sys.stdout = io.TextIOWrapper (sys.stdout.buffer, encoding ='utf-8') and
  4. The compiler scolded me that "the attribute buffer does not exist (in sys.stdout)".
  5. I searched for the following information but could not find it easily

Isn't it? Apparently ** sys.stdout.buffer seems to be deprecated from 3.8 ** (?). on that evidence, If you ask people who work this way, they will say 3.7 or earlier, and if you ask people who fail this way, they will say 3.8. (It may be a coincidence because I haven't investigated it properly. I'm sorry if that is the case.)

It's confusing because the information older than 3.8 says "for 3.x" or "for 3." At the time when 3.6 and 3.7 were the latest, I didn't think that the specifications would change with 3.8, so it seems that it has become generalized. From now on, I would like you to say something like "for python3.0-3.7" instead of saying "for python3".

Why an error

According to https://qiita.com/butada/items/33db39ced989c2ebf644, the standard output consists of shift_jis in windows10 and earlier. Because of that, if you run python3.8.2 or earlier on windows, ** whatever the character code inside python, it will be automatically converted to shift_jis at the time of output **. An error occurs during this process. (By the way, in python3 ~ 3.8.2, the str type character code is fixed to utf-8. There was also a bulletin board that suggested " # coding: utf-8 ", but this is the source encoding specification)

solution

In python3.7-3.8.2, when you start python from the console, you can start it in" UTF-8 mode "by adding the option -X utf8. Here, the imitation such as "automatically convert to shift_jis" is stopped.

・ By the way, if you want to use http.server, this is not enough. Set the environment variable to the variable name PYTHONIOENCODING and the value ʻutf-8`.

Recommended Posts

A real way for people using python 3.8.0-2 from windows to work with multibyte characters
From building a Python environment for inexperienced people to Hello world
From preparation for morphological analysis with python using polyglot to part-of-speech tagging
Create a tool to automatically furigana with html using Mecab from Python3
Build an environment to execute C ++ functions from Python with Pybind11 (for Windows & Visual Studio Code people)
Searching for an efficient way to write a Dockerfile in Python with poetry
Easy way to scrape with python using Google Colab
I want to work with a robot in python.
From buying a computer to running a program with python
I want to tell people who want to import from a higher directory with Python direnv
Execute C ++ functions from Python with Pybind11 (for Windows & Visual Studio Code people) Debugging
Convert from Pandas DataFrame to System.Data.DataTable using Python for .NET
[Linux] Copy data from Linux to Windows with a shell script
Probably the easiest way to create a pdf with Python3
Experiment to make a self-catering PDF for Kindle with Python
A simple way to avoid multiple for loops in Python
Introduction to Python for VBA users-Calling Python from Excel with xlwings-
Execute C ++ functions from Python with Pybind11 (for Windows & Visual Studio Code people) Environment construction
Using Rstan from Python with PypeR
Python OCR System Raise characters from images to improve work efficiency
What to do if Python doesn't work on Git for Windows
Create folders from '01' to '12' with python
Building a Windows 7 environment for getting started with machine learning with Python
Try to extract a character string from an image with Python3
The simplest way to build a Spleeter usage environment using Windows
I tried to make a todo application using bottle with python
[Note] Using 16x2-digit character LCD (1602A) from Python with Raspberry Pi
Pass a list by reference from Python to C ++ with pybind11
[Python] Try to recognize characters from images with OpenCV and pyocr
Everything from building a Python environment to running it on Windows
[Introduction to Udemy Python3 + Application] 47. Process the dictionary with a for statement
How to deal with OAuth2 error when using Google APIs from Python
I came up with a way to make a 3D model from a photo.
Steps to create a Python virtual environment with VS Code on Windows
How to get a value from a parameter store in lambda (using python)
A python script for Mac that zips without garbled characters on Windows
Data integration from Python app on Windows to Amazon Redshift with ODBC
Rock-paper-scissors with Python Let's run on a Windows local server for beginners
Notes for using OpenCV on Windows10 Python 3.8.3.
Memo to ask for KPI with python
[50 counts] Key transmission using Python for Windows
Output color characters to pretty with python
"Python AI programming" starting from 0 for windows
Send a message from Python to Slack
# 5 [python3] Extract characters from a character string
Building a Python 3.6 environment with Windows + PowerShell
How to work with BigQuery in Python
Tips for using python + caffe with TSUBAME
Python> Output numbers from 1 to 100, 501 to 600> For csv
To work with timestamp stations in Python
A way to understand Python duck typing
Call a command from Python (Windows version)
Try to beautify with Talking Head Anime from a Single Image [python preparation]
Notes from installing Homebrew to building an Anaconda environment for Python with pyenv
Invert pandas DataFrame upside down with just 15 characters without using a for statement
When I tried to create a virtual environment with Python, it didn't work
SSH login to the target server from Windows with a click of a shortcut
WEB scraping with python and try to make a word cloud from reviews
Build a local development environment with WSL + Docker Desktop for Windows + docker-lambda + Python
[Introduction to Python] How to get the index of data with a for statement
Send a message from Slack to a Python server