I stumbled on the character code when converting CSV to JSON in Python

Overview

When I made a tool to convert a CSV file to a JSON file with Python, the contents of the JSON file were garbled, so I will leave the correction method.

resource

CSV file → JSON file conversion source (before modification)

• Reference → http://hktech.hatenablog.com/entry/2018/09/10/235725

csv_to_json.py

import json
import csv

json_list = []

#Read CSV file
with open('./csv_sample.csv', 'r') as f:
    for row in csv.DictReader(f):
        json_list.append(row)

#Write to JSON file
with open('./json_sample.json', 'w') as f:
    json.dump(json_list, f)
# with codecs.open('./json_sample.json', 'w', 'utf-8') as f:
#     json.dump(json_list, f, ensure_ascii=False)

#Load JSON file
with open('./json_sample.json', 'r') as f:
    json_output = json.load(f)

CSV file

csv_sample.csv

name,Ranking,Country of origin
Nadal,2,Spain
Federer,3,Switzerland
Djokovic,1,Serbia

Run csv_to_json.py to generate a JSON file

Run csv_to_json.py

(csv_to_json_tool) bash-3.2$ python csv_to_json.py

JSON file generated after execution

[{"\u540d\u524d": "\u30ca\u30c0\u30eb", "\u9806\u4f4d": "2", "\u51fa\u8eab\u56fd": "\u30b9\u30da\u30a4\u30f3"}, {"\u540d\u524d": "\u30d5\u30a7\u30c7\u30e9\u30fc", "\u9806\u4f4d": "3", "\u51fa\u8eab\u56fd": "\u30b9\u30a4\u30b9"}, {"\u540d\u524d": "\u30b8\u30e7\u30b3\u30d3\u30c3\u30c1", "\u9806\u4f4d": "1", "\u51fa\u8eab\u56fd": "\u30bb\u30eb\u30d3\u30a2"}]

The characters have been garbled. Check the character code of the JSON file with the [file --mime] command.

• About the [file --mime] command → https://qiita.com/pugiemonn/items/106749351991037fb606

(csv_to_json_tool) bash-3.2$ file --mime json_sample.json 
json_sample.json: application/json; charset=us-ascii

The character code is us-ascii, and it seems that it was written in a Unicode escaped state.

• About character code and decoding method → https://ferret-plus.com/7007
• About escape → https://blog.ohgaki.net/what-is-escape#i-2

Fixed garbled JSON file

By specifying "ensure_ascii = False" in the 3rd argument of json.dump when writing a JSON file, it was possible to write with utf-8 and avoid escaping.

• About json.dump of Python → https://docs.python.org/ja/3/library/json.html#json.dump If> ensure_ascii is true (the default value), then the output guarantees that all non-ASCII characters entered are escaped. If ensure_ascii is false, these characters will be printed as is.

csv_to_json.py

import json
import csv

json_list = []

#Read CSV file
with open('./csv_sample.csv', 'r') as f:
    for row in csv.DictReader(f):
        json_list.append(row)

#Write to JSON file
with codecs.open('./json_sample.json', 'w') as f:
    json.dump(json_list, f, ensure_ascii=False)      #"ensure_ascii=False"Specify

#Load JSON file
with open('./json_sample.json', 'r') as f:
    json_output = json.load(f)
    print(json_output)

(csv_to_json_tool) bash-3.2$ file --mime json_sample.json 
json_sample.json: application/json; charset=utf-8

JSON file generated after execution

json_sample.json

[{"name": "Nadal", "Ranking": "2", "Country of origin": "Spain"}, {"name": "Federer", "Ranking": "3", "Country of origin": "Switzerland"}, {"name": "Djokovic", "Ranking": "1", "Country of origin": "Serbia"}]

It was displayed as it was!