Precautions when inputting from CSV with Python and outputting to json to make it an exe

As the title says, I will write a memorandum of notes when inputting from CSV with Python and outputting to json to make it an exe. The implementation example is a completely personal tool, so I'll just give you an overview. The important thing is the error message and the remedy.

CSV causing a decoding error

First, load csv. Prepare the data in csv called ʻinput.csv` and read it.

import csv

csv_file = open("./input.csv", "r")
c = csv.DictReader(csv_file)

Suddenly, when I try to look at the contents of csv here, I get the following error (probably).

Exception has occurred: UnicodeDecodeError
'cp932' codec can't decode byte 0xef in position 0: illegal multibyte sequence
  File "C:\Users\hoge\fuga.py", line 19, in <module>
    for row in c:

You will be scolded for not being able to decode. This is because python tries to make the ʻutf-8fileshift-jis`. It needs to be modified as follows.

csv_file = open("input.csv", "r",encoding="utf-8-sig")

All the same data json

The part that was the most annoying this time.

#Excerpt of only parameters from csv
params = []
for row in c:
    points.append(row)
n = len(params)

#Read json and secure the parameter frame
import json
json_open = open('sample.json', 'r')
j = json.load(json_open)
for i in range(n-1):
    j["Movements"].append(j["Movements"][0])

I did something awkward in the bottom row, which was the source of the trouble, but sample.json has a data template, which is extended to the number of parameter lines n. Therefore, I took the method of copying the template n-1 times. As a result, n frames have been successfully created in the Movements key of the json data j, so after that, we will just turn the for statement and insert the data.

…… But after I finished putting in the data, I was troubled by the mysterious result that ** for some reason all the rows have the same data **.

After a bit of debugging, I found that every input rewrites every row, which means that ** every row references the same data **. Even if the apparent data is in the n line, it only refers to one data, and if you modify any one of them, all the data will be affected.

Why did this happen?

For types like ʻintandstr, even if you copy a variable with = , it will be considered as" another variable "and assigned an ID when you make any changes, but list If you edit the variable after copying the variable with = (actually, it could not be copied), the original data will be edited. I knew it as knowledge, but so was dict. The only solution is to copy the variables using a technique called hard copy.

#Excerpt of only parameters from csv
params = []
for row in c:
    points.append(row)
n = len(params)

#Read json and secure the parameter frame
import json
import copy
json_open = open('sample.json', 'r')
j = json.load(json_open)
for i in range(n-1):
    new_j = copy.deepcopy(j["Movements"][0])
    j["Movements"].append(new_j)

In this case, it had to be deepcopy (deep copy) instead of justcopy (shallow copy).

Exe that doesn't work

There is a module called pyinstaller that makes it an exe so that even people who do not have Python can run it. When I tried to pip this, I couldn't install it due to an error, and if I went around, I should downgrade pip (2018), so I succeeded. When I tried to reproduce it now, for some reason even the latest pip succeeded, so the error cannot be reproduced. I'm sorry.

So I made it into an exe using pyinstaller safely, but even if I execute it, it ends immediately. It doesn't even spit out the first print that I put in for debugging, so it seems that there is a problem at the time of ʻimport`. So, I will hunt witches one by one for the following modules that I actually put in.

import json
import pprint
import csv
import cmath
import math
import numpy as np
import copy
import os

As a result, it was caused by numpy. Fortunately, I didn't use the part that needed numpy in the program (I used np.pi, but I don't need it anymore), so I just deleted the module. If you have to exe a program that uses numpy to calculate rigorously, think again ...

Summary

I'm sorry that there are many contents that "I managed to do it, but I didn't understand", but I will forget more and more if I do not accumulate such things, so I wrote it for myself. I've been afraid to skip this kind of thing until now, so from now on I'll actively write program records like this diary.

Code

It's a completely personal tool, so I don't think it's useful to anyone, but for the time being.


import json
import pprint
import csv
import cmath
import math
import copy
import os

json_open = open('./sample.json', 'r')
j = json.load(json_open)

print("sample.loaded json")

csv_file = open("./input.csv", "r",encoding="utf-8-sig")
c = csv.DictReader(csv_file)

points = []

for row in c:
    points.append(row)

n = len(points)

for i in range(n-1):
    new_j = copy.deepcopy(j["Movements"][0])
    j["Movements"].append(new_j)

for i,p in enumerate(points):
    x = float(p['X'])
    y = float(p['Y'])
    z = float(p['Z'])
    duration = float(p['Duration'])
    j["Movements"][i]["Duration"] = duration
    j["Movements"][i]["StartPos"]["x"] = x
    j["Movements"][i]["StartPos"]["y"] = y
    j["Movements"][i]["StartPos"]["z"] = z
    j["Movements"][i-1]["EndPos"]["x"] = x
    j["Movements"][i-1]["EndPos"]["y"] = y
    j["Movements"][i-1]["EndPos"]["z"] = z
    y -= 1.5
    c = complex(-x,-z)
    rad = cmath.phase(c)
    deg = -math.degrees(rad)+90
    #print(deg)
    j["Movements"][i]["StartRot"]["y"] = deg 
    j["Movements"][i-1]["EndRot"]["y"] = deg 
    c2 = complex(abs(c),y)
    rad2 = cmath.phase(c2)
    deg2 = math.degrees(rad2)
    #print(-deg2)
    j["Movements"][i]["StartRot"]["x"] = deg2 
    j["Movements"][i-1]["EndRot"]["x"] = deg2

pprint.pprint(j)

with open('output.json', 'w') as f:
    json.dump(j, f, indent=4)

print("Completed the work")
end = input()

Recommended Posts

Precautions when inputting from CSV with Python and outputting to json to make it an exe
Scraping tabelog with python and outputting to CSV
Make JSON into CSV with Python from Splunk
[Python / Ruby] Understanding with code How to get data from online and write it to CSV
Fractal to make and play with Python
2. Make a decision tree from 0 with Python and understand it (2. Python program basics)
Read CSV file with Python and convert it to DataFrame as it is
Make a decision tree from 0 with Python and understand it (4. Data structure)
Output the report as PDF from DB with Python and automatically attach it to an email and send it
Read JSON with Python and output as CSV
Generate an insert statement from CSV with Python.
WEB scraping with python and try to make a word cloud from reviews
Tips and precautions when porting MATLAB programs to Python
How to convert JSON file to CSV file with Python Pandas
Get mail from Gmail and label it with Python3
Read json file with Python, format it, and output json
How to make an arbitrary DictCursor with PyMySQL and not return None when NULL
Try to make it using GUI and PyQt in Python
When it is troublesome to set up an SMTP server locally when sending mail with Python.
Associate Python Enum with a function and make it Callable
Try to generate a cyclic peptide from an amino acid sequence with Python and RDKit
I made a server with Python socket and ssl and tried to access it from a browser
I tried to make a generator that generates a C # container class from CSV with Python
Write to csv with Python
[Python Kivy] How to create an exe file with pyinstaller
I tried to make GUI tic-tac-toe with Python and Tkinter
[Python] Convert CSV file uploaded to S3 to JSON file with AWS Lambda
How to not escape Japanese when dealing with json in python
Join csv normalized by Python pandas to make it easier to check
How to import CSV and TSV files into SQLite with Python
How to make a surveillance camera (Security Camera) with Opencv and Python
How to avoid duplication of data when inputting from Python to SQLite.
Precautions when passing def to sorted and groupby functions in Python? ??
I tried to make a periodical process with Selenium and Python
How to deal with errors when installing Python and pip with choco
Throw something to Kinesis with python and make sure it's in
Extract database tables with CSV [ODBC connection from R and python]
[Python] Try to recognize characters from images with OpenCV and pyocr
I tried to make an image similarity function with Python + OpenCV
Make apache log csv with python
[Python] Write to csv file with Python
Create folders from '01' to '12' with python
Output to csv file with Python
JSON encoding and decoding with python
Make Scrapy an exe with Pyinstaller
Precautions when using phantomjs from python
Precautions when using six with Python 2.5
I want to pass an argument to a python function and execute it from PHP on a web server
[TCP / IP] After studying, try to make an HTTP client-like with Python
[Data science basics] I tried saving from csv to mysql with python
Try to make foldl and foldr with Python: lambda. Also time measurement
Use Python from Java with Jython. I was also addicted to it.
Convert the spreadsheet to CSV and upload it to Cloud Storage with Cloud Functions
Precautions and error handling when calling .NET DLL from python using pythonnet
Error due to UnicodeDecodeError when reading CSV file with Python [For beginners]
[Zaif] I tried to make it easy to trade virtual currencies with Python
[python] Send the image captured from the webcam to the server and save it
Create a decision tree from 0 with Python and understand it (5. Information Entropy)
From Python to using MeCab (and CaboCha)
How to convert Python to an exe file
Convert from PDF to CSV with pdfplumber