Parse and visualize JSON (Web application ⑤ with Python + Flask)
Click here for an overview of our efforts.
- Environment preparation (OS setup)
- Environment preparation (setup in OS)
- Trace the contents of Flask's QuickStart (installation and minimal setup)
- Trace the contents of Flask's Tutorial (learning how to make a basic application)
- Make original content ★
Last time I've finished up to 4. So I'll do the original, which is the original purpose. .. What we are creating is an application that reads a JSON file that assumes a specific configuration and visualizes the contents. There are some products that can output a configuration file in JSON for work, so it would be convenient if I could visualize it (eventually Excel output). I feel that it exists even if I don't make it, but I will make it as a practice.
Requirements
--Has a mechanism to upload JSON files --Parse the uploaded JSON file and visualize (a part of) the contents
Let's create an application that meets this requirement. Actually, it is easier to imagine using the json file that you actually want to use, but since it contains information that can not be released outside the company, use the one published on the Internet instead. http://docs.aws.amazon.com/ja_jp/amazondynamodb/latest/gettingstartedguide/samples/moviedata.zip A JSON file that records movie titles, ratings, genres, etc.
Check the contents of the JSON file
Even if you want to visualize it, you first need to understand what kind of structure it is (I think ...). I don't know what professionals will do, but as an amateur, I'm using this Chrome extension. http://jsoneditoronline.org/
There is a guimon who wonders if this is all right for visualization, but there is a desire to make it more Excel-like, or finally to Excel.
image
The sample JSON is a file that summarizes movie information. I would like to make it an image of the following output.
| Title | Rating | Genre | Actors |
|---|---|---|---|
| Prisoners | 8.2 | Crime | Hugh Jackman |
| Drama | Jake Gyllenhaal | ||
| Thriller | Viola Davis |
The number of Genres and Actors is variable.
environment
As the environment, use the same VM as the one used up to previous, and divide only the Python virtual environment (virtualenv).
setup
Directory structure
Use the following directory structure.
- jsonparser
- db/ --env / (created with the virtualenv command)
- static/
- js/
- jquery-3.1.1.min.js
- style.css
- js/
- templates/ --layout.html (created from now on) --inputfile.html (created from now on) --output.html (created from now on)
- uploads/ --app.py (created from now on) --config.py (created from now on) --parse_json.py (created from now on)
The part that is almost the same as the last time is omitted.
Create virtualenv
abridgement
Create config file
This time, the part that sets the variable is separated into another file.
config.py
# -*- coding: utf-8 -*-
# configuration
DATABASE = '/root/jsonparser/db/jsonparser.db'
DEBUG = True
UPLOAD_FOLDER = '/root/jsonparser/uploads/'
ALLOWED_EXTENSIONS = set(['json'])
SECRET_KEY = 'development key'
The variables set here can be accessed from the py file that imported this file by the following procedure.
- Create an instance of Flask app = Flask(name)
- Read variables from file app.config.from_pyfile('config.py')
- Assign the contents of the variable UPLOAD_FOLDER to the variable temp temp = app.config['UPLOAD_FOLDER']
Create a database to store file information
Create the following SQL statement and call it from app.py as before to initialize the database. The process of initializing app.py at this stage is the same as last time, so it is omitted.
schema.sql
drop table if exists file_entries;
create table file_entries (
id integer primary key autoincrement,
filename string not null,
desc string,
created string
);
Creation of the main body
View function
Add the following function to app.py. A function that checks the extension of uploaded files.
def allowed_file(filename):
return '.' in filename and \
filename.rsplit('.', 1)[1].lower() in app.config[
'ALLOWED_EXTENSIONS']
Next, the function to upload a file and the function to delete (entry) the uploaded file.
@app.route('/', methods=['GET', 'POST'])
def inputfile():
cur = g.db.execute(
'select filename,desc,created,id from file_entries order by created desc')
entries = [dict(filename=row[0], desc=row[1], created=row[
2], id=row[3]) for row in cur.fetchall()]
if request.method == 'POST':
# check if the post request has the file part
if 'file' not in request.files:
flash('No file part')
return redirect(url_for('inputfile'))
file = request.files['file']
# if user does not select file, browser also
# submit a empty part without filename
if file.filename == '':
flash('No selected file')
return redirect(url_for('inputfile'))
if file and allowed_file(file.filename):
filename = secure_filename(file.filename)
file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
current = datetime.now()
g.db.execute('insert into file_entries (filename,desc,created) values (?, ?, ?)',
[file.filename, request.form['desc'], current])
g.db.commit()
message = "File upload finished successfully."
return redirect(url_for('inputfile', message=message))
current = datetime.now().strftime('%Y/%m/%d %H:%M')
message = request.args.get('message', '')
if not message:
message = "Current time is " + current
return render_template('inputfile.html', message=message, entries=entries)
@app.route('/delete', methods=['GET', 'POST'])
def delete():
id = request.args.get('value')
g.db.execute("delete from file_entries where id = ?", [id])
g.db.commit()
return redirect(url_for('inputfile'))
HTML creation
Template to be the basis
layout.html
<!doctype html>
<html>
<head>
<title>JSON Parser</title>
<script type="text/javascript" src="../static/js/jquery-3.1.1.min.js"></script>
<link rel=stylesheet type=text/css href="{{ url_for('static', filename='style.css') }}">
{% block head %}{% endblock %}
</head>
<div class=page>
<h1>JSON Parser</h1>
<div class=metanav>
<a href="{{ url_for('inputfile') }}">Home</a>
</div>
{% for message in get_flashed_messages() %}
<div class=flash>{{ message }}</div>
{% endfor %}
{% block body %}{% endblock %}
</div>
</html>
A screen for uploading, listing, and deleting files.
inputfile.html
{% extends "layout.html" %}
{% block body %}
<p>{{ message }} </p>
<form action="{{ url_for('inputfile') }}" method=post class=add-entry enctype="multipart/form-data">
File: <input type="file" name="file" size="30"/><br>
Description: <input type="text" name="desc" size="30" placeholder="Description"/><br>
<input type="submit" />
</form>
<ul class=entries>
{% for entry in entries %}
<li><h2>{{ entry.filename }}</h2>
<p><a href="{{url_for('delete',value=entry.id)}}">Delete</a></p>
Description: {{ entry.desc }}<br>
Created: {{ entry.created }}<br>
{% else %}
<li><em>There are no entries yet.</em>
{% endfor %}
</ul>
{% endblock %}
Upload the file from the action of the form element. Use url_for to jump to the inputfile function (corresponding URL) in app.py.
CSS creation
The style sheet is omitted because the tutorial one is used as it is for the time being.
Results so far
When you come to this point, start the application and access it from the browser,
[root@cnenarnupgd1c jsonparser]# . env/bin/activate
(env) [root@cnenarnupgd1c jsonparser]#
(env) [root@cnenarnupgd1c jsonparser]# python app.py
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger pin code: 173-472-212
When you upload the file,
Up to this point is OK.
Perspective, display
The kick will be the screen that displays the list of uploaded files created earlier. Add a link called "Configure Confirmation" to the screen and skip it to the function called output along with the target file name.
inputfile.html(part)
<ul class=entries>
{% for entry in entries %}
<li><h2>{{ entry.filename }}</h2>
<p><a href="{{url_for('output',value=entry.filename)}}">Configuration check</a></p>
<p><a href="{{url_for('delete',value=entry.id)}}">Delete</a></p>
Description: {{ entry.desc }}<br>
Created: {{ entry.created }}<br>
{% else %}
<li><em>There are no entries yet.</em>
{% endfor %}
</ul>
The received function is as follows. Based on the file name passed as value, read the json file, pass it to the module for parsing, and pass the returned object to the template.
app.py(part)
from parse_json import parse_json
@app.route('/output')
def output():
fname = request.args.get('value').replace(' ','_')
fpath = app.config['UPLOAD_FOLDER'] + fname
jsonfile = open(fpath,'r')
config = json.load(jsonfile)
output = parse_json(config)
return render_template('output.html', entries=output)
Next is the module for parsing.
parse_json.py
import sys
def parse_json(config):
# Initialize array
# Create (num of movie) * empty {}
arr = []
for i in range(len(config)):
try:
arr[i].append({})
except IndexError:
arr.append({})
except:
print 'Unexpected Error:',sys.exc_info()[0]
# Populate arr
for idx,movie in enumerate(config):
try:
arr[idx].update({'title': movie.get('title')})
arr[idx].update({'rank': movie.get('info').get('rank')})
arr[idx].update({'genres': movie.get('info').get('genres')})
arr[idx].update({'actors': movie.get('info').get('actors')})
if movie.get('info').get('genres') and movie.get('info').get('actors'):
arr[idx].update({'rowspan': max(len(movie.get('info').get('genres')),len(movie.get('info').get('actors')))})
elif not movie.get('info').get('genres') and movie.get('info').get('actors'):
arr[idx].update({'rowspan': len(movie.get('info').get('actors'))})
elif movie.get('info').get('genres') and not movie.get('info').get('actors'):
arr[idx].update({'rowspan': len(movie.get('info').get('genres'))})
else:
arr[idx].update({'rowspan': 1})
except:
print 'Unexpected Error:', sys.exc_info()[0]
pass
return arr
To be honest, I feel that Flask code (on HTML) can handle it at this level, but I think about it later. Also, a variable called rowspan is calculated on the way, but this is a variable that indicates how many rows are required for each movie.
Click here for a template to pass the generated object.
output.html
{% extends "layout.html" %}
<h1>Movies</h1>
{% block body %}
<table ~~~ style="table-layout:fixed;width:100%;" border = "3">
<colgroup>
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
<col style="width:25%;">
</colgroup>
<tr bgcolor="skyblue">
<td ~~~ style="word-wrap:break-word;">Title</td>
<td ~~~ style="word-wrap:break-word;">Rank</td>
<td ~~~ style="word-wrap:break-word;">Genres</td>
<td ~~~ style="word-wrap:break-word;">Actors</td>
</tr>
{% for entry in entries %}
{% for i in range(entry.rowspan) %}
<tr>
{% if i == 0 %}
<td ~~~ style="word-wrap:break-word;" rowspan = "{{ entry.rowspan }}">{{ entry.title }}</td>
<td ~~~ style="word-wrap:break-word;" rowspan = "{{ entry.rowspan }}">{{ entry.rank }}</td>
{% endif %}
{% if entry.genres[i] %}
<td ~~~ style="word-wrap:break-word;">{{ entry.genres[i] }}</td>
{% else %}
<td ~~~ style="word-wrap:break-word;" bgcolor="black">Empty</td>
{% endif %}
{% if entry.actors[i] %}
<td ~~~ style="word-wrap:break-word;">{{ entry.actors[i] }}</td>
{% else %}
<td ~~~ style="word-wrap:break-word;" bgcolor="black">Empty</td>
{% endif %}
</tr>
{% endfor %}
{% else %}
<tr><td ~~~ style="word-wrap:break-word;" columnspan="4">No entry here.</td></tr>
{% endfor %}
</table>
{% endblock %}
For entries that require 4 lines, display the title and Rank together in 4 lines. Fill in the remaining cells.
Also, the cell width is fixed so that long character strings are wrapped. It refers to the following. http://qiita.com/n_s_y_m/items/cb29d730e63772b02475
result
The output will be as follows. Regardless of the number of items, it is close to the image.
Actually, it would be nice to be able to sort by column, but once this is done. The entire source etc. are published below. https://github.com/KI1208/jsonparser.git
Supplement, memorandum
At first glance, there are many problems and requests at present, but I will close my eyes and make it a future issue. --The file is apparently deleted but not actually deleted --Does not consider conflicts between files with the same name --Files cannot be grouped (cannot be updated) --I want to touch the display dynamically --Other types of JSON need to be rewritten each time (general purpose is low). And so on
Recommended Posts