A dentist (!?) Created a tongue coating amount judgment app with flask + keras
Introduction
With keras + flask, I created a web application (app name: Tanpic) that judges the degree of dirt from the image of the tongue.
It is said that the reason for the development is that dirt on the surface of the tongue called tongue coating is related to bad breath (* 1), and I thought it would be interesting if there was an app that could visualize bad breath on a smartphone. is.
Udemy's <a href="https://px.a8.net/svt/ejp?a8mat=35RZ3H+G23XY2+3L4M+BW8O2&a8ejpredirect=https%3A%2F%2Fwww.udemy.com%2Fcourse%2Ftensorflow-advanced % 2F "rel =" nofollow "> [Image Judgment AI App Development / Part 1] Introduction to Image Judgment AI App Development Made with TensorFlow / Python / Flask <img border =" 0 "width =" 1 "height = I referred to "1" src = "https://www13.a8.net/0.gif? A8mat = 35RZ3H + G23XY2 + 3L4M + BW8O2" alt = "">. The explanations are easy to understand and you can ask questions, so I think it is a recommended teaching material for those who are making an image classification application for the first time.
By the way, I'm currently in the 5th grade of Dentistry and I'm not really a dentist yet. .. ..
Deliverables
When I upload an image of my tongue ...
The degree of dirt on the tongue is judged in 3 stages. By the way, it will give you a little bit of knowledge about bad breath. I hope you can overlook the bad behavior and lack of design sense. .. ..
Tanpic:https://tanpic-b86b4.firebaseapp.com/
Data collection
In this phase, we did the following:
- Collecting images with icrawler
- Elimination of unnecessary images
- Trimming
- Label in 3 stages according to the degree of dirt on the tongue
Image data was collected using icrowler. Please refer to the document for details.
Reference: icrawler Documentation
After removing unnecessary images, only the tongue part is manually trimmed one by one. After that, the degree of dirt on the tongue was classified into three.
Degree of dirt 1 (label name: tongue0)
Degree of dirt 2 (label name: tongue1)
Degree of dirt 3 (label name: tongue2)
Processing the data was time consuming and laborious and quite difficult sweat
Learning
I used google colabratoly GPU for learning.
After inflating and training the collected images, save the model as tongue_cnn_aug.h5. I will omit the details.
Test Accuracy
It was 66.66%.
Cooperation with flask
Here's a quick summary of what we're doing.
- Load model
- Convert image to 3 colors with RGB
- Resize image to 50 * 50
- Convert image to Numpy Array
- Pass the image to the model and predict
- The argmax function returns the predicted label value
- Pass the label value to the html file with the render_template function
predict_web.py
import os
from flask import Flask, flash, request, redirect, url_for, render_template
from werkzeug.utils import secure_filename
from keras.models import load_model
import numpy as np
from PIL import Image
import tensorflow as tf
class_1 = "It is 1."
class_content_1 = "There is almost no tongue coating on it, and it seems to be in a fairly beautiful state."
classes_solution_1 = "Let's do our best in oral care as before. As a caveat, over-cleaning your tongue will not only hurt your tongue, but will also increase the amount of tongue coating that adheres to it, so be gentle with it once a day."
class_2 = "It is 2."
class_content_2 = "You don't have to worry about dirt on your tongue."
classes_solution_2 = "Dirty tongue(Tongue coating)May cause bad breath, so if you are worried about it, you should gently clean your tongue once a day. In addition, bad breath tends to be stronger during the evening hours when saliva production is low. Please gargle and rehydrate to take measures."
class_3 = "It is 3."
class_content_3 = "Your tongue may be slightly dirty. Tongue coating (dirt on the tongue) can also cause bad breath, so it's a good idea to clean your tongue in addition to brushing your teeth."
classes_solution_3 = "To clean the tongue coating, use gauze or a tongue brush and gently rub the surface of the tongue from the back to the front. Do it only once a day."
classes = [class_1,class_2,class_3]
classes_content = [class_content_1, class_content_2, class_content_3]
classes_solution = [classes_solution_1, classes_solution_2, classes_solution_3]
num_classes = len(classes)
image_size = 50
UPLOAD_FOLDER = './uploads'
ALLOWED_EXTENSIOS = set(['png', 'jpg', 'gif','heic'])
app = Flask(__name__)
app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER
model = load_model('./tongue_cnn_aug.h5')
graph = tf.get_default_graph()
def allowed_file(filename):
#if.が含まれたる かつ if拡張子前の.Separated by, true if the extension is lowercase or lowercase
return '.' in filename and filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIOS
# @app.route('/', methods=['GET', 'POST'])
@app.route('/')
def upload_file():
return render_template('test.html')
@app.route('/predict',methods=['GET', 'POST'])
def predict_file():
global graph
with graph.as_default():
if request.method == 'POST':
file = request.files['file'] #add to
if file and allowed_file(file.filename):
filename = secure_filename(file.filename)
file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename)
image = Image.open(filepath)
image = image.convert('RGB')
image = image.resize((image_size, image_size))
data = np.asarray(image)
X = []
X.append(data)
X = np.array(X)
result = model.predict([X])[0]
predicted = result.argmax()
return render_template('predict.html', result = classes[predicted], result_content = classes_content[predicted], result_solution = classes_solution[predicted], result_title="Analysis result",img_name=file.filename)
from flask import send_from_directory
@app.route('/uploads/<filename>')
def uploaded_file(filename):
return send_from_directory(app.config['UPLOAD_FOLDER'], filename)
predict.html
<!DOCTYPE! html>
<html lang="ja">
<head>
<meta charset="utf-8">
<title>Tanpic</title>
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="/static/css/style.css">
<link rel="shortcut icon" href="/static/favicon.ico">
<!--Bootstrap CSS loading-->
<link rel="stylesheet" href="/static/css/bootstrap.min.css">
<!--manifest.loading json-->
<!--<link rel="manifest" href="./manifest.json">-->
</head>
<body>
<!--navi bar-->
<nav class="navbar navbar-expand-lg navbar-light bg-light fixed-top">
<a class="navbar-brand" href="#">Tanpic</a>
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarTogglerDemo03" aria-controls="navbarTogglerDemo03" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
<div class="collapse navbar-collapse" id="navbarTogglerDemo03">
<ul class="navbar-nav mr-auto mt-2 mt-lg-0">
<li class="nav-item active">
<a class="nav-link" href="#">Home <span class="sr-only">(current)</span></a>
</li>
<li class="nav-item">
<a class="nav-link" href="#">Link</a>
</li>
<li class="nav-item">
<a class="nav-link disabled" href="#" tabindex="-1" aria-disabled="true">Disabled</a>
</li>
</ul>
</div>
</nav>
<!--header area-->
<header class = "bg-primary text-center">
<div class = "bg-mask">
<div class="container">
<h1>Tanpic</h1>
<h2>Please put a picture of your tongue</h2>
<ul>
{% for entry in entries %}
<li>{{entry.title}} / {{entry.text}}</li>
{% endfor %}
</ul>
</div>
</div>
</header>
<!--sorting Grid section-->
<section class = "sorting">
<div class = "container text-center">
{%if img_name%}
<img src="uploads/{{img_name}}" style="width:300px;height:300px;">
{%endif%}
</div>
</section>
<!-- prediction session-->
<div class = "container">
<p>
{%if result_title%}
{{result_title}}
{%else%}
{%endif%}
</p>
<p class="lead">
<span>
{% if result %}
Your tongue is dirty in 3 levels,{{ result }}
{% else %}
{% endif %}
</span>
<br>
<br>
<span>
{%if result_content%}
{{result_content}}
{%else%}
{%endif%}
</span>
<br>
<br>
<span>
{%if result_solution%}
{{result_solution}}
{%else%}
{%endif%}
</span>
</p>
<a href="/">Return</a>
</div>
<!-- Footer -->
<footer class="page-footer font-small cyan darken-3">
<!-- Footer Elements -->
<div class="container">
<!-- Grid row-->
<div class="row">
<!-- Grid column -->
<div class="col-md-12 py-5">
<div class="mb-5 flex-center">
<!-- Facebook -->
<a class="fb-ic">
<i class="fab fa-facebook-f fa-lg white-text mr-md-5 mr-3 fa-2x"> </i>
</a>
<!-- Twitter -->
<a class="tw-ic">
<i class="fab fa-twitter fa-lg white-text mr-md-5 mr-3 fa-2x"> </i>
</a>
<!-- Google +-->
<a class="gplus-ic">
<i class="fab fa-google-plus-g fa-lg white-text mr-md-5 mr-3 fa-2x"> </i>
</a>
<!--Linkedin -->
<a class="li-ic">
<i class="fab fa-linkedin-in fa-lg white-text mr-md-5 mr-3 fa-2x"> </i>
</a>
<!--Instagram-->
<a class="ins-ic">
<i class="fab fa-instagram fa-lg white-text mr-md-5 mr-3 fa-2x"> </i>
</a>
<!--Pinterest-->
<a class="pin-ic">
<i class="fab fa-pinterest fa-lg white-text fa-2x"> </i>
</a>
</div>
</div>
<!-- Grid column -->
</div>
<!-- Grid row-->
</div>
<!-- Footer Elements -->
<!-- Copyright -->
<div class="footer-copyright text-center py-3">© 2018 Copyright:
<a href="https://mdbootstrap.com/education/bootstrap/"> MDBootstrap.com</a>
</div>
<!-- Copyright -->
</footer>
<script type="text/javascript" src="/static/js/jquery-3.3.1.min.js"></script>
<!--js script-->
<script type="text/javascript" src="/static/js/text.js"></script>
<!--Bootstrap JS loading-->
<script type="text/javascript" src="/static/js/bootstrap.min.js"></script>
<!-- Firebase App (the core Firebase SDK) is always required and must be listed first -->
<script src="https://www.gstatic.com/firebasejs/6.2.0/firebase-app.js"></script>
<!-- Add Firebase products that you want to use -->
<script src="https://www.gstatic.com/firebasejs/6.2.0/firebase-firestore.js"></script>
<script src="https://www.gstatic.com/firebasejs/6.2.0/firebase-storage.js"></script>
</body>
</html>
Works with firebase Cloud Storage
I decided to use Firebase Cloud Storage to collect the uploaded images.
Reference article: [Implementing an image upload application to Firebase Cloud Storage](https://kapi-travel.com/programing/firebase-cloud-storage%E3%81%B8%E3%81%AE%E7%94% BB% E5% 83% 8F% E3% 82% A2% E3% 83% 83% E3% 83% 97% E3% 83% AD% E3% 83% BC% E3% 83% 89% E3% 82% A2% E3% 83% 97% E3% 83% AA% E3% 82% 92% E5% AE% 9F% E8% A3% 85 /)
text.js
// Your web app's Firebase configuration
var firebaseConfig = {
apiKey: "",
authDomain: "",
databaseURL: "",
projectId: "",
storageBucket: "",
messagingSenderId: "",
appId: "",
measurementId: ""
};
// Initialize Firebase
firebase.initializeApp(firebaseConfig);
const storage = firebase.storage();
var file_name;
var blob;
$('#f1').on('submit',function(e){
//Processing when the image is submitted without being uploaded
if($('#myfile').val()===""){
$('#error').text("Please select a file");
e.preventDefault();
}
//Process of uploading to Cloud Storage
var uploadRef = storage.ref().child(file_name);
uploadRef.put(blob).then(snapshot => {
console.log(snapshot.state);
});
//value reset
file_name = '';
blob = '';
});
When I checked the console screen, the image was saved firmly. It was convenient because it was very easy to implement.
Deploy to heroku
I deployed it on heroku referring to the following article.
Reference: Qiita Flask Tutorial + Deploy to heroku
After logging in to heroku, follow the steps below to commit.
$ git add .
$ git commit
$ git push heroku master
I stumbled upon an R14 error due to lack of memory during git push heroku master. With the Free plan, the upper limit of memory is 512MB, and it seems that an R14 error occurred because it exceeded this. The solution was to use a smaller model and keep the memory usage below the upper limit.
Reference: What to do when R14 / R15 error occurs on Qiita Heroku
Source code
github:https://github.com/salmon0511/TongueCoatingClassificationApp
Impressions
It was fun to implement it while referring to various articles in order to realize the idea. Also, I was impressed when I was able to deploy after overcoming many errors.
However, I didn't care about readability (to be exact, I still don't know what code is highly readable), so I'd like to try to write code that is easy to read from now on.
reference
-Prototype image classification service with Qiita Flask & Keras
-Create a handwriting recognition application with Flask + Keras
Recommended Posts