[Flask + Keras] How to infer multiple models at high speed on the server

Conclusion

keras==2.2.4 tensorflow=1.14.0 numpy==1.16.4

Test code

from flask import Flask
import time

import numpy as np
import tensorflow as tf
from keras.models import load_model
from keras.preprocessing.image import img_to_array, load_img

app = Flask(__name__)

model_path1 = "mnist.h5"
model1 = load_model(model_path1)
label1 = ["l0", "l1", "l2", "l3", "l4", "l5", "l6", "l7", "l8", "l9"]
model1._make_predict_function()#<Very important> Speeding up predict
graph1 = tf.get_default_graph()


model_path2 = "mnist.h5"
model2 = load_model(model_path2)
label2 = ["l0", "l1", "l2", "l3", "l4", "l5", "l6", "l7", "l8", "l9"]
model2._make_predict_function()
graph2 = tf.get_default_graph()

def model1_predict(img_path):
    img = img_to_array(load_img(img_path, target_size=(28, 28), grayscale=True))
    img_nad = img_to_array(img) / 255
    img_nad = img_nad[None, ...]
    global graph1
    with graph1.as_default():
        pred = model1.predict(img_nad, batch_size=1, verbose=0)
    score = np.max(pred)
    pred_label = label1[np.argmax(pred[0])]
    print("Score:", score, "label:", pred_label)

def model2_predict(img_path):
    img = img_to_array(load_img(img_path, target_size=(28, 28), grayscale=True))
    img_nad = img_to_array(img) / 255
    img_nad = img_nad[None, ...]
    global graph2
    with graph2.as_default():
        pred = model2.predict(img_nad, batch_size=1, verbose=0)
    score = np.max(pred)
    pred_label = label2[np.argmax(pred[0])]
    print("Score:", score, "label:", pred_label)

@app.route("/", methods=['GET', 'POST'])
def webapp():
    start1 = time.time()
    model1_predict("mnist_test.jpg ")
    end1 = time.time()-start1
    print("processing time<model1>: ", end1, "Seconds")

    start2 = time.time()
    model2_predict("mnist_test.jpg ")
    end2 = time.time() - start2
    print("processing time<model2>: ", end2, "Seconds")

    output = "<p>model1:"+str(round(end1, 3))+"Seconds</p><br><p>model2:"+str(round(end2, 3))+"Seconds</p>"
    return output

if __name__ == "__main__":
    app.run(port=5000, debug=False)

Important part

model1 = load_model(model_path1)
model1._make_predict_function()#<Very important> Speeding up predict
graph1 = tf.get_default_graph()

def model1_predict():
    global graph1
    with graph1.as_default():
        pred = model1.predict(***, batch_size=1, verbose=0)

Recommended Posts

[Flask + Keras] How to infer multiple models at high speed on the server
How to create large files at high speed
Tutorial to infer the model learned in Tensorflow with C ++/OpenVINO at high speed
A note on how to check the connection to the license server port
[Python] How to get divisors of natural numbers at high speed
How to set the server time to Japanese time
[Python] How to save images on the Web at once with Beautiful Soup
How to assign multiple values to the Matplotlib colorbar
How to make multiple kernels selectable on Jupyter
How to measure line speed from the terminal
Think about how to program Python on the iPad
How to put Takoyaki Oishikunaru on the segment tree
PostgreSQL-For those who want to INSERT at high speed
How to enjoy Python on Android !! Programming on the go !!
How to run Django on IIS on a Windows server
How to register the same data multiple times with one input on the Django management screen
How to use GitHub on a multi-person server without a password
How to increase the processing speed of vertex position acquisition
[Hyperledger Iroha] Notes on how to use the Python SDK
How to deploy the easiest python textbook pybot on Heroku
How to scrape at speed per second with Python Selenium
Notes on how to use marshmallow in the schema library
Flask Web server cannot be published to the outside [VScode]
How to print characters to the console before booting on ARM