A story that I had a hard time trying to create an "app that converts images like paintings" with the first web application

Introduction

This is a story in which the author, who has little knowledge of networks, tried to create an image processing web application as the first web application and had trouble handling images. Even if I searched the net, there were surprisingly few articles about ** image processing web application using Python, GCP, Flask **, so ** I struggled with the procedure for creating an image processing web application ** that works for the time being. I will introduce it.

Finished product

** 1. When you open the website, select the image you want to process and send it, ** ** Selected original image **

** 2. An image converted like a painting will be returned. ** **

Web application that converts images like paintings

• Please do not worry because the domain is not taken and the url is attached to GCP without permission. robust-dragon ... robust-dragon? ?? ??

Web app overview

I tried to summarize the outline of the created web application in the figure. This time, we used Google's cloud ** GCP ** as a web server within the scope of the free service. The actual image processing part was created using ** Python **, and the appearance of the website was easily created using only ** HTML **. Then, I created a link between Python and HTML using ** Flask **, which is a kind of Python API.

1. GCE (Google Cloud Engine) environment construction

First, we will build an environment to publish web applications using a service called GCE in GCP. ** Google account and credit card are required to use GCP **, so have them ready. Also, in this article, ** SSH connection from Google Chrome to GCE ** is used to remotely control the virtual environment in the cloud. In order to use the GCE service, you need to do the following three tasks, but please refer to the article that explains it in an easy-to-understand manner. Since I connect from Google Cloud Platform Console (browser is Google Chrome), I skipped the SSH authentication key part of the article. Settings are required if you want to make an SSH connection from a Mac terminal. [Reference article] GCP (GCE) to start from now on, safely use the free tier

[Required work] ・ Register for a free trial of GCP ・ Creating a project · Creating a virtual machine instance

2. How to start Cloud shell

After building the GCE environment, open the Instances page (https://console.cloud.google.com/compute/instances) and click Activate cloud shell in the upper right to launch the terminal. .. ** Once the terminal is up and running, you can work with the GCE instance you built using Linux commands. ** By the way, according to the Official Site, Compute Engine will automatically connect to the VM instance for the first time. It seems that an SSH authentication key pair has been generated in.

3. How to transfer data to and from the local environment

To run websites and image processing created in your local environment on GCE, you need to share your local data with GCE. There is a way to put it in a cool way with commands, but since it is troublesome to set up, I chose ** the method of using Cloud Storage, which is the easiest to understand **. From the GCP tab, select Strage → Browser. After creating a bucket with a suitable name in the storage browser, upload the file you want to share with GCE locally.

And you can save the data saved in the storage to any location of GCE by executing the following code in the cloud shell. gsutil is an application provided to access Cloud Storage from the command line.

gsutil cp -r gs://{Bucket name}/ ./{GCE folder name, etc.}

[Reference article] [Free] The story of the Qiita Hall of Fame [Easy]

4. Web application file structure (when using GCE / Python / Flask)

When creating a web application using Python and Flask on GCE, it is necessary to have the following file and directory structure. ** Conversely, you can save the created program with the following configuration and execute some commands on GCP to create an application immediately. ** If you name each file and folder other than my folder name different from the following, an error will occur, so you need to be careful.

my folder name (any name is OK) / 　├ static/ 　├ templates/ 　│　 └ index.html 　├ app.yaml 　├ main.py 　└ requirements.txt

I will briefly describe each role. -** static : Enter static data (css, Javascript, images, trained DL model, etc.) that you want to access while the app is running. This time it's empty. - templates : Save the html file. - app.yaml : This is an instruction sheet that describes the behavior of GAE. - main.py : A python file that describes data processing and cooperation with html. - requirements.txt **: A file containing the python library and version required to execute main.py. It will build the environment as it is at the time of deployment.

5. Creating a web app

Once you have created the above file structure on GCE, creating a web application is easy. Execute the following command.

To deploy your app, you must first create it in your region.

• If you have already created the app once, you can skip this command.

gcloud app create

Deploy the app.

gcloud app deploy app.yaml --project {my project name}

Enter your project name in the {my project name} part. I think it's written in the Cloud Shell. If you copy and paste the source code in Chapter 9 of this article to create a program file and the directory structure is as in Chapter 4, it will work. If you get an error, you can check the contents from App Engine Dashboard.

6. Image processing method

I adopted the ** Kuwahara filter ** introduced in the article below and used the code as it is. ** It's amazing that you can get a painting-like image that looks like an elaborate deep learning image processing, even though it can be expressed by a simple formula **. I think it's quite interesting. See the article below for details. I came across an image filter with a clearly Japanese name called Kuwahara filter, and when I tried it, it was amazing, so I will introduce it.

The original image

** After applying the filter **

7. Struggling with how to pass images from Python to HTML

I had a hard time passing the image after image processing in Python to HTML. At first, I usually saved the image once in a static file, passed the path to HTML, and read the image from HTML. ** However, while the app is running in the GCE environment, writing from Python to the folder seems to be restricted, resulting in an error. ** According to Official Site, the following four are recommended for saving data in the runtime environment. Was there. · Cloud Firestore in Datastore mode ・ Cloud SQL for MySQL ・ Cloud SQL for PostgreSQL ・ Cloud Storage

I tried Cloud Storage, which seems to be able to save images, but I couldn't get it to work well with my current knowledge of networks. Besides, I found an article that data can be saved with tempfile, which is a Python API, and tried it, and I was able to save data from Python, but on the contrary, I could not access from HTML and got stuck. It was.

When I think about it, I don't need to save the image somewhere in the first place, so I searched for a way to pass image data directly from Python to HTML **. ** There was a method of encoding image data with base64 on the Python side, passing it to HTML as a character string, and decoding the character string on the HTML side to create an image! ** When I actually tried it with reference to the article below, it worked. Image generated using pillow is displayed in html without saving to a file [Django]

8. Problems due to lack of memory

** When I tried various images, it seems that depending on the image size, the maximum memory (256MB) of the free version GCE was exceeded and an error occurred (it was useless if it was larger than 1000 x 600). ** Maybe I can improve the program and data structure without upgrading the machine, but I haven't done it yet. ** GCP has $ 300 free credit for the first year ** so you might want to upgrade and give it a try. By the way, you can check the details of the error from the App Engine dashboard in the GCP tab.

9. Source code

app.yaml

runtime: python37

main.py

#Load the required module
#Flask related
from flask import Flask, render_template, request, redirect, url_for, abort
import base64
import tempfile
from PIL import Image
import io
import numpy as np
import cv2

app = Flask(__name__)

def kuwahara(pic,r=5,resize=False,rate=0.5): #Original image, one side of a square area, ratio when resizing or resizing
    h,w,_=pic.shape
    if resize:pic=cv2.resize(pic,(int(w*rate),int(h*rate)));h,w,_=pic.shape
    pic=np.pad(pic,((r,r),(r,r),(0,0)),"edge")
    ave,var=cv2.integral2(pic)
    ave=(ave[:-r-1,:-r-1]+ave[r+1:,r+1:]-ave[r+1:,:-r-1]-ave[:-r-1,r+1:])/(r+1)**2 #Batch calculation of average value
    var=((var[:-r-1,:-r-1]+var[r+1:,r+1:]-var[r+1:,:-r-1]-var[:-r-1,r+1:])/(r+1)**2-ave**2).sum(axis=2) #Bulk calculation of variance
    
    def filt(i,j):
        return np.array([ave[i,j],ave[i+r,j],ave[i,j+r],ave[i+r,j+r]])[(np.array([var[i,j],var[i+r,j],var[i,j+r],var[i+r,j+r]]).argmin(axis=0).flatten(),j.flatten(),i.flatten())].reshape(w,h,_).transpose(1,0,2)
    filtered_pic = filt(*np.meshgrid(np.arange(h),np.arange(w))).astype(pic.dtype) #Color determination
    return filtered_pic
    
@app.route("/", methods=["GET", "POST"])
def upload_file():
    if request.method == "GET":
        return render_template("index.html")
    if request.method == "POST":
        #Save the uploaded file once
        f = request.files["file"]
        filepath = tempfile.mkdtemp()+"/a.png "
        #filepath = "{}/".format(tempfile.gettempdir()) + datetime.now().strftime("%Y%m%d%H%M%S") + ".png "
        
        f.save(filepath)
        image = Image.open(filepath)
        
        #Image processing part
        image = np.asarray(image)
        filtered = kuwahara(image, r=7)        
        filtered = Image.fromarray(filtered)
        
        #Encode with base64
        buffer = io.BytesIO()
        filtered.save(buffer, format="PNG")
        img_string = base64.b64encode(buffer.getvalue()).decode().replace("'", "")
        
        result = "image size {}×{}".format(len(image[0]), len(image))
        return render_template("index.html", filepath=filepath, result=result, img_data=img_string)
    
    
if __name__ == "__main__":
     app.run(host="127.0.0.1", port=8080, debug=True)

The following article was used as a reference when writing Flask code. Python x Flask x PyTorch Easy construction of number recognition web application

requirements.txt

Flask==1.1.2
Pillow==7.2.0
Numpy==1.16.4
opencv-python==4.2.0.34

index.html

<html>
    <body>
        {% if result %}
	<IMG SRC="data:image/png;base64,{{img_data}}" alt="img_data"  id="imgslot"/>
        <div>{{result}}</div>
	<HR>
        {% endif %}
Please select a file and send<BR>
        <form action = "./" method = "POST" 
           enctype = "multipart/form-data">
           <input type = "file" name = "file" />
           <input type = "submit"/>
        </form>
     </body>
</html>

10. Finally

Thank you for reading the article until the end. Creating a web application wasn't straightforward, but I managed to make something that works. ** LGTM will be one of the motivations to write an article, so thank you. ** **