Understand the function of convolution using image processing as an example

Introduction

As an exercise in "Learn from Mosaic Removal: State-of-the-art Deep Learning" written by koshian2, the convolution function was taken up as an example. This time, I would like to summarize what I understood about this convolution function. https://qiita.com/koshian2/items/aefbe4b26a7a235b5a5e

This book is understandable even for people like me who have started machine learning. It is explained in an easy-to-understand order from the basics. In particular, it was good that the latest papers were reviewed. The latest papers distributed in general bookstores are about a year or two ago. It's a book that makes you feel passionate about deep learning, and I'm glad you bought it.

What is convolution

Convolution is used in the convolutional neural network, which is famous for deep learning. Somehow, it's a kitchen-like word, and I have a feeling that I want to say it in words for the time being.

image.png

The calculation of convolution is done by the method shown in the image. Take a 3x3 cell from the input matrix, multiply it with a matrix called a convolution kernel, and add the sum as the output.   This is different from the so-called forward propagation network in which the units in the adjacent layer are fully connected (see the figure below).

002.png

Convolution calculation


X=\left(
\begin{array}{cc} 
0 & 1 & 2 & 3 & 4 \\
5 & 6 & 7 & 8 & 9 \\
10 & 11 & 12 & 13 & 14 \\
15 & 16 & 17 & 18 & 19 \\
20 & 21 & 22 & 23 & 24 \\
\end{array}
\right)

\\
kernel=\left(
\begin{array}{cc} 
0 & 1 & 2  \\
3 & 4 & 5 \\
6 & 7 & 8 \\

\end{array}
\right)

This time, we define the input as a 5x5 line of X and the kernel factor as a 3x3 line of kernel.

c.ipynp



import numpy as np

def conv(inputs, kernel):
  outputs = np.zeros((3,3),inputs.dtype)
  for i in range(3): #Calculate in the row direction three times.
    for j in range (3): #Calculate in the column direction three times.
      patch= X[i:i+3,j:j+3]
      prod = patch * kernel #Multiply the mass and kernel.
      sum =  np.sum(prod) #Add the hangs.
      outputs[i,j] = sum #Put a value in the output layer.
  return outputs

#Define an X matrix. The point is to convert it to 5x5 by reshape.
X = np.arange(25,dtype=np.float32).reshape(5,5)

#Put a value in the output layer.
kernel = np.arange(9, dtype = np.float32).reshape(3,3)
conv(X,kernel)

Outputs=\left(
\begin{array}{cc} 
312 & 348 & 384  \\
492 & 528 & 564 \\
672 & 708 & 744 \\

\end{array}
\right)


I was able to calculate.

Image convolution using Tensorflow

Image processing can be used to fade, black and white, and enhance edges. This is done by the convolution process we just summarized.

Here is the original photo. This is a picture I took when I went to the aquarium last year.

004.png

Process the image with an edge enhancement filter. When using Tensorflow, the order of the tensor axes makes sense. In the image, the order is basically batch, vertical resolution, horizontal resolution, and channel. Therefore, when adding axes (dimensions), it is necessary to pay attention to this order. Also on the program as below

c.ipynp


float_img[ :, :, :, i:i+1]

Since there are 3 channels (R, G, B) for color images, it is necessary to turn i for the 4th channel.

c.ipynp


kernel = []
kernel = np.array([0,0,0,0,10,0,0,0,0]).reshape(3,3,1,1)
kernel = 0.5* kernel.astype(np.float32)


outputs = []
float_img = tf.cast(img,tf.float32)/255.0
for i in range(3):
  conv_result = tf.nn.conv2d(float_img[:,:,:,i:i+1],kernel,1,'SAME')
  outputs.append(conv_result)
outputs = tf.concat(outputs, axis = -1)

fig = plt.figure(figsize=(14,14))
ax = fig.gca()
ax.imshow(outputs[0].numpy())

The kernel of the edge enhancement filter used is as follows.

kernel=\frac{1}{2}\left(
\begin{array}{cc} 
-1 & -1 & -1  \\
-1 & 10 & -1 \\
-1 & -1 & -1 \\

\end{array}
\right)

image.png

In this way, we were able to obtain the same effect as ImageFilter.EDGE_ENHANCE in PIL, a library often used in image processing. From this, we were able to confirm that what is being done in the convolution operation and image processing is the same. The kernel can be blurred, black and white, or the output can be changed as specified. Also, if you change this value depending on the input position, you will get another effect. I would like to learn more about this kernel.

The full program can be found here. https://github.com/Fumio-eisan/convol_20200307

Recommended Posts

Understand the function of convolution using image processing as an example
The image display function of iTerm is convenient for image processing.
Matrix Convolution Filtering-Reinventor of Python Image Processing-
I tried to get the batting results of Hachinai using image processing
Example of taking Python> function> * args as arguments
Machine learning of sports-Analysis of J-League as an example-②
Image processing? The story of starting Python for
I tried using the image filter of OpenCV
Python> function> Example of taking function as an argument> map (lambda x: 2 ** x, [1, 2, 3]) / locals () ['myprint'] (3.1415, 2.718)
I tried to understand the support vector machine carefully (Part 1: I tried the polynomial / RBF kernel using MakeMoons as an example).
Drawing on Jupyter using the plot function of pandas
Note: The meaning of specifying only * (asterisk) as an argument in the Python function definition.
Save screenshot of [Python] [Windows] screen as an image
Tweet the probability of precipitation as part of the function of the bot
Example of using lambda
[Pokemon Sword Shield] I tried to visualize the judgment basis of deep learning using the three family classification as an example.
Bayesian estimation using coin throwing as an example See the posterior distribution for each trial
Save the input video from the capture board as an image
Django returns the contents of the file as an HTTP response
Paste the image into an excel file using Python's openpyxl
Try adding a simple image baking function as an add-on
100 language processing knock-29: Get the URL of the national flag image
A method of converting the style of an image while preserving the color
I tried to get the index of the list using the enumerate function
Flow of getting the result of asynchronous processing using Django and Celery
Calculate the editing distance as an index of pronunciation similarity [python]
Consider the speed of processing to shift the image buffer with numpy.ndarray
A function that measures the processing time of a method in python
[Python] Explains how to use the format function with an example
Utilization of lambda (when passing a function as an argument of another function)
Try using Elasticsearch as the foundation of a question answering system
Finding the optimum value of a function using a genetic algorithm (Part 1)
[Fabric] I was addicted to using boolean as an argument, so make a note of the countermeasures.
The story of making a tool to load an image with Python ⇒ save it as another name
Image capture of firefox using python
Judgment of backlit image using OpenCV
Calculation of normal vector using convolution
Understand the contents of sklearn's pipeline
Extract feature points of an image
Get the attributes of an object
Precautions when using the urllib.parse.quote function
Environmentally friendly scraping using image processing
Notation of template matching using convolution
Image recognition of fruits using VGG16
I tried to transform the face image using sparse_image_warp of TensorFlow Addons
100 Language Processing Knock-93 (using pandas): Calculate the accuracy rate of analogy tasks
Let's talk about the tone curve of image processing ~ LUT is amazing ~
[Python] [Excel] Operate an Excel sheet from Python using openpyxl (using a test sheet as an example)
The eval () function that calculates a string as an expression in python
About the main tasks of image processing (computer vision) and the architecture used
Python: I want to measure the processing time of a function neatly
An example of the answer to the reference question of the study session. In python.
Be careful of the type when making an image mask with Numpy
I tried to understand the learning function of neural networks carefully without using a machine learning library (first half).