Vertical Tower of Pisa using OpenCV

Overview

Recently, I learned image processing in class and learned the technique of transforming a plane by matrix transformation.

Then I suddenly got the urge to make the Leaning Tower of Pisa vertical, so I will try to practice Python and OpenCV.

Well, I can hardly understand it, so I'm a piggyback ride on the useful functions of OpenCV.

Let's Vertical!

environment

windows10 Python 3.7.7 OpenCV 3.4.2 Pillow 7.1.2 numpy 1.18.4

Implementation

Click here for the Leaning Tower of Pisa to use

Since white is strong overall, this time I will try to detect the outline by color tone instead of edge (Not to mention the background that didn't work at the edge)

pisa

#White detection
def detect_white_color(img):
    #Convert to HSV color space
    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

    #White HSV range
    hsv_min = np.array([0,0,100])
    hsv_max = np.array([180,45,255])
    mask = cv2.inRange(hsv, hsv_min, hsv_max)

    #Masking process
    masked_img = cv2.bitwise_and(img, img, mask=mask)

    return mask, masked_img

The left is mask and the right is masked_img

Next, binarize and detect the rectangle

pisa

imgray = cv2.cvtColor(blurred_img,cv2.COLOR_BGR2GRAY)
ret,thresh = cv2.threshold(imgray,115,255,0)
im, contours, hierarchy = cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)

for cnt in contours:
    rect = cv2.minAreaRect(cnt)
    box = cv2.boxPoints(rect)
    box = np.int0(box)
    cv2.drawContours(img, [box], 0, (0,255,0), 2)

First execute minAreaRect (contours) on rect minAreaRect () returns a tuple with Box2D structure The structure of Box2D is (upper left point (x, y), horizontal and vertical size (width, height), rotation angle).

In order to draw a rectangle, we need to convert this information into four coordinates, so execute boxPoints () that can do that.

It is a flow to draw at the end (Reference Reference)

It seems to work, but I'll blur it with a Gaussian filter just in case so that it can handle other photos well!

pisa

#Blur with Gaussian filter
blurred_img = cv2.GaussianBlur(white_masked_img,(5,5),10)

The edge has calmed down quite a bit

Only the largest rect is enough to pick up

pisa

#A function that returns the index of the rectangle with the largest area
def detect_max_rect(img,contours):
    #Index of contours taking the maximum area
    max_idx = 0
    
    for i in range(len(contours)):
        if cv2.contourArea(contours[max_idx]) < cv2.contourArea(contours[i]):
            max_idx = i
    
    return max_idx

contour Area (contours) You can get the area of the rectangle surrounded by the corners

Use this to define a function that returns the index of the coordinates that maximize the area

And by setting minAreaRect (max_idx), only the quadrangle with the maximum area can be extracted.

You're leaning!

Let's Vertical

I'm sorry I'm gudaguda because I'm doing it without writing a flowchart

Rotating rectangle trimming

First, create a function that trims the square area that you may need later.

pisa

#Cutout function
def crop_rect(img, rect):
    center, size, angle = rect
    center = tuple(map(int, center))  # float -> int
    size = tuple(map(int, size))  # float -> int
    h, w = img.shape[:2]  #Image height and width

    #Get height and width of rect
    height,width=rect[1]
    height,width=int(height),int(width)
    
    #Since angle indicates the angle formed by the horizontal line,+90
    if angle<0:
        angle+=90
    
    #Rotate the image using the affine matrix
    M = cv2.getRotationMatrix2D(center, angle, 1)
    rotated = cv2.warpAffine(img, M, (w, h))

    #cut out
    cropped = cv2.getRectSubPix(rotated, (w,h), center)

    return cropped

I wrote the details in the comments The image of the execution result is as follows

It's unnatural and interesting

Otateru

It ’s easy because the angle is obtained.

Create a function to rotate the image

pisa

#Function to rotate
def rotate_img(img,rect):
    #Calculate the center of the screen from the height and width
    _height = img.shape[0]                         
    _width = img.shape[1]                       
    _center = (int(_width/2), int(_height/2))
    
    _angle=rect[2]
    #Since angle indicates the angle formed by the horizontal line, it is +90.
    if _angle<0:
        _angle+=90
    
    #Rotate the image using the affine matrix
    _M = cv2.getRotationMatrix2D(_center, _angle, 1)
    _rotated_img = cv2.warpAffine(img, _M, (_width, _height))
    
    return _rotated_img

Take the center of the image and rotate it around it by the angle of Pisa.

High perfect! !! !!

.. .. .. I'm kidding, I'm sorry Please throw away the previous function

As a policy, I will complement the image hollowed out from the original image and paste the trimmed image at the same coordinates.

Image complement

pisa

#Image complement
def inpaint_img(img):
    
    _mask = cv2.imread('img/pisa1.jpg')

    #Fill with white
    cv2.drawContours(_mask, [box], 0, (255,255,255), -1)
    _maskGray = cv2.cvtColor(_mask,cv2.COLOR_BGR2GRAY)
    ret,_thresh = cv2.threshold(_maskGray,254,255,0)
    
    #Image correction
    dst = cv2.inpaint(img, _thresh, 3, cv2.INPAINT_TELEA)
    
    cv2.imwrite('img/pisa_rect_inpaint.jpg', dst)

inpaint is the input image and the mask image of the same size. Is required Pixels with non-zero values in this mask image represent where to repair In other words, it can be complemented by making the part to be repaired white and the rest black.

There are two types of algorithms, INPAINT_TELEA and INPAINT_NS, and this time I tried to execute both.

INPAINT_TELEA on the left and INPAINT_NS on the right

I feel that the empty part of NS is a little cleaner, so I adopted this one this time

Otateru (Take 2)

Pasting images seems to be troublesome if only OpenCV is used, so use the Pillow library

pisa

#Paste the trimmed pisa into the complementary background image
def paste_pisa(rect):
    #x,The center coordinates of the trimmed Pisa are entered in y
    x,y=rect[0]
    
    #I want to bring it to the upper left, so half the width w and height h x,Subtract from y
    _w,_h=rect[1]
    #print(h,w)
    x -= _h/2
    y -= _w/2
    
    x,y=int(x),int(y)
    
    source_img = Image.open('img/pisa_rect_crop.jpg')
    canvas_img = Image.open('img/pisa_rect_inpaint.jpg')
    
    #Paste the trimmed pisa on the inpainted image
    canvas_img.paste(source_img, (x,y))
    #Save image
    canvas_img.save('img/vertical_pisa.jpg')

Execute

** Pisa, Pisa, Pisa stood up! !! !! ** **

Try to make it rush

at the end

The complementary part is visible and rough, and it is not very vertical, but overall I feel like I said it well.

Personally, I was surprised at the accuracy of the inpaint function, which is only an algorithm, without any learning. Also, the dynamic typing of python is too convenient, and it's a perfect phrase. .. .. Honestly, I don't understand it, but I'm impressed with the relaxed specifications because there are places where I proceeded

This time, the area is detected by color, so if you can do it with the edge next time, I feel that the degree of freedom will increase.

Everyone, ** Let'vertical! ** **

bonus

** Vertical Michael **

This is normal gravity

Reference material

minAreaRect function https://www.it-swarm.dev/ja/python/minarearect-opencv%E3%81%AB%E3%82%88%E3%81%A3%E3%81%A6%E8%BF%94%E3%81%95%E3%82%8C%E3%82%8B%E5%9B%9B%E8%A7%92%E5%BD%A2%E3%81%AE%E3%83%88%E3%83%AA%E3%83%9F%E3%83%B3%E3%82%B0python/824441051/

Contour extraction https://hk29.hatenablog.jp/entry/2020/02/01/162533

White extraction https://temari.co.jp/blog/2017/11/13/opencv-4/

trimming https://teratail.com/questions/219340

inpaint function https://lp-tech.net/articles/kb4bO/view?page=2

Pillow library https://water2litter.net/rum/post/python_pil_paste/