I tried to correct the keystone of the image

Motivation

――In the shopping mall in the neighborhood, many plants are planted in the outer passage, and there is a plate with a description of the plants together. I was wondering what kind of plants (I think there are more than 100 kinds), so I decided to aggregate the information on the plate. ――However, it is troublesome to input the information on the plate into your smartphone and collect the information. .. .. I decided to take a picture, correct it, and then perform OCR for the sake of simplicity.

Image before correction

What i did

1. Acquisition of plate image
2. Mask and binarize parts other than the plate
3. Noise removal
4. Get / approximate contour Approximate
5. Plate image correction
6. Keystone correction

1. Acquisition of plate image

1. Mask and binarize parts other than the plate

The image is converted to HSV format, and what Hue (hue) is included in the range of the plate is output as white, and the others are output as black, and masking and binarization are performed.

preprocess.py

cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

The mask looks good.

Supplement

HSV is a method of expressing color with three elements.

--Hue --Color types (eg red, blue, yellow) --Saturation --Color vividness --Brightness --Color brightness

2. Noise removal

After closing, opening Closing first because the pixels of the plate may be missing

Supplement

--Dilation --If there is even one white pixel around the pixel of interest, replace the pixel of interest with white. --Erosion --If there is even one black pixel around the pixel of interest, replace the pixel of interest with black. --Closing --Processing that expands and contracts the same number of times --Opening --Processing that contracts and expands the same number of times

3. Obtain contour ・ Approximate plate contour

Get contour

Detects and draws the contours of objects in the denoised image.

preprocess.py

img, contours, hierarchy = cv2.findContours(img, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPL)
img = cv2.drawContours(img, contour, -1, (0, 0, 255), 30)
cv2.imwrite(output_path, img)

The outline was drawn for this.

Get the contour of the plate

In this image, there is no outline other than the plate, Get the contour (plate) with the largest area in case there is a contour other than the plate.

preprocess.py

contour_areas = {}
for i, contour in enumerate(contours):
    area = cv2.contourArea(contour)
    contour_areas[i] = area

max_area = max(contour_areas.values())
max_area_idx = [i for i, v in contour_areas.items() if v == max_area][0]
max_contour = contours[max_area_idx]

Get the coordinates of the four corners of the plate

Approximate the shape of the area with a small number of points and get the coordinates of the four corners of the plate.

preprocess.py

arc_len = cv2.arcLength(max_contour, True)
approx_contour = cv2.approxPolyDP(max_contour, epsilon=0.1 * arc_len, closed=True)
img = cv2.drawContours(img, approx_contour, -1, (0, 0, 255), 30)
cv2.imwrite(output_path, img)

I was able to get the coordinates of the four corners with a good feeling.

2. Plate image correction

1. Keystone correction

The coordinates of the four corners are divided into upper left, lower left, upper right, and lower right, and keystone correction is performed to draw the image.

preprocess.py

approx = approx_contour.tolist()

left = sorted(approx, key=lambda x: x[0])[:2]
right = sorted(approx, key=lambda x: x[0])[2:]
left_down = sorted(left, key=lambda x: x[0][1])[0]
left_up = sorted(left, key=lambda x: x[0][1])[1]
right_down = sorted(right, key=lambda x: x[0][1])[0]
right_up = sorted(right, key=lambda x: x[0][1])[1]

perspective_base = np.float32([left_down, right_down, right_up, left_up])
perspective = np.float32([[0, 0], [700, 0], [700, 500], [0, 500]])

psp_matrix = cv2.getPerspectiveTransform(perspective_base, perspective)
plate_img = cv2.warpPerspective(org_img, psp_matrix, (700, 500))
cv2.imwrite(output_path, img)

Keystone correction result

Summary

Stumble

――At first, straight line detection was performed on the image before correction, but even if the line of the plant was acquired or the straight line of the plate was acquired, it was cut off, which was not the intended result. ――After that, I tried to mask based on the RGB values, but I couldn't detect the outline of the plate well because most of the plate was missing or the part unrelated to the plate could not be masked.

Good thing

――Since I started by groping completely, I didn't know which method to use, and as a result, I was able to try various image processing methods. (Although the theory has not caught up.) --Binarization (fixed value, Otsu, adaptive threshold processing) --Line detection (Hough transform, stochastic Hough transform) --Edge detection (Canny method, LSD) --Smoothing (moving average, Gaussian)

Task

If plants overlap the plate and the area of the plate is divided as shown in the image below, the keystone correction will not work. (I wonder if the plants will not cover the plate when taking pictures ...)

ending

By the way, in Google Cloud Vision, the OCR result is almost the same before and after correction, and OCR is almost perfect even before correction. Google's style! !! !!

Other

Click here for source code https://github.com/ChihiroHozono/Plate-Text-Detector