Former image processing engineer tried to solve Saizeriya's spot the difference with OpenCV

Introduction

Since I was making settings for image processing used for industrial machines in my previous job, I was curious about the algorithm of Spot the Difference in Saizeriya, so I tried out it. ..

This theme

As expected, there is no trick to use the same thing, so this time I will search for mistakes with the image below. body.png Copyright Saizeriya Co,. Ltd All rights reserved.

From here, you can find information on the answers to finding mistakes. If you don't like spoilers, try it from Saizeriya's homepage.

Changes ... Before implementing

Confirmation of assumptions

First, let's check Saizeriya's spot the difference. There are two points to watch when looking for mistakes in Saizeriya.

  1. Use the normally printed menu
  2. Differences in color are included in the answer

1 means that the image file is likely to be converting the resolution from print to display. Therefore, there is no guarantee that the pixels in the same place will be the same color, and some noise countermeasures are required. 2 means that we need a method to clearly describe and detect the difference in color. It seems that better results will be obtained by comparing HSV (hue / brightness / saturation) instead of RGB.

change point

Confirmation before change

First, let's check the current situation. The difference in the initial state is as follows. 最初.png

Noise removal 1

Try blurring to see if you can remove fine noise.

    #Remove image margins
    img = img_src[:, padding:-padding]
    #Split the image left and right
    height, width, channels = img.shape[:3]
    img1 = img[:, :width//2]
    img2 = img[:, width//2:]

    #Add blur to remove fine noise
    blue_size = (5, 5)
    img1 = cv2.blur(img1, blue_size)
    img2 = cv2.blur(img2, blue_size)
    
    cv2.imshow("img2",img2)

ぼかし元画像.png ぼかし.png

There is no atmosphere that has improved so much. Let's try another method later.

HSV conversion

Get diffs and convert to grayscale

Being aware that it is printed matter, convert it to HSV and then take the difference. Also, the answer should be that one of H, S, and V has changed significantly, so check the maximum value of the HSV difference to see if it is a mistake.

#Since the original target is printed matter, try converting it to HSV.
img1_hsv = cv2.cvtColor(img1, cv2.COLOR_RGB2HSV)
img2_hsv = cv2.cvtColor(img2, cv2.COLOR_RGB2HSV)

#Calculate the difference between two images
img_diff = cv2.absdiff(img2_hsv, img1_hsv)
diff_h = img_diff[:, :, 0]
diff_s = img_diff[:, :, 1]
diff_v = img_diff[:, :, 2]

#Get the biggest change in the HSV difference
diff_glay = np.maximum(diff_h, diff_s, diff_v)

ぼかし+HSV変換.png

Especially, there is a lot of noise near the characters, but since the characters are black and white, the hue (H) changes greatly with a slight variation in RGB.

Also, I'm worried that the color of the sheep's clothes, which should have been wrong, is quite dark. Consider the later processing, and change it so that the change in brightness is detected as over.

Low-saturation hue measures

Simply avoid seeing changes in hue in desaturated areas. At the same time, the difference between brightness and saturation is standardized so that it is in the range of 0 to 255.


#Calculate the difference between two images
img_diff = cv2.absdiff(img2_hsv, img1_hsv)
diff_h = img_diff[:, :, 0]
diff_s = img_diff[:, :, 1] * 3
diff_v = img_diff[:, :, 2]

#Saturation below a certain level(V)The part of is the shade(H)Do not consider the difference of
H_THRESHOLD = 70
_, diff_h_mask = cv2.threshold(diff_v, H_THRESHOLD, 255, cv2.THRESH_BINARY)
diff_h = np.minimum(diff_h, diff_h_mask)

#Normalize the difference between brightness and saturation
diff_s = cv2.normalize(diff_s, _, 255, 0, cv2.NORM_MINMAX)
diff_v = cv2.normalize(diff_v, _, 255, 0, cv2.NORM_MINMAX)

#Get the biggest change in the HSV difference
diff_glay = np.maximum(diff_h, diff_s, diff_v)

H虫.png

It has become much easier to understand.

Binarization and noise removal

Next, let's make it a little easier to understand what has changed. Binarize and display the difference in white.

Also, since it was full of noise as it was, cancel the blur and use the opening to remove the noise.


#Get candidates for unusual locations by binarization
DIFF_THRESHOLD = 60
_, diff_bin = cv2.threshold(diff_glay, DIFF_THRESHOLD, 255, cv2.THRESH_BINARY)

#Noise removal
diff_bin = cv2.morphologyEx(diff_bin, cv2.MORPH_OPEN, cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3)))

Binarization only 2値化のみ.png

Opening included オープニング.png

By arranging them in this way, the power of noise removal by the opening is obvious.

Display format improvement

In the linked article, I couldn't visually find the color depth. Let's change it so that you can easily see the change between a place where there is no difference and a certain place.

That said, the places where there are already mistakes are shown in black and white. It will be enough to incorporate this in a nice way.

#Image composition
diff_bin_rgb = cv2.cvtColor(diff_bin, cv2.COLOR_GRAY2RGB)
add = np.maximum(np.minimum(img2, diff_bin_rgb), (img2 // 3))
cv2.imshow("add", add)

The final output looks like this: The fireplace is difficult to understand, but I think you can understand it enough by looking at the data at the time of binarization. Or I think that it is easy to see if you use it by expanding (dilating) the binarized data before RGB conversion.

最終結果.png

Source code

There were some other parts of the source of the original article that I was interested in, but this time it is not essential, so I will omit it. After putting in my knowledge of image processing, I ended up with a source like this.

import cv2
import numpy as np


# --------------------------------------------------- #
#Image composition#
# --------------------------------------------------- #

def FitImageSize_small(img1, img2):
    # height
    if img1.shape[0] > img2.shape[0]:
        height = img2.shape[0]
        width = img1.shape[1]
        img1 = cv2.resize(img1, (width, height))
    else:
        height = img1.shape[0]
        width = img2.shape[1]
        img2 = cv2.resize(img2, (width, height))

    # width
    if img1.shape[1] > img2.shape[1]:
        height = img1.shape[0]
        width = img2.shape[1]
        img1 = cv2.resize(img1, (width, height))
    else:
        height = img2.shape[0]
        width = img1.shape[1]
        img2 = cv2.resize(img2, (width, height))
    return img1, img2


img = cv2.imread('file name')

if img is None:
    print('Unable to read the file')
    import sys

    sys.exit()

cv2.imshow("img", img)

#Find the appropriate padding width so that the two images best match when the margins are removed
img_src = img
padding_result = []
for padding in range(0, 50):
    #Remove image margins
    # (Consider the possibility of no margins)
    if padding:
        img = img_src[:, padding:-padding]

    #Split the image left and right
    height, width, channels = img.shape[:3]
    img1 = img[:, :width // 2]
    img2 = img[:, width // 2:]

    #Match image size(To the smaller one)
    img1, img2 = FitImageSize_small(img1, img2)

    #Calculate the difference between two images
    img_diff = cv2.absdiff(img2, img1)
    img_diff_sum = np.sum(img_diff)

    padding_result.append((img_diff_sum, padding))

#Choose the one with the least difference
_, padding = min(padding_result, key=lambda x: x[0])

#Remove image margins
if padding:
    img = img_src[:, padding:-padding]

#Split the image left and right
height, width, channels = img.shape[:3]
img1 = img[:, :width // 2]
img2 = img[:, width // 2:]
cv2.imshow("img2", img2)

#Since the original target is printed matter, try converting it to HSV.
img1_hsv = cv2.cvtColor(img1, cv2.COLOR_RGB2HSV)
img2_hsv = cv2.cvtColor(img2, cv2.COLOR_RGB2HSV)

#Calculate the difference between two images
img_diff = cv2.absdiff(img2_hsv, img1_hsv)
diff_h = img_diff[:, :, 0]
diff_s = img_diff[:, :, 1] * 3
diff_v = img_diff[:, :, 2]

#Saturation below a certain level(V)The part of is the shade(H)Do not consider the difference of
H_THRESHOLD = 70
_, diff_h_mask = cv2.threshold(diff_v, H_THRESHOLD, 255, cv2.THRESH_BINARY)
diff_h = np.minimum(diff_h, diff_h_mask)

#Normalize the difference between brightness and saturation
diff_s = cv2.normalize(diff_s, _, 255, 0, cv2.NORM_MINMAX)
diff_v = cv2.normalize(diff_v, _, 255, 0, cv2.NORM_MINMAX)

#Get the biggest change in the HSV difference
diff_glay = np.maximum(diff_h, diff_s, diff_v)

#Get candidates for unusual locations by binarization and opening
DIFF_THRESHOLD = 60
_, diff_bin = cv2.threshold(diff_glay, DIFF_THRESHOLD, 255, cv2.THRESH_BINARY)
diff_bin = cv2.morphologyEx(diff_bin, cv2.MORPH_OPEN, cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3)))

#Match image size(To the smaller one)
img2, diff_bin = FitImageSize_small(img2, diff_bin)
cv2.imshow("img_diff", diff_bin)

#Image composition
diff_bin_rgb = cv2.cvtColor(diff_bin, cv2.COLOR_GRAY2RGB)
add = np.maximum(np.minimum(img2, diff_bin_rgb), (img2 // 3))
cv2.imshow("add", add)

cv2.waitKey(0)
cv2.destroyAllWindows()

Summary

Recommended Posts

Former image processing engineer tried to solve Saizeriya's spot the difference with OpenCV
I made a program to solve (hint) Saizeriya's spot the difference
I tried to process the image in "sketch style" with OpenCV
I tried to process the image in "pencil style" with OpenCV
I tried "smoothing" the image with Python + OpenCV
I tried "differentiating" the image with Python + OpenCV
I tried "binarizing" the image with Python + OpenCV
I tried to solve the soma cube with python
I tried to solve the problem with Python Vol.1
Python OpenCV tried to display the image in text.
I tried "gamma correction" of the image with Python + OpenCV
Real-time image processing basics with opencv
Try blurring the image with opencv2
I tried to solve the ant book beginner's edition with python
Consider the speed of processing to shift the image buffer with numpy.ndarray
Python3 Engineer Certification Basic Exam-I tried to solve the mock exam-
I tried to make an image similarity function with Python + OpenCV
I tried to get the batting results of Hachinai using image processing
How to crop the lower right part of the image with Python OpenCV
I tried to solve the 2020 version of 100 language processing [Chapter 3: Regular expressions 25-29]
Image processing with Python & OpenCV [Tone Curve]
I tried playing with the image with Pillow
I tried to solve TSP with QAOA
AtCoder Green tried to solve with Go
Light image processing with Python x OpenCV
Image processing with Lambda + OpenCV (gray image creation)
I tried to extract named entities with the natural language processing library GiNZA
I tried to solve the 2020 version of 100 language processing knocks [Chapter 1: Preparatory movement 00-04]
The front engineer tried to automatically start go's API server with systemd quickly
I tried to solve the 2020 version of 100 language processing knocks [Chapter 1: Preparatory movement 05-09]
XavierNX accelerates OpenCV image processing with GPU (CUDA)
Crop the image to rounded corners with pythonista
I tried to save the data with discord
Try to solve the man-machine chart with Python
I tried to detect motion quickly with OpenCV
How to crop an image with Python + OpenCV
I tried to correct the keystone of the image
The easiest way to use OpenCV with python
I tried using the image filter of OpenCV
I tried simple image processing with Google Colaboratory.
I tried to solve the virtual machine placement optimization problem (simple version) with blueqat
I tried to compare the processing speed with dplyr of R and pandas of Python
The 15th offline real-time I tried to solve the problem of how to write with python
I tried to learn the sin function with chainer
Try to solve the programming challenge book with python3
Try to solve the internship assignment problem with Python
I tried to touch the CSV file with Python
[OpenCV / Python] I tried image analysis of cells with OpenCV
NW engineer tried to aggregate addresses with python netaddr
Convert the image in .zip to PDF with Python
I tried to compress the image using machine learning
I tried to solve AOJ's number theory with Python
How to write offline real time I tried to solve the problem of F02 with Python
Sentiment analysis with natural language processing! I tried to predict the evaluation from the review text
Image processing with MyHDL
Image processing with Python
Image Processing with PIL
I tried to analyze the whole novel "Weathering with You" ☔️
I wanted to solve the Panasonic Programming Contest 2020 with Python
I tried to find the average of the sequence with TensorFlow
I tried to notify the train delay information with LINE Notify