Detect mosaic points in the image

(Added on May 24, 2017) I registered the code on github. It is the same as what is described on this page. https://github.com/summer4an/mosaic_detector

Premise

https://twitter.com/stsiizk/status/858517661889646592 that? Maybe this is ... So I decided to prepare (?) For the coming day.

There are multiple types of mosaics, but this time we will focus on a large number of boxes painted in a single color.

I tried google, but it was all about mosaic processing, and I couldn't find a way to detect the mosaic part, so I did it myself.

The goal is to detect the mosaic part of the image and paint it white.

I used python3.5.2 and OpenCV3.2.0.

policy

After trial and error by building code with image processing software and python, we obtained satisfactory results with the following procedure.

1. Convert to grayscale
2. Edge detection with Canny
3. Black and white inversion
4. Blur a little
5. Pattern matching for grid images of various sizes
6. Fill the matched area

The reason why I blur a little with 4 is that the result of 3 is that the border line of the box does not become a straight line with a width of 1 pixel, but it goes back and forth between the widths of 2 pixels, so pattern matching is difficult It's because of it. The reason for using grid images of various sizes in 5 is that pattern matching is vulnerable to scaling. I used a grid image of 11 to 20 pixels square, which seems to be common.

code

First, create a grid image for pattern matching of each size below.

python

#!/usr/bin/env python
# -*- coding: utf-8 -*-

#Generate a pattern image file for when the mosaic size is from 11 to 20.
#The following reference materials.
#  http://qiita.com/suto3/items/5181b4a3b9ebc206f579

from PIL import Image

def make_image(masksize, filename):
    picturesize = 2+masksize+masksize-1+2
    screen = (picturesize, picturesize)

    img = Image.new('RGB', screen, (0xff,0xff,0xff))

    pix = img.load()

    for i in range(2,picturesize,masksize-1):
        for j in range(2,picturesize,masksize-1):
            for k in range(0,picturesize):
                pix[i, k] = (0,0,0)
                pix[k, j] = (0,0,0)

    img.save(filename)
    return

for i in range(11, 20+1):
    make_image(i, "pattern"+str(i)+"x"+str(i)+".png ")

If you execute it, you can get 10 images like this. 　

The following is detected using this grid image.

python

#!/usr/bin/env python
# -*- coding: utf-8 -*-

#Detect the mosaic part and paint it white.
#The following reference materials.
#・ Template Matching
#    http://docs.opencv.org/3.2.0/d4/dc6/tutorial_py_template_matching.html
#    http://opencv.jp/cookbook/opencv_img.html#id32

import cv2
import numpy as np

import sys
args = sys.argv

if len(args) != 2:
    print("too few argument.")
    sys.exit(1)

img_rgb = cv2.imread(args[1])

img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY) #To grayscale
img_gray = cv2.Canny(img_gray,10,20) #Edge detection
img_gray = 255-img_gray #Black and white inversion
img_gray = cv2.GaussianBlur(img_gray,(3,3),0) #Blur a little

cv2.imwrite('output_gray.png', img_gray)

for i in range(11,20+1):
    pattern_filename = "pattern"+str(i)+"x"+str(i)+".png "
    template = cv2.imread(pattern_filename, 0)
    w, h = template.shape[::-1]

    img_kensyutu_kekka = cv2.matchTemplate(img_gray,template,cv2.TM_CCOEFF_NORMED)
    threshold = 0.3
    loc = np.where(img_kensyutu_kekka >= threshold)
    for pt in zip(*loc[::-1]):
        #cv2.rectangle(img_rgb, pt, (pt[0] + w, pt[1] + h), (255,255,255), 1)
        cv2.rectangle(img_rgb, pt, (pt[0] + w, pt[1] + h), (255,255,255), -1)
    cv2.imwrite('output_progress_'+str(i)+'.png', img_rgb)

cv2.imwrite('output_result.png', img_rgb)

cv2.imshow('window1', img_rgb)
cv2.imshow('window2', img_gray)
cv2.waitKey(0)

cv2.destroyAllWindows()

result

Process images with mosaics in various sizes below. The image is from http://gahag.net/011032-cat-sunflower-smell/.

The result is below.

I was able to fill in all the areas where the mosaic was applied. The extra parts are also filled in, but it will be complemented anyway, so it will be OK.

I am satisfied with the results of the other images as well.

Reference material

--How to use the Image library http://qiita.com/suto3/items/5181b4a3b9ebc206f579

• Template Matching http://docs.opencv.org/3.2.0/d4/dc6/tutorial_py_template_matching.html http://opencv.jp/cookbook/opencv_img.html#id32