A story that supports electronic scoring of exams with image recognition

Welcome to the 23rd day of Yuruyuru New Graduate Advent Calendar 2019. I'm @NamedPython, a freshman freshman with a loose margin: joy:

Today is about my hidden feat ** Image recognition with OpenCV **.

I had decided to write about image recognition in advance, but it was hard to find something new, so I tried my best to write it in the school class, but I didn't give a memorial service anywhere ~~ ** Answer sheet I will leave a record of area extraction ** here: pencil2:

What were you trying to make

I said it was a class, but the class name is ** Programming Application **, and let's work on problem solving by software development as a team, involving what we usually learn in class and the skills of students. is.

Then, what is the theme of problem solving?

** Ease the electronic scoring of the exams your teacher is doing **: pray:

It was that. In the first place, at the time of electronic scoring, it is quite now and young, but if you summarize the requirements

--Scanning student exam answers and importing them to ** iPad ** (** Now ) ――The answer sheet does not have an answer column, so you can write it freely. ――Because of the above, the position of the answer is ** different depending on the student ** ――So, the answer area of the captured image is surrounded by ** color-coded ** with Apple Pencil ( Now **) ――I want to score the same question (same color coding) at once --You guys, ** please do something **: pray:

It was like that.

team

Actually, one year before this class,

--Backend API, web front @NamedPython --iOS @Taillook --Android @ youta1119 --Python and RasPi control, natural language processing @Kyoskk

We formed the strongest team and received great criticism. As a result, the number of team members was limited: frowning2:

Even so, I pulled @Taillook and another close classmate to form a team of three people.

solution

With the team team mentioned above, I have been pulling ** the guy who can write iOS **.

Since it was a big deal, I proposed a system with the following configuration that allows you to ride iOS from anything to anything (extracted from the presentation of the final announcement). Thanks for the implementation around iOS, @Taillook.

I did my best with Keynote for this figure. The fonts are ** M + ** and ** Tsukushi B Maru Gothic **.

Recognition overview

--Since there is already a color-coded operation, it seems good to extract only the saturated part. --Convert from BGR space to HSV space and extract only S (saturation channel)

• Binarization --Since it is reasonably binarized at the time of S channel extraction, a simple discriminant analysis method is OK. --Cut out each area --Edge detection by findContours --Since various edges are detected, edge structuring by cv2.RETR_TREE --Those with more sides inside ʻOR Skip the area below a certain level --Determine which color the area is painted in --Extract about 20 pixel colors diagonally to the upper left (x -1, y -1`) of the extracted side --Take the average and use the color of that side

Target image

At that time, I received an actual answer sheet for operation verification, but I had a feeling that the rights were gray to bring it here, so I used the power of ** Pages craftsman ** to generate a sample.

Implementation

Well, it's almost boring, so Dawn and the source code. I prepared Dockerfile which is FROM jjanzic / docker-python3-opencv so that I can develop it anywhere. In the end, the image recognition part was overkill because I developed it fully.

The contents of Docker are as follows.

• Python 3.7.0
• OpenCV 3.4.1

source.py

import os
import cv2
import numpy as np

DIR = os.path.dirname(os.path.abspath(__file__))

image = cv2.imread(f'{DIR}/image/sample_answer_sheet.png')
if image is None:
    print("File not found.")
    exit()


def write_out(img, name):
    cv2.imwrite(f'{DIR}/image/result/{name}.png', img)


def extract_inside(contours, hierarchy, debug=False):
    extracted = []
    contours_drawed = image.copy()
    if debug:
        print(f'len: {len(contours)}')
        print(hierarchy)

    for index in range(len(contours)):
        if hierarchy[0, index, 2] != -1 or cv2.contourArea(contours[index]) < 8000:
            continue
        extracted.append(contours[index])

        if debug:
            cv2.drawContours(contours_drawed, contours, index, (255, 0, 255), 10)

        print(f'{index} : {cv2.contourArea(contours[index])}')

    if debug:
        write_out(contours_drawed, 'out_contours')

    return extracted


width, height = image.shape[:2]

extracted = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)[:, :, 1]

write_out(extracted, 'after_cvt')

ret, thrshld_ex = cv2.threshold(extracted, 0, 255, cv2.THRESH_OTSU)

write_out(thrshld_ex, 'thrshld')

kernel = np.ones((3, 1), np.uint8)
thrshld_ex = cv2.morphologyEx(thrshld_ex, cv2.MORPH_OPEN, kernel)

write_out(thrshld_ex, 'thrshld_ex')

_, contours, hierarchy = cv2.findContours(thrshld_ex, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)

inside_contours = extract_inside(contours, hierarchy, True)

print(f'extracted {len(inside_contours)} countours')

clr_img_cp = image.copy()
for circle in inside_contours:
    colors = []
    av_color = 0
    M = cv2.moments(circle)
    center_of_circle = (int(M['m10']/M['m00']), int(M['m01']/M['m00']))
    cnt = 0
    for approx in circle:
        if cnt == 20:
            break
        cv2.circle(clr_img_cp, (approx[0, 0] - 1, approx[0, 1] - 1), 1, (255, 0, 255), 1)
        colors.append(image[approx[0, 1] - 1, approx[0, 0] - 1])
        cnt += 1
    cav = np.mean(colors, axis=0)
    cv2.circle(clr_img_cp, center_of_circle, 100, cav, 20)

write_out(clr_img_cp, 'out')

It looks like this when I drop what I wrote in the recognition summary earlier into the source code. After prototyping with Python like this, I ported it to ʻObjective-C` and had @Taillook bridge it with Swift.

result

--Convert to HSV color space and extract only S --Binarization by discriminant analysis --Edge detection --Extract the color of the side

The results of each of the four processes are posted.

Convert to HSV color space and extract only S

extract_s_of_hsv.py

cv2.cvtColor(image, cv2.COLOR_BGR2HSV)[:, :, 1] # 0: H, 1: S, 2: V

At this point, it's almost binarized. It's too much.

Binarization by discriminant analysis

Just specify cv2.THRESH_OTSU. Easy.

thresold_with_otsu.py

cv2.threshold(extracted, 0, 255, cv2.THRESH_OTSU)

If there is a little noise, then apply morphologyEX to remove the noise.

Edge detection

OpenCV is already amazing. findContours is amazing. I read the paper for the time being, but it's too metamorphic to recognize even the structural information of the sides.

find_contours_with_structure.py

_, contours, hierarchy = cv2.findContours(thrshld_ex, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)

In hierarchy, which index side in contours is ** next to the index side ** or ** is included ** is stored. It's too much.

The sides are represented by a set of points and are drawn in purple.

Extract edge color + draw

This is a push. Thanks to numpy for making it so easy to average colors.

sampling_color_and_averaging_and_plotting.py

clr_img_cp = image.copy()
for circle in inside_contours:
    colors = []
    av_color = 0
    M = cv2.moments(circle)
    center_of_circle = (int(M['m10']/M['m00']), int(M['m01']/M['m00']))
    cnt = 0
    for approx in circle:
        if cnt == 20:
            break
        cv2.circle(clr_img_cp, (approx[0, 0] - 1, approx[0, 1] - 1), 1, (255, 0, 255), 1)
        colors.append(image[approx[0, 1] - 1, approx[0, 0] - 1])
        cnt += 1
    cav = np.mean(colors, axis=0)
    cv2.circle(clr_img_cp, center_of_circle, 100, cav, 20)

The center of the figure is determined, and a circle is drawn there with the average color. (Actually, if you look closely at this image, the target pixels of 20 samples are purple.) It feels good with almost the same color as the sides.

Summary

So, I made a memorial service here for the image processing program I made when I was a student: coffin:

From my point of view, it was a reflection, but I hope you read this and feel the possibility of Python + ʻOpenCV`.

The only caveat is that ʻOpenCV` can be used by learning from many cases without knowing the theory, but I feel that the theory is the first to guide the shortest path of application of means to problem solving. Therefore, ** information engineering is important **. I have only the last-minute theory because I was light and light.

However, the following can be said for sure.

―― ʻOpenCV is amazing --Python + ʻOpenCV is easy to do --Good compatibility with numpy

about it. Actually, before writing this article, I tried ʻOpenCV by Rust, but it was not good compared to the usability of Python + ʻOpenCV. Well, I'm not used to implementing Rust, so it may not be helpful, but: yum:

It's Python, isn't it slow? There seems to be a story, but excluding drawing, it is about ** 0.3 [sec] **. It's just a C-binding. Great. Even at the bicycle shopping site cyma-cyma-, which I am involved in development, I feel more motivated to recognize images, so I would like to show my strength.

@shimura_atsushi wrote 2 articles in Ateam cyma Advent Calendar 2019, so if you are interested, please.

-Challenge the challenges of Cyma using the OCR service of Google Could Platform -Continued to challenge Cyma's challenges using Google Could Platform's OCR service

in conclusion

Yuruyuru New Graduate 1st Grade Advent Calendar 2019, how was your 23rd day?

The 24th day is the turn of the front-end engineer @ cheez921, who also understands the back end. Duel, standby!

Good ** Christmas **, Good ** Year-end **, Good ** New Year **, Good ** Image recognition **.