Judgment of backlit image using OpenCV

Introduction

Hello, My name is Kano. At ABEJA, I am in charge of customer support for a retail service called Insight for Retail.

Recently, I am living a vegan life under the influence of a documentary called Game Changer that I saw on Netflix. However, I sometimes eat Jiro, Iekei Ramen, and Kotteri ramen, so I can't lose weight.

I'm not an engineer, but I'll do my best to write. This is the article on the 13th day of ABEJA Advent Calendar 2019.

Trigger

At ABEJA, I feel that engineers are really enjoying making things.

Of course, so is work, but there are also things that are not related to work. For example, stock AI or Shiitake cultivation using IoT devices.

As I joined ABEJA in 2018 and worked with them every day, I became more and more eager to make something.

In addition, ABEJA values technopreneurship, and the idea that even business members can innovate by utilizing technology permeates. There was also the fact that it was done.

Customer support role for a service called Insight for Retail

By the way, if I explain the service called Insight for Retail that I am in charge of very roughly,

・ Install IoT devices such as cameras in retail stores to acquire data ・ AI analyzes the acquired data ・ Aggregate the analyzed data and visualize it in an easy-to-understand manner ・ And realize data-driven store management

That is.

Therefore, when introducing the service, the installation of IoT devices in the store always occurs, but the construction work is also included in the role of customer support. In other words, one of the important missions of Insight for Retail customer support is to "provide AI with clean data suitable for analysis from IoT devices."

(Although it is not related to this article, customer support also does research and verification of new IoT devices. Of course, it also handles inquiries and release announcements, which is the main part of customer support, so ordinary SaaS companies I feel that the range of defense is wide compared to the customer support of. I would like to introduce this area if there is an opportunity.)

I want to detect back light

By the way, when I handle the data acquired by IoT devices every day as customer support, I realize that it is really difficult to acquire data in real space. For example, backlit.

Below is a picture of when I climbed Mt. Fuji with a friend this summer. The back light of the sunrise is amazing.

One of the services of Insight for Retail is that AI estimates age, gender, and whether it is a repeater based on the face image of the visitor, but with such a black face image, no matter how good the AI model is. The accuracy doesn't come out.

It is rare for an actual store to have such a back light, but the back light sometimes darkens the face. Moreover, the backlight may or may not come out depending on the time, so it is difficult to detect it at the time of installation.

Therefore, I decided to think that if I could create a mechanism to automate the detection of backlight, it would be useful for my work, and it would be useful for studying programming, and even for studying image processing.

What is the state of backlight?

What is the state of backlight in the first place? To see this, try drawing a histogram of the backlit image and the non-backlit image.

You can see OpenCV Tutorial for the explanation of the histogram. It's easy, but roughly speaking, it shows the distribution of the pixel values of each pixel in the image. The pixel value is large when it is bright and small when it is dark, so you can see the distribution of light and dark in the image by looking at the histogram.

(I wrote it briefly, but I read an image processing book to get here, and I was told by an engineer that there is a convenient image processing / CV library called OpenCV.)

The code that draws the histogram is below.

import cv2
import numpy as np
from matplotlib import pyplot as plt

#Standard input
filename = input()

#Load image
img = cv2.imread(filename)

#Draw histogram of image
plt.hist(img.ravel(),256,[0,256]); plt.show()

First, the histogram of the backlit image. The image is a cropped face part of the photo of Mt. Fuji. Since the face of the backlit image becomes dark, the pixel value is biased toward the smaller one.

Next is a histogram of the image that is not backlit. I am 7 years ago. Compared to the backlit image, this image has evenly distributed pixel values.

Therefore, I hypothesized that if the median or average value of the pixel values of an image is smaller than a certain value, it can be judged as backlight.

There is also a back light that brightens the face

However, when I searched for backlit images on the Web, I found that there was such a case. (I didn't have a good image on hand, so I bought it at PIXTA.)

On the contrary, this backlit image makes the face too bright, which seems to affect the judgment accuracy.

The histogram looks like this:

This time, the pixel value is biased toward the larger one. It seems that the strategy of judging by the amount of the average value of the pixel values mentioned earlier is impossible. What do you do now?

Let's take a look at the histogram of each image again.

Looking at this, the backlit images in both cases have a biased distribution of pixel values compared to non-backlit images. So, I thought that the back light could be judged by this bias. Specifically, it seems good to calculate the standard deviation of the pixel values of the image (a value that indicates the degree of data scattering).

Backlight judgment based on the standard deviation of the pixel value of the image

So, let's calculate the standard deviation of the pixel values of the image using OpenCV.

Click here for the code. OpenCV is very convenient because you can write such calculations with just a few lines of code.

from imutils import paths
import argparse
import cv2
import os
import numpy as np

#Standard input
filename = input()

#Load the image.
img = cv2.imread(filename)

#Make the image grayscale.
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

#Calculate the standard deviation.
mean, stddev = cv2.meanStdDev(gray)
print('stddev',stddev)

The results are as follows.

First of all, from an image that is not backlit. The standard deviation here was 55.6 </ font>.