Face estimation using OpenCV

Last time, I extracted the face area using Haar-like feature classifier (Face recognition using OpenCV (Haar-like feature classifier). ). This time, as an application, the face image extracted by the Haar-like feature classifier was trained by the OpenCV face estimator (Eigenface, Fisherface, LBPH), and there were differences in facial expressions, disguise, and lighting conditions. Let's guess who the unlearned face is.

The face estimation algorithm installed in OpenCV is as follows. (Click here for details)

** Eigenface ** The process of Principal Component Analysis (PCA), which converts a face image into an eigenface, is as follows.

Prepare an image for training (same lighting conditions, scaling at eye and nose positions, same resolution).
Calculate the average of the training images and subtract the average image from each image.
Calculate the covariance matrix of the subtracted image.
Calculate the eigenvectors and eigenvalues from the covariance matrix.
Select the main component.

Fisherface
An improved version of Eigenface. Compared to Eigenface, it is less susceptible to differences in lighting and angles.
Local Binary Patterns Histogram(LBPH)
Divide the face into smaller cells and compare the histograms for each area. Compared to Eigenface and Fisherface, this algorithm has the feature that it can detect different face sizes and shapes with higher accuracy.

All images are from the OpenCV tutorial (link)

This time, we will recognize the face, but these algorithms are also applied to handwriting assessment, medical image processing, and lip reading. In an interesting place, it is possible to identify a person in disguise.

OpenCV OpenCV (Open Source Computer Vision Library) is a collection of BSD-licensed video / image processing libraries. There are many algorithms for image filtering, template matching, object recognition, video analysis, machine learning, and more.

■ Example of motion tracking using OpenCV (OpenCV Google Summer of Code 2015) https://www.youtube.com/watch?v=OUbUFn71S4s

■ Click here for installation and easy usage Install OpenCV 3 (core + contrib) in Python 3 environment & Difference between OpenCV 2 and OpenCV 3 & simple operation check ★ The predict () method did not work as intended in the face module of opencv_contrib in OpenCV 3.1. This time, I installed OpenCV 2 on Anaconda 2 and run the following program.

■ Click here for still image processing Try edge detection with OpenCV Perform various filters with OpenCV (gradient, high-pass, Laplacian, Gaussian) Extract feature points with OpenCV (AgastFeature, FAST, GFTT, MSER, AKAZE, BRISK, KAZE, ORB, SimpleBlob) Face recognition using OpenCV (Haar-like feature classifier)

■ Click here for video processing Try converting videos in real time with OpenCV Try converting webcam / camcorder video in real time with OpenCV Draw optical flow in real time with OpenCV (Shi-Tomasi method, Lucas-Kanade method) Object tracking using OpenCV (tracking feature points specified by mouse by Lucas-Kanade method Motion template analysis using OpenCV (recognizing objects and their moving directions in real time)

Face database

Introducing the face database that is often used in image processing.

The Yale face database(Yalefaces)：
http://vision.ucsd.edu/content/yale-face-database
It is a face image in GIF format.
The AT&T：
http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html
It is a face image in PBM format.

This time, we will use the Yale face database (Yale faces).

Training data and test data

Yalefaces contains 15 different facial expressions. Each person has a normal face, a face wearing glasses, a happy face, a winking face, a sad face, a sad face, a surprised face, and a light from the right / left. I have a face. As appropriate, extract any one face image from each person's face image for testing (do not include the test image in the training image). This time, I stored the training images in the yalefaces directory and the test images in the test directory.

Faces with various facial expressions

program

The program flow is as follows.

Load the training image
Extract facial areas using Haar-like feature classifier
Resize the face area to a certain size
Train images and labels as training data with FaceRecognizer (train ())
Repeat steps 1 to 4 for all training images
Load the test image
Extract facial areas using Haar-like feature classifier
Resize the face area to a certain size
Prediction of test image with FaceRecognizer (predict ()) ⇒ [Label, Accuracy]
Repeat 6-9 for all test images

There are some points in the program.

Labels are specified as numbers (ints).
The training image and test image must be resized to the same image size.
Confidence is calculated based on the variance, so the closer it is to 0, the higher the accuracy. If you are learning the same image as the test image during training, the confidence will be 0 because the feature points will match.
OpenCV does not support GIF images. Yalefaces GIF images are processed with Pillow (PIL).
The Haar-like feature classifier template uses haarcascade_frontalface_default.xml that comes with OpenCV.
If you want to switch the face estimator algorithm to EigenFace, FisherFace, LBPH, switch the commented out recognizer.

`face_predict.py`


#!/usr/bin/python
# -*- coding: utf-8 -*-

import cv2, os
import numpy as np
from PIL import Image

#Training image
train_path = './yalefaces'

#Test image
test_path = './test'

# Haar-like feature classifier
cascadePath = "haarcascade_frontalface_default.xml"
faceCascade = cv2.CascadeClassifier(cascadePath)

#Build a face recognizer for OpenCV 2
#* In OpenCV3, FaceRecognizer is cv2.It will be a face module
# EigenFace
#recognizer = cv2.createEigenFaceRecognizer()
# FisherFace
#recognizer = cv2.createFisherFaceRecognizer()
# LBPH
recognizer = cv2.createLBPHFaceRecognizer()

#Get the image in the specified path
def get_images_and_labels(path):
    #Array to store images
    images = []
    #Array to store labels
    labels = []
    #Array to store file names
    files = []
    for f in os.listdir(path):
        #Image path
        image_path = os.path.join(path, f)
        #Import images in grayscale
        image_pil = Image.open(image_path).convert('L')
        #Stored in an array of NumPy
        image = np.array(image_pil, 'uint8')
        # Haar-face detected with like feature classifier
        faces = faceCascade.detectMultiScale(image)
        #Processing of detected face image
        for (x, y, w, h) in faces:
            #Resize face to 200x200 size
            roi = cv2.resize(image[y: y + h, x: x + w], (200, 200), interpolation=cv2.INTER_LINEAR)
            #Store images in an array
            images.append(roi)
            #Get label from filename
            labels.append(int(f[7:9]))
            #Store filenames in an array
            files.append(f)

    return images, labels, files


#Get training image
images, labels, files = get_images_and_labels(train_path)

#Training implementation
recognizer.train(images, np.array(labels))

#Get test image
test_images, test_labels, test_files = get_images_and_labels(test_path)

i = 0
while i < len(test_labels):
    #Prediction implementation for test images
    label, confidence = recognizer.predict(test_images[i])
    #Console output of prediction results
    print("Test Image: {}, Predicted Label: {}, Confidence: {}".format(test_files[i], label, confidence))
    #Display test image
    cv2.imshow("test image", test_images[i])
    cv2.waitKey(300)
    
    i += 1

#End processing
cv2.destroyAllWindows()

This program ran on Python 2.7.12 and OpenCV 2.4.13. When running with Python 3 and OpenCV 3, change the FaceRecognizer part in the program from cv2 to cv2.face. Also, in OpenCV 3.1, the FaceRecognizer predict () method does not work as intended. For details, please refer to Differences between OpenCV2 and OpenCV3 & etc. ⇒ Some methods do not work due to bugs.

Execution result

The training is executed by setting the "XX" part of subjectXX as a numerical value in the label. The correct label (Predicted Label) can be estimated for the test image (Image).

When estimated with Eigenface, the estimation of light-related label = 5, 6, 8, 9 failed, and other estimations were successful.

Test Image: subject01.happy, Predicted Label: 1, Confidence: 4383.25505059
Test Image: subject02.wink, Predicted Label: 2, Confidence: 6947.75053221
Test Image: subject03.happy, Predicted Label: 3, Confidence: 4145.80848328
Test Image: subject04.glasses, Predicted Label: 4, Confidence: 5420.9213318
Test Image: subject05.leftlight, Predicted Label: 12, Confidence: 7722.72936213
Test Image: subject06.leftlight, Predicted Label: 1, Confidence: 10086.4101755
Test Image: subject07.glasses, Predicted Label: 7, Confidence: 7043.70495967
Test Image: subject08.leftlight, Predicted Label: 2, Confidence: 10275.9545456
Test Image: subject09.rightlight, Predicted Label: 15, Confidence: 7481.31094502
Test Image: subject10.sleepy, Predicted Label: 10, Confidence: 2317.22633915
Test Image: subject11.centerlight, Predicted Label: 11, Confidence: 8077.42380817
Test Image: subject12.glasses, Predicted Label: 12, Confidence: 5233.03342586
Test Image: subject13.surprised, Predicted Label: 13, Confidence: 6516.98395617
Test Image: subject14.normal, Predicted Label: 14, Confidence: 0.0
Test Image: subject15.surprised, Predicted Label: 15, Confidence: 7165.71597327

When I estimated it with Fisherface, all the estimations were successful.

Test Image: subject01.happy, Predicted Label: 1, Confidence: 801.784987691
Test Image: subject02.wink, Predicted Label: 2, Confidence: 2368.90429845
Test Image: subject03.happy, Predicted Label: 3, Confidence: 826.018934498
Test Image: subject04.glasses, Predicted Label: 4, Confidence: 1080.94198758
Test Image: subject05.leftlight, Predicted Label: 5, Confidence: 2137.42013849
Test Image: subject06.leftlight, Predicted Label: 6, Confidence: 2092.53092982
Test Image: subject07.glasses, Predicted Label: 7, Confidence: 2042.67529443
Test Image: subject08.leftlight, Predicted Label: 8, Confidence: 2239.45348941
Test Image: subject09.rightlight, Predicted Label: 9, Confidence: 2875.2788263
Test Image: subject10.sleepy, Predicted Label: 10, Confidence: 662.762591569
Test Image: subject11.centerlight, Predicted Label: 11, Confidence: 1703.80515728
Test Image: subject12.glasses, Predicted Label: 12, Confidence: 1480.18770297
Test Image: subject13.surprised, Predicted Label: 13, Confidence: 1690.12255703
Test Image: subject14.normal, Predicted Label: 14, Confidence: 0.0
Test Image: subject15.surprised, Predicted Label: 15, Confidence: 1887.42538269

When estimated by LBPH, the estimation of light-related labels = 6 and 9 failed. Others were successful.

Test Image: subject01.happy, Predicted Label: 1, Confidence: 34.9751422497
Test Image: subject02.wink, Predicted Label: 2, Confidence: 37.8730262399
Test Image: subject03.happy, Predicted Label: 3, Confidence: 35.1183059319
Test Image: subject04.glasses, Predicted Label: 4, Confidence: 37.5886492389
Test Image: subject05.leftlight, Predicted Label: 5, Confidence: 48.2634869014
Test Image: subject06.leftlight, Predicted Label: 14, Confidence: 64.5502245279
Test Image: subject07.glasses, Predicted Label: 7, Confidence: 54.5043891288
Test Image: subject08.leftlight, Predicted Label: 8, Confidence: 84.4281976817
Test Image: subject09.rightlight, Predicted Label: 12, Confidence: 75.3254674542
Test Image: subject10.sleepy, Predicted Label: 10, Confidence: 17.8806440153
Test Image: subject11.centerlight, Predicted Label: 11, Confidence: 74.8238311755
Test Image: subject12.glasses, Predicted Label: 12, Confidence: 31.8721301084
Test Image: subject13.surprised, Predicted Label: 13, Confidence: 40.3420527188
Test Image: subject14.normal, Predicted Label: 14, Confidence: 0.0
Test Image: subject15.surprised, Predicted Label: 15, Confidence: 33.2920487407

The comparison result of the three algorithms shows that the Fisherface is less susceptible to light, as it was said before. Also, since the success and failure of the estimation corresponded to the value of Confidence to some extent, it seemed good to set a threshold for Confidence and classify the unreliable estimation result into an unestimable item.

(Bonus) GIF image → PNG image conversion

OpenCV does not support GIF images to avoid patent issues. Reading a GIF image with cv2.imread () does not read anything. If you want to use Yalefaces, a face image database in GIF format, with OpenCV, it is convenient to convert it to a face image in PNG format in advance with the following script. The script below converts the GIF image in the "yalefaces" directory to a PNG image and saves it in the "png" directory. If you need it, please use the script below.

`gif2png.py`


# -*- coding: utf-8 -*-

import os
from PIL import Image
import shutil

#Directory containing files to be converted
org_dir = 'yalefaces'
#Extension of the file to be converted
org_ext = 'gif'
#Directory to store the converted files
conv_dir = 'png'
#File extension after conversion
conv_ext = 'png'

#Delete existing directories including files
if os.path.exists(conv_dir):
    shutil.rmtree(conv_dir)
#Create directory
os.mkdir(conv_dir)

# 「.Character string length including the extension
org_ext_len = len(org_ext) + 1

for dirname, dirnames, filenames in os.walk(org_dir):
    for filename in filenames:
        #Path of the file to be converted
        org_path = org_dir + '/' + filename

        #File path after return
        if len(filename) > org_ext_len and \
            filename[-org_ext_len:] == '.' + org_ext:
            filename = filename[0:-org_ext_len]
        conv_path = conv_dir + '/' + filename + '.' + conv_ext

        try:
            #Convert execution
            Image.open(org_path).save(conv_path)
        except IOError:
            print('cannot convert :', org_path)

If you do not have the Python image processing library and PIL module installed, please install pillow. Example)

$ conda install pillow

Estimate who's face using OpenCV (Eigenface, Fisherface, LBPH)