[JAVA] I tried to make a simple face recognition Android application using OpenCV

Introduction

Face recognition technology itself has existed for a long time, but recently, smartphone apps that use face recognition such as Snow have appeared, and it is becoming more familiar. How did it happen before? I'm also interested in that, and if I take a face photo myself, I'd like to make an app that replaces the face of the person in the photo with another one! I thought. However, when I actually looked up face recognition and image analysis on the net, many of them focused on machine learning, and I was overwhelmed by the difficulty ... However, when I made a face recognition function part using OpenCV with an Android application this time, it was easier than I imagined, so I would like to introduce it. What I found by actually making it was that it was more difficult to implement the camera function than to recognize the face from the image using OpenCV.

What is OpenCV in the first place?

OpenCV (Open Source Computer Vision Library) is an open source library for processing videos and images. For OpenCV, this article was helpful.

What I wanted to do

Use OpenCV to determine if a face is present in a photo taken with the Android camera function

  1. Perform machine learning to acquire features
  2. Implement the camera function in the Android app
  3. Implemented face recognition function for image data taken with a camera ← Part to be introduced this time
  4. Image processing and judgment using recognition results (face size, position information)

What is used to realize the face recognition function with OpenCV

OpenCV provides interfaces for C, C ++, Python and Java. When embedding in a smartphone app, there is an SDK prepared for iOS and Android, and by importing this SDK into the project, you can easily prepare an environment to use OpenCV. Also, in face recognition, the result of machine learning is required to judge "face", but there are some that OpenCV has already prepared. The result of this machine learning * is enclosed in the SDK package.

Usage environment / library

Process flow

Load OpenCV library

--Write the following code in the class that uses OpenCV and load the OpenCV library imported into Android Studio. --Refer to the following site for how to import to Android Studio - https://qiita.com/kodai100/items/6c9e8a34d0714913c017

FaceClassifier.java



  //Load OpenCV library
  static {
    System.loadLibrary("opencv_java3"); 
  }

Convert image data to Mat format

--Convert image data to OpenCV Mat format. --In the sample, we converted from Bitmap format to Mat format.

FaceClassifier.java



    //Convert image data (Bitmap Mat file conversion)
    Mat matImg = new Mat();
    Utils.bitmapToMat(image,matImg); 

Creating a CascadeClassifier instance for face recognition

--Use "haarcascade_frontalface_alt.xml" which is a face cascade. Since the feature file is also downloaded when the SDK is downloaded, store the target file under res / raw of the Android project. ――I got stuck here for a moment ... --The CascadeClassifier constructor cannot take an InputStream as an argument, so you must take the path of the xml file as an argument. Therefore, it is necessary to follow the procedure "Read the xml file once, output it with FileOutputStream, and get the path of the output file."

FaceClassifier.java



    //Generate a cascade classifier instance that performs face recognition (write the file once and get the path of the file)
    //Get the xml file once stored under raw
    InputStream inStream = this.activity.getResources().openRawResource(R.raw.haarcascade_frontalface_alt);
    File cascadeDir = this.activity.getDir("cascade", Context.MODE_PRIVATE);
    File cascadeFile = new File(cascadeDir, "haarcascade_frontalface_alt.xml");
    //Output the acquired xml file to a specific directory
    FileOutputStream outStream = new FileOutputStream(cascadeFile);
    byte[] buf = new byte[2048];
    int rdBytes;
    while ((rdBytes = inStream.read(buf)) != -1) {
      outStream.write(buf, 0, rdBytes);
    }
    outStream.close();
    inStream.close();
    //Use the path of the output xml file as an argument of CascadeClassifier
    CascadeClassifier faceDetetcor = new CascadeClassifier(cascadeFile.getAbsolutePath());
    //When the CascadeClassifier instance is created, the output file is unnecessary, so delete it.
    if (faceDetetcor.empty()) {
      faceDetetcor = null;
    } else {
      cascadeDir.delete();
      cascadeFile.delete();
    }

Face recognition by giving image data to Cascade Classifier

--Mat type image data and MatOfRect instance are passed to CascadeClassifier.detectMultiScale method. The authentication result is stored in MatOfRect.

FaceClassifier.java



    //Face recognition by giving image data to the cascade classifier
    MatOfRect faceRects = new MatOfRect();
    faceDetetcor.detectMultiScale(matImg, faceRects);

Get the number of faces recognized in the image and the position of the face from the result stored in the MatOfRect instance.

--MatOfRect instance can be changed to Rect type. --The sample source is only log output, but if you want to replace the face in the image with another one, you can do it by overlaying the images according to the coordinates of the face and the size of the face.

FaceClassifier.java



    //Confirmation of face recognition results
    Log.i(TAG ,"Number of recognized faces:" + faceRects.toArray().length); 
    if (faceRects.toArray().length > 0) {
      for (Rect face : faceRects.toArray()) {
        Log.i(TAG ,"Vertical width of the face" + face.height);
        Log.i(TAG ,"Width of face" + face.width);
        Log.i(TAG ,"Face position (Y coordinate)" + face.y);
        Log.i(TAG ,"Face position (X coordinate)" + face.x);
      }
      return true;
    } else {
      Log.i(TAG ,"No face was detected");
      return false;
    }

Source code

FaceClassifier.java



import org.opencv.android.Utils;
import org.opencv.core.Mat;
import org.opencv.core.MatOfRect;
import org.opencv.core.Rect;
import org.opencv.objdetect.CascadeClassifier;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;

public class FaceClassifier {
  //Load OpenCV library
  static {
    System.loadLibrary("opencv_java3"); 
  }
  private Activity activity;
  public FaceClassifier (Activity activity) {
    this.activity = activity;
  }
  public boolean checkFaceExistence (Bitmap image) throws IOException {
    //Convert image data (Bitmap Mat file conversion)
    Mat matImg = new Mat();
    Utils.bitmapToMat(image,matImg); 
    //Generate a cascade classifier instance that performs face recognition (write the file once and get the path of the file)
    //Get the xml file once stored under raw
    InputStream inStream = this.activity.getResources().openRawResource(R.raw.haarcascade_frontalface_alt);
    File cascadeDir = this.activity.getDir("cascade", Context.MODE_PRIVATE);
    File cascadeFile = new File(cascadeDir, "haarcascade_frontalface_alt.xml");
    //Output the acquired xml file to a specific directory
    FileOutputStream outStream = new FileOutputStream(cascadeFile);
    byte[] buf = new byte[2048];
    int rdBytes;
    while ((rdBytes = inStream.read(buf)) != -1) {
      outStream.write(buf, 0, rdBytes);
    }
    outStream.close();
    inStream.close();
    //Use the path of the output xml file as an argument of CascadeClassifier
    CascadeClassifier faceDetetcor = new CascadeClassifier(cascadeFile.getAbsolutePath());
    //When the CascadeClassifier instance is created, the output file is unnecessary, so delete it.
    if (faceDetetcor.empty()) {
      faceDetetcor = null;
    } else {
      cascadeDir.delete();
      cascadeFile.delete();
    }
    //Face recognition by giving image data to the cascade classifier
    MatOfRect faceRects = new MatOfRect();
    faceDetetcor.detectMultiScale(matImg, faceRects);
    //Confirmation of face recognition results
    Log.i(TAG ,"Number of recognized faces:" + faceRects.toArray().length); 
    if (faceRects.toArray().length > 0) {
      for (Rect face : faceRects.toArray()) {
        Log.i(TAG ,"Vertical width of the face" + face.height);
        Log.i(TAG ,"Width of face" + face.width);
        Log.i(TAG ,"Face position (Y coordinate)" + face.y);
        Log.i(TAG ,"Face position (X coordinate)" + face.x);
      }
      return true;
    } else {
      Log.i(TAG ,"No face was detected");
      return false;
    }
  }
}

Summary

The OpenCV SDK is much easier to use, so it's easier than you think it would be. If you want to recognize something other than the face, you need to check machine learning using OpenCV, but since there are various published results (XML files) of machine learning for OpenCV, once It is recommended to check. If you want to make an app that uses face recognition technology like me, please use OpenCV.

Recommended Posts

I tried to make a simple face recognition Android application using OpenCV
I tried to build a simple application using Dockder + Rails Scaffold
Java beginner tried to make a simple web application using Spring Boot
I tried to make a talk application in Java using AI "A3RT"
I tried using Hotwire to make Rails 6.1 scaffold a SPA
I tried to create a simple map app in Android Studio
[Unity] I tried to make a native plug-in UniNWPathMonitor using NWPathMonitor
I tried to make an Android application with MVC now (Java)
java I tried to break a simple block
[iOS] I tried to make a processing application like Instagram with Swift
I tried to implement a server using Netty
I tried to make a machine learning application with Dash (+ Docker) part3 ~ Practice ~
I tried to make a simple game with Javafx ① "Let's find happiness game" (unfinished)
[Android] I tried to make a material list screen with ListView + Bottom Sheet
I tried using a database connection in Android development
I tried to decorate the simple calendar a little
I tried to make a login function in Java
[Introduction to Android application development] Let's make a counter
I tried to make a simple game with Javafx ① "Let's find happiness game" (unfinished version ②)
I tried to make an application in 3 months from inexperienced
I tried adding a separator line to TabLayout on Android
I tried to modernize a Java EE application with OpenShift.
I tried to implement a buggy web application in Kotlin
I tried to make a client of RESAS-API in Java
Try to introduce OpenCV to Android application
Try to make a simple callback
I tried to make the sample application into a microservice according to the idea of the book "Microservice Architecture".
[Android] I tried using Coordinator Layout.
I tried to make it an arbitrary URL using routing nesting
[Java] I tried to make a maze by the digging method ♪
I tried to make a group function (bulletin board) with Rails
[Rails] Implementation of multi-layer category function using ancestry "I tried to make a window with Bootstrap 3"
I tried to make a machine learning application with Dash (+ Docker) part2 ~ Basic way of writing Dash ~
After learning Progate, I tried to make an SNS application using Rails in the local environment
I want to develop a web application!
I tried using OpenCV with Java + Tomcat
Face recognition app with OpenCV + Android Studio
I tried to make a parent class of a value object in Ruby
I tried to summarize the stumbling points when developing an Android application
I tried to make a Web API that connects to DB with Quarkus
I tried to make my own transfer guide using OpenTripPlanner and GTFS
I tried to build a Firebase application development environment with Docker in 2020
I made a virtual currency arbitrage bot and tried to make money
I tried to make a machine learning application with Dash (+ Docker) part1 ~ Environment construction and operation check ~
I tried to make a sample program using the problem of database specialist in Domain Driven Design
Using the face detection function of Watson Visual Recognition, I tried to process a whole body image of a person into an image of only the face part
[LINE @] I tried to make a Japanese calendar Western calendar conversion BOT [Messaging API]
I tried to make Basic authentication with Java
I tried to implement a function equivalent to Felica Lite with HCE-F of Android
A story when I tried to make a video by linking Processing and Resolume
I tried to develop a man-hour management tool
I did Java to make (a == 1 && a == 2 && a == 3) always true
I tried to develop a DUO3.0 study website.
[Android] I quit SQLite and tried using Realm
[Java] I tried to connect using a connection pool with Servlet (tomcat) & MySQL & Java
I wanted to make (a == 1 && a == 2 && a == 3) true in Java
I tried to create a LINE clone app
Try to make a music player using Basic Player
I tried to clone a web application full of bugs with Spring Boot
[Azure] I tried to create a Java application for free-Web App creation- [Beginner]
I tried to develop an application in 2 languages