Replace your face with Twitter icon with openCV face recognition and do ZOOM

This is kimo_0tak, which is the first post in a long time. You can easily make babiniku with facerig etc., but I couldn't find a way to replace only my face with another image, so I made it quickly. There is no guarantee that it will replace the face 100%, so please use it as an inner ring material.

The displayed image looks like this. ANf8EAbN.png

The usage environment is Windows 10, Python 3.6.5. Put the image you want to replace with this program and the xml file of the face recognition model in the same file.

Here is the site that I used as a reference.

Face detection with Haar Cascades http://labs.eecs.tottori-u.ac.jp/sd/Member/oyamada/OpenCV/html/py_tutorials/py_objdetect/py_face_detection/py_face_detection.html Draw another image on top of the image with OpenCV https://note.com/npaka/n/nddb33be1b782 A story about capturing webcam images with Python + OpenCV, processing them, and displaying them. https://ensekitt.hatenablog.com/entry/2017/12/19/200000

Let's make it. The completed code is at the bottom, so if you want to see that much, scroll down.


Import required libraries
#OpenCV import
import cv2
import numpy as np
from PIL import Image

If you have any libraries that you have not installed, please install them yourself. Open a command prompt

pip install opencv-python
pip install numpy
pip install Pillow

You should be able to enter it by typing.

Various settings
#Camera resolution setting
WIDTH = 1920
HEIGHT = 1080

#Specifying the image to read
img = "vtuber_may.jpg "
cv2_img = cv2.imread(img, cv2.IMREAD_UNCHANGED)

#Designation of face recognition model
face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_alt2.xml')

#You can select the camera with the argument.
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, WIDTH)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, HEIGHT)

You can create the face recognition cascade classifier yourself, but it was troublesome, so I used the classifier provided by Opencv. It is included when you install opencv with pip install opencv. In my environment it was in the C: \ Users \ {username} \ AppData \ Local \ Programs \ Python \ Python36-32 \ Lib \ site-packages \ cv2 \ data folder.


Function that synthesizes images
def overlayImage(src, overlay, location, size):
    overlay_height, overlay_width = overlay.shape[:2]

    #Convert webcam images to PIL format
    src = cv2.cvtColor(src, cv2.COLOR_BGR2RGB)
    pil_src = Image.fromarray(src)
    pil_src = pil_src.convert('RGBA')

    #Convert the image you want to combine to PIL format
    overlay = cv2.cvtColor(overlay, cv2.COLOR_BGRA2RGBA)
    pil_overlay = Image.fromarray(overlay)
    pil_overlay = pil_overlay.convert('RGBA')
    #Resize to fit the size of your face
    pil_overlay = pil_overlay.resize(size)

    #Combine images
    pil_tmp = Image.new('RGBA', pil_src.size, (255, 255, 255, 0))
    pil_tmp.paste(pil_overlay, location, pil_overlay)
    result_image = Image.alpha_composite(pil_src, pil_tmp)

    #Convert to OpenCV format
    return cv2.cvtColor(np.asarray(result_image), cv2.COLOR_RGBA2BGRA)

Since images cannot be combined with Opencv, create a function to combine images after converting to PIL format. I also resize it according to the height and width of my face.

Main function
def main(): 
    #A variable that stores the coordinates of the upper left corner of face recognition
    x = 0
    y = 0

    #A variable that stores the width and height of the face
    w = 0
    h = 0
    while True:
        #Load 1 frame from VideoCapture
        ret, frame = cap.read()

        #Convert frame to gray format for face recognition
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        face = face_cascade.detectMultiScale(gray, 1.3, 5)

        #If the face is recognized x, y, w,update h
        if face != ():
            (x, y, w, h) = face[0]

        #If the variable is other than the initial value, the image is combined with the frame
        if w != 0:
            frame = overlayImage(frame, cv2_img, (x, y), (w, h))


        #Display the processed frame
        cv2.imshow('Frame', frame)

        #Wait 1ms for key input and break when ESC is pressed
        k = cv2.waitKey(1)
        if k == 27:
            break

    #Release the capture and close the window
    cap.release()
    cv2.destroyAllWindows()

One frame is read from the webcam and the image is processed. Depending on the frame, the face may not be recognized, so if it is not recognized, it is combined with the previously recognized position. Note that if two or more faces are recognized on the screen, the image will be combined with only one of the faces. I thought I'd improve it, but I didn't improve it because it's unlikely that two people would appear in ZOOM at the same time.

Processing to execute the main function
if __name__ == '__main__':
    main()



In summary, it looks like this
#OpenCV import
import cv2
import numpy as np
from PIL import Image

#Camera resolution setting
WIDTH = 1920
HEIGHT = 1080

#Specifying the image to read
img = "vtuber_may.jpg "
cv2_img = cv2.imread(img, cv2.IMREAD_UNCHANGED)

#Designation of face recognition model
face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_alt2.xml')

#You can select the camera with the argument.
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, WIDTH)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, HEIGHT)

def overlayImage(src, overlay, location, size):
    overlay_height, overlay_width = overlay.shape[:2]

    #Convert webcam images to PIL format
    src = cv2.cvtColor(src, cv2.COLOR_BGR2RGB)
    pil_src = Image.fromarray(src)
    pil_src = pil_src.convert('RGBA')

    #Convert the image you want to combine to PIL format
    overlay = cv2.cvtColor(overlay, cv2.COLOR_BGRA2RGBA)
    pil_overlay = Image.fromarray(overlay)
    pil_overlay = pil_overlay.convert('RGBA')
    #Resize to fit the size of your face
    pil_overlay = pil_overlay.resize(size)

    #Combine images
    pil_tmp = Image.new('RGBA', pil_src.size, (255, 255, 255, 0))
    pil_tmp.paste(pil_overlay, location, pil_overlay)
    result_image = Image.alpha_composite(pil_src, pil_tmp)

    #Convert to OpenCV format
    return cv2.cvtColor(np.asarray(result_image), cv2.COLOR_RGBA2BGRA)


def main():
    #A variable that stores the coordinates of the upper left corner of face recognition
    x = 0
    y = 0

    #A variable that stores the width and height of the face
    w = 0
    h = 0
    while True:
        #Load 1 frame from VideoCapture
        ret, frame = cap.read()

        #Convert frame to gray format for face recognition
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        face = face_cascade.detectMultiScale(gray, 1.3, 5)

        #If the face is recognized x, y, w,update h
        if face != ():
            (x, y, w, h) = face[0]

        #If the variable is other than the initial value, the image is combined with the frame
        if w != 0:
            frame = overlayImage(frame, cv2_img, (x, y), (w, h))


        #Display the processed frame
        cv2.imshow('Frame', frame)

        #Wait 1ms for key input and break when ESC is pressed
        k = cv2.waitKey(1)
        if k == 27:
            break

    #Release the capture and close the window
    cap.release()
    cv2.destroyAllWindows()

if __name__ == '__main__':
    main()

All you have to do is load the webcam image displayed by this program in OBS Studio and output it as a virtual camera with virtualcam. I will not explain it here, but please see this site etc. and set it.

How to use VirtualCam to recognize OBS as a virtual camera https://loumo.jp/archives/24912

43LjUnYe.png It looks like this when used with Zoom. You can hide your nerd face with a Twitter icon and have a meeting.

Saigo ni

I posted the code little by little with the intention of explaining the code more, but most of the explanation was written in the comments, and other than the comments, I wrote only the location of the opencv cascade classifier. Also recently, I'm addicted to Hololive's vtuber Shisaki Sion-chan , so please take a look.

Recommended Posts

Replace your face with Twitter icon with openCV face recognition and do ZOOM
Face recognition with Python's OpenCV
Face recognition / cutting with OpenCV
Face recognition with camera with opencv3 + python2.7
I tried face recognition with OpenCV
[python, openCV] base64 Face recognition with images
Hello World and face detection with OpenCV 4.3 + Python
Face recognition with Edison
[Super easy] Simultaneous face recognition and facial expression recognition in real time with Python and OpenCV!
Easy face recognition try with Jetson Nano and webcam
Face recognition with Amazon Rekognition
Face detection with Python + OpenCV
Try face recognition with Python
Image recognition with Keras + OpenCV
Anime face detection with OpenCV
Shining life with Python and OpenCV
Neural network with OpenCV 3 and Python 3
Try face recognition with Generated Photos
[OpenCV] Personal identification with face photo
First Anime Face Recognition with Chainer
Cut out face with Python + OpenCV
Object recognition with openCV by traincascade
Draw shapes with OpenCV and PIL
Face detection from multiple image files with openCV, cut out and save
Create your own virtual camera with Python + OpenCV and apply original effects
Let's make an image recognition model with your own data and play!
You can do it in 5 minutes !? Create a face detection API with FastAPI and OpenCV and publish it on Heroku