Detect mask wearing status with OpenCV and Raspberry Pi

Overview

• Previously, I used OpenCV to visualize the viewing situation of art works. https://qiita.com/cami_oshimo/items/ce33491c07f9625ed121

• I thought that if I applied this and changed to mouth detection instead of eyes, I could use the mask to judge wearing.

• First, simply determine if your mouth is exposed and then determine if you are wearing a mask.

What to prepare

• Raspberry Pi3 Model B (4 is probably okay)
• Case for Raspberry Pi
• Heat sink for Raspberry Pi
• Raspberry Pi3 Model B B + compatible power supply set (5V 3.0A)
• Camera for raspberry Pi
• Monitor with HDMI input
• 100-yen shop tripod
• Remote operation of Raspberry Pi from a PC using VNC makes development easier.

Preparation

• Set up Raspberry Pi.
• Keep the Raspberry Pi camera ready for use. https://www.raspberrypi.org/downloads/
• Make the OpenCV library and cascade file available. haarcascade_frontalface_default.xml (face detection) haarcascade_mcs_mouth.xml (mouth detection)

programming

• Overlay a message prompting you to frame in the face, a face / mouth detection frame, and a warning when the mask is not worn on the camera image.
• If a face is detected and a mouth is detected, it is judged that the mask is not worn.

import sys

sys.path.append('/home/pi/.local/lib/python2.7/site-packages')

import numpy as np

import cv2

import time

import datetime

import ambient

args = sys.argv

face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

mouth_cascade = cv2.CascadeClassifier('haarcascade_mcs_mouth.xml')

cap = cv2.VideoCapture(0)

while(True):


    ret, frame = cap.read()

    height = frame.shape[0]

    width = frame.shape[1]

    import sys

    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    faces = face_cascade.detectMultiScale(gray, 1.3, 5)

    cv2.putText(frame, 'Please face in frame.', (350, 20), cv2.FONT_HERSHEY_PLAIN, 1.5, (0,255,0), thickness=2)

    for (x,y,w,h) in faces:

        roi_gray = gray[y:y+h, x:x+w]

        roi_color = frame[y:y+h, x:x+w]

        cv2.rectangle(frame,(x,y),(x+w,y+h),(0,255,0),2)

        mouth = mouse_cascade.detectMultiScale(roi_gray)

        for (ex,ey,ew,eh) in mouth:

            cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,255),2)

            cv2.putText(frame, 'Not masked!!', (10, 460), cv2.FONT_HERSHEY_DUPLEX, 1.0, (0,0,255), thickness=2)

    resized_img = cv2.resize(frame,(width*2, height*2))

    cv2.imshow('maskcheck', resized_img)

    if cv2.waitKey(1) & 0xFF == ord('q'):

        break

cap.release()

cv2.destroyAllWindows()

• Place a shell script on the desktop of Raspberry Pi so that the Python program can be started.
• Make sure to start by double-clicking the icon.

maskcheck.sh

#! /bin/bash
cd /home/pi/opencv_programs/
sudo python3 detect_face_camera.py

Task

• It may be difficult to detect due to external factors. (Distance to face, angle, amount of light, etc.) The curtain is closed because noise was introduced due to backlight and it became undetectable.
• Depending on the type of mask and how it is worn, it may not be detected as a face.
• A place other than the mouth may be recognized as a mouth, probably because of the cascading file of mouth detection. It seems that the cascade file can be modified, so it may be possible to solve it by tuning over time.
• Face and recognition → Since the logic is to recognize the presence or absence of the mouth, it is necessary to make the face recognized in the first place. I tried mouth recognition only logic, but it didn't work. (OpenCV problem?)
• I'm curious about how much performance will be improved with Raspberry Pi 4, which has a noticeable drawing delay.