How to process camera images with Teams or Zoom

`This article is also introduced here. `` https://cloud.flect.co.jp/entry/2020/03/31/162537

Hello everyone.

At present, at the request of the Governor of Tokyo, our company works from home in principle to prevent the spread of new coronavirus infection. There are many people who are taking similar measures, and I think there are various difficulties, but I would like to work together to overcome this difficult song.

By the way, in principle, if you work from home for a long time, you may get stressed, such as not being able to have casual conversations that were held on a daily basis. In such a situation, I would like to create a situation where you can laugh and take a break, so I would like to introduce one small story.

The content is a method of hooking a webcam, processing it, and delivering it in a video conference such as Microsoft Teams or Zoom. Since I am a Linux user, this time I will introduce it on Linux. I think other platforms will be introduced somewhere.

In addition, I choose the time and the case to create a "situation where I can laugh and take a break", so please do so at your own risk (^ _ ^) /.

Premise

I think it works fine on most Linux systems, but the environment I've worked in is Debian Buster.

$ cat /etc/debian_version
10.3

Also, if python3 is not included, please install it.

$ python3 --version
Python 3.7.3

Installation of related software

Virtual webcam device

This time we will use a virtual webcam device called v4l2loopback. https://github.com/umlaeute/v4l2loopback

We need to identify the virtual webcam device from the real webcam, so first check the actual webcam device file. In the example below, it seems that video0 and video1 are assigned to the actual webcam.

$ ls /dev/video*
/dev/video0  /dev/video1

Now let's introduce v4l2 loopback. First, git clone, make and install.

$ git clone https://github.com/umlaeute/v4l2loopback.git
$ cd v4l2loopback
$ make
$ sudo make install

Then load the module. At this time, it seems that it is necessary to add exclusive_caps = 1 in order to recognize it especially with chrome. [https://github.com/umlaeute/v4l2loopback/issues/78]

sudo modprobe v4l2loopback exclusive_caps=1

Now that the module is loaded, let's check the device file. In the example below, video2 has been added.

$ ls /dev/video*
/dev/video0  /dev/video1  /dev/video2

ffmpeg The easiest way to send data to a virtual webcam device is to use ffmpeg. Please install it quickly with apt-get etc.

Webcam hook and video distribution

This time, if a smile is detected, I will try to process the image. When a smile is detected, a smile mark is displayed on the video.

First, clone the files in the following repository to install the module.

$ git clone https://github.com/dannadori/WebCamHooker.git
$ cd WebCamHooker/
$ pip3 install -r requirements.txt

Get the cascade file from here. Please check the opencv official for details on the cascade file. https://github.com/opencv/opencv/tree/master/data/haarcascades

$ wget https://raw.githubusercontent.com/opencv/opencv/master/data/haarcascades/haarcascade_frontalface_default.xml -P models/
$ wget https://raw.githubusercontent.com/opencv/opencv/master/data/haarcascades/haarcascade_smile.xml -P models/

Let's borrow the smile mark from Mr. Toya.

$ wget https://4.bp.blogspot.com/-QeM2lPMumuo/UNQrby-TEPI/AAAAAAAAI7E/cZIpq3TTyas/s160/mark_face_laugh.png -P images/

It would be nice to have a folder structure like this.

$ ls -1
haarcascade_frontalface_default.xml
haarcascade_smile.xml
mark_face_laugh.png
webcamhooker.py

The execution is as follows. Enter the actual webcam device number in --input_video_num. For / dev / video0, enter the trailing 0. For --output_video_dev, specify the device file of the virtual webcam device. In addition, please use ctrl + c to end it.

$ python3 webcamhooker.py --input_video_num 1 --output_video_dev /dev/video2

When you execute the above command, ffmpeg will run and the video will start to be delivered to the virtual camera device.

Let's have a video chat!

When you have a video chat, you will see dummy ~ ~ in the list of video devices, so select it. This is an example of Teams. Think of the left and right as the screens of each participant. The user on the left is using this virtual camera device. The right side is the receiving side. When you smile, you will see a smile mark. Great success (^ _ ^) /.

Finally

Now that face-to-face communication is difficult, I hope you can enjoy more using video chat. This time, I showed an example of detecting a smile and processing the image, but I think that various processing can be done by using opencv and other tools depending on the device. Please try various things!

reference

I referred to here for smile detection with opencv. https://qiita.com/fujino-fpu/items/99ce52950f4554fbc17d

Please refer to here for pasting images with opencv. https://qiita.com/a2kiti/items/9672fae8e90c2da6f352