How to process camera images with Teams and Zoom Sentiment analysis with Tensorflow

`This article is also introduced here. `` https://cloud.flect.co.jp/entry/2020/04/01/201158

Hello everyone.

Last time, I introduced how to process camera images with Teams and Zoom. We have introduced a demo that detects a smile and displays a smiley face. https://qiita.com/wok/items/0a7c82c6f97f756bde65

This time, I would like to introduce a slightly expanded version of this, as I tried to analyze emotions in real time with AI (Tensorflow). Specifically, as shown below, emotions such as sadness and anger are read from the facial expressions of the person in the image, and the image that matches them is displayed on the screen. It seems that you can now convey your emotions without having to say words in a video conference. (No, wait ...)

Premise

Please refer to Previous article and set v4l2loopback etc.

Webcam hook extension

This time, we will analyze the emotions of the person taken by the camera and display the corresponding image in the video on the video stream. We will use Tensorflow for sentiment analysis, but let's use the trained model provided under the MIT license at the following site.

https://github.com/oarriaga/face_classification

First, clone the script from the repository below and install the required modules as before.

$ git clone https://github.com/dannadori/WebCamHooker.git
$ cd WebCamHooker/
$ pip3 install -r requirements.txt

Next, get a trained model for sentiment analysis from the previous site. In addition, this time, in order to display an appropriate image, I will also judge the gender at the same time.

$ wget https://github.com/oarriaga/face_classification/raw/master/trained_models/emotion_models/fer2013_mini_XCEPTION.110-0.65.hdf5 -P models #Model for sentiment analysis
$ wget https://github.com/oarriaga/face_classification/raw/master/trained_models/gender_models/simple_CNN.81-0.96.hdf5 -P models/ #Model for gender determination

Also, let's borrow the image from Mr. Toya again.

$ wget https://4.bp.blogspot.com/-8DirG_alwXo/V5Xc1SMykvI/AAAAAAAA8u4/krI2n_SWimUBGEyMWCw5kZZ-HzoUKrY8ACLcB/s800/pose_sugoi_okoru_woman.png -P images/
$ wget https://4.bp.blogspot.com/-EBpxVigkCCY/V5Xc1CHSeEI/AAAAAAAA8u0/9XIAzDJaQNU3HIiXi4PCPK3aMip3aoGyACLcB/s800/pose_sugoi_okoru_man.png -P images/

$ wget https://4.bp.blogspot.com/-HJ0FUQz67AA/XAnvUxSRsLI/AAAAAAABQnM/3XzIWzvW6L80aGB-geaHvAQETlJTAwkYQCLcBGAs/s800/business_woman2_4_think.png -P images/
$ wget https://3.bp.blogspot.com/-S7iQQCOgfWY/XAnvQWwBGtI/AAAAAAABQmc/z7yIqGjIQr88Brc_QNdOGsrJRLvqY1hcQCLcBGAs/s800/business_man2_4_think.png -P images/

$ wget https://4.bp.blogspot.com/-PQQV4wfGlNI/XAnvQBMeneI/AAAAAAABQmU/lN7zIROor9oi3q-JZOBJiKKzfklzPE1hwCLcBGAs/s800/business_man2_2_shock.png] -P images/
$ wget https://3.bp.blogspot.com/-QcDbWqQ448I/XAnvUT4TMDI/AAAAAAABQnE/_H4XzC4E93AEU2Y7fHMDBjri1drdyuAPQCLcBGAs/s800/business_woman2_2_shock.png -P images/

$ wget https://3.bp.blogspot.com/-dSPRqYvIhNk/XAnvPdvjBFI/AAAAAAABQmM/izfRBSt1U5o7eYAjdGR8NtoP4Wa1_Zn8ACLcBGAs/s800/business_man1_4_laugh.png -P images/
$ wget https://1.bp.blogspot.com/-T6AOerbFQiE/XAnvTlQvobI/AAAAAAABQm8/TYVdIfxQ5tItWgUMl5Y0w8Og_AZAJgAewCLcBGAs/s800/business_woman1_4_laugh.png -P images/

$ wget https://4.bp.blogspot.com/-Kk_Mt1gDKXI/XAnvS6AjqyI/AAAAAAABQm4/LQteQO7TFTQ-KPahPcAqXYannEArMmYfgCLcBGAs/s800/business_woman1_3_cry.png -P images/
$ wget https://4.bp.blogspot.com/-3IPT6QIOtpk/XAnvPCPuThI/AAAAAAABQmI/pIea028SBzwhwqysO49pk4NAvoqms3zxgCLcBGAs/s800/business_man1_3_cry.png -P images/

$ wget https://3.bp.blogspot.com/-FrgNPMUG0TQ/XAnvUmb85VI/AAAAAAABQnI/Y06kkP278eADiqvXH5VC0uuNxq2nnr34ACLcBGAs/s800/business_woman2_3_surprise.png -P images/
$ wget https://2.bp.blogspot.com/-i7OL88NmOW8/XAnvQacGWuI/AAAAAAABQmY/LTzN4pcnSmYLke3OSPME4cUFRrLIrPsYACLcBGAs/s800/business_man2_3_surprise.png -P images/

$ cp images/lN7zIROor9oi3q-JZOBJiKKzfklzPE1hwCLcBGAs/s800/business_man2_2_shock.png]  images/lN7zIROor9oi3q-JZOBJiKKzfklzPE1hwCLcBGAs/s800/business_man2_2_shock.png

Of the above, the last command just removes the garbage (the key brackets at the end) in the file name.

The execution is as follows. One option has been added.

--Enter the actual webcam device number in input_video_num. For / dev / video0, enter the trailing 0. --Specify the device file of the virtual webcam device for output_video_dev. --Set emotion_mode to True.

In addition, please use ctrl + c to end it.

$ python3 webcamhooker.py --input_video_num 0 --output_video_dev /dev/video2 --emotion_mode True

When you execute the above command, ffmpeg will run and the video will start to be delivered to the virtual camera device.

Let's have a video conference!

As before, when you have a video conference, you will see something like dummy ~ ~ in the list of video devices, so select it. This is an example of Teams. The character string at the top of the screen changes according to the facial expression, and the corresponding image is displayed accordingly. It's a great success.

Finally

It may be difficult to communicate casually because working from home is prolonged, but I think it would be good to bring this kind of playfulness to video conferencing and activate conversations. I think we can do more, so please give it a try.

reference

For sentiment analysis by Tensorflow, I referred to the following site. (This site is introduced in tensorflowjs)

https://book.mynavi.jp/manatee/detail/id=99887