How to process camera images with Teams and Zoom Sentiment analysis with Tensorflow

`This article is also introduced here. `` https://cloud.flect.co.jp/entry/2020/04/01/201158

Hello everyone.

Last time, I introduced how to process camera images with Teams and Zoom. We have introduced a demo that detects a smile and displays a smiley face. https://qiita.com/wok/items/0a7c82c6f97f756bde65

This time, I would like to introduce a slightly expanded version of this, as I tried to analyze emotions in real time with AI (Tensorflow). Specifically, as shown below, emotions such as sadness and anger are read from the facial expressions of the person in the image, and the image that matches them is displayed on the screen. It seems that you can now convey your emotions without having to say words in a video conference. (No, wait ...)

out2.gif

Premise

Please refer to Previous article and set v4l2loopback etc.

Webcam hook extension

This time, we will analyze the emotions of the person taken by the camera and display the corresponding image in the video on the video stream. We will use Tensorflow for sentiment analysis, but let's use the trained model provided under the MIT license at the following site.

https://github.com/oarriaga/face_classification

First, clone the script from the repository below and install the required modules as before.

$ git clone https://github.com/dannadori/WebCamHooker.git
$ cd WebCamHooker/
$ pip3 install -r requirements.txt

Next, get a trained model for sentiment analysis from the previous site. In addition, this time, in order to display an appropriate image, I will also judge the gender at the same time.

$ wget https://github.com/oarriaga/face_classification/raw/master/trained_models/emotion_models/fer2013_mini_XCEPTION.110-0.65.hdf5 -P models #Model for sentiment analysis
$ wget https://github.com/oarriaga/face_classification/raw/master/trained_models/gender_models/simple_CNN.81-0.96.hdf5 -P models/ #Model for gender determination

Also, let's borrow the image from Mr. Toya again.

$ wget https://4.bp.blogspot.com/-8DirG_alwXo/V5Xc1SMykvI/AAAAAAAA8u4/krI2n_SWimUBGEyMWCw5kZZ-HzoUKrY8ACLcB/s800/pose_sugoi_okoru_woman.png -P images/
$ wget https://4.bp.blogspot.com/-EBpxVigkCCY/V5Xc1CHSeEI/AAAAAAAA8u0/9XIAzDJaQNU3HIiXi4PCPK3aMip3aoGyACLcB/s800/pose_sugoi_okoru_man.png -P images/

$ wget https://4.bp.blogspot.com/-HJ0FUQz67AA/XAnvUxSRsLI/AAAAAAABQnM/3XzIWzvW6L80aGB-geaHvAQETlJTAwkYQCLcBGAs/s800/business_woman2_4_think.png -P images/
$ wget https://3.bp.blogspot.com/-S7iQQCOgfWY/XAnvQWwBGtI/AAAAAAABQmc/z7yIqGjIQr88Brc_QNdOGsrJRLvqY1hcQCLcBGAs/s800/business_man2_4_think.png -P images/

$ wget https://4.bp.blogspot.com/-PQQV4wfGlNI/XAnvQBMeneI/AAAAAAABQmU/lN7zIROor9oi3q-JZOBJiKKzfklzPE1hwCLcBGAs/s800/business_man2_2_shock.png] -P images/
$ wget https://3.bp.blogspot.com/-QcDbWqQ448I/XAnvUT4TMDI/AAAAAAABQnE/_H4XzC4E93AEU2Y7fHMDBjri1drdyuAPQCLcBGAs/s800/business_woman2_2_shock.png -P images/

$ wget https://3.bp.blogspot.com/-dSPRqYvIhNk/XAnvPdvjBFI/AAAAAAABQmM/izfRBSt1U5o7eYAjdGR8NtoP4Wa1_Zn8ACLcBGAs/s800/business_man1_4_laugh.png -P images/
$ wget https://1.bp.blogspot.com/-T6AOerbFQiE/XAnvTlQvobI/AAAAAAABQm8/TYVdIfxQ5tItWgUMl5Y0w8Og_AZAJgAewCLcBGAs/s800/business_woman1_4_laugh.png -P images/

$ wget https://4.bp.blogspot.com/-Kk_Mt1gDKXI/XAnvS6AjqyI/AAAAAAABQm4/LQteQO7TFTQ-KPahPcAqXYannEArMmYfgCLcBGAs/s800/business_woman1_3_cry.png -P images/
$ wget https://4.bp.blogspot.com/-3IPT6QIOtpk/XAnvPCPuThI/AAAAAAABQmI/pIea028SBzwhwqysO49pk4NAvoqms3zxgCLcBGAs/s800/business_man1_3_cry.png -P images/

$ wget https://3.bp.blogspot.com/-FrgNPMUG0TQ/XAnvUmb85VI/AAAAAAABQnI/Y06kkP278eADiqvXH5VC0uuNxq2nnr34ACLcBGAs/s800/business_woman2_3_surprise.png -P images/
$ wget https://2.bp.blogspot.com/-i7OL88NmOW8/XAnvQacGWuI/AAAAAAABQmY/LTzN4pcnSmYLke3OSPME4cUFRrLIrPsYACLcBGAs/s800/business_man2_3_surprise.png -P images/

$ cp images/lN7zIROor9oi3q-JZOBJiKKzfklzPE1hwCLcBGAs/s800/business_man2_2_shock.png]  images/lN7zIROor9oi3q-JZOBJiKKzfklzPE1hwCLcBGAs/s800/business_man2_2_shock.png

Of the above, the last command just removes the garbage (the key brackets at the end) in the file name.

The execution is as follows. One option has been added.

--Enter the actual webcam device number in input_video_num. For / dev / video0, enter the trailing 0. --Specify the device file of the virtual webcam device for output_video_dev. --Set emotion_mode to True.

In addition, please use ctrl + c to end it.

$ python3 webcamhooker.py --input_video_num 0 --output_video_dev /dev/video2 --emotion_mode True

When you execute the above command, ffmpeg will run and the video will start to be delivered to the virtual camera device.

Let's have a video conference!

As before, when you have a video conference, you will see something like dummy ~ ~ in the list of video devices, so select it. This is an example of Teams. The character string at the top of the screen changes according to the facial expression, and the corresponding image is displayed accordingly. It's a great success. out2.gif

Finally

It may be difficult to communicate casually because working from home is prolonged, but I think it would be good to bring this kind of playfulness to video conferencing and activate conversations. I think we can do more, so please give it a try.

reference

For sentiment analysis by Tensorflow, I referred to the following site. (This site is introduced in tensorflowjs)

https://book.mynavi.jp/manatee/detail/id=99887

Recommended Posts

How to process camera images with Teams and Zoom Sentiment analysis with Tensorflow
How to process camera images with Teams or Zoom
How to process camera images with Teams and Zoom Volume of processing in animation style
How to share folders with Docker and Windows with tensorflow
[How to!] Learn and play Super Mario with Tensorflow !!
How to make a surveillance camera (Security Camera) with Opencv and Python
How to search using python's astroquery and get fits images with skyview
How to display images continuously with matplotlib Note
Learn how to inflate images from TensorFlow code
[Linux] How to display CPU usage, display header, and not display grep process with ps command
How to extract null values and non-null values with pandas
How to loop and play gif video with openCV
[TensorFlow] I want to process windows with Ragged Tensor
I tried to implement Grad-CAM with keras and tensorflow
[TF] How to save and load Tensorflow learning parameters
How to insert a specific process at the start and end of spider with scrapy
How to make a Cisco Webex Teams BOT with Flask
How to kill a process instantly with Python's Process Pool Executor
[TensorFlow 2 / Keras] How to run learning with CTC Loss in Keras
How to not load images when using PhantomJS with Selenium
[Python] How to play with class variables with decorator and metaclass
Send experiment results (text and images) to slack with Python
How to learn TensorFlow for liberal arts and Python beginners
Convert garbled scanned images to PDF with Pillow and PyPDF
How to do Bulk Update with PyMySQL and notes [Python]
How to create dataframes and mess with elements in pandas
How to log in to AtCoder with Python and submit automatically
Real-time object detection Android app with TensorFlow and Camera X
Sentiment analysis with Python (word2vec)
How to update with SQLAlchemy?
How to cast with Theano
How to Alter with SQLAlchemy?
How to separate strings with','
How to RDP with Fedora31
How to Delete with SQLAlchemy?
Text sentiment analysis with ML-Ask
How to set a shortcut to switch full-width and half-width with IBus
Creating a Tensorflow Sequential model with original images added to MNIST
How to import CSV and TSV files into SQLite with Python
How to deal with errors when installing whitenoise and deploying to Heroku
Correspondence analysis of sentences with COTOHA API and save to file
How to run Jupyter and Spark on Mac with minimal settings
How to install pandas on EC2 (How to deal with MemoryError and PermissionError)
[C language] How to create, avoid, and make a zombie process
I tried to make a periodical process with Selenium and Python
How to deal with errors when installing Python and pip with choco
Explain how to use TensorFlow 2.X with implementation of VGG16 / ResNet50
Node.js: How to kill offspring of a process started with child_process.fork ()
How to use Python with Jw_cad (Part 2 Command explanation and operation)
How to build Python and Jupyter execution environment with VS Code
The fastest way to get camera images regularly with python opencv
[Python] Try to recognize characters from images with OpenCV and pyocr
How to put OpenCV in Raspberry Pi and easily collect images of face detection results with Python