Facial expression recognition using Pepper's API
By using the ** ALFaceCharacteristics API ** introduced in Age recognition using Pepper's API, facial expressions such as smiles You can recognize it. Here, we will try two methods, smile detection provided by the API and other facial expression recognition methods.
Note that ** ALFaceCharacteristics API does not have a means to check the operation of virtual robots, and an actual Pepper machine is required. ** I would like you to experiment with the actual Pepper machine at Aldebaran Atelier Akihabara. (Reservation URL: http://pepper.doorkeeper.jp/events)
How to get the project file
The project file for this tutorial is available on GitHub https://github.com/Atelier-Akihabara/pepper-face-characteristics-example. There are several ways to get the code on GitHub, but one of the easiest ways is to get the archive from the Download ZIP link. There are various other methods such as using git related tools, so please check according to the situation.
Judgment of smile
By using the ʻALFaceCharacteristics` API, it is possible to judge the degree of smile on the recognized face from the image features such as the fineness of the eyes and whether the corners of the mouth are raised. You can use this value to determine if the person Pepper recognizes is smiling.
Using the memory event FaceCharacteristics / PersonSmiling
The easiest way to make a smile check is to use the FaceCharacteristics / PersonSmiling event.
First, let's check the contents of the document.
Check the document
More information about this event can be found in the Choregraphe documentation at NAOqi Developer guide> NAOqi Framework> NAOqi API> NAOqi PeoplePerception> ALFaceCharacteristics> ALFaceCharacteristics API. If you click on the FaceCharacteristics / PersonSmiling link, you will find the following explanation.
Event: "FaceCharacteristics/PersonSmiling" callback(std::string eventName, int id, std::string subscriberIdentifier)
Raised when a person has a smile degree above the current threshold (default = 0.7). Parameters:
- eventName (std::string) – “FaceCharacteristics/PersonSmiling”
- id – ID of the person as defined by the ALPeoplePerception API.
- subscriberIdentifier (std::string) –
It is explained that this event occurs when there is a person whose smile degree (expressed as 0.0 to 1.0) exceeds the threshold value (default is 0.7).
Also, the value passed by this event is ʻid. This is [ALPeoplePerception: Get List of People](http://qiita.com/Atelier-Akihabara/items/4162192129f366da1240#alpeopleperception-%E4%BA%BA%E3%81%AE%E4%B8%80% E8% A6% A7% E3% 81% AE% E5% 8F% 96% E5% BE% 97) This is the person's identifier that we have already seen. In other words, based on the value obtained from this FaceCharacteristics / PersonSmiling, it is possible to obtain additional information about the person who detected that he / she is smiling by acquiring PeoplePerception / Person /
Checking the operation of memory events on the memory watcher
Before creating the application, let's check the behavior of the event with Memory Watcher. First, set the content of the event you want to check in the memory watcher.
-
Connect Choregraphe to the actual Pepper machine
-
From the Memory Watcher panel, double-click **
-
First, I want to check
FaceCharacteristics / PersonSmiling, so enter PersonSmiling in ** [Filter] [A] **, check ** [PersonSmiling] underFaceCharacteristics[B] **, and then click [ Click the OK button
This will add FaceCharacteristics / PersonSmiling to the memory watcher.
-
Next, also monitor
PeoplePerception / VisiblePeopleListto see the value of the person's ID. In the same way as in 3., enter VisiblePeopleList in ** [Filter] [A] **, check ** [VisiblePeopleList] underPeoplePerception[B] **, and click the [OK] button.
Make sure that FaceCharacteristics / PersonSmiling and PeoplePerception / VisiblePeopleList are added to the Memory Watcher panel as shown below.
Now, let's check the operation of the memory event. Standing in front of Pepper, you should see the following changes in the contents of the memory watcher:
The number [6602] is displayed in PeoplePerception / VisiblePeopleList. This indicates that Pepper recognizes a person in the area visible to the camera and that person has the identifier 6602. (This value depends on the internal state of Pepper at that time.)
Next, let's laugh in front of Pepper. Make a smile by raising the corners of your mouth, squinting your eyes, and showing your teeth.
Then, 6602 was displayed in FaceCharacteristics / PersonSmiling. This indicates that Pepper fired this event when a person (identifier 6602) had a smile level greater than 0.7, and that person's identifier was printed to FaceCharacteristics / PersonSmiling.
Try to make an application
Now that you have confirmed the movement of FaceCharacteristics / PersonSmiling, let's create an application using this event.
Here, as a sample, let's create an application ** that says "Can you laugh at me?" At startup, and when it recognizes a smile, it says "(ID) -san, you have a nice smile!" I will.
The sample project is ** subscribe-person-smiling **. Of the files obtained from GitHub, you can open them by double-clicking subscribe-person-smiling.pml in the subscribe-person-smiling folder.
It is created by the following procedure.
-
Click the ** Add Memory Event [+] button on the left side of the flow diagram **, enter ** PersonSmiling [A] ** in the filter, and check ** FaceCharacteristics / PersonSmiling [B] ** [OK] ] Button
-
Place the following boxes on the flow diagram
- standard box library * Audio > Voice > Say
- advanced box library * Audio > Voice > Say Text
-
Connect each box to the input
FaceCharacteristics / PersonSmilingcreated in 1. and set the contents as follows.
-
Customize Say Text Box (http://qiita.com/Atelier-Akihabara/items/8df3e81d286e2e15d9b6#%E8%A3%9C%E8%B6%B3say-text%E3%83%9C%E3%83 % 83% E3% 82% AF% E3% 82% B9% E3% 81% AE% E3% 82% AB% E3% 82% B9% E3% 82% BF% E3% 83% 9E% E3% 82% A4 % E3% 82% BA) and change what you say
sentence += str(p) + "Mr., a nice smile!"
When you run this application and turn your smile on it, you will say, "12743-san, you have a nice smile!" Once you move away from Pepper and approach again, or when another person approaches and smiles, you can also see that he speaks another identifier, such as "16234, you have a nice smile!".
This time, I'm talking about the person's identifier as it is for the sake of explanation, but I think it would be interesting to simply take a picture with FaceCharacteristics / PersonSmiling as a trigger without using this identifier. Please take advantage of it.
Using SmileProperties for each People ID
Similar to age recognition, you can get the smile level by using the person's identifier as a key. In the previous example, it may not have been recognized as a smile unexpectedly, but if you use this sample to check your own smile level, you may be able to see the characteristics of smile level judgment in Pepper.
The basic structure is the [Get Age Box](http://qiita.com/Atelier-Akihabara/items/4162192129f366da1240#alfacecharacteristics-%E5%B9%B4%E9%BD%A2%] introduced in the age acquisition example. Same as E3% 81% AE% E6% 8E% A8% E5% AE% 9A).
From the person's ID obtained by the Basic Awareness box, try to reference the value of PeoplePerception / Person / <ID> / SmileProperties.
Here, as an example, let's make a person who is being tracked by Basic Awareness get a smile level at 10-second intervals and talk about that value **.
The sample project is ** get-smile **. Of the files obtained from GitHub, you can open them by double-clicking get-smile.pml in the get-smile folder.
It is created by the following procedure.
-
Create a Get Smile box as an empty Python box. Similar to the Get Age box, it has the following input / output configuration. Please refer to Python Box Concept for how to create it.
-
Double-click the Get Smile box to open the script editor and write a Python script like the one below.
class MyClass(GeneratedClass): def __init__(self): GeneratedClass.__init__(self) def onLoad(self): self.memory = ALProxy("ALMemory") self.faceChar = ALProxy("ALFaceCharacteristics") def onUnload(self): pass def onInput_onPeopleDetected(self, peopleId): if peopleId < 0: return r = self.faceChar.analyzeFaceCharacteristics(peopleId) if not r: self.onUnknown() return smileData = self.memory.getData("PeoplePerception/Person/%d/SmileProperties" % peopleId) self.logger.info("Smile Properties: %d => %s" % (peopleId, smileData)) if smileData and len(smileData) == 2: self.onSmile(smileData[0]) else: self.onUnknown()
The code structure is similar to the Get Age box. The only difference is that the key obtained from ʻALMemory is SmileProperties instead of ʻAgeProperties.
- Connect to the Basic Awareness box using the same procedure as for the Get Age box. Here, I used the Timer w / Number box, which is a modified Timer box, so that the value finally output from Basic Awareness is entered in the Get Smile box every 10 seconds.
Refer to the contents of the project file for the values of each box.
When you run the get-smile project and Pepper can see his face, he will talk about his smile every 10 seconds as shown below. If you try to speak the decimal point value as it is, it will be difficult to understand because it will be spoken as "Zerotenichi", so I am speaking an integer value multiplied by 100.
Your smile is about 13
Your smile is about 34
As you change your facial expression, you can see that the value of smile level changes. Try different facial expressions and experiment with what factors Pepper uses to calculate your smile.
If you look at the Log Viewer panel, you can also see the raw value of the smile level and the value that indicates the accuracy.
[INFO ] behavior.box :onInput_onPeopleDetected:20 _Behavior__lastUploadedChoregrapheBehaviorbehavior_1892204656__root__GetSmile_5: Smile Properties: 47915 => [0.12999999523162842, 0.09800000488758087]
[INFO ] behavior.box :onInput_onPeopleDetected:20 _Behavior__lastUploadedChoregrapheBehaviorbehavior_1892204656__root__GetSmile_5: Smile Properties: 47915 => [0.3400000035762787, 0.8450000286102295]
Two numbers enclosed in [] are output, such as [0.3400000035762787, 0.8450000286102295], the first number is the smile degree and the second is the smile degree accuracy. This accuracy is a numerical value that indicates the degree of confidence in the set smile level value, and the closer it is to 0, the less confident it is, and the closer it is to 1, the more confident it is.
In this way, by referring to PeoplePerception / Person / <ID> / SmileProperties, more detailed values can be obtained and fine control can be performed.
Judgment of facial expressions other than smiles
Although the recognition and format of smiles are different, by using the values of PeoplePerception / Person / <ID> / ExpressionProperties, human facial expressions are expressionless (Neutral), happy (Happy), surprised (Surprised), and angry (Surprised). It is possible to determine whether it looks like Angry) or Sad.
The information obtained by SmileProperties was in the form[smile, accuracy], but the value of ʻExpressionProperties was [expressionless, happy, surprise, anger, sadness] `, respectively. The degree of judgment as a facial expression is stored in the range of 0.0 to 1.0. Also, the sum of these values is set to 1.0.
A sample using this value is provided as a sample project ** get-expression **. Of the files obtained from GitHub, you can open them by double-clicking get-expression.pml in the get-expression folder.
It is created by the following procedure.
-
Create a Get Expression box as an empty Python box
The values output to the onMax output are 0 ... neutral, 1 ... happy, 2 ... surprised, 3 ... angry, 4 ... sad.
-
Write the following code as a Get Expression box
class MyClass(GeneratedClass): def __init__(self): GeneratedClass.__init__(self) def onLoad(self): self.memory = ALProxy("ALMemory") self.faceChar = ALProxy("ALFaceCharacteristics") def onUnload(self): pass def onInput_onPeopleDetected(self, peopleId): if peopleId < 0: return r = self.faceChar.analyzeFaceCharacteristics(peopleId) if not r: self.onUnknown() return exprData = self.memory.getData("PeoplePerception/Person/%d/ExpressionProperties" % peopleId) self.logger.info("Expression Properties: %d => %s" % (peopleId, exprData)) if exprData and len(exprData) == 5: self.onNeutral(exprData[0]) self.onHappy(exprData[1]) self.onSurprised(exprData[2]) self.onAngry(exprData[3]) self.onSad(exprData[4]) self.onMax(exprData.index(max(exprData))) else: self.onUnknown() -
For testing, write the following flow, for example. Each output of onNeutral to onSad is a switch case that changes the contents of the Log box that outputs the log for checking the operation, such as "You look happy" or "Are you surprised?" Depending on the value of onMax. Place the box
In this way, you can get each value of ʻExpressionProperties`, talk according to the item with the highest value, and so on.
For example, if you run a sample project, open your mouth, open your eyes, and make a surprised face, the log will output: You can see that the Surprised item has a large value.
[INFO ] behavior.box :onInput_onPeopleDetected:20 _Behavior__...: Expression Properties: 60331 => [0.14000000059604645, 0.0, 0.85999995470047, 0.0, 0.0]
[INFO ] behavior.box :onInput_message:27 _Behavior__...: Neutral: 0.140000000596
[INFO ] behavior.box :onInput_message:27 _Behavior__...: Happy: 0.0
[INFO ] behavior.box :onInput_message:27 _Behavior__...: Surprised: 0.8599999547
[INFO ] behavior.box :onInput_message:27 _Behavior__...: Angry: 0.0
[INFO ] behavior.box :onInput_message:27 _Behavior__...: Sad: 0.0
Also, since the third value of ʻExpressionProperties, the surprising value, is the maximum value, 2` is output in the onMax output.
This value is entered in the Switch Case box and the process of saying "Are you surprised?" Is executed.
In this way, by using PeoplePerception / Person / <ID> / ExpressionProperties, it is possible to judge facial expressions other than smiles. By using the facial expression information such as smiles that we have seen so far, it is possible to make Pepper move according to the user's emotions.
Recommended Posts