The story of making a sound camera with Touch Designer and ReSpeaker

I made a sound camera using ReSpeaker to learn TouchDesigner. I made it because I thought it might be interesting to visualize the sound by performing SSL (Sound Source Localization) with ReSpeaker and adding effects in the direction of the arrival of the sound. I decided to call this a sound camera.

Touch Designer side

This has been a problem for a long time, but how can I share this kind of visual programming product on Qiita so that it's easy to see? For the time being, I will go with the image this time.

The content is very simple. All you have to do is read the image from the camera with videodevicein and overlay the following gif image at the specified position.

Since SSL is done in raw Python, the result is received in OSC from that process.

ReSpeaker side

ReSpeaker used v2.0. I also have 1.0, which seems to be more accurate, but for some reason I gave up because I could not connect with my Windows 10. I used to be connected ...

ReSpeaker v2.0 https://github.com/respeaker/usb_4_mic_array

From this library, from the place where DoA (Direction of Arrival) is performed, only the value of the sound source direction is extracted and sent by OSC. Please note that you are quite addicted to writing firmware. By the way, this time it is done on the Python side, but since the LED of the sound source direction lights up even with the microphone array alone, it seems that the device side is also estimating the sound source direction. At first glance, this one is more accurate, so if you can extract the data, you may want to use it.

doa.py

from pythonosc import osc_message_builder
from pythonosc import udp_client
from tuning import Tuning
import usb.core
import usb.util
import time

osc_client = udp_client.SimpleUDPClient('127.0.0.1', 50000)

from infi.devicemanager import DeviceManager
dm = DeviceManager()
devices = dm.all_devices
for i in devices:
    try:
        print ('{} : address: {}, bus: {}, location: {}'.format(i.friendly_name, i.address, i.bus_number, i.location))
    except Exception:
        pass


import usb.backend.libusb1
backend = usb.backend.libusb1.get_backend(find_library=lambda x: "./libusb-1.0.dll")
dev = usb.core.find(idVendor=0x2886, idProduct=0x0018)
direction = 0

if dev:
    Mic_tuning = Tuning(dev)
    direction = Mic_tuning.direction
    print(Mic_tuning.direction)
    while True:
        try:
            if direction != Mic_tuning.direction:
                direction = Mic_tuning.direction
                osc_client.send_message("/direction", direction)
                print(direction)
            time.sleep(0.5)
        except KeyboardInterrupt:
            break

Setting

Arrange so that the origin of the microphone array and the origin of the camera match. This time, I used a webcam called c920 from logitech. Check the angle of view of the camera in advance.

Operation check

When I try to move it, it looks like this. There is a little lag, but can you see the effect coming in after chasing the smartphone?

pic.twitter.com/NYk4rkXXf5

— hatbot (@hatbot3) November 13, 2020

Also, here is a video of a stapler ticking at hand.

pic.twitter.com/tdwNgsyNZ1

— hatbot (@hatbot3) November 13, 2020

This time, the microphone array was arranged flat, so it was a one-dimensional effect, but if you arrange the microphone array three-dimensionally or use two ReSpeakers, you should be able to add effects in spots.