The story of making a sound camera with Touch Designer and ReSpeaker

I made a sound camera using ReSpeaker to learn TouchDesigner. I made it because I thought it might be interesting to visualize the sound by performing SSL (Sound Source Localization) with ReSpeaker and adding effects in the direction of the arrival of the sound. I decided to call this a sound camera.

Touch Designer side

スクリーンショット 2020-11-13 191013.png This has been a problem for a long time, but how can I share this kind of visual programming product on Qiita so that it's easy to see? For the time being, I will go with the image this time.

The content is very simple. All you have to do is read the image from the camera with videodevicein and overlay the following gif image at the specified position. 2 - コピー.gif

Since SSL is done in raw Python, the result is received in OSC from that process.

ReSpeaker side

ReSpeaker used v2.0. I also have 1.0, which seems to be more accurate, but for some reason I gave up because I could not connect with my Windows 10. I used to be connected ...

ReSpeaker v2.0 https://github.com/respeaker/usb_4_mic_array

From this library, from the place where DoA (Direction of Arrival) is performed, only the value of the sound source direction is extracted and sent by OSC. Please note that you are quite addicted to writing firmware. By the way, this time it is done on the Python side, but since the LED of the sound source direction lights up even with the microphone array alone, it seems that the device side is also estimating the sound source direction. At first glance, this one is more accurate, so if you can extract the data, you may want to use it.

doa.py


from pythonosc import osc_message_builder
from pythonosc import udp_client
from tuning import Tuning
import usb.core
import usb.util
import time

osc_client = udp_client.SimpleUDPClient('127.0.0.1', 50000)

from infi.devicemanager import DeviceManager
dm = DeviceManager()
devices = dm.all_devices
for i in devices:
    try:
        print ('{} : address: {}, bus: {}, location: {}'.format(i.friendly_name, i.address, i.bus_number, i.location))
    except Exception:
        pass


import usb.backend.libusb1
backend = usb.backend.libusb1.get_backend(find_library=lambda x: "./libusb-1.0.dll")
dev = usb.core.find(idVendor=0x2886, idProduct=0x0018)
direction = 0

if dev:
    Mic_tuning = Tuning(dev)
    direction = Mic_tuning.direction
    print(Mic_tuning.direction)
    while True:
        try:
            if direction != Mic_tuning.direction:
                direction = Mic_tuning.direction
                osc_client.send_message("/direction", direction)
                print(direction)
            time.sleep(0.5)
        except KeyboardInterrupt:
            break

Setting

Arrange so that the origin of the microphone array and the origin of the camera match. This time, I used a webcam called c920 from logitech. Check the angle of view of the camera in advance.

Image from iOS.jpg

Operation check

When I try to move it, it looks like this. There is a little lag, but can you see the effect coming in after chasing the smartphone?

Also, here is a video of a stapler ticking at hand.

This time, the microphone array was arranged flat, so it was a one-dimensional effect, but if you arrange the microphone array three-dimensionally or use two ReSpeakers, you should be able to add effects in spots.

Recommended Posts

The story of making a sound camera with Touch Designer and ReSpeaker
The story of making a question box bot with discord.py
The story of making a module that skips mail with python
The story of making a university 100 yen breakfast LINE bot with Python
The story of making a lie news generator
The story of making a mel icon generator
The story of making a box that interconnects Pepper's AL Memory and MQTT
The story of making a web application that records extensive reading with Django
The story of making a music generation neural network
A server that returns the number of people in front of the camera with bottle.py and OpenCV
The story of Python and the story of NaN
The story of writing a program
A story stuck with the installation of the machine learning library JAX
A story about predicting prefectures from the names of cities, wards, towns and villages with Jubatus
The story of making a tool that runs on Mac and Windows at the game development site
The story of making Python an exe
The story of making an immutable mold
The story of trying deep3d and losing
The story of blackjack A processing (python)
A story that visualizes the present of Qiita with Qiita API + Elasticsearch + Kibana
The story of a Parking Sensor in 10 minutes with GrovePi + Starter Kit
Calculate the shortest route of a graph with Dijkstra's algorithm and Python
The story of having a hard time introducing OpenCV with M1 MAC
Make a DNN-CRF with Chainer and recognize the chord progression of music
The story of making a package that speeds up the operation of Juman (Juman ++) & KNP
The story of making a tool to load an image with Python ⇒ save it as another name
The story of doing deep learning with TPU
What I investigated in the process of expressing (schematicizing) containers in a nested frame with Jupyter and making it
Story of making a virtual planetarium [Until a beginner makes a model with a script and manages to put it together]
A story about calculating the speed of a small ball falling while receiving air resistance with Python and Sympy
The story of making the Mel Icon Generator version2
Gently explain the process of making a simple serverless surveillance camera using Raspberry Pi, Gmail API and Line API
Try out the touch of data-driven testing with Selenium Python Bindings and py.test
The story of making a Line Bot that tells us the schedule of competitive programming
A discussion of the strengths and weaknesses of Python
The story of making soracom_exporter (I tried to monitor SORACOM Air with Prometheus)
Get the stock price of a Japanese company with Python and make a graph
A story about getting the Atom field (XML telegram) of the Japan Meteorological Agency with Raspberry Pi and tweeting it
The story of Airflow's webserver and DAG, which takes a long time to load
The result of making a map album of Italy honeymoon in Python and sharing it
The story of trying to contribute to COVID-19 analysis with AWS free tier and failing
The story of launching a Minecraft server from Discord
A story that reduces the effort of operation / maintenance
Try making a simple website with responder and sqlite3
A story about making 3D space recognition with Python
The story of Python without increment and decrement operators.
[Touch Designer] Change the appearance color of Button COMP
The story of stopping the production service with the hostname command
Specify the volume on linux and make a sound
The story of replacing Nvidia GTX 1650 with Linux Mint 20.1.
The story of sharing the pyenv environment with multiple users
A story of trying out pyenv, virtualenv and virtualenvwrapper
Take a screenshot of the LCD with Python-LEGO Mindstorms
Visualize the range of interpolation and extrapolation with python
A story about making Hanon-like sheet music with Python
Create a batch of images and inflate with ImageDataGenerator
Visualize the characteristic vocabulary of a document with D3.js
A story about changing the master name of BlueZ
Zip 4 Gbyte problem is a story of the past
A story that analyzed the delivery of Nico Nama.
A memorandum when making a surveillance camera with Raspberry Pi