I tried to make a real-time sound source separation mock with Python machine learning

This time, we will actually use non-negative matrix factorization (NMF) for sound. The goal is to create a mock that anyone can easily perform a sound source separation demo.

The result of making

The result first ... I wrote a simple graph drawing program and tried to draw it! 代替テキスト

The demo is available as Video.

What is NMF?

I hope you can take a look at the [article] I wrote earlier (https://qiita.com/sumita_v09/items/d22850f41257d07c45ea).

How do you process sound with NMF?

The basic form of NMF is


V \approx WH

It can be expressed by an expression such as. When separating sound sources, we assume sparseness in the spectrogram matrix V and decompose it so that it becomes the product of the dictionary matrix W and the excitation matrix H. The dictionary matrix expresses the power spectrum of the timbres contained in V, and the excitation matrix expresses how many timbres expressed in W are included in the time axis. Sound source separation is possible without a teacher, but this time we want to separate arbitrary tones from the input sound, so we will use supervised NMF. Specifically, an arbitrary tone power spectrum is registered in the dictionary matrix, and only the excitation matrix is updated when updating. In addition, in order to perform sound source separation in real time, the input V is not the spectrogram matrix (frequency x time) but the power spectrum matrix (frequency x 1) of only the latest frame is V.

What kind of thing do you make?

The sound source separation mock was created with the following requirements. --Work on CUI --Sound source separation is possible in real time --You can change / add the sound source you want to detect during execution. --Sound source separation result is transmitted by OSC

As for the implementation itself, the NMF implementation described in the above link article has been modified for sound so that it can be operated on the CUI.

Source code

The implementation is not posted in the article this time, but it is published on Github. Detailed environment settings and CUI operations are described in the README, so we hope you will find it helpful. https://github.com/T-Sumida/RealTimeSoundSeprator

Summary

I created a sound source separation mock that operates in real time. It's not as insanely good as deep learning, but I think it's a benefit to be able to add sounds to separate without a learning process. NMF is also fun, but I'm thinking of writing about machine learning-related stories that I'm doing at work in the future.

Recommended Posts

I tried to make a real-time sound source separation mock with Python machine learning
Machine learning beginners tried to make a horse racing prediction model with python
[5th] I tried to make a certain authenticator-like tool with python
[2nd] I tried to make a certain authenticator-like tool with python
[3rd] I tried to make a certain authenticator-like tool with python
I tried to make a periodical process with Selenium and Python
I tried to make a 2channel post notification application with Python
I tried to make a todo application using bottle with python
[4th] I tried to make a certain authenticator-like tool with python
[1st] I tried to make a certain authenticator-like tool with python
I want to make a game with Python
Python: I tried to make a flat / flat_map just right with a generator
I tried to make a traffic light-like with Raspberry Pi 4 (Python edition)
Source code of sound source separation (machine learning practice series) learned with Python
A beginner of machine learning tried to predict Arima Kinen with python
I tried to move machine learning (ObjectDetection) with TouchDesigner
I tried to draw a route map with Python
I tried to automatically generate a password with Python3
I tried to make a periodical process with CentOS7, Selenium, Python and Chrome
I tried to make a simple mail sending application with tkinter of Python
[Patent analysis] I tried to make a patent map with Python without spending money
I tried to make various "dummy data" with Python faker
I tried machine learning with liblinear
Mayungo's Python Learning Episode 3: I tried to print numbers with print
I tried to make a stopwatch using tkinter in python
I tried to make GUI tic-tac-toe with Python and Tkinter
I tried to divide with a deep learning language model
I tried to build an environment for machine learning with Python (Mac OS X)
[1 hour challenge] I tried to make a fortune-telling site that is too suitable with Python
I tried to make a generator that generates a C # container class from CSV with Python
Rubyist tried to make a simple API with Python + bottle + MySQL
I tried to make deep learning scalable with Spark × Keras × Docker
I tried to make a regular expression of "amount" using Python
[Python] I tried to implement stable sorting, so make a note
I tried to make a regular expression of "time" using Python
[Python] A memo that I tried to get started with asyncio
I tried to create a list of prime numbers with python
I tried to make a regular expression of "date" using Python
[Introduction] I want to make a Mastodon Bot with Python! 【Beginners】
I tried to make a strange quote for Jojo with LSTM
I tried to make an image similarity function with Python + OpenCV
I tried to make a mechanism of exclusive control with Go
I installed Python 3.5.1 to study machine learning
I tried a functional language with Python
I tried to make a Web API
I tried to make Othello AI with tensorflow without understanding the theory of machine learning ~ Introduction ~
I tried to make Othello AI with tensorflow without understanding the theory of machine learning ~ Implementation ~
I tried to communicate with a remote server by Socket communication with Python.
I tried to create a program to convert hexadecimal numbers to decimal numbers with python
Mayungo's Python Learning Episode 6: I tried to convert a character string to a number
I tried to make a calculator with Tkinter so I will write it
Mayungo's Python Learning Episode 2: I tried to put out characters with variables
I tried to make "Sakurai-san" a LINE BOT with API Gateway + Lambda
[AWS] [GCP] I tried to make cloud services easy to use with Python
I tried to discriminate a 6-digit number with a number discrimination application made with python
(Machine learning) I tried to understand Bayesian linear regression carefully with implementation.
[Outlook] I tried to automatically create a daily report email with Python
I tried to visualize the model with the low-code machine learning library "PyCaret"
I tried to build a Mac Python development environment with pythonz + direnv
[Zaif] I tried to make it easy to trade virtual currencies with Python
I started machine learning with Python (I also started posting to Qiita) Data preparation