A story about making 3D space recognition with Python

Introduction

First from things Here is it

As you can see, it recognizes 3D space from the image.

algorithm

Simply put, use ** vector difference ** What does that mean?

First, find the feature points on the image and match the two images. Then, assuming that the image plane is in real space, find the distance between the center of the image plane and the camera.

Then, from the coordinates of the feature point with the center of the image as the origin and the distance between the origin and the camera obtained earlier, the vector with the camera as the origin is derived from each of the two images.

This is the preparation.

Of the vectors derived in the process up to this point, the vector derived from the first image

\vec{a}=(a_1,a_2,a_3)

The vector derived from the second image

\vec{b}=(b_1,b_2,b_3)

The vector that the camera moved between these two images

\vec{c}=(c_1,c_2,c_3)

Then, the intersection $ O $ of $ \ vec {a} $ and $ \ vec {b} $ and the respective shooting points $ A (\ vec {a}) and B (\ vec {b}) $ are on the same plane. Because it's on top

-l\vec{a}=\vec{c}-k\vec{b}\\
k\vec{b}-l\vec{a}=\vec{c}

Will be From this formula

\left\{
\begin{array}{l}
c_1=kb_1-la_1\\
c_2=kb_2-la_2\\
c_3=kb_3-la_3
\end{array}
\right.

The simultaneous equations are established, but if you try to solve using a matrix as it is,

\begin{vmatrix}
b_{1}&-a_{1}\\
b_{2}&-a_{2}\\
b_{3}&-a_{3}\\
\end{vmatrix}
\begin{vmatrix}
k\\
l\\
\end{vmatrix}
=
\begin{vmatrix}
c_{1}\\
c_{2}\\
c_{3}
\end{vmatrix}\\
\begin{vmatrix}
k\\
l\\
\end{vmatrix}
=
\begin{vmatrix}
c_{1}\\
c_{2}\\
c_{3}
\end{vmatrix}
\begin{vmatrix}
b_{1}&-a_{1}\\
b_{2}&-a_{2}\\
b_{3}&-a_{3}\\
\end{vmatrix}^{-1}

And

\begin{vmatrix}
b_{1}&-a_{1}\\
b_{2}&-a_{2}\\
b_{3}&-a_{3}\\
\end{vmatrix}

Is not an invertible matrix, so the equation cannot be solved. Therefore, the simultaneous equations are transformed.

\left\{
\begin{array}{l}
(c_1+c_2)=k(b_1+b_2)-l(a_1+a_2)\\
(c_2+c_3)=k(b_2+b_3)-l(a_2+a_3)\\

\end{array}
\right.

As a result, $ k and l $ are derived using a matrix.

And $ k \ vec {b} $ is the distance from $ B $.

program

The program is available here GitHub For the time being, the core part of the code

match.py


def convert(data):
    data_converted=[]
    for loc in data:
        data_converted.append(convert_3d(loc))
    A=np.matrix([
        [-1*sum(data_converted[1][:2]),sum(data_converted[0][:2])],
        [-1*sum(data_converted[1][1:]),sum(data_converted[0][1:])],
    ])
    Y=np.matrix([
        [-0.6],
        [0],
    ])
    coe = np.linalg.solve(A,Y).reshape(-1,).tolist()
    
    return data_converted[1][0]*coe[0][0],data_converted[1][1]*coe[0][0],data_converted[1][2]*coe[0][0]

problem

As long as you are the person who made it, you have to raise the problem. As you can see in the tweet at the beginning, outliers are inevitable. It seems that the accuracy of $ \ vec {c} $ has a strong influence on this outlier.

The reason why such a problem occurs is because it is assumed that they are in the same plane **. In other words, it seems that outliers will appear if $ \ vec {c} $ shifts even a little.

I have no idea how to correct this. I will add it as soon as it is corrected.

Well, it was a story like this. Well then

Recommended Posts

A story about making 3D space recognition with Python
A story about making Hanon-like sheet music with Python
A story about an amateur making a breakout with python (kivy) ②
A story about an amateur making a breakout with python (kivy) ①
A story about trying a (Golang +) Python monorepo with Bazel
Solve ABC166 A ~ D with Python
A story about making a tanka by chance with Sudachi Py
A story about a python beginner stuck with No module named'http.server'
A story about adding a REST API to a daemon made with Python
A story about machine learning with Kyasuket
The story of making a standard driver for db with python.
A story about developing a soft type with Firestore + Python + OpenAPI + Typescript
Solve AtCoder ABC168 with python (A ~ D)
A story about Python pop and append
The story of making a module that skips mail with python
The story of making a university 100 yen breakfast LINE bot with Python
A story about a beginner making a VTuber notification bot from scratch in Python
[Python3] A story stuck with time zone conversion
Solve A ~ D of yuki coder 247 with python
A story stuck with handling Python binary data
A story about implementing a login screen with django
A story about running Python on PHP on Heroku
A story about modifying Python and adding functions
Automatic Zakuzaku, Bitcoin. A story about a Python beginner making a coin check 1-minute chart
A story about making an x86 bootloader that can boot vmlinux with Rust
A story about using Resona's software token with 1Password
A story about predicting exchange rates with Deep Learning
A story about how theano was moved with TSUBAME 2.0
A memo about building a Django (Python) application with Docker
[Python] How to create a 2D histogram with Matplotlib
Create a 2d CAD file ".dxf" with python [ezdxf]
A story about how Windows 10 users created an environment to use OpenCV3 with Python 3.5
Create 3d gif with python3
Stumble story with Python array
A memorandum about correlation [Python]
[Python] A story about making a LINE Bot with a practical manned function on its own without using Salesforce [Messaging API]
Make a fortune with Python
A memorandum about Python mock
AtCoder ABC 182 Python (A ~ D)
Try face recognition with Python
Automate sushi making with Python
Create a directory with python
A note about [python] __debug__
A note about hitting the Facebook API with the Python SDK
[Small story] [Python] Replace strings in 2D arrays with numbers
A story about how to specify a relative path in python.
A story about competing with a friend in Othello AI Preparation
A story about installing matplotlib using pip with an error
A story about how to deal with the CORS problem
[Python] The first step to making a game with Pyxel
A story about trying to implement a private variable in Python.
The story of making a question box bot with discord.py
A story about building a PyPI cache server (with Docker) and making me a little happy again
[Note] A story about trying to override a class method with two underscores in Python 3 series.
Machine learning A story about people who are not familiar with GBDT using GBDT in Python
[Python, Selenium, PhantomJS] A story when scraping a website with lazy load
Number recognition in images with Python
[Python] What is a with statement?
Solve ABC163 A ~ C with Python
Operate a receipt printer with python
A python graphing manual with Matplotlib.