I tried to easily create a high-precision 3D image with one photo [1]. (Depth can now be edited in PNG.)

Overview

** One photo ** ** High-precision 3D image ↓ Easy creation **.

(↓ It doesn't make any sense, but this pizza sharpens the PNG of depth. There is a part where the image is dragged, but I wonder if it is a pizza ...)

Below are the papers and github.

https://arxiv.org/pdf/2004.04727.pdf Paper "3D Photography using Context-aware Layered Depth Inpainting" Meng-Li Shih1 and others

github is below. https://github.com/vt-vl-lab/3d-photo-inpainting

Paper abstract (Google Translate mom)

738/5000 We suggest how to convert a single RGB-D input image to a 3D photo. This is a new view composition multi-layer representation that includes hallucinatory color and depth structures in areas blocked by the original view. Using a layered depth image with explicit pixel connections as a basic representation, we present a learning-based repair model that synthesizes new local color and depth content into a closed area in a spatial context-aware way. .. The resulting 3D photo can be efficiently rendered with motion parallax using a standard graphics engine. Verify the effectiveness of the method in a variety of challenging everyday situations and reduce artifacts compared to state-of-the-art technology.

It will be an additional note, 2020/05/23 ** About 3 days ago, depth could be edited in PNG. ** ** I tried it.

I tried it (work)

(The code on github worked easily in my windows environment. I may need a GPU. I don't need a good GPU. The processing time was a minute or two.)

As a creation procedure, Put the image you want to process in the image folder Just type the following command.

python main.py --config argument.yml

For publication here, the original photo is "High Quality Free Image Material" https://pixabay.com/ja/ Obtained from. (Similar to King Kong in the overview)

Move some objects from one photo? Since it is (shaking), of course, there is a part where there is no information **, but it seems that it is forcibly generated. great.

** I'm sorry, it may take about 30 seconds to display. .. .. ** **

** All inputs are one still image ** 1 sheet!

ダウンロードgr.gif

ダウンロードgirl.gif

ダウンロードcar.gif

ダウンロード66a.gif

ダウンロードbeach.gif

I tried it (additional work)

** Metropolitan Museum of Art ** Use the paintings in the collection.

ダウンロードart2.gif

ダウンロードart3.gif

ダウンロードart1.gif

I tried it (failure, CG work, etc.)

Again, the original photo is "high quality free image material" https://pixabay.com/ja/ Obtained from.

↓ ** Failure? **. (Rather, even when computers are incompatible ...)

ダウンロードdoragon.gif

↓ ** Failure **. ** Escher system . Also, balloons are difficult, aren't they? You may name it " Balloon Problem **". However, the picture is beautiful. .. ..

ダウンロードeee.gif

↓ Somehow the texture. .. .. Failure. ダウンロードkuma.gif

↓ Normal failure. ダウンロードdog.gif

(2020/05/23) About 3 days ago, depth could be edited with PNG.

I edited the depth.

Step 1: In argument.yml Change depth_format to png and execute.


・ ・ ・ depth_format: '.png' ・ ・ ・


Then the depth will be saved as PNG. (I don't want to display this png ↓ in a large size at all, but ... It is probably displayed in a large size along the road ...)

![graffiti-745071 -copy.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/41ecdffa-b58a-ffbc-da3c-3dfecd66025f.png)



Step 2:
Edit PNG.
(At the bottom of the car, make the part where the depth is shallow deeper (= black).)
See the results shown below to see if such a rough edit is okay.

![graffiti-745071.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/e0f227c8-37a6-b6e0-68df-1f6d206c95ec.png)



Step 3:
require_Set midas to False so that the edited PNG is used.

require_midas: False


Then re-execute the conversion.

![Download edit.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/64bb82f9-d9c7-3e90-0b39-07766e2b011d.gif)

As intended, the depth at the bottom of the car is deeper. However, it looks like PNG,
Rough PNG editing didn't clean it up.
If you can do paint processing normally, I think it's okay. I can't. With a brush
I painted it properly. .. ..
You may not be able to tell what you have done because the work is complicated, but when you are dissatisfied with the automatically detected depth, you can edit the depth expressed as PNG as an image.

#Information that may be useful
・ GFORCE GTX1050Ti (Dedicated GPU memory 4).0GB)Then
  **It may not work due to insufficient memory of GPU**。
this is,**Rather than relying on image size**、
Perhaps it's related to the amount of memory required to represent the 3D in the image content or something.
I haven't found a solution. It may be possible to manage by changing parameters.
Red car, so dedicated GPU memory 3.About 4GB, white one is 2.About 7GB was used.
For this memory problem, why not use Google Colab for the time being?
-I showed editing depth information with PNG, but surely the default is numpy?。
That data(numpy)If you can handle the rattling, you may edit it.

#Summary

I don't understand how great it is technically.
However, I wonder if it feels like a force.
(In the first place, people's perception when looking at a photo is also forcible, so ...)
I will study technology.
(For the time being, I understand the habit of processing, so I think I can make a little more fun work.)
At this stage, I have no idea what it can be used for.
**Please try it yourself. "Indoor selfie" is recommended**

#Summary 2
Described after "I tried it (additional work)".
Unfortunately,**I got tired of seeing it in no time. .. .. ..**
It's tough if you don't have a purpose and output for it. .. .. ..
mere**effect**I mean. .. .. Then you get tired of it. Furthermore, we expect the next level of technology.
I'm tired of it, but I'm interested in technology, though I don't understand it yet. .. ..
It was easy to edit the depth.

#from now on
I think it's better to display the name of the author of pixabay, separately.
Is there a different way to show the processed image? ?? ?? I think it's simple.
If you have any comments, please.

reference.
[I tried to easily create a high-precision 3D image with one photo [2].(Try processing depth with numpy)](https://qiita.com/torinokaijyu/items/e761c00c87d6a00b8c30)
[I tried to easily create a high-precision 3D image with one photo [3]. Only depth without permission feat. intel-of isl.](https://qiita.com/torinokaijyu/items/6889598b732851c2e8fd)


Recommended Posts

I tried to easily create a high-precision 3D image with one photo [1]. (Depth can now be edited in PNG.)
I tried to easily create a high-precision 3D image with one photo [2]. (Try processing depth with numpy)
I tried to easily create a high-precision 3D image with one photo [3]. MiDaS of feat. Intel-isl only depth without permission.
Creating a highly accurate 3D image easily with one photo! (You can now edit the depth in PNG.) [1-E]
I tried to easily create a high-precision 3D image with one photo [0]. (Confirmed how to capture the space, put a net)
I tried to create a class that can easily serialize Json in Python
I want to create a priority queue that can be updated in Python (2.7)
I tried to easily create a fully automatic attendance system with Selenium + Python
I came up with a way to create a 3D model from a photo Part 02 Image loading and vertex drawing
I tried to create a table only with Django
I tried to create a class to search files with Python's Glob method in VBA
I tried to automatically create a report with Markov chain
I tried to create an article in Wiki.js with SQLAlchemy
I came up with a way to create a 3D model from a photo Part 04 Polygon generation
I tried using PI Fu to generate a 3D model of a person from one image
I tried to process the image in "sketch style" with OpenCV
I came up with a way to create a 3D model from a photo Part 01 Creating an environment
I tried to create Bulls and Cows with a shell program
I tried to process the image in "pencil style" with OpenCV
How to easily create a CG image like a toy photo "Ubuntu / Linux" ・ Photo realistic rendering with LeoCAD & Blender
I tried to create a program to convert hexadecimal numbers to decimal numbers with python
Create a web app that can be easily visualized with Plotly Dash
I came up with a way to make a 3D model from a photo.
I tried to create a plug-in with HULFT IoT Edge Streaming [Development] (2/3)
I tried to create a plug-in with HULFT IoT Edge Streaming [Execution] (3/3)
[Outlook] I tried to automatically create a daily report email with Python
ConSinGAN: I tried using GAN that can be generated from one image
I tried to create a plug-in with HULFT IoT Edge Streaming [Setup] (1/3)
I tried to create a linebot (preparation)
I wanted to quickly create a mail server that can be used freely with postfix + dovecot on EC2
I wanted to create a program in Reverse Polish Notation in Python (determining if a string can be converted to a number)
When I tried to create a virtual environment with Python, it didn't work
I tried to predict the horses that will be in the top 3 with LightGBM
I made a familiar function that can be used in statistics with Python
[Azure] I tried to create a Linux virtual machine in Azure of Microsoft Learn
I tried to extract a line art from an image with Deep Learning
I tried to create a button for Slack with Raspberry Pi + Tact Switch
I tried to implement what seems to be a Windows snipping tool in Python
A machine learning beginner tried to create a sheltie judgment AI in one day
I tried to make a simple image recognition API with Fast API and Tensorflow
I tried to create a model with the sample of Amazon SageMaker Autopilot
I want to easily create a Noise Model
I want to create a window in Python
I tried to integrate with Keras in TFv1.1
You can easily create a GUI with Python
You can drink Zoom! !! I tried to make a just one style cooperative theme guessing game with Flask + Heroku
I tried to automatically generate a character string to be input to Mr. Adjustment with Python
I tried to create a reinforcement learning environment for Othello with Open AI gym
I came up with a way to make a 3D model from a photo. 0 Projection to 3D space
[Python] I tried to automatically create a daily report of YWT with Outlook mail
I tried to make a memo app that can be pomodoro, but a reflection record
I tried scraping food recall information with Python to create a pandas data frame
I want to transition with a button in flask
I tried to draw a route map with Python
I tried to implement a pseudo pachislot in Python
I want to manually create a legend with matplotlib
[Python] How to create a 2D histogram with Matplotlib
I tried to implement SSD with PyTorch now (Dataset)
I tried to automate "one heart even if separated" using a genetic algorithm in Python
I want to see the graph in 3D! I can make such a dream come true.
I tried to create a Python script to get the value of a cell in Microsoft Excel