[Titan Craft] I made a tool to summon a giant to Minecraft

What is TitanCraft is?

A Minecraft .schematic file is generated from a single full-body image taken with a smartphone etc. via a 3D object. You can summon a giant by loading the generated .schematic file into the world of Micra using mod such as WorldEdit. I will.

The source code is available here

You can do this

titan1.png

  • Humanity remembered that day *

titan2.png

titans.png

Process flow

Let's follow the processing flow using the following image. (The image was taken from here: pray_tone1:) man_2.jpg

1. Extract the foreground from the input image

I use a repository called remove_bg that I created to extract the foreground. At that time, a mask image is required for PI Fu in the next process, so create it together. In remove_bg, first apply Semantic Segmentation to the input image to extract the target object. After that, a ternary image called trimap is generated and input to FBA_Matting together with the input image to extract the foreground.

[Processing result] man_2_remove_bg.png

It is not very suitable because the sample image has almost no background, but you can see that the person is cut out well. (There are some places where the background remains, such as near the crotch and neck.) I thought it would be more accurate if I used API of remove.bg, but there was a charge. So I didn't use it this time.

2. Create a 3D object from an image with PIFu

Create a 3D object from an image using a technique called PIFu. This process infers not only the 3D shape but also the texture of clothes and hairstyles that are not shown in the image. I don't really understand the details of PLFu, but you can find it in a little more detail in this article.

[Processing result] man_1_obj.jpg (Visualized with * meshlab * *)

The back and buttocks that are not shown in the input image are also well reproduced. It's amazing. At present, it may be a little difficult to use it as a 3D object as it is, but it seems that it will not be long before a beautiful 3D object can be generated from one image if the technology advances further in the future: ok_hand_tone2:

3. Generate a .schematic file for Minecraft from a 3D object

Using the repository I created obj2schematic, I used the .obj file generated by [2.](# 2.-Generate 3D object from image with PIFu) to create a .schematic for Minecraft. Create a file. It reads the .obj file generated by PIFu, analyzes the contents and converts it into a block of Minecraft. You can summon a giant by loading the generated .schematic file into the world with mod such as WorldEdit!

[Processing result] man_schematic.png

How about that⁉ I think it's done well: smile_cat:

Below is the gallery of the summoned giants.

2020-08-03_16.53.29.png

2020-08-03_16.55.51.png

2020-08-03_16.56.12.png

2020-08-03_16.58.22.png

2020-08-03_17.01.51.png

2020-08-03_17.05.14.png

Summary

I made a tool to summon a giant in Minecraft from one image. I made it possible to run it on Google Colab, so please use it. Open In Colab

If you want to run it locally, you need a python environment, but if you can clear it, you can use it immediately. Even in my environment without GPU (MacBook Pro (2019)), one image (512x512) can be processed in 2 to 3 minutes.

I made it because I wanted my son to say "Wow!", But I'm happy that I could do it as I imagined: joy_cat: This time I made a 3D object from a full-body image, but I think that there are many other things that can be done by switching the model to be used. If you have any ideas, please let us know in the comments etc .: rolling_eyes:

Also, I am using public models such as PIFu and FBA_Matting, but it is amazing that it is in a state where it can be used without understanding the details of each method. I felt it again.

Recommended Posts

[Titan Craft] I made a tool to summon a giant to Minecraft
I made a tool to compile Hy natively
I made a tool to get new articles
I made a tool to create a word cloud from wikipedia
I made a script to display emoji
I made a tool to automatically browse multiple sites with Selenium (Python)
I made a CLI tool to convert images in each directory to PDF
I made a tool to convert Jupyter py to ipynb with VS Code
I made a useful tool for Digital Ocean
I made a router config collection tool Config Collecor
I made a tool to notify Slack of Connpass events and made it Terraform
I made a tool to easily display data as a graph by GUI operation.
I made a tool to generate Markdown from the exported Scrapbox JSON file
I made a tool to automatically back up the metadata of the Salesforce organization
〇✕ I made a game
I made a user management tool for Let's Chat
I made a library to separate Japanese sentences nicely
I made a cleaning tool for Google Container Registry
I made a script to put a snippet in README.md
I made a Python module to translate comment outs
I made a code to convert illustration2vec to keras model
I made a command to markdown the table clipboard
I made a python library to do rolling rank
I made a repeating text data generation tool "rpttxt"
I made a toolsver that spits out OS, Python, modules and tool versions to Markdown
I made a tool that makes it a little easier to create and install a public key.
I made a box to rest before Pepper gets tired
I made a command to generate a table comment in Django
I made a function to check the model of DCGAN
I made a python text
I made a tool to automatically generate a simple ER diagram from the CREATE TABLE statement
I made a tool that makes it convenient to set parameters for machine learning models.
I made a plug-in that can "Daruma-san fell" with Minecraft
I made you to execute a command from a web browser
I made a script to say hello at my Koshien
[5th] I tried to make a certain authenticator-like tool with python
I made a program to solve (hint) Saizeriya's spot the difference
[2nd] I tried to make a certain authenticator-like tool with python
I made a web server with Raspberry Pi to watch anime
[3rd] I tried to make a certain authenticator-like tool with python
I made a client / server CLI tool for WebSocket (like Netcat for WebSocket)
[4th] I tried to make a certain authenticator-like tool with python
I made a scaffolding tool for the Python web framework Bottle
[1st] I tried to make a certain authenticator-like tool with python
I made a library that adds docstring to a Python stub file.
I made a command to display a colorful calendar in the terminal
I made a CUI-based translation script (2)
I made a wikipedia gacha bot
I made a fortune with Python.
I made a CUI-based translation script
A tool to convert Juniper config
I made a daemon with Python
I made a Docker container to use JUMAN ++, KNP, python (for pyKNP).
I made a plugin to generate Markdown table from csv in Vim
I made a password generator to teach Python3 to children (bonus) * Completely remade
I made a web application in Python that converts Markdown to HTML
[Django] I made a field to enter the date with 4 digit numbers
I made a kitchen timer to be displayed on the status bar!
I made a kind of simple image processing tool in Go language.
I tried to discriminate a 6-digit number with a number discrimination application made with python
I made a program to notify you by LINE when switches arrive