A story of using rembg to make a white background + person video into a black background video
Overview
Using ffmpeg, rembg, opencv, I changed the video of white background + person to black background + person video.
background
For the wedding entertainment, I decided to make a parody video of perfume. I wanted to make a video of people singing on a black background like a music video for perfume, but I didn't want to spend time buying a black cloth. She wanted to make the white background somehow black because she could shoot anywhere on a white background.
What it was like
This is,
It looks like this. Great accuracy. rembg is amazing. I tried various other person cropping methods, but this was the most natural.
- For mosaic, https://www.facepixelizer.com/jp/ I used.
Premise
--python is installed. --rembg is installed. (This installation is troublesome. Maybe I can explain it separately) --opencv is installed. --ffmpeg is installed and PATH is in place.
procedure
--Preparation of video files (MP4, etc., do not raise FPS more than necessary (processing becomes heavy)) --Disassemble video files into image files.
Command example (video frame image is output to raw folder): ffmpeg -i ~~ .mp4 -vcodec png raw \ image_% 05d.png
--Crop the person in the image file using rembg. It takes a lot of time because it targets all the images in the video.
Command example: rembg -a -ae 15 -o output1 \ image_252.png raw \ image_00001.png
https://github.com/danielgatis/rembg However, it was quite difficult to introduce. Difficult version of pytorch, torchvision, etc. I think it is better to build it in the virtual environment of pyenv or conda.
--Fill the transparent part after cropping the person with a different image or any color. I did it with opencv.
--Animate the image
Command example: ffmpeg -f image2 -r 30 -i image_% 03d.png -r 30 -an -vcodec libx264 -pix_fmt yuv420p video.mp4
Script used
Folder structure
The folder structure of the script used this time is as follows.
root/
├ raw_video/ #Folder to save raw videos
├ formatted_video/ #toMP4.Folder to save FPS and video quality adjusted with py
├ output/ #Folder to save the final generated video
├ mask/ #Folder to save the black background image
└ black.png #A 1920x1080 black image is prepared according to the size of the video. It doesn't matter if it's not black.
├ tmp1/ # formatted_Folder to save the image of the video
├ tmp2/ #Folder to save the image of tmp1 with the white background transparent with rembg
├ tmp3/ #tmp2 image and mask/black.Folder to save png images with opencv and black background
├ toMP4.py # raw_Adjust the FPS and image quality of the video video, formatted_Script to put in the video folder
└ main.py #Script that performs a series of processing
Code to make a full HD, 30FPS MP4 file (toMP4.py)
A script that generates ffmpeg commands in python and executes them.
import os
base_dir = "raw_video"
output_dir = "formatted_video"
fns = os.listdir(base_dir)
print(len(fns))
for f in fns:
print(f)
cmd = "ffmpeg.exe -i {} -s hd1080 -c:v libx264 -c:a copy -r 30 {} -y".format(
base_dir + "\\" + f, output_dir + "\\" + f + "__.mp4")
os.system(cmd)
Code that performs a series of processing (main.py)
--Disassemble video files into image files. --Crop the person in the image file using rembg. It takes a lot of time because it targets all the images in the video. --Fill the transparent part after cropping the person with a different image or any color. --Animate the image
import os
import shutil
import cv2
import matplotlib.pylab as plt
base_dir = "formatted_video"
tmp1_dir = "tmp1"
tmp2_dir = "tmp2"
tmp3_dir = "tmp3"
output_dir = "output"
fns = os.listdir(base_dir)
print(len(fns))
for f in fns:
bn = os.path.basename(f)
print(f)
#Create folder
shutil.rmtree(tmp1_dir)
os.makedirs(tmp1_dir)
shutil.rmtree(tmp2_dir)
os.makedirs(tmp2_dir)
shutil.rmtree(tmp3_dir)
os.makedirs(tmp3_dir)
# os.makedirs(output_dir)
# #Break down video files into image files
p1 = base_dir + "\\" + f
p2 = tmp1_dir + "\\" + bn + "_%05d.png "
cmd1 = "ffmpeg -i {} -vcodec png {}".format(p1, p2)
print(cmd1)
os.system(cmd1)
#Crop a person in an image file using rembg
fn2s = os.listdir(tmp1_dir)
for f2 in fn2s:
if(f2[-3:] != "png"):
continue
p3 = tmp1_dir + "\\" + f2
p4 = tmp2_dir + "\\" + f2
cmd2 = "rembg -a -ae 15 -o {} {}".format(p4, p3)
print(cmd2)
os.system(cmd2)
#Fill the transparent part after cropping the person with a different image or any color
fn2s = os.listdir(tmp2_dir)
for f2 in fn2s:
if(f2[-3:] != "png"):
continue
print('{}/{}'.format(tmp2_dir, f2))
frame = cv2.imread('mask/black1.png')
png_image = cv2.imread('{}/{}'.format(tmp2_dir, f2),
cv2.IMREAD_UNCHANGED) #Read with Alfa channel included
# png_image[:, :, 3:] = np.where(png_image[:, :, 3:] > 200, 255, 0)
x1, y1, x2, y2 = 0, 0, png_image.shape[1], png_image.shape[0]
frame[y1:y2, x1:x2] = frame[y1:y2, x1:x2] * (1 - png_image[:, :, 3:] / 255) + \
png_image[:, :, :3] * (png_image[:, :, 3:] / 255)
# plt.imshow(frame)
# plt.show()
cv2.imwrite('{}/{}'.format(tmp3_dir, f2), frame)
#Animate the image
cmd3 = "ffmpeg -f image2 -r 30 -i {} -r 30 -an -vcodec libx264 -pix_fmt yuv420p {}.mp4 -y".format(
tmp3_dir + "\\" + bn + "_%05d.png ", output_dir + "\\" + bn + "_bg_blk")
os.system(cmd3)
# break
It also contains extra code for debugging.
install rembg
--Create a virtual environment with Anaconda. (Python = 3.8)
- conda install pytorch===1.7.0 torchvision===0.8.1 torchaudio cpuonly -c pytorch
- pip install rembg==1.0.18 --pip install numpy == 1.19.3 → When installing rembg, numpy 1.19.4 was installed, but it didn't work well, so I changed it to 1.19.3. ――In addition, opencv is also installed. pip install opencv-python --In the virtual environment, execute rembg -o 〇〇.png ✕✕.png, and it is OK if the 〇〇.png file that is transparent except for the person in the ✕✕.jpg image is output.
Finally
Please note that the content is only about your own memo, so it may be difficult to understand.
Recommended Posts