Video frame interpolation by deep learning Part1 [Python]

I am dealing with ** video frame interpolation by deep learning ** at the university, and I will output the implementation I am trying in the process. I will continue to post the continuation of the implementation of frame interpolation for videos, so please follow LGTM & follow me if you like.

What I did this time is to build a network that uses actual video frames to generate 1 intermediate frame from 6 frames before and after.

Implementation environment

Google Colab https://colab.research.google.com/notebooks/welcome.ipynb?hl=ja

Implementation overview

** Deep learning that generates an intermediate frame from the front and rear frames (3 front and 3 rear). ** The network is DnCNN [1]. I have this network at hand, so I am using it. ([1] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang, “Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising”, https://arxiv.org/abs/1608.03981)

image.png

The network of DnCNN is as follows. Originally, it was intended to remove noise. For input, the frame size is 160 * 90 and the number of channels is 18 channels (6 frames * RBG). For output, the frame size is similar and the number of channels is 3.

I messed with the parameters of the blue middle layer. It has 15 layers, a kernel size of 3 * 3, and 72 channels.

image.png

data set

I used MOT17 which took a picture of the city. https://motchallenge.net/ The number of sets is train 1320, test 1285.

result

The first image is the front 2 frames, the generated intermediate frame, and the rear 2 frames from the top. Actually, there is another input for each frame, but I omitted it because the image becomes small.

no5-2_series.PNG

The following image is a comparison with the correct middle frame.

no5-2_midonly.PNG

The result is that it cannot be said that it can be interpolated because it is pulled back and forth and the color is changing.

Performance evaluation

Click here for a graph of generalization performance. Since the values are close to each other, there seems to be no problem here. no5-1_hanka.PNG Click here for numerical data such as loss values. image.png

The numerical value is not so bad. It is close to the loss value and average PSNR when one image was previously cropped and pseudo frame interpolation was performed. However, this seems to be because the front and back images are almost the same. The mid-top rate is low, so you need to raise it here.

Consideration

The reason why it could not be interpolated ・ Small number of data sets ・ Input 6 frames are not working well ・ Network (DnCNN) problems I think there are three points.

There are about 1300 sets of training and test data. The number of original images is large, but it is difficult to earn the amount because one set consumes 7 frames. I'm in the process of creating my own dataset, so I'd like to keep an eye on the number of datasets.

What about 6 input frames? .. Every paper looks at the interpolation in 2 frames before and after, so I'm worried if it will continue to work in 6 frames. I started thinking that I should go back to 2 frames for comparison.

Plan from now on

-Increase the number of sets with your own data set.

Finally

Thank you for reading until the end. Please do not hesitate to point out any improvements. I will continue to post this system, so please follow LGTM & follow me if you like!

Recommended Posts

Video frame interpolation by deep learning Part1 [Python]
Python Deep Learning
Deep learning × Python
Python: Gender Identification (Deep Learning Development) Part 1
Python: Gender Identification (Deep Learning Development) Part 2
Python: Deep Learning Practices
Python: Deep Learning Tuning
Python: Supervised Learning: Hyperparameters Part 1
Python: Supervised Learning: Hyperparameters Part 2
Try deep learning with TensorFlow Part 2
Machine learning summary by Python beginners
Deep learning learned by implementation 1 (regression)
Interval scheduling learning memo ~ by python ~
(python) Deep Learning Library Chainer Basics Basics
Deep learning learned by implementation 2 (image classification)
[Learning memo] Basics of class by python
Python vs Ruby "Deep Learning from scratch" Chapter 2 Logic circuit by Perceptron
Produce beautiful sea slugs by deep learning
Deep Understanding Object Detection by Deep Learning by Keras
python learning
Deep Learning
Python & Machine Learning Study Memo ④: Machine Learning by Backpropagation
"Deep Learning from scratch" Self-study memo (Part 12) Deep learning
Basic data frame operations written by beginners in a week of learning Python
I tried to classify Oba Hana and Emiri Otani by deep learning (Part 2)
[Python / Machine Learning] Why Deep Learning # 1 Perceptron Neural Network
Classification of guitar images by machine learning Part 1
Machine learning starting with Python Personal memorandum Part2
Machine learning starting with Python Personal memorandum Part1
Pattern recognition learning in video Part 1 Field of Pattern Recognition
Deep learning learned by implementation ~ Anomaly detection (unsupervised learning) ~
EV3 x Python Machine Learning Part 2 Linear Regression
99.78% accuracy with deep learning by recognizing handwritten hiragana
[Note] Python, when starting machine learning / deep learning [Links]
Stock Price Forecast Using Deep Learning (TensorFlow) -Part 2-
Python: Deep Learning in Natural Language Processing: Basics
Parallel learning of deep learning by Keras and Kubernetes
Python vs Ruby "Deep Learning from scratch" Summary
Classification of guitar images by machine learning Part 2
Deep learning learned by implementation (segmentation) ~ Implementation of SegNet ~
First deep learning in C #-Imitating implementation in Python-
QGIS + Python Part 2
[Python] Learning Note 1
Python learning notes
Report_Deep Learning (Part 2)
Report_Deep Learning (Part 1)
Report_Deep Learning (Part 1)
QGIS + Python Part 1
python learning output
Start Deep learning
Report_Deep Learning (Part 2)
Python learning site
Python learning day 4
Python learning (supplement)
Python: Scraping Part 1
python learning notes
Python3 Beginning Part 1
Python: Scraping Part 2
Stock investment by deep reinforcement learning (policy gradient method) (1)
[Programming learning] Logic comparison by language part.1 ("Hello World")
[Programming learning] Logic comparison by language part.3 (conditional branching)