Try Common Representation Learning with chainer

Overview

"[Joint multi-modal representations for e-commerce catalog search driven by visual attributes](https://kddfashion2016.mybluemix.net/kddfashion_finalSubmissions/Joint%20multi-modal%]" from IBM Research, India at the KDD2016 workshop. 20representations% 20for% 20e-commerce% 20catalog% 20search% 20driven% 20by% 20visual% 20attributes.pdf) ”is to be implemented by chainer.

The original dissertation method is here. The Implementation of this paper is written in Theano.

The content of the dissertation feels like a rough read, and when text and images are given as a pair, the common space is found using Neural Net (method name is "Correlational Neural Net, abbreviated as Corr Net") and is useful for search engines It is something like that.

I think that it is common to use CCA when seeking a common space of two different modals, but in practice scikit-learn CCA takes time to train as soon as the data size becomes large & it may not be usable due to Memory Error. There is, and this time I tried to implement it with chainer quickly.

The code written in jupyter notebook in python3 is ** here **.

It seems that the point is to train the correlation coefficient so that it becomes large in the common space.

Image diagram スクリーンショット 2017-03-27 11.29.47.png

The loss function consists of the loss of restoring both when two modals are given, the loss of restoring both when one modal is given, and the loss that makes the correlation in the hidden layer high. I have. スクリーンショット 2017-03-27 11.35.52.png

Data used

I wanted to try it easily, so I used MNIST to find a common space for 28x28 images and label information in one-hot-vector format.

I will try different data in the future.

result

Original image

スクリーンショット 2017-03-24 17.07.01.png

Image after restoration ("Image" → "Image")

スクリーンショット 2017-03-24 17.06.36.png

It seems that it can be restored successfully

Image after restoration using label ("Label" → "Image")

スクリーンショット 2017-03-24 17.06.52.png

Image after restoration of (0 and 8) by combining label information ("label" → "image")

スクリーンショット 2017-03-27 11.52.26.png

An image that looks like the middle between 0 and 8 is generated properly.

Transition of loss

Total loss.png

Detail

detail_loss.png

Learning is progressing so that the correlation is high in the hidden layer properly!

Consideration

It seems that you can restore just from the label information!

In any case, chainer is much easier to write than Theano and can output logs, which is convenient ^^

Recommended Posts

Try Common Representation Learning with chainer
Try deep learning with TensorFlow
Try implementing RBM with chainer.
Try Deep Learning with FPGA
Try machine learning with Kaggle
Now, let's try face recognition with Chainer (learning phase)
Try Deep Learning with FPGA-Select Cucumbers
Reinforcement learning 13 Try Mountain_car with ChainerRL.
Try deep learning with TensorFlow Part 2
Try horse racing prediction with Chainer
[Chainer] Learning XOR with multi-layer perceptron
Try machine learning with scikit-learn SVM
Reinforcement learning 8 Try using Chainer UI
Classify anime faces with deep learning with Chainer
Try Bitcoin Price Forecasting with Deep Learning
Try deep learning of genomics with Kipoi
Reinforcement learning 11 Try OpenAI acrobot with ChainerRL.
Seq2Seq (1) with chainer
Try scraping with Python.
Learning Python with ChemTHEATER 03
"Object-oriented" learning with python
Learning Python with ChemTHEATER 05-1
Use tensorboard with Chainer
Try to predict forex (FX) with non-deep machine learning
Now, let's try face recognition with Chainer (prediction phase)
Learning Python with ChemTHEATER 02
Learning Python with ChemTHEATER 01
Try SNN with BindsNET
Use scikit-learn training dataset with chainer (for learning / prediction)
Try regression with TensorFlow
Try to build a deep learning / neural network with scratch
[Evangelion] Try to automatically generate Asuka-like lines with Deep Learning
I tried to implement ListNet of rank learning with Chainer
Try to predict if tweets will burn with machine learning
Let's move word2vec with Chainer and see the learning progress
Machine learning learned with Pokemon
Try to factorial with recursion
Try function optimization with Optuna
Try using PythonTex with Texpad.
Play with reinforcement learning with MuZero
Try edge detection with OpenCV
Ensemble learning summary! !! (With implementation)
Try Google Mock with C
Try using matplotlib with PyCharm
Try programming with a shell!
Try GUI programming with Hy
Try Python output with Haxe 3.2
Try matrix operation with NumPy
Reinforcement learning 6 First Chainer RL
Reinforcement learning starting with Python
Try implementing XOR with PyTorch
Learn elliptical orbits with Chainer
Try running CNN with ChainerRL
About learning with google colab
Try various things with PhantomJS
Deep Kernel Learning with Pyro
Reinforcement learning 5 Try programming CartPole?
Use chainer with Jetson TK1
Linux fastest learning with AWS
Machine learning Minesweeper with PyTorch
Try running Python with Try Jupyter