Where are matrix products and inner products used in deep learning?

About matrix product and inner product

When I was studying mathematics, the word "inner product" came up. I investigated how this "inner product" is used in neural networks. The inner product is the product of vectors, but the matrix product is the product of matrices. The inner product seems to be called "dot product".

What is inner product?

-Meaning of inner product -[Mathematics] If you understand the meaning of "inner product" graphically, you can see various things. 1 -[Dot product](https://ja.wikipedia.org/wiki/dot product)

What is matrix product?

-Definition of matrix product and its reason

Example

Consider the following two-layer neural network.

Unknown-2.tiff

Matrix product formula

If you replace the output of the above Nural network with the matrix product formula, it looks like this.

\begin{bmatrix}
w_{11} & w_{12} & w_{13}\\
w_{21} & w_{22} & w_{23}
\end{bmatrix}

\cdot 

\begin{bmatrix}
x_{1} \\
x_{2} \\
x_{3} \\
\end{bmatrix}

Manual calculation

Apply the numbers specifically

\begin{bmatrix}
1 & 2 & 3\\
4 & 5 & 6
\end{bmatrix}

\cdot 

\begin{bmatrix}
1 \\
2 \\
3 \\
\end{bmatrix}

If you calculate by hand

\begin{bmatrix}
1 \times 1 + 2 \times 2 + 3 \times 3  \\
4 \times 1 + 5 \times 2 + 6 \times 3  \\
\end{bmatrix}

=

\begin{bmatrix}
14  \\
32  \\
\end{bmatrix}
 

In Numpy

import numpy as np
w = np.array([[1,2,3],[4,5,6]])
x = np.array([[1],[2],[3]])

w.dot(x) #=> array([[14],[32]])

w.dot (x) represents the matrix product of w and x. By the way, the method name is changed from "dot product" to dot.

In Tensorflow

import tensorflow as tf
w = tf.get_variable("weight" , initializer=tf.constant(np.array([[1,2,3],[4,5,6]]).astype("int32")))
x = tf.get_variable("x" , initializer=tf.constant(np.array([[1],[2],[3]]).astype("int32")))

Session initialization

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
sess.run(tf.matmul(w, x))  #=> array([[14],[32]])

tf.matmul (w, x) represents the matrix product of w and x. By the way, the method name is matmul from" matrix multiplication ".

Recommended Posts

Where are matrix products and inner products used in deep learning?
I installed and used the Deep Learning library Chainer
Used in machine learning EDA
(Important inner product in deep learning.) About the relationship between inner product, outer product, dot product, and numpy dot function.
numpy's matrix and mat are different
Classification and regression in machine learning
Organize machine learning and deep learning platforms
Deep learning / matrix product error backpropagation
Graph of the history of the number of layers of deep learning and the change in accuracy
Put your own image data in Deep Learning and play with it
Deep Learning from scratch-Chapter 4 tips on deep learning theory and implementation learned in Python
Modules and packages in Python are "namespaces"
Introduction to Deep Learning ~ Convolution and Pooling ~
Meaning of deep learning models and parameters
[python] Frequently used techniques in machine learning
Image recognition model using deep learning in 2016
Find and check inverse matrix in Python
"Deep Learning from scratch" in Haskell (unfinished)
I want to visualize where and how many people are in the factory
List of main probability distributions used in machine learning and statistics and code in python