y_1 = w_{1, 1}x_1 + w_{2, 1}x_2 + b_1 \\
y_2 = w_{1, 2}x_1 + w_{2, 2}x_2 + b_2

I think this itself is obvious if you look at the figure. Let's express this in a matrix representation.

\left(
  \begin{array}{c}
    y_1 \\
    y_2
  \end{array}
\right)
=
\left(
  \begin{array}{cc}
    w_{1, 1} & w_{2, 1} \\
    w_{1, 2} & w_{2, 2}
  \end{array}
\right)
\left(
  \begin{array}{c}
    x_1 \\
    x_2
  \end{array}
\right)
+
\left(
  \begin{array}{c}
    b_1 \\
    b_2
  \end{array}
\right)

I feel like this. If you understand matrix multiplication, you will find that it is an equivalent expression. By the way, regarding $ w_ {i, j} $, pay attention to the subscripts. You usually read subscripts like ** $ i $ row $ j $ column **, right? However, in the above formula, it looks like ** $ j $ row $ i $ column **. Let's transpose it as it is because it is difficult to handle theoretically and implementationally.

\left(
  \begin{array}{c}
    y_1 \\
    y_2
  \end{array}
\right)
=
\left(
  \begin{array}{cc}
    w_{1, 1} & w_{1, 2} \\
    w_{2, 1} & w_{2, 2}
  \end{array}
\right)^{\top}
\left(
  \begin{array}{c}
    x_1 \\
    x_2
  \end{array}
\right)
+
\left(
  \begin{array}{c}
    b_1 \\
    b_2
  \end{array}
\right) \\
\Leftrightarrow
\boldsymbol{Y} = \boldsymbol{W}^{\top}\boldsymbol{X} + \boldsymbol{B}

Transpose is also mentioned in here. For the time being, this completes the mathematical expression of the layer object with 2 inputs and 2 outputs. It's easy. Let's generalize this. That doesn't change. Mathematically

\boldsymbol{Y} = \boldsymbol{W}^{\top}\boldsymbol{X} + \boldsymbol{B}

It remains. Let's take a closer look at this. Considering the $ M $ input $ N $ output layer

\underbrace{\boldsymbol{Y}}_{N \times 1} = \underbrace{\boldsymbol{W}^{\top}}_{N \times M}\underbrace{\boldsymbol{X}}_{M \times 1} + \underbrace{\boldsymbol{B}}_{N \times 1}

It looks like. $ \ Boldsymbol {W} ^ {\ top} $ shows the shape after transposition. Before transposition, it's $ \ underbrace {\ boldsymbol {W}} _ {M \ times N} $. That is all for the theory. Now let's move on to implementation.

Forward propagation implementation in matrix

The implementation location is [baselayer.py](https://qiita.com/kuroitu/items/884c62c48c2daa3def08#layer module code preparation) as in the scalar implementation. Rewrite the implementation in scalar.

    def forward(self):
        """
Implementation of forward propagation
        """
        #Remember your input
        self.x = x.copy()

        #Forward propagation
        y = self.w.T @ x + self.b
        self.y = self.act.forward(y)
        
        return self.y

baselayer.py

        y = self.w * x + self.b

But

baselayer.py

        y = self.w.T @ x + self.b

It has become. For numpy arrays, transpose can be done with ndarray.T. The @ operator may be unfamiliar to some, but it is the same as np.dot. ** It is available in Numpy version 1.10 and above, so please be careful if you are using a lower version. ** **

x = np.array([1, 2])
w = np.array([[1, 0], [0, 1]])
b = np.array([1, 1])

y = w.T @ x + b
print(y)
print(y == np.dot(w.T, x) + b)

#----------
#The output is
# [2 3]
# [ True  True]
#It will be.

x = np.matrix([1, 2]).reshape(2, -1)
w = np.matrix([[1, 0], [0, 1]])
b = np.matrix([1, 1]).reshape(2, -1)

y = w.T @ x + b
print(y)
print(y == np.dot(w.T, x) + b)

#----------
#The output is
# [[2]
#  [3]]
# [[ True]
#  [ True]]
#It will be.

Implementation of the __init__ method

By the way, there are some members that the layer object should have. Let's implement this. In addition, I will have some members.

    def __init__(self, *, prev=1, n=1, 
                 name="", wb_width=1,
                 act="ReLU",
                 **kwds):
        self.prev = prev  #Number of outputs of the previous layer=Number of inputs to this layer
        self.n = n        #Number of outputs in this layer=Number of inputs to the next layer
        self.name = name  #The name of this layer
        
        #Set weight and bias
        self.w = wb_width*np.random.randn(prev, n)
        self.b = wb_width*np.random.randn(n)
        
        #Activation function(class)Get
        self.act = get_act(act)

About matrix operation

Here, we will briefly introduce matrix operations. Please note that I will not explain anything mathematically (I can not do it) just because it is calculated like this.

Matrix sum

First is the matrix sum.

\left(
  \begin{array}{cc}
    a & b \\
    c & d
 \end{array}
\right)
+
\left(
  \begin{array}{cc}
    A & B \\
    C & D
  \end{array}
\right)
=
\left(
  \begin{array}{cc}
    a + A & b + B \\
    c + C & d + D
  \end{array}
\right)

Well, it's a natural result. Add for each element. Not to mention addition ** The shape of the matrix must match exactly. ** **

Matrix multiplication

I will also touch on the element product that is not mentioned at all in this article. The element product is also called the Hadamard product.

\left(
  \begin{array}{cc}
    a & b \\
    c & d
 \end{array}
\right)
\otimes
\left(
  \begin{array}{cc}
    A & B \\
    C & D
  \end{array}
\right)
=
\left(
  \begin{array}{cc}
    aA & bB \\
    cC & dD
  \end{array}
\right)

By the way, the Hadamard product symbol can be written as \ otimes. There are many other symbol-related items on here.

Matrix product

This is a matrix product, not an element product. It is a big difference from the element product that there are restrictions on the shape and that it is not commutative.

\left(
  \begin{array}{cc}
    a & b \\
    c & d
 \end{array}
\right)
\left(
  \begin{array}{cc}
    A & B \\
    C & D
  \end{array}
\right)
=
\left(
  \begin{array}{cc}
    aA + bC & aB + bD \\
    cA + dC & cB + dD
  \end{array}
\right)

As a calculation method, it feels like rolling horizontally $ \ times $. It is like this. In order to be able to calculate in this way, the number of elements of ** horizontal ** </ font> and the number of elements of ** vertical ** </ font> Must match. In other words, the ** column ** </ font> of the first matrix and the ** row ** </ font> of the second matrix are one. You need to do it. Generalized, the result of the matrix product of the ** $ L \ times M $ matrix and the $ M \ times N $ matrix is $ L \ times N $. ** **

\left(
  \begin{array}{cc}
    a & b & c \\
    d & e & f
 \end{array}
\right)^{\top}
=
\left(
  \begin{array}{cc}
    a & d \\
    b & e \\
    c & f
 \end{array}
\right)