Perceptron basics and implementation

Learning Perceptron

Network to use

-A network consisting of three layers: an input layer, an intermediate layer, and an output layer. ・ The layers are fully connected. -All neurons output $ 1 $ or $ 0 $.

image.png

The purpose of the perceptron is to learn the weight (synaptic weight) between the intermediate layer and the output layer and generate an output pattern corresponding to the input pattern.

Input layer

Suppose you have $ M $ neurons. It receives the input from the outside as it is and outputs it. The output of the $ i $ th neuron is expressed by the following formula. $output_i = input_i$

Middle layer

Suppose you have $ N $ neurons. The input given to the $ j $ th neuron in the middle layer is the sum of the output values of all input layer neurons multiplied by the synaptic weight $ w_ {i, j} . $input_j=\sum_{i=1}^{M}{w_{i,j}output_i}$$

Then, set the threshold $ \ theta_j $ for the signal received by each middle layer neuron, and reduce it by that amount. $input_j-\theta_j$

Since the output is only $ 0 $ or $ 1 $, set the output function $ f (x) $. Therefore, the output value of the $ j $ th neuron in the middle layer is expressed by the following formula. $output_j = f(input_j-\theta_j)$

f(u) = \left\{
\begin{array}{ll}
1 & (u \gt 0) \\
0 & (u \leq 0)
\end{array}
\right.

Output layer

The inputs of the output layer, like the inputs of the middle layer, are the output values of all neurons in the previous layer multiplied by synaptic weights. Let the number of neurons in the output layer be $ 1 $.

input_o=\sum_{j=1}^{N}{w_{j,o}output_j}

And the output of the output layer is the result of applying $ f (x) $ to the value obtained by subtracting the threshold value from the input, as in the intermediate layer.

output_o=f(input_o-\theta_o)
f(u) = \left\{
\begin{array}{ll}
1 & (u \gt 0) \\
0 & (u \leq 0)
\end{array}
\right.

Learning

During training, only the synaptic weight $ w_ {j, o} $ between the output layer and the intermediate layer is updated without changing other parameters.

\Delta w_{j,o}=\eta(t_o-output_o)output_j
w^{t+1}=w^{t}+\Delta w_{j,o}

$ \ eta $ is the learning rate, and it is common to set a small positive value. $ t_o-output_o $ is the difference between the teacher data output $ t_o $ and the actual output data $ output_o $. Therefore, it can be seen that the synaptic weight is updated only when the calculated result and the teacher signal are different.

The explanation and proof of Perceptron's convergence theorem are omitted here. If you are interested, please check it out.

Implementation

parameter settings

code

Library installation and input path settings

import numpy as np
import matplotlib.pyplot as plt

PATH_X = "./../input_x.npy"
PATH_Y = "./../input_y.npy"

Convert input data from $ (x, y) $ to $ 0 $ and $ 1 $ columns of length $ 8 $

def to_input(data):
    x = data[0]
    y = data[1]
    n = x * 16 + y
    return np.array([int(k) for k in format(n, '08b')])

Perceptron class __Caution! __ The load calculation in the program is calculated by considering the formula explained above as a vector.

class Perceptron:
    def __init__(self, m, n, o):
        # decide initial weight [-0.005,0.005)
        #I added 1 to handle the threshold easily
        self.w_IM = np.random.rand(n,m+1) - 0.5
        self.w_IM = self.w_IM / 100
        self.w_MO = np.random.rand(o,n+1) - 0.5
        self.w_MO = self.w_MO / 100

    # calculate accuracy
    def get_acc(self, x, y):
        ok = 0
        for i in range(len(x)):
            #I'm adding a neuron that always outputs 1
            mid_in = np.inner(np.append(x[i],1.), self.w_IM)
            mid_out = np.array([int(k > 0) for k in mid_in])
            #I'm adding a neuron that always outputs 1
            out_in = np.inner(np.append(mid_out,1.), self.w_MO)
            ok += int(int(out_in[0] > 0) == y[i])
        return ok / len(x)

    def learn(self, train_x, train_y, eta = 0.00001):
        #I'm adding a neuron that always outputs 1
        mid_in = np.inner(np.append(train_x,1.), self.w_IM)
        mid_out = np.array([int(k > 0) for k in mid_in])
        #I'm adding a neuron that always outputs 1
       out_in = np.inner(np.append(mid_out,1.), self.w_MO)
        out = int(out_in[0] > 0)

        #Updating loads from output and teacher data values
        self.w_MO[0,:-1] = self.w_MO[0,:-1] + eta * (train_y - out) * mid_out

Parameter setting and result graph drawing

def main():
    # read datas
    x = np.load(PATH_X)
    y = np.load(PATH_Y)
    # split datas
    train_x, test_x = np.split(x, 2)
    train_y, test_y = np.split(y, 2)
    # preprocess - transfer data into inputs
    datas = np.array([to_input(k) for k in train_x])
    tests = np.array([to_input(k) for k in test_x])
    # number of neurons input layer
    m = 8
    # number of neurons mid layer
    n = 10
    # number of neurons output layer
    o = 1
    # define the perceptron
    P = Perceptron(m,n,o)
    
    # learning time
    N = 10
    cnt = 0

    x = np.linspace(0,200,200)
    acc_train = np.copy(x)
    acc_test = np.copy(x)
    while True:
        acc = P.get_acc(datas, train_y)
        acc_train[cnt] = acc
        acc = P.get_acc(tests, test_y)
        acc_test[cnt] = acc
        print("Try ", cnt, ": ", acc)
        cnt += 1
        for i in range(len(datas)):
            P.learn(datas[i], train_y[i])
        if cnt >= 200:
            break
    plt.plot(x,acc_train,label="train")
    plt.plot(x,acc_test,label="test")
    plt.savefig("result.png ")

if __name__ == "__main__":
    main()

It is also uploaded to Github. https://github.com/xuelei7/NeuralNetwork/tree/master/Perceptron

result

For $ 30 $ middle layer neurons:

For $ 100 $ middle layer neurons:

in conclusion

If there are any improper points, I would like to correct them. We apologize for the inconvenience, but please contact the author.

Reference material

"Neural Network", Yasunari Yoshitomi, Asakura Shoten,

Recommended Posts

Perceptron basics and implementation
Search basics and Java implementation / measurement
Explanation and implementation of simple perceptron
Basics of Perceptron Foundation
Explanation and implementation of SocialFoceModel
Normalizing Flow theory and implementation
Python basics: conditions and iterations
Maxout description and implementation (Python)
[Day 5] View and template basics
Explanation and implementation of PRML Chapter 4
Introduction and Implementation of JoCoR-Loss (CVPR2020)
Explanation and implementation of ESIM algorithm
Introduction and implementation of activation function
Simple neural network theory and implementation
Image recognition environment construction and basics
perceptron
Open3D basics and point cloud voxels
Implementation and experiment of convex clustering method
Statistical basics and Python, graphing, etc. (memo)
Implementation module "deque" in queue and Python
PointNet theory and implementation (point cloud data)
Implementation and explanation using XGBoost for beginners
Machine learning algorithm classification and implementation summary
Matplotlib Basics / Differences between fig and axes
Gaussian process regression Numpy implementation and GPy
Learn while implementing with Scipy Logistic regression and the basics of multi-layer perceptron