Python: Deep Learning Practices

Overview of deep learning

Code is written using a "framework" that can streamline programming. For deep learning frameworks "TensorFlow" developed by Google in the United States There are various types such as "PyTorch" developed by Facebook in the United States.

Here, we made TensorFlow and TensorFlow even easier to use. Use the framework "Keras".

Examples of deep learning

Number of epochs (represents how many times one learning data is repeatedly trained) As it goes up

Correct answer rate acc for training data Correct answer rate val_acc for test data Let's check how it is going up.

import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import mnist
from keras.layers import Activation, Dense, Dropout
from keras.models import Sequential, load_model
from keras import optimizers
from keras.utils.np_utils import to_categorical

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train.reshape(X_train.shape[0], 784)[:6000]
X_test = X_test.reshape(X_test.shape[0], 784)[:1000]
y_train = to_categorical(y_train)[:6000]
y_test = to_categorical(y_test)[:1000]

model = Sequential()
model.add(Dense(256, input_dim=784))
model.add(Activation("sigmoid"))
model.add(Dense(128))
model.add(Activation("sigmoid"))
model.add(Dropout(rate=0.5))
model.add(Dense(10))
model.add(Activation("softmax"))

sgd = optimizers.SGD(lr=0.1)
model.compile(optimizer=sgd, loss="categorical_crossentropy", metrics=["accuracy"])

#Specify 5 for the number of epochs
history = model.fit(X_train, y_train, batch_size=500, epochs=5, verbose=1, validation_data=(X_test, y_test))

#acc, val_acc plot
plt.plot(history.history["acc"], label="acc", ls="-", marker="o")
plt.plot(history.history["val_acc"], label="val_acc", ls="-", marker="x")
plt.ylabel("accuracy")
plt.xlabel("epoch")
plt.legend(loc="best")
plt.show()

What is deep learning? (1)

Deep learning, also known as deep learning, is a type of machine learning that goes one step further. Referencing biological neural networks to classify and regress data We use a model called a deep neural network.

However, the purpose of deep learning is not to reproduce the neural network of the brain. Research is being conducted with the goal of improving the recognition accuracy of images and sounds.

Realized by automatic driving technology using image recognition and video analysis, and voice analysis Smart speakers such as "Google Home" and "Siri" This technology is used in many aspects of our daily lives.

One of the reasons why deep learning is attracting attention is to automate the work that people have traditionally done. It saves time and effort and improves accuracy.

For example, when creating a program that detects a "car" from an image In machine learning, detailed information such as the characteristics of various types of car tires and windshields is prepared in advance. I had to make a model that trained it.

However, deep learning can automatically find the features of car parts without such a model.

What is deep learning? (2)

The figure below shows the neurons that make up a neural network. x1 and x2 are the input values and w1 and w2 are the weight parameters.

In this model, if the value of x1w1 + x2w2 is higher than the threshold θ, the neuron fires. Outputs 1 and 0 otherwise.

When the input exceeds the threshold, the neuron transmits information to the next neuron.

This is called neuron firing.

Neural networks receive input values such as vectors and matrices It finally outputs values such as scalars and vectors through a chain of firing.

If you compare this with image recognition, you can input pixel data of an animal image. It is an image that outputs information about which category (cat, dog, bird, etc.) has a high probability of belonging.

Then, as shown in the figure below, the layers of neurons were stacked so that more complex problems could be handled. It is a deep neural network. In particular, it often refers to a structure with three or more layers. It was named "Deep" because of the depth of its structure.

Actually, the mathematical model called neural network It has already existed since the 1950s.

And in recent years, a computing environment that can utilize deep neural networks has been set up. As research on learning methods has progressed, it has attracted considerable attention.

In deep learning, the weight parameters of each neuron are mechanically adjusted. Create a classification model or regression model.

Classification flow using deep learning

Let's look at the flow of classification using deep learning.

① Make a network model

By stacking layers that bundle several neurons as shown in the figure Build a deep network model.

Initially, the neuron responds randomly to the input, outputting meaningless values.

② Give the model training data and perform learning

The model takes the input value X and outputs the output value y.

At this time, make sure that ΔE, which is the difference between y and the correct data (teacher label) T, becomes small. Neuron weights are automatically adjusted using a method called error backpropagation.

Then, the weight is repeatedly adjusted by giving the raw data X such as an image and the correct answer data T. Gradually you will be able to obtain the desired output value.

If the learning progresses well, you will have a model that returns appropriate predicted values.

③ Pass the classification data to the model

Data can be classified by deep learning according to the flow shown in the figure. In short, deep learning allows you to model classifications and regressions without having to understand the details of the procedure.

Classification of handwritten numbers

Flow to classification

Let's implement the following neural network model using a library called Keras. First, the following procedure is used to classify handwritten numbers, which can be said to be a standard in deep learning.

1, prepare the data 2, Build a neural network model 3, give data to the model and train it 4, evaluate the classification accuracy of the model Finally, give an image of handwritten numbers to see the predicted values.

Deep neural network

A neural network with a certain depth

Called a deep neural network

Input layer: Input layer Output layer: Output layer Hidden layer: Between the input layer and the output layer It's called.

This time, all neurons were connected to neurons in the previous layer

Fully connected layer

Create a neural network with a simple structure with two.

When the input handwritten image data is "7" The correct label of the teacher data is 1 for the element corresponding to 7, and 0 for the other elements.

Data with all zero values except one element

one-It is called a hot vector.

Since this time we are recognizing numbers, the 0th to 9th elements represent the letters "0" to "9" respectively.

If this is image recognition such as "what kind of animal is in the picture" The 0th to Nth elements correspond to each animal that is a recognition option. Such classifications (classes) are assigned in order and expressed numerically, which is called a "class label".

Node unit

The data to be input is a monochrome image with 28 pixels in height and width. Even an image is called a vector because it looks like a collection of 28 x 28 numbers to a computer.

The element represented by "○" that receives a vector and performs calculation

It is called a "node" or "unit".

Counting from the input side, the units arranged vertically in the above figure

It is called a "layer" or "layer".

This is how it is called when it is organized so that it is easy for humans to see. From where to where the units are in the same layer will change from time to time.

This time

model.add(Dense(256, input_dim=784))
#I made an input layer. 256 is the number of units.
# input_In dim, the number of elements of the input vector is specified.
# input_dim may be treated as the number of columns.

Since 28 × 28 = 784, we specify that a vector with 784 elements will be input.

I decided to express which character the output is with probability. Probability is output for each of 10 types from 0 to 9. Prepare 10 units that accept all the values output by the hidden layer units and output the probability.

Introduction of Keras

Survey conducted by data mining information site KD Nuggets In Deep Learning Framework Power Scores 2018 The results show that Keras is the second most popular after TensorFlow.

TensorFlow is an open source software library for machine learning provided by Google. Keras is a TensorFlow wrapper library that allows you to write code more intuitively and concisely.

To put it plainly, Keras is somewhere between TensorFlow and code, rather than writing TensorFlow directly. You can reduce the code to a fraction.

The wrapper in this case means that the system is included to make it easier to use.

Data preparation

Learn handwritten numbers on the Yann LeCun's website

MNIST(Emnist)Use the dataset.

MNIST can also be downloaded by running the code below in Keras.

from keras.datasets import mnist
(X_train, y_train), (X_test, y_test) = mnist.load_data()

In the code, X is the image data and y is the teacher label data. train is the training data, and test is the data used to evaluate the performance of the model.

However, there is no essential difference between train data and test data. You can also read the data by executing MNIST in the downloaded state.

Let's output the size of each of X_train, y_train, X_test, and y_test.

from keras.datasets import mnist

(X_train, y_train), (X_test, y_test) = mnist.load_data()

print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)

#Output result

(60000, 28, 28) (60000,) (10000, 28, 28) (10000,)

Model generation

In Keras, first create an instance to manage the model

Define it layer by layer with the add method.

Create an instance.

from keras.models import Sequential
from keras.layers import Dense, Activation

model = Sequential()

This time we define a fully connected layer of 128 units. Define the model layer by layer using the add method as shown below.

model.add(Dense(128))

A function called the activation function is applied to the output of the fully connected layer.

model.add(Activation("sigmoid"))

This is a mechanism equivalent to the firing of nerves in animals.

Set the sigmoid function sigmoid and the ReLU function relu.

And finally

Compile method compile()The training process is set in, and the model generation is completed.

model.compile(optimizer=sgd, loss="categorical_crossentropy", metrics=["accuracy"])
#We will discuss these functions and parameters later.

If you give one example

First, get an image of building a network model. In the code below, after generating a network model with one hidden layer. Generate a model like the one below, which has two hidden layers. Use the ReLU function relu as the activation function.

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Activation
from keras.utils.vis_utils import plot_model
from keras.utils.np_utils import to_categorical
import matplotlib.pyplot as plt

(X_train, y_train), (X_test, y_test) = mnist.load_data()

# (◯, 28, 28)Data(◯, 784)Dimensionality reduction.(Reduce the number of data for simplicity)
shapes = X_train.shape[1] * X_train.shape[2]
X_train = X_train.reshape(X_train.shape[0], shapes)[:6000]
X_test = X_test.reshape(X_test.shape[0], shapes)[:1000]
y_train = to_categorical(y_train)[:6000]
y_test = to_categorical(y_test)[:1000]

model = Sequential()
#The number of input units is 784,The number of output units in the first fully connected layer is 256
model.add(Dense(256, input_dim=784))
model.add(Activation("sigmoid"))

#The number of output units in the second fully connected layer is 128. The activation function is relu.
#　---------------------------
#It is creating two hidden layers.
model.add(Dense(128))
model.add(Activation("relu"))

#　---------------------------

#The number of output units in the third fully connected layer (output layer) is 10.
model.add(Dense(10))
model.add(Activation("softmax"))

model.compile(optimizer="sgd", loss="categorical_crossentropy", metrics=["accuracy"])

#Model structure output
plot_model(model, "model125.png ", show_layer_names=False)
#Visualization of model structure
image = plt.imread("model125.png ")
plt.figure(dpi=150)
plt.imshow(image)
plt.axis('off')
plt.show()

Model learning

#Training is done by using the fit method and passing training data to the model.
model.fit(X_train, y_train, verbose=1, epochs=3)

In the fit method, the training data is input to the model in order. Update the weight of each neuron so that the difference between the output and the teacher data is small We will improve the prediction accuracy of the model.

(1) As a return value, a History object that holds a record of learning is returned.
② Argument X_train is training data, y_train is teacher data.
③ 1 for verbose(Or True)If is specified, the learning progress is output, and if it is 0, it is not output.
④ epochs specifies the number of trainings to be performed on the same data set.

Here is an example.

from keras.datasets import mnist
from keras.layers import Activation, Dense
from keras.models import Sequential
from keras import optimizers
from keras.utils.np_utils import to_categorical
import matplotlib.pyplot as plt

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train.reshape(X_train.shape[0], 784)[:6000]
X_test = X_test.reshape(X_test.shape[0], 784)[:1000]
y_train = to_categorical(y_train)[:6000]
y_test = to_categorical(y_test)[:1000]

model = Sequential()
model.add(Dense(256, input_dim=784))
model.add(Activation("sigmoid"))
model.add(Dense(128))
model.add(Activation("sigmoid"))
model.add(Dense(10))
model.add(Activation("softmax"))

model.compile(optimizer="sgd", loss="categorical_crossentropy", metrics=["accuracy"])
#---------------------------
history = model.fit(X_train, y_train, verbose=True, epochs=3)
#---------------------------
#acc, val_acc plot
plt.plot(history.history["acc"], label="acc", ls="-", marker="o")
plt.ylabel("accuracy")
plt.xlabel("epoch")
plt.legend(loc="best")
plt.show()

Evaluation of the model

Even if the tuning by learning is completed and the accuracy of the model is improved It is possible that its performance can only be demonstrated with training data. Therefore, the trained data cannot be used to correctly evaluate the performance of the model.

Therefore, the model is evaluated using test data that has not been used for training. The accuracy at this time

Generalization accuracy(Accuracy for new data)It's called.

For calculation of generalization accuracy

Use the evaluate method.

score = model.evaluate(X_test, y_test, verbose=1)

X_test is the input data for evaluation (test), and y_test is the teacher data. The value of the loss function and the accuracy rate obtained by the evaluate method are stored in score.

The test data is for calculating generalization accuracy and is not used for training.

import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import mnist
from keras.layers import Activation, Dense
from keras.models import Sequential, load_model
from keras import optimizers
from keras.utils.np_utils import to_categorical

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train.reshape(X_train.shape[0], 784)[:6000]
X_test = X_test.reshape(X_test.shape[0], 784)[:1000]
y_train = to_categorical(y_train)[:6000]
y_test = to_categorical(y_test)[:1000]

model = Sequential()
model.add(Dense(256, input_dim=784))
model.add(Activation("sigmoid"))
model.add(Dense(128))
model.add(Activation("sigmoid"))
model.add(Dense(10))
model.add(Activation("softmax"))

model.compile(optimizer="sgd", loss="categorical_crossentropy", metrics=["accuracy"])

model.fit(X_train, y_train)

# ---------------------------
score = model.evaluate(X_test, y_test, verbose=True)
# ---------------------------
print("evaluate loss: {0[0]}\nevaluate acc: {0[1]}".format(score))

Classification by model

Use the predict method to get the predicted value of the input data.

predict(self, x, batch_size=None, verbose=0, steps=None)

argument

①　x:Input data. The format of the Numpy array.
②　batch_size:integer. The default is 32.
③　verbose:Progress message output mode. 0 or 1.
④　steps:Total number of steps (sample batch) before declaring the end of the evaluation round. If None (default value), it will be ignored.

Return value

Numpy array containing predicted values

For example, when predicting the numbers of two images of X_test, it will be as follows.

import numpy as np
#Prepare a trained model.
model = ...

#Predict
pred = np.argmax(model.predict(X_test[0:2]))
print("Predicted value:" + str(pred))

Note that predict assumes multiple images. It is necessary to pay attention to the dimension when predicting only one sheet. If you want to predict only one It is necessary to devise to describe the argument as X_test [0] .reshape (1, -1).

In MNIST, the output of the predict method is 10 dimensions Use the argmax function to return the index of the maximum value of the array Gets the location of the neuron that returns the highest value.

import numpy as np
x = np.array([[0, 1, 2], [-1, 1000, 1]])
print(np.argmax(x)) #Returns 4.
print(np.argmax(x, axis=0)) # [0, 1, 0]Returns.
print(np.argmax(x, axis=1)) # [2, 1]Returns.

The argmax function

(1) Specify the axis with the optional argument axis and give it with the first argument
Check the index of the maximum value for the array.

② If axis is not specified, the array will be one-dimensional.
Returns the index of the maximum value when reshaped.

After learning, let's output the predicted value of X_test [0:10].

import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import mnist
from keras.layers import Activation, Dense
from keras.models import Sequential, load_model
from keras.utils.np_utils import to_categorical

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train.reshape(X_train.shape[0], 784)[:6000]
X_test = X_test.reshape(X_test.shape[0], 784)[:1000]
y_train = to_categorical(y_train)[:6000]
y_test = to_categorical(y_test)[:1000]

model = Sequential()
model.add(Dense(256, input_dim=784))
model.add(Activation("sigmoid"))
model.add(Dense(128))
model.add(Activation("sigmoid"))
model.add(Dense(10))
model.add(Activation("softmax"))

model.compile(optimizer="sgd", loss="categorical_crossentropy", metrics=["accuracy"])

model.fit(X_train, y_train, verbose=True)

score = model.evaluate(X_test, y_test, verbose=False)
print("evaluate loss: {0[0]}\nevaluate acc: {0[1]}".format(score))

#Display the first 10 sheets of test data
for i in range(10):
    plt.subplot(1, 10, i+1)
    plt.imshow(X_test[i].reshape((28,28)), "gray")
plt.show()

# X_Shows the first 10 predicted labels of test
#---------------------------
pred = np.argmax(model.predict(X_test[0:10]), axis=1)
print(pred)
#---------------------------