[Python] I tried the same calculation as LSTM predict with from scratch [Keras]

1. Overview

2. Data and model preparation

data

X = np.arange(24).reshape(4,3,2)
y = np.array([[0,1],[0,1],[0,1],[1,0]])
print(X.shape)
#=> (4, 3, 2)
print(y.shape)
#=> (4, 2)
print(X[0])
#=> [[0 1]
#    [2 3]
#    [4 5]]

model

from keras.layers import Input, Dense
from keras.models import Model
from keras.layers.recurrent import LSTM
import tensorflow as tf
from keras import backend

tf.reset_default_graph()
backend.clear_session()

inputs = Input(shape=[3,2])
x = LSTM(8, activation='tanh', recurrent_activation='sigmoid')(inputs)
outputs = Dense(2, activation='softmax')(x)
model = Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='adam',loss='categorical_crossentropy')
model.summary()
Screen Shot 2020-01-18 at 18.56.23.png
history = model.fit(X, y, epochs=2, verbose=1)

3. Understanding get_weights ()

for weight in model.get_weights():
    print(weight.shape)
#=> (2, 32)
#   (8, 32)
#   (32,)
#   (8, 2)
#   (2,)

model.get_weights (): Returns a list of all model weight tensors with Numpy arrays as elements.

W = model.get_weights()[0]
U = model.get_weights()[1]
b = model.get_weights()[2]
dense_W = model.get_weights()[3]
dense_b = model.get_weights()[4]

W_i, W_f, W_tC, W_o = W[:,:8], W[:,8:16], W[:,16:24], W[:,:24:]
U_i, U_f, U_tC, U_o = U[:,:8], U[:,8:16], U[:,16:24], U[:,:24:]
b_i, b_f, b_tC, b_o = b[:8], b[8:16], b[16:24], b[24:]

4. Calculate!

#We will calculate using the first sample.
_X = X[0]

#Define the activation function.
def sigmoid(x):
    return(1.0/(1.0+np.exp(-x)))
def relu(x):
    ret_x = x
    ret_x[ret_x<0] = 0
    return ret_x

#First C,The values of h are all 0.
C = np.zeros((1,8))
h = np.zeros((1,8))

#LSTM part
for i in range(len(_X)):
    x_t = _X[i]
    i_t = sigmoid(np.dot(x_t,W_i) + np.dot(h,U_i) + b_i)
    f_t = sigmoid(np.dot(x_t,W_f) + np.dot(h,U_f) + b_f)
    tC = np.tanh(np.dot(x_t,W_g) + np.dot(h,U_g) + b_g)
    o_t = sigmoid(np.dot(x_t,W_o) + np.dot(h,U_o) + b_o)
    C = f_t*C + i_t*tC
    h = np.tanh(C) * o_t

#Dense part
output = np.dot(h,dense_W) + dense_b

#softmax calculation
E = []
Esum = 0
for i in range(2):
    E.append(np.exp(output[0,i]))
    Esum += np.exp(output[0,i])
result = []
for i in range(2):
    result.append(E[i]/Esum)
    
print(result)
#=> [0.5211381547054326, 0.4788618452945675]
print(model.predict(_X.reshape(1,3,2)))
#=> [[0.5211382  0.47886187]]

5. [Extra] activation and recurrent_activation

Check activation and recurrent_activation.
  • First, check activation.
#Create a model with activation changed to relu.
tf.reset_default_graph()
backend.clear_session()

inputs = Input(shape=[3,2])
x = LSTM(8, activation='relu', recurrent_activation='sigmoid')(inputs) # relu!
outputs = Dense(2, activation='softmax')(x)
model = Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='adam',loss='categorical_crossentropy')
history = model.fit(X, y, epochs=2, verbose=1)


#Compute the output of the forecast using numpy.
C = np.zeros((1,8))
h = np.zeros((1,8))
_X = X[0]

W = model.get_weights()[0]
U = model.get_weights()[1]
b = model.get_weights()[2]
dense_W = model.get_weights()[3]
dense_b = model.get_weights()[4]

W_i, W_f, W_g, W_o = W[:,:8], W[:,8:16], W[:,16:24], W[:,24:]
U_i, U_f, U_g, U_o = U[:,:8], U[:,8:16], U[:,16:24], U[:,24:]
b_i, b_f, b_g, b_o = b[:8], b[8:16], b[16:24], b[24:]

for i in range(len(_X)):
    x_t = _X[i]
    i_t = sigmoid(np.dot(x_t,W_i) + np.dot(h,U_i) + b_i)
    f_t = sigmoid(np.dot(x_t,W_f) + np.dot(h,U_f) + b_f)
    tC = relu(np.dot(x_t,W_g) + np.dot(h,U_g) + b_g) # relu!
    o_t = sigmoid(np.dot(x_t,W_o) + np.dot(h,U_o) + b_o)
    C = f_t*C + i_t*tC
    h = relu(C) * o_t # relu!

output = np.dot(h,dense_W) + dense_b

E = []
Esum = 0
for i in range(2):
    E.append(np.exp(output[0,i]))
    Esum += np.exp(output[0,i])
result = []
for i in range(2):
    result.append(E[i]/Esum)

#The output looks like this:
print(result)
#=> [0.5606417941538421, 0.4393582058461578]

# model.predict()Check the output of.
print(model.predict(_X.reshape(1,3,2)))
#=> [[0.5606418 0.4393582]]
  • Next, let's look at recurrent_activation.
#Create a model with activation changed to relu.
tf.reset_default_graph()
backend.clear_session()

inputs = Input(shape=[3,2])
x = LSTM(8, activation='tanh', recurrent_activation='relu')(inputs) # relu!
outputs = Dense(2, activation='softmax')(x)
model = Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='adam',loss='categorical_crossentropy')
history = model.fit(X, y, epochs=2, verbose=1)


#Compute the output of the forecast using numpy.
C = np.zeros((1,8))
h = np.zeros((1,8))
_X = X[0]

W = model.get_weights()[0]
U = model.get_weights()[1]
b = model.get_weights()[2]
dense_W = model.get_weights()[3]
dense_b = model.get_weights()[4]

W_i, W_f, W_g, W_o = W[:,:8], W[:,8:16], W[:,16:24], W[:,24:]
U_i, U_f, U_g, U_o = U[:,:8], U[:,8:16], U[:,16:24], U[:,24:]
b_i, b_f, b_g, b_o = b[:8], b[8:16], b[16:24], b[24:]

for i in range(len(_X)):
    x_t = _X[i]
    i_t = relu(np.dot(x_t,W_i) + np.dot(h,U_i) + b_i) # relu!
    f_t = relu(np.dot(x_t,W_f) + np.dot(h,U_f) + b_f) # relu!
    tC = np.tanh(np.dot(x_t,W_g) + np.dot(h,U_g) + b_g)
    o_t = relu(np.dot(x_t,W_o) + np.dot(h,U_o) + b_o) # relu!
    C = f_t*C + i_t*tC
    h = np.tanh(C) * o_t

output = np.dot(h,dense_W) + dense_b

E = []
Esum = 0
for i in range(2):
    E.append(np.exp(output[0,i]))
    Esum += np.exp(output[0,i])
result = []
for i in range(2):
    result.append(E[i]/Esum)

#The output looks like this:
print(result)
#=> [0.5115599582737976, 0.4884400417262024]

# model.predict()Check the output of.
print(model.predict(_X.reshape(1,3,2)))
#=> [[0.51155996 0.48844004]]

Recommended Posts

[Python] I tried the same calculation as LSTM predict with from scratch [Keras]
I tried using the Python library from Ruby with PyCall
I tried "smoothing" the image with Python + OpenCV
I tried "differentiating" the image with Python + OpenCV
I tried "binarizing" the image with Python + OpenCV
I tried to touch the CSV file with Python
I tried to solve the soma cube with python
I tried to solve the problem with Python Vol.1
I tried hitting the API with echonest's python client
I tried fp-growth with python
I tried scraping with Python
"Deep Learning from scratch" Self-study memo (No. 16) I tried to build SimpleConvNet with Keras
I tried scraping with python
I tried to find the entropy of the image with python
I tried "gamma correction" of the image with Python + OpenCV
I tried to simulate how the infection spreads with Python
I tried sending an email from Amazon SES with Python
I tried changing the python script from 2.7.11 to 3.6.0 on windows10
I tried to divide the file into folders with Python
I tried the same data analysis with kaggle notebook (python) and Power BI at the same time ②
I tried the same data analysis with kaggle notebook (python) and Power BI at the same time ①
Sentiment analysis with natural language processing! I tried to predict the evaluation from the review text
Introduction to AI creation with Python! Part 1 I tried to classify and predict what the numbers are from the handwritten number images.
I tried web scraping with python.
I liked the tweet with python. ..
I tried running prolog with python 3.8.2.
I tried SMTP communication with Python
I tried scraping the ranking of Qiita Advent Calendar with Python
I tried using the python module Kwant for quantum transport calculation
I tried sending an email from the Sakura server with flask-mail
I tried to improve the efficiency of daily work with Python
[Python] I tried to visualize the night on the Galactic Railroad with WordCloud!
I replaced the numerical calculation of Python with Rust and compared the speed
I tried to refer to the fun rock-paper-scissors poi for beginners with Python
[Data science basics] I tried saving from csv to mysql with python
I tried scraping Yahoo News with Python
I tried sending an email with python.
Business efficiency starting from scratch with Python
I tried non-photorealistic rendering with Python + opencv
I tried to get the authentication code of Qiita API with Python.
I tried using UnityCloudBuild API from Python
From scratch! Introductory Python3! The same person has published these two books.
I tried a functional language with Python
I tried recursion with Python ② (Fibonacci sequence)
I tried using "Streamlit" which can do the Web only with Python
Python: I tried the traveling salesman problem
I tried with the top 100 PyPI packages> I tried to graph the packages installed on Python
I tried playing with the image with Pillow
I tried to streamline the standard role of new employees with Python
I tried to get the movie information of TMDb API with Python
I tried to refactor the template code posted in "Getting images from Flickr API with Python" (Part 2)
I tried to predict the behavior of the new coronavirus with the SEIR model.
I tried the Python Tornado Testing Framework
#I tried something like Vlookup with Python # 2
I tried to predict the number of people infected with coronavirus in consideration of the effect of refraining from going out
Introduction to AI creation with Python! Part 2 I tried to predict the house price in Boston with a neural network
I tried to implement merge sort in Python with as few lines as possible
[Python] I tried to judge the member image of the idol group using Keras
[Python] [Natural language processing] I tried Deep Learning ❷ made from scratch in Japanese ①
I tried to predict the horses that will be in the top 3 with LightGBM
Deep Learning from scratch The theory and implementation of deep learning learned with Python Chapter 3