Rabbit Challenge 3DAY
Section1 Concept of Recurrent Neural Network
The big picture of RNN
What is RNN?
Neural network that can support ** time series data **
Data in chronological order
A series of data that are observed at regular intervals in chronological order and that have statistical dependencies on each other.
--Voice data --Text data --Stock price data
About RNN
In forward propagation, the calculation result of the previous intermediate layer is processed by weight and bias, and then used as a part of the input to the intermediate layer.
To work with a time series model, we need a recursive structure that holds the initial state and the state of the past time t-1 and recursively finds t at the next time.
BPTT
What is BPTT?
Backpropagation Through Time A type of backpropagation method Parameter adjustment method in RNN A form in which the backpropagation method is implemented including the flow in time series. It is not suitable for long-term use because the amount of calculation is large.
BPTT mathematical notation
np.dot(X.T, delta[:,t].reshape(1,-1))
$ \frac{\partial E }{\partial W*{in}} = \frac{ \partial E}{ \partial v^t} \big[ \frac{ \partial u^t}{ \partial W*{(out)}}\big] ^t = \delta ^{out,t} [ z^t] ^T$
np.dot(z[:,t+1].reshape(-1,1), delta_out[:,t].reshape(-1,1))
np.dot(z[:,t].reshape(-1,1), delta[:,t].reshape(1,-1))
$ \frac{ \partial E}{ \partial u^t} = \frac{\partial E}{ \partial v^t} \frac{\partial v^t}{\partial i^t} = f'(u^t)W_{out}^T \delta^{out,t} = \delta ^t$
delta[:,t] = (np.dot(delta[:,t+1].T, W.T) + np.dot(delta_out[:,t].T, W_out.T)) * functions.d_sigmoid(u[:,t+1])
Parameter update
$ W*{(in)}^{t+1} = W*{(in)}^t - \epsilon \frac {\partial E}{ \partial W*{(in)}} = W*{(in)}^t - \epsilon \sum_{z=0}^{T_t} \delta ^{t-z} \big[ x^{t-z} \big] ^T$
W_in -= learning_rate * W_in_grad
$ W*{(out)}^{t+1} = W*{(out)}^t - \epsilon \frac {\partial E}{ \partial W*{(out)}} = W*{(out)}^t - \epsilon \delta ^{out,t} \big[ z^t \big] ^T$
W_out -= learning_rate * W_out_grad
$ W^{t+1} = W^t - \frac{\partial E}{ \partial W} = W*{(in)}^t - \epsilon \sum*{z=0}^{T_t} \delta ^{t-z} \big[ x^{t-z-1} \big] $
W -= learning_rate * W_grad
The big picture of BPTT
E^t = loss(y^t,d^t) \\
= loss(g(w_{(out)}z^t + c),d^t) \\
= loss(g(W_{(out)}f(W_{(in)}x^t + Wz^{t-1} + b)+c),d^t)
Section2 LSTM
RNN Challenges The more you go back in time series, the more the gradient disappears → It is difficult to learn long time series
solution The one that solves the problem by changing the structure is called LSTM.
Big picture of LSTM
CEC
Gradient disappearance and gradient explosion can be solved with a gradient of 1
Task Input data has a uniform weight regardless of time dependence → The learning characteristics of the neural network are lost in the first place
solution Input gate and output gate
Input gate and output gate
By adding an input gate and an output gate The weight of the input value to each gate can be changed by the weight matrix W and U. → Solve CEC problems
Input gate
Output gate
Current status All past information is stored All information
Task Cannot delete past information when it is no longer needed Always pulled by past information
solution Oblivion gate
Oblivion gate
Gate for deleting CEC information that is no longer needed
Task I want to propagate the past information stored in CEC to other nodes at any time, or forget it at any time. The value of CEC itself does not affect gate control.
solution Peephole connection
Except hole connection
A structure that makes it possible to propagate the value of CEC itself via a weight matrix.
GRU
In LSTM, there was a problem that there were many parameters and the calculation load was high. →GRU
Images show that GRU has reduced that parameter
Section4 Bidirectional RNN
Use future information as well as past information in time series data
Section5 Seq2Seq
Encoder-A type of Decoder model
It is used as a model for machine translation and dialogue.
Encoder RNN
A structure in which text data input by the user is divided into tokens such as words and passed.
Taking: Decompose sentences into tokens such as words by morphological analysis and convert them into IDs for each token.
Embedding: Convert to distributed representation vector based on ID
Encoder RNN: Input vectors to RNN in order
Input vec1 to RNN and output hidden state. Repeat this and the next input vec2 into the RNN ... and so on.
Set the hidden state when the last vec is inserted as the final state. This is a vector that represents the meaning of the input
Decoder RNN
A structure in which the system generates output data for each token such as a word. 1.Decoder RNN: Output the generation probability of each token from the final state (thought vector) of Encoder RNN Set final state as the initial state of Decoder RNN and enter Embedding. 2.Sampling: Randomly select tokens based on generation probability 3. Embedding: token as the next input to the Decoder RNN 4.Detokenize: Repeat 1-3 to convert the token obtained in 2 into a character string.
HRED
seq2seq can only answer one question at a time → There is no context to the question, just the response continues As a solution → HRED
What is HRED Generate the next utterance from the past n-1 utterances More human response Seq2Seq + Context RNN (Context RNN: A structure that converts a conventional conversation context into a vector.)
HERD Challenges There is no diversity in the flow of conversation Tends to learn short common answers
VHRED
A solution to the HREAD problem by adding the concept of VAE latent variables to HREAD.
VAE
Autoencoder
One of unsupervised learning. Teacher data is not used. The network that converts the input data to the latent variable z is the Encoder. Decoder is a neural network that restores the original image using the latent variable z as input.
merit Dimensionality reduction can be performed.
VAE
In the case of a normal autoencoder, data is pushed into the latent variable z, but the state of its structure is unknown.
In the case of VAE, the probability distribution z to N (0,1) is assumed for the latent variable z. Allows data to be pushed into a structure called the probability distribution of the latent variable z
Section6 Word2Vec
RNN Challenges
You can't give NN a variable-length string like a word → You can only study with a fixed length
solution
Word2Vec
Word2vec
For converting variable-length character strings to fixed-length format. Create a vocabulary from training data Create one-hot-vector for input data with vocabulary as dimension → Distributed representation
merit
Learning the distributed representation of large-scale data has become feasible with realistic calculation speed and memory capacity.
Until now, a weight matrix only for "vocabulary x vocabulary" was born. In Word2vec, a weight matrix is born with "vocabulary x arbitrary word vector dimension".
Section7 AttentionMechanism
Challenges of seq2seq
Difficult to handle long sentences. The longer the sentence, the larger the dimension of the internal representation of the input.
solution
Learn "Which words in the input and output are related" →Attention Mechanism
Confirmation test
Answer the size of the output image when the input image of size 5x5 is folded with the filter of size 3x3. The stride is 2 and the padding is 1.
answer
3×3
RNN networks have three main weights. One is the weight applied when defining the current middle layer from the input, and the other is the weight applied when defining the output from the middle layer. Explain the remaining one weight.
answer
Weights from one middle layer to the next
Find dz/dx using the principle of chain rule.
$ z=t^2$
answer
$ dz/dt = 2t$
Express y1 as a mathematical formula in the figure below.
$ x ・ s_0: Use s_1 ・ w_ {in} ・ w ・ w_ {out} $
- Define the bias with any character
- Apply the sigmoid function g (x) to the output of the intermediate layer.
answer
$ z1 = sigmoid(S_0 w + x_1 w{(in)} + b)$
$ y1= sigmoid(z_1 W{(out)} + c)$
When the sigmoid function is differentiated, it takes the maximum value when the input value is 0. Select the correct value from the options.
（1）0.15 （2）0.25 （3）0.35 （4）0.45
answer
(2)
Suppose you want to enter the following sentence into an LSTM and predict the words that fit in the blanks. The word "very" in the text is not considered to have any effect if it disappears in the blank prediction. Which gate is considered to work in such a case?
"The movie was interesting. By the way, I'm so hungry that something \ _ \ _."
answer
Oblivion gate
Briefly describe the challenges facing LSTMs and CECs.
answer
LSTM
The number of parameters is large and the calculation load is high.
CEC
Gradient continues to be passed at 1 and loses weight The concept of learning is compromised
Briefly describe the difference between LSTMs and GRUs.
answer
LSTMs have more parameters than GRUs
Choose from the options below that describe seq2seq.
(1) RNNs in the forward and reverse directions with respect to time are constructed, and these two intermediate layer representations are used as features.
(2) A type of Encoder-Decoder model that uses RNN, and is used for models such as machine translation.
(3) A neural network that recursively performs an operation of creating an expression vector (phrase) from adjacent words on a tree structure such as a syntax tree (with the same weight) and obtains the expression vector of the entire sentence.
(4) A type of RNN that solves the vanishing gradient problem, which is a problem in simple RNNs, by introducing the concept of CEC and gate.
answer
(2)
(1) Bidirectional RNN (2)seq2seq (3)RNN (4)LSTM
Briefly describe the difference between seq2seq and HRED and between HRED and VHRED.
answer
ses2seq
I can only answer one question at a time
HRED
Can answer in context
VHRED
You can answer according to the context, and you can answer in a variety of ways
Answer the blanks in the VAE description below.
Introducing \ _ \ _ \ _ \ _ into the latent variable of the self-encoder.
answer
Probability distribution
Briefly describe the difference between RNN and word2vec, seq2seq and Attention.
answer
RNN and word2vec
Compared to RNN, word2vec can now be calculated with a realistic amount of resources
seq2se1 and seq2se1 + Attention
seq2se1 can only learn fixed length ones seq2se1 + Attention can translate long sentences
Exercise results
https://github.com/Tomo-Horiuchi/rabbit/blob/master/part2/3Day/3_1_simple_RNN.ipynb
Consideration
simple_RNN The accuracy is improved when the number of nodes in the middle layer is around 64. Increasing the learning rate causes a gradient explosion When changing the sigmoid function to the ReLU function, a gradient explosion occurs and learning does not proceed at all. Learning does not proceed when changing to tanh The accuracy improves when the number of hidden layers is increased, but the error is large at the beginning of learning. If the weight is increased, the error is small at the beginning of learning, but the gradient explodes from there. There is an existing problem, and new models are being created to solve it, and the accuracy of learning is increasing.
Recommended Posts