Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 4

Introduction

I suddenly started studying "Deep Learning from scratch ❷ --- Natural language processing" Note that I stumbled in Chapter 4. is.

The execution environment is macOS Catalina + Anaconda 2019.10, and the Python version is 3.7.4. For details, refer to Chapter 1 of this memo.

Chapter 4 Speeding up word2vec

This chapter is a speedup of the word2vec CBOW model created in Chapter 3.

4.1 Improvement of word2vec ①

First is the speedup from the input layer to the intermediate layer. This part plays the role of embedding that converts words into distributed expressions, but since the MatMul layer is wasteful, replace it with the Embedding layer.

The Embedding layer is simple, but the part that adds $ dW $ when idx is duplicated in the backpropagation implementation may be a bit confusing. In the book, it is taken up in Figure 4-5, and the explanation is omitted, "Let's think about why we add."

Therefore, I thought about it by comparing it with the backpropagation calculation for the MatMul layer. This is because the Embedding layer must have the same result as the MatMul layer.

First, change $ idx $ in Figure 4-5 back to $ x $ in the MatMul layer.

\begin{align}
idx &= 
\begin{pmatrix}
0\\
2\\
0\\
4\\
\end{pmatrix}\\
\\
x &=
\begin{pmatrix}
1 & 0 & 0 & 0 & 0 & 0 & 0\\
0 & 0 & 1 & 0 & 0 & 0 & 0\\
1 & 0 & 0 & 0 & 0 & 0 & 0\\
0 & 0 & 0 & 0 & 1 & 0 & 0\\
\end{pmatrix}
\end{align}

The backpropagation formula for the MatMul layer is $ \ frac {\ partial L} {\ partial W} = x ^ T \ frac {\ partial L} {\ partial y} $ (see page 33), so Figure 4-5 If you replace it with the notation of, it becomes $ dw = x ^ Tdh $. Applying $ x $ and $ dh $ in Figure 4-5 here to calculate $ dW $, we get: Actually, I wanted to make $ dh $ as shown in Fig. 4-5, but I can't express the shade of ● like a book, so here I express it with $ ●, ◆, a, b $.

\begin{align}
dW &= x^Tdh\\
\\
\begin{pmatrix}
? & ? & ? \\
○ & ○ & ○ \\
●_1 & ●_2 & ●_3 \\
○ & ○ & ○ \\
◆_1 & ◆_2 & ◆_3 \\
○ & ○ & ○ \\
○ & ○ & ○ \\
\end{pmatrix}
&=
\begin{pmatrix}
1 & 0 & 1 & 0\\
0 & 0 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 0 & 0\\
0 & 0 & 0 & 1\\
0 & 0 & 0 & 0\\
0 & 0 & 0 & 0\\
\end{pmatrix}
\begin{pmatrix}
a_1 & a_2 & a_3 \\
●_1 & ●_2 & ●_3 \\
b_1 & b_2 & b_3 \\
◆_1 & ◆_2 & ◆_3 \\
\end{pmatrix}\\
\end{align}

As you can see from the calculation, the second line ($ ● _1 ● _2 ● _3 ) and the fourth line ( ◆ _1 ◆ _2 ◆ _3 $) of $ dh $ are $ dW $ as they are as shown in Fig. 4-5. It will be the 3rd and 5th lines of. And the $? $ On the first line of the $ dW $ in question looks like this:

\begin{align}
\begin{pmatrix}
a_1 + b_1 & a_2 + b_2 & a_3 + b_3 \\
○ & ○ & ○ \\
●_1 & ●_2 & ●_3 \\
○ & ○ & ○ \\
◆_1 & ◆_2 & ◆_3 \\
○ & ○ & ○ \\
○ & ○ & ○ \\
\end{pmatrix}
&=
\begin{pmatrix}
1 & 0 & 1 & 0\\
0 & 0 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 0 & 0\\
0 & 0 & 0 & 1\\
0 & 0 & 0 & 0\\
0 & 0 & 0 & 0\\
\end{pmatrix}
\begin{pmatrix}
a_1 & a_2 & a_3 \\
●_1 & ●_2 & ●_3 \\
b_1 & b_2 & b_3 \\
◆_1 & ◆_2 & ◆_3 \\
\end{pmatrix}
\end{align}

In other words, you can see that we are adding the first and third lines of $ dh $. The same thing as the calculation of this MatMul layer must be implemented in the Embedding layer, so it is necessary to add it.

4.2 Improvement of word2vec ②

Next is the improvement from the intermediate layer to the output layer. Negative Sampling's idea of boldly reducing learning with negative examples is interesting.

I didn't have a big stumbling block, but in the book, the explanation of backpropagation of the Embedding Dot layer is omitted because it is "not a difficult problem, so let's think about it for yourself", so I will summarize here a little. to watch.

In Figure 4-12, if you cut out only the part of the Embedding Dot layer, it will be as follows.

What the dot node is doing is multiplying each element and adding the results. Therefore, consider back propagation by decomposing into a multiplication node (see "1.3.4.1 Multiplication node" in Chapter 1) and a Sum node (see "1.3.4.4 Sum node" in Chapter 1). Then it will have the following form. Blue letters are backpropagation.

Returning this to the previous Dot node diagram, it looks like this:

If you implement it as shown in this figure, it is OK, but as it is, the shape of dout does not match with h and target_W, so the product of each element is not calculated by*of NumPy. Therefore, first match the shapes with dout.reshape (dout.shape [0], 1) and then calculate the product. If you implement it in this way, you can see that it becomes the code of ʻEmbeddingDot.backwad ()` in the book.

4.3 Learning improved word2vec

The learning implementation is not particularly stumbling. I use the PTB corpus in the book, but I like Japanese after all, so I tried learning with Aozora Bunko's divided text as in Chapter 2. It was.

To get the corpus, use the modified version of dataset / aozorabunko.py instead of dataset / ptb.py. For more information on this source and mechanism, see [Chapter 2 Memo "Improvement of Count-Based Method"](https://qiita.com/segavvy/items/52feabbf7867020e117d#24-Improvement of Count-Based Method). I have written it, so please refer to it.

ch04 / train.py has also been changed to use the corpus of Aozora Bunko as follows. The changes are those with ★ in the comments.

ch04/train.py

# coding: utf-8
import sys
sys.path.append('..')
from common import config
#When executing on GPU, delete the comment out below (cupy required)
# ===============================================
# config.GPU = True
# ===============================================
from common.np import *
import pickle
from common.trainer import Trainer
from common.optimizer import Adam
from cbow import CBOW
from skip_gram import SkipGram
from common.util import create_contexts_target, to_cpu, to_gpu
from dataset import aozorabunko  #★ Changed to use the corpus of Aozora Bunko

#Hyperparameter settings
window_size = 5
hidden_size = 100
batch_size = 100
max_epoch = 10

#Data reading
corpus, word_to_id, id_to_word = aozorabunko.load_data('train')  #★ Change corpus
vocab_size = len(word_to_id)

contexts, target = create_contexts_target(corpus, window_size)
if config.GPU:
    contexts, target = to_gpu(contexts), to_gpu(target)

#Generation of models etc.
model = CBOW(vocab_size, hidden_size, window_size, corpus)
# model = SkipGram(vocab_size, hidden_size, window_size, corpus)
optimizer = Adam()
trainer = Trainer(model, optimizer)

#Start learning
trainer.fit(contexts, target, max_epoch, batch_size)
trainer.plot()

#Save the data you need for later use
word_vecs = model.word_vecs
if config.GPU:
    word_vecs = to_cpu(word_vecs)
params = {}
params['word_vecs'] = word_vecs.astype(np.float16)
params['word_to_id'] = word_to_id
params['id_to_word'] = id_to_word
pkl_file = 'cbow_params.pkl'  # or 'skipgram_params.pkl'
with open(pkl_file, 'wb') as f:
    pickle.dump(params, f, -1)

In addition, it took about 8 hours to study in the environment at hand. Next is the confirmation of the result. I changed ch04 / eval.py a little so that I can try various words from standard input. ★ is the changed part.

ch04/eval.py

# coding: utf-8
import sys
sys.path.append('..')
from common.util import most_similar, analogy
import pickle


pkl_file = 'cbow_params.pkl'
# pkl_file = 'skipgram_params.pkl'

with open(pkl_file, 'rb') as f:
    params = pickle.load(f)
    word_vecs = params['word_vecs']
    word_to_id = params['word_to_id']
    id_to_word = params['id_to_word']

#most similar task ★ Changed the query to standard input
while True:
    query = input('\n[similar] query? ')
    if not query:
        break
    most_similar(query, word_to_id, id_to_word, word_vecs, top=5)


#analogy task ★ Changed the query to standard input
print('-'*50)
while True:
    query = input('\n[analogy] query? (3 words) ')
    if not query:
        break
    a, b, c = query.split()
    analogy(a, b, c,  word_to_id, id_to_word, word_vecs)

Below are the results of various trials.

First, check for similar words. For comparison, I also listed the count-based one I tried in Chapter 2. Also, the window size of CBOW was 5 in the book code, but I also tried 2 which is the same as the count base.

It's quite confusing as it was in Chapter 2. No superiority or inferiority can be given. The area where "cats" appear in "myself" shows the bias of the corpus. It seems that the reason is that the size of the corpus is too small, as it uses only the works of Soseki Natsume, Kenji Miyazawa, and Ranpo Edogawa.

Next is the analogy problem.

Suddenly the first problem is a mixture of low scores but distressing results. Insufficient training data can be scary. I feel that I have a glimpse of the background of the recent demand for "explainable AI."

Other results are tattered, but barely the correct answers were mixed in "body: face = car :?" And "go: come = speak :?". The appearance of "cops" and "inspectors" in "cars" is probably due to Ranpo Edogawa.

After all, I should have used the Japanese version of Wikipedia obediently, but it's an amanojaku: sweat: There are many people who are trying Wikipedia, so if you are interested, "wikipedia Japanese corpus" Please try google with.

4.4 Remaining themes for word2vec

Negative / positive judgment of email is explained as an example of transfer learning, but with the knowledge up to this chapter, even if words can be converted to fixed-length vectors, sentences such as email cannot be converted to fixed-length vectors. .. Therefore, we cannot challenge such a task yet.

Also, regarding the quality of distributed expressions, in the case of Japanese, the quality of prior word-separation seems to have a large effect. Some Japanese distributed expression models have been released, but when considering transfer learning of them, I think that it is a prerequisite to use the same word-separation mechanism (logic, dictionary contents, parameters, etc.). .. In that case, does it mean that transfer learning cannot be easily performed with tasks that deal with technical terms specific to the industry or individual companies, for example? Japanese is a lot of work.

4.5 Summary

The first half of this book is finally over, but considering that Chapter 1 was a review of the first volume, it may still be about 1/3. The destination seems to be long ...

That's all for this chapter. If you have any mistakes, I would be grateful if you could point them out.