2-2

As a model, I use BERT, which is a hot natural language processing model recently (or rather a while ago?). I didn't really understand what BERT was, so I referred to the following site. https://udemy.benesse.co.jp/ai/bert.html https://ainow.ai/2019/05/21/167211/ http://deeplearning.hatenablog.com/entry/transformer

The model used this time is the BERT + CNN model. The BERT part is quite difficult to train on my own, so when I was wondering what to do, I found the following very grateful ones. https://yoheikikuta.github.io/bert-japanese/ God. Thank you very much.

By the way, since this is a Sentence piece, I have to create learning data ...? https://qiita.com/hideki/items/1ec1c21c33326ad5615f

Eh, is it too convenient? Thank you very much. As a result, the accumulation of great ancestors has made it possible to instantly acquire vectors with the monster model BERT.

So, I will make the head part. As a poor student this time, I couldn't prepare enough computing resources to fine-tune each BERT, so I tune only the head part with Google Colablatory. (Thanks to google)

The vector obtained by BERT is word by word. Enter vectors for two documents and solve the problem as a two-class classification of the same or different meanings.

The point that I devised a little here is that the input vector was created by inserting the vectors of two documents alternately. This is because I thought that it would be better to have a vector with position information that puts words that are close to each other closer to each other than to arrange them in parallel.

The above vectors are classified by CNN. The configuration of CNN is like this with keras.

def make_network():
    x_in = Input((256,768,1))
    x = Conv2D(64,(3,3),padding="valid")(x_in)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = Conv2D(64,(3,3),padding="valid")(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(3,3),strides=2,padding="same")(x)

    x = Conv2D(64,(2,2),padding="valid")(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = Conv2D(128,(2,2),padding="valid")(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)
    x = MaxPooling2D(pool_size=(3,3),strides=2,padding="same")(x)

    x = GlobalAveragePooling2D()(x)

    x = Dense(2)(x)
    x = Activation("softmax")(x)
    model = Model(x_in,x)
    return model

Well, it's just CNN with no particular changes.

def data_gen(data_X,data_y,vectorian,batchsize=32):
    d_cnt = 0
    x = []
    y = []
    while True:
        for i in range(len(data_X)):
            t_l = str(data_X[i][0])
            t_r = str(data_X[i][1])

            t_l = vectorian.fit(t_l).vectors
            t_r = vectorian.fit(t_r).vectors


            itp = range(t_l.shape[1])
            in_x = np.insert(t_r,itp,t_l[:,itp],axis=1)
            
            x.append(in_x)
            
            if data_y[i] == 1:
                y.append([0,1])
            else:
                y.append([1,0])
            
            d_cnt += 1
            
            if d_cnt == batchsize:
                inputs = np.array(x).astype(np.float32)
                inputs = inputs.reshape(batchsize,256,768,1)
                targets = np.array(y)
                x = []
                y = []
                d_cnt = 0
                yield inputs,targets

Well, it's not interesting. So I started learning

It took 1 epoch an hour, so learning at Google Colablatory was a pain. The accuracy of 3 to 5 epoch is like this (I erased the output of 1 to 2 epoch due to the GPU limitation on the way and the new runtime.)

Epoch 1/1
1955/1955 [==============================] - 3846s 2s/step - loss: 0.1723 - categorical_accuracy: 0.9396 - val_loss: 0.1632 - val_categorical_accuracy: 0.9398
Epoch 1/1
1955/1955 [==============================] - 3852s 2s/step - loss: 0.1619 - categorical_accuracy: 0.9436 - val_loss: 0.1477 - val_categorical_accuracy: 0.9487
Epoch 1/1
1955/1955 [==============================] - 3835s 2s/step - loss: 0.1532 - categorical_accuracy: 0.9466 - val_loss: 0.1462 - val_categorical_accuracy: 0.9482

It seems that the loss will still drop, but it is difficult to stick it in front of the PC, so please forgive me ... By the way, this time it is forced, but since I made equilibrium data, I will not calculate F1 ~~ (Troublesome) ~~

If it is equilibrium data, I think that accuracy will not give such a strange result.

output

You can use the above model and dictionary to make a sentence paraphrase judgment and a difficulty score. Prepare an output function as appropriate.

def texts_to_inputs(text1,text2):
    t_l = text1
    t_r = text2

    t_l = vectorian.fit(t_l).vectors
    t_r = vectorian.fit(t_r).vectors

    itp = range(t_l.shape[1])
    in_x = np.insert(t_r,itp,t_l[:,itp],axis=1).reshape(1,256,768,1)
    return in_x

def easy_trans_scores(test1,test2,vocab_diff_dic):
    test_x = texts_to_inputs(test1,test2)
    p = model.predict(test_x)

    result = jumanpp.analysis(test1)
    d1 = 0
    d1_cnt = 0
    for mrph in result.mrph_list(): #Access each morpheme
        if mrph.hinsi in indep:
            if mrph.genkei in vocab_diff_dic:
                d1 += vocab_diff_dic[mrph.genkei]
                d1_cnt += 1
    print("Implication probability:{:.3g}Total difficulty:{}Number of independent words:{}Average difficulty:{}".format(p[0][1],d1,d1_cnt,d1/d1_cnt))
    print("total score:{:.5g}".format(p[0][1]/d1))

For now, let's use implication / (total difficulty) as the score. The reason for the total rather than the average difficulty is that I thought it was better to evaluate short and concise sentences well. Well, there is a bug that the value always becomes inf when the total difficulty level is 0 and there is no implication relationship ... This time it's okay ...

result

The paraphrases that are judged to have the highest score are in bold. Isn't it a reasonably convincing result?

By the way, Mr. Aizen ... Please give me some kinder words ...

Result 2 (Addition) In other words, good battle and power of Kim Roger

Using this system, I paraphrased my friend and played a good king deciding match (1vs1 game, but if I win, I will be the king) Does it look like this when made into a game? It's like that.

Click here for the theme.

** The man who gained wealth, fame, power, and all of the world'Gold Roger, the Pirate King'The words he gave at his death drove people to the sea. ** **

This is the mouth of the OP of a certain national anime that everyone knows.

First move (I) 1st turn

** Money, position, power The man who put everything in this world, Gold Roger, the pirate king The words he said before he died made people want to go to the sea **

Score: Implication probability: 0.696 Total difficulty: 8 ** total score: 0.087022 **

Well, I think it's a good first move. Since the implication probability is a little low, I feel that I attacked too much to lower the difficulty level.

Gote (opposite) 1st turn

** "The man who got all the money, popularity, power, and the world,'Pirate King Gold Roger' The word he said just before his death sent people to the sea" **

Score: Implication probability: 0.974 Total difficulty: 14 ** total score: 0.069588 **

It's talked about with implication probabilities, but I'm still better at the total score. If you don't attack by paraphrasing words, it will be like this. It is surprisingly strategic.

First move (I) 2nd turn

** Money, popularity, power, the man who got everything in the world,'Pirate King Gold Roger' The words he said before he died made people want to go to the sea **

Score: Implication probability: 0.917 Total difficulty: 9 ** total score: 0.10194 **

Since the other party had a higher implication probability, we were able to greatly improve the implication probability and total score by incorporating it as a model. This is ** won ** (flag)

Gote (opposite) 2nd turn (winning sentence)

** A man who got all of money, popularity, power, and the world'Pirate King Kim Roger'The word he said when he died sent everyone to the sea **

Score: Implication probability: 0.952 Total difficulty: 9 ** total score: 0.10581 **

e···?

lost. Although it is a close margin, losing is losing. Probably the winner was ** Kim Roger **. This greatly reduces the difficulty level while maintaining the implication probability. (When I saw the result, I was laughing.)

Well, it is also an interesting part of this system that you can have a fun paraphrasing good king deciding match like this. ~~ Because it's sloppy here and there ~~ How do you interpret this system? You can compete for points that are competitive, including meta-competition, and points that are within the bounds of common sense, and each has its own fun.

Summary and future prospects

Although there were some kettles and looseness, I was able to somehow judge whether or not a simple paraphrase was made by combining the implication judgment and the difficulty judgment.

** This is pretty interesting to do by myself ** (It is also interesting to set the question, "Is it paraphrased well?" Or isn't such a sentence difficult?) So, I want to make this into a game or a service. (I want to implement online ranking, etc.) (I can't do it now because I don't have the knowledge to build a service)

As I mentioned at the beginning, if you enjoy this as a game, you may be able to enjoy it, and at the same time, you may be able to create a data set of easy paraphrases made by people, and by using that data you can make more interesting considerations. I'm thinking about it.

This model etc. (for those who want to try it)

https://github.com/MosasoM/Aizen

It happened in. I wrote it in the repository,

• JUMAN++
• text-vectorian
• BERT Japanese learned model (https://yoheikikuta.github.io/bert-japanese/)

If you have a look at Aizen.ipynb, you can (probably) try it quite easily. If you have any interesting results or if you like this scoring system, I would appreciate it if you could comment.

Acknowledgments

This article is based on the borrowing of BERT trained models and various data sets. As mentioned in the text, thank you to all the people who maintained these.