Natural language processing 3 Word continuity

Aidemy　2020/10/30

Introduction

Hello, it is Yope! I am a liberal arts student, but I was interested in the possibilities of AI, so I went to the AI-specialized school "Aidemy" to study. I would like to share the knowledge gained here with you, and I am summarizing it on Qiita. I am very happy that many people have read the previous summary article. Thank you! This is the third post on natural language processing. Nice to meet you.

• This article is a summary of what you learned in "Aidemy" "in your own words". It may contain mistakes and misunderstandings. Please note.

What to learn this time ・ ・

Preparation for word continuity

Creating a word dictionary

-In order to perform "word continuity analysis", the word data is quantified as a preparation. -After dividing the data set, first create a __word dictionary (list) __ with the ID set in order to assign an ID to each word and quantify it. Of these, we want to number the words in descending order of appearance, so we count the number of occurrences of __words and sort them in descending order __.

-Count the number of occurrences of words using __Counter () __ and __itertools.chain () __. __Counter (itertools.chain (* list of word data to count)) __

-Counter () counts the number of elements, but since the result is returned in multiple dimensions, each element cannot be accessed individually. This is returned to one dimension by using __itertools.chain () __. Add __ "*" __ to the multidimensional list passed to this itertools.chain ().

-Use __most_common (n) __ to sort in descending order. If n is specified, the number of tuples will be returned in descending order.

・ Once you have done this, you can create a word dictionary by assigning an ID to each of the sorted appearance count lists and storing them in an empty dictionary.

・ Code

Convert word data to numeric data

-Once the dictionary is created, do the purpose __ "quantify the data set" __. -Refer to the ID part of the word dictionary created in the previous section, and convert the utterance data set "wakatiO" to a new array called "wakatiO_n" with only numerical data.

-The code is as follows.

・ This code is easy to understand if you look at it from behind. First, the __ "for waka in wakati O" __ part indicates that each word list (separated by one sentence) of the dataset "wakatiO" is stored in waka. Next, __ "for word in waka" __ indicates that the word list is divided into each word and stored in word. -And, each word is stored in "wakatiO_n" by referring to the dictionary ID with __ "dic_inv [word]" __.

Feature extraction from word continuity

N-gram -N-gram is a model used when extracting topics from text, and uses the method of dividing __text into N consecutive characters. ・ For N divisions, if there is a character string "aiueo", N=1 "1-For "gram", "Ah|I|U|e|Divided into "O" and "2-For "gram", "AhI|IU|Ue|eお」と分割される。 -I tried to extract topics from the "word document matrix" that appeared in "Natural Language Processing 2", but that was __ "word co-occurrence (does it appear in the same sentence)" __ In contrast, N-gram represents __ "word continuity (in what order)" __.

-Since there is no method for creating N-gram, the method of storing a list of character strings divided into an empty list by iterative processing is adopted.

list = []
word = ['Good','weather','is','Ne','。']
#3-Creating a gram model
for i range(len(word)-2):
    list.append([word[i],word[i+1],word[i+2]])
print(list)
#[['Good','weather','is']['weather','is','Ne']['is','Ne','。']]

-For __ "len (word) -2" __, specify the same number so that "i + 2" below it does not exceed the length of the word.

2-Creating a gram list

-For the above N-gram, by creating a 2-gram list from "wakatiO_n" __, the number of occurrences (weights) of two nodes is calculated __. -First, apply the 2-gram model to "wakatiO_n" to create the 2-gram array "bigramO". Convert it to __DataFrame __ (df_bigramO), group by "'node1' and'node2'" (two nodes that calculate continuous values) (__groupby () __), and finally "__sum () __ ] To complete the total number of appearances.

·code

• __ "as_index = False" __ of group by () specifies that the __ key is output to column instead of index __. In other words, if nothing is specified, the key node1 will be output as an index, so it will not be output in the same way as the column node2. In order to avoid this, it is specified like this.

-Output result

2-Creating a gram network

-In Chapter 2, an undirected graph was created with the similarity of words as edges (weights). This time, __ Create a directed graph with the number of occurrences of word pairs as edges (weights) __. The data of the creation source is df_bigramO created in the previous section. -A directed graph refers to a graph that has the concept of "direction" at the __ edge. Regarding the number of occurrences of word pairs, the directed graph also has meaning in the information "in order of appearance", that is, "which word comes first". -The creation method is exactly the same as for undirected graphs, just specify __ "nx.DiGraph" __ in the argument of __nx.from_pandas_edgelist () __.

·code

・ Result

2-gram network features

-Similar to the previous Chapter, it is difficult to understand the characteristics just by looking at the graph, so calculate __ "average cluster coefficient" __ and __ "mediation centrality" __ to grasp the characteristics quantitatively. ・ (Review) The average cluster coefficient is calculated by __nx.average_clustering () __, and the mediation centrality is calculated by __nx.betweenness_centrality () __.

See the effect of each word

・ To see how each word affects each other, visualize __ "order distribution" __. -In the case of a directed graph, it is divided into "influenced by other words" __in-degree __ and "influenced in other words" __out-degree __. -Check the input degree using the __in_degree (weight) __ method. It is returned in the form of (node number, degree). -Similarly, check the degree by using the __out_degree (weight) __ method.

·code

·result

Summary

・ To understand the characteristics from __word continuity __, first divide the __ utterance text __ and then create a __word dictionary __ to convert the __ data into numerical values. __. -Convert the data converted to numerical values to N-gram list, calculate the number of occurrences of each word combination __, and create a directed graph from it __. ・ Since it is difficult to understand the characteristics if the graph remains directed, the characteristics are quantitatively grasped by calculating __ "average cluster coefficient" __ and __ "mediation centrality" __. -Also, quantitative features can be made __visualized by doing __order distribution __.

This time is over. Thank you for reading until the end.