Model using convolutional neural network in natural language processing
Introduction
Recently, natural language processing using convolutional neural networks (CNN) has attracted attention. Compared to RNNs, CNNs are easier to parallelize, and by using a GPU, convolution operations can be performed at high speed, which has the advantage of overwhelmingly high processing speed.
This article is a compilation of models using convolutional neural networks in natural language processing. We hope this will help you to get a bird's eye view of the progress of research on natural language processing using CNN.
Sentence classification (reputation analysis, topic classification, question type classification)
Convolutional Neural Networks for Sentence Classification(2014/08)
A paper proposing a CNN that performs sentence classification such as reputation analysis and question type classification.
Specifically, sentences are represented as a sequence of word vectors, and features are extracted and classified using CNN. In the paper, it is reported that the performance was improved by using the pre-learned word vector (Google News learned with word2vec). It is interesting that each of the two channels represents a word vector, one is updated during learning and the other is not updated to improve performance. When evaluated on seven document classification tasks, including reputation analysis and question type classification, four out of seven tasks gave the best results ever.
The author's implementation of Theano and Google Brain's implementation of TensorFlow by Denny Britz: https://github.com/yoonkim/CNN_sentence https://github.com/dennybritz/cnn-text-classification-tf
A brief explanation and implementation in Japanese by ichiroex [Chainer] Document classification by convolutional neural network
Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts(2014/08)
A paper proposing CNN (CharSCNN) that analyzes movie reviews and reputation on Twitter.
Reputation analysis for short texts such as Twitter has the problem that it is difficult because contextual information is limited. To deal with this problem, we improved performance by constructing character-level vector representations in addition to the word-level vector representations commonly used in reputation analysis, and using them to obtain sentence vector representations. I did that. Experiments with datasets for movie reviews (SSTb) and Twitter (STS) have yielded better results than previous methods.
Theano implementation by hogefugabar: https://github.com/hogefugabar/CharSCNN-theano
I also wrote a commentary article: Sentiment analysis of tweets by deep learning
#TAGSPACE: Semantic Embeddings from Hashtags(2014/10)
A paper proposing a CNN that learns short text expressions using hashtags used in SNS as teachers.
Specifically, using CNN, the score is output for the pair of hashtags corresponding to the input text, and the expression of the text is learned in the process of ranking the hashtags. Hashtag prediction and document recommendation tasks evaluated the results better than the baseline approach.
Effective Use of Word Order for Text Categorization with Convolutional Neural Networks(2014/12)
A paper that proposes a CNN for text classification considering word order.
There are various tasks in document classification, but in tasks such as reputation analysis, high performance cannot be obtained unless the word order is taken into consideration. In order to deal with this problem, we are proposing a CNN that can classify documents in consideration of word order. Specifically, most CNN methods input word embedding as input, but in this research, we input a high-dimensional one-hot vector as it is to learn embedding in a small text area. The effectiveness of the proposed method was shown as a result of comparison with the SOTA method using three datasets related to reputation analysis (including IMDB) and topic classification.
Implementation by the author: http://riejohnson.com/cnn_download.html
Semi-supervised Convolutional Neural Networks for Text Categorization via Region Embedding(2015/04)
A story that proposed a semi-supervised learning framework using CNN for text classification. In the conventional model, pre-learned word embedding was used for the input of the convolution layer, but in this study, embedding is learned from a small text area without supervised learning and used as a part of the input of the convolution layer in supervised CNN. Experiments with reputation analysis (IMDB, Elec) and topic classification (RCV1) showed higher performance than previous studies.
Implementation by the author: http://riejohnson.com/cnn_download.html
Character-level Convolutional Networks for Text Classification(2015/09)
A story about using a character-level convolutional neural network for text classification. The data is increased by replacing words in the text with synonyms using a thesaurus. Comparisons are made for bow, bag-of-ngram, bag-of-means as a traditional method, and word-based CNN and LSTM as a deep learning method. Eight datasets were created and compared with the base method, and some datasets showed effectiveness.
Lua implementation by the author: https://github.com/zhangxiangxiao/Crepe
The CNN model gives good results in sentence classification, but it requires an expert to decide the architecture and set hyperparameters. I'm not sure what the consequences of these changes will be, so I've verified it with more CNN. Finally, he gives practical advice on how to set the model architecture and hyperparameters when classifying sentences by CNN.
Series labeling (part of speech tagging, named entity recognition, chunking)
Natural Language Processing (almost) from Scratch(2011/03)
A story that proposed a neural network that can learn part-of-speech tagging, chunking, named entity recognition, and semantic role assignment. Although the performance was lower than the benchmark by simply training, it was shown that a good word vector contributes to the performance improvement by training the language model in advance using unlabeled data. Furthermore, it was shown that the performance can be further improved by sharing parameters between models for solving each task and performing multitask learning.
Summary slides in Japanese: Natural Language Processing (Almost) from Scratch (6th Deep Learning Study Group Material; Sakaki)
Learning Character-level Representations for Part-of-Speech Tagging(2014/07)
A story about part-of-speech tagging using CNN (CharWNN). Specifically, we constructed a vector representation of words by integrating word-level and character-level embeddings, and constructed a CNN that outputs the part-speech score by inputting the constructed vector. Experiments with datasets for English and Portuguese (WSJ and Mac-Morpho) resulted in SOTA results.
Language model
Language Modeling with Gated Convolutional Networks(2016/12)
It is said that CNN achieved accuracy equal to or higher than LSTM in the language model task. The folded result is processed by a mechanism similar to GRU so that past information is not lost. The Google Billion Word dataset has the same accuracy as LSTM, but the calculation efficiency has been improved by about 20 times.
Implementation by TensorFlow: Language-Modeling-GatedCNN
in conclusion
The latest summary of paper information on machine learning, natural language processing, and computer vision is distributed on the following Twitter accounts. We are waiting for you to follow us as we are delivering interesting content for those who read this article.
arXivTimes
reference
Recommended Posts