This article is the 21st day of Natural Language Processing # 2 Advent Calendar 2019. By the way, it's my birthday today. Please celebrate. ~~ M's book, such as setting a deadline for a birthday ~~

Introduction

In the word embedding world, BERT has been rampant in the past year, and even ELMo is becoming less common. You may still want to use legacy distributed representations such as word2vec and GloVe. In addition, you may want to learn with the data you have (at least for me). So, I made a learning kit for word2vec / doc2vec / GloVe / fastText for myself, so I will publish it.

word2vec / doc2vec / fastText can learn the model of gensim and GloVe can learn the model of official implementation ..

I wrote how to use it in the README of each package, so here I will write about the design concept of the learning kit.

1. API standardization of model learning functions

There are various libraries / packages of word distributed expressions, The assumed format of the dataset is different for each library, Whenever I write a preprocessing script that shapes it into a form suitable for the library The code gets dirty more and more. Therefore, we shared the iterator for reading the text data set. I try to format the data format suitable for each library in the function.

def train_*****_model(
    output_model_path,
    iter_docs,
    **kwargs
)

For word2vec:

def train_word2vec_model(
    output_model_path,
    iter_docs,
    size=300,
    window=8,
    min_count=5,
    sg=1,
    epoch=5
):
    """
    Parameters
    ----------
    output_model_path : string
        path of Word2Vec model
    iter_docs : iterator
        iterator of documents, which are raw texts
    size : int
        size of word vector
    window : int
        window size of word2vec
    min_count : int
        minimum word count
    sg : int
        word2vec training algorithm (1: skip-gram other:CBOW)
    epoch : int
        number of epochs
    """

iter_docs is an iterator of word lists for each document.

2. Allow learning from any text dataset

Prepare an abstract class TextDatasetBase that defines a dataset read API. Arbitrary data set can be handled by implementing the read class of the data set that the user wants to use in a form that inherits this class.

class TextDatasetBase(ABC):
    """
    a bass class for text dataset
    
    Attributes
    ----------
    """
    @abstractmethod
    def iter_docs(self):
        """
        iterator of documents
        
        Parameters
        ----------
        """
        yield None

Example dataset class for MARD

class MARDDataset(TextDatasetBase):
    def __init__(self, word_tokenizer):
        self.root_path = None
        self.word_tokenizer = word_tokenizer

    def iter_docs(self, dataset_path):
        """
        get iterator of texts in one document
        
        Parameters
        ----------
        dataset_path: string
            path to dataset
        """
        self.root_path = Path(dataset_path)
        reviews_json_fn = self.root_path / "mard_reviews.json"
        with open(reviews_json_fn, "r") as fi:
            for line in fi:
                review_dict = json.loads(line, encoding="utf-8")
                title = review_dict["reviewerID"]
                text = review_dict["reviewText"]
                yield self.word_tokenizer.tokenize(text)

I feel that pytorch's DataLoader is about 200 million times more sophisticated than this, but this is what I came up with. Please let me know if you have a better design.

How to use

Installation

Take word2vec as an example

git clone [email protected]:stfate/word2vec-trainer.git
cd word2vec-trainer

git submodule init
git submodule update
pip install -r requirements.txt

Execution of learning

python train_text_dataset.py -o $OUTPUT_PATH --dictionary-path=$DIC_PATH --corpus-path=$CORPUS_PATH --size=100 --window=8 --min-count=5

How to use the model

model_path = "model/word2vec.gensim.model"
model = Word2Vec.load(model_path)

important point

When training with a large data set such as Wikipedia, it may eat up memory and fall. investigation in progress.

in conclusion

It's fun to think about the API of the library