I tried running BERT with Sakura VPS (without GPU)

Introduction

This article Explanation about BERT! Let's make a prediction using the Japanese model! If you follow the street, you can easily experience BERT on Google cola boratory.

I wondered if it could be used publicly on a cheap server instead of on Google colaboratory, so I tried running it on Sakura VPS1G (800 yen per month).

The point is that the GPU that can be used with Google colaboratory cannot be used with Sakura VPS, so it is necessary to install the CPU version of the library.

Installation environment

Sakura VPS Memory 1 GB, storage SSD 50 GB, CPU 2 cores (800 yen per month) Standard installation CentOS8 x86_64

Introduction

First of all, referring to the following article, I created an "anaconda" user and created an environment "ml_env" for machine learning. In the article, it was built with Centos 7.7, but I had no problems with CentOS 8. [Part 1] Build a Python machine learning development environment on CentOS (Anaconda installation) --Qiita

Continuing from the above article, we will activate the machine learning environment "ml_env" as an "anaconda" user and introduce BERT in this environment.

Installing the CPU version of Transformers

pip install transformers[tf-cpu]

Installing the CPU version of Torch

pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

You have now installed the required libraries.

BERT Japanese Pretrained model download

Next, [Kurobashi / Kawahara / Murawaki Laboratory](http://nlp.ist.i.kyoto-u.ac.jp/index.php?BERT%E6%97%A5%E6%] of Kyoto University 9C% AC% E8% AA% 9EPretrained% E3% 83% A2% E3% 83% 87% E3% 83% AB) is available for download, BERT Japanese Pretrained model.

Assuming you are currently in / home / anaconda / as an anaconda user, create and move the directory / home / anaconda / bert / japan_test to store your Japanese model.

mkdir bert
cd bert
mkdir japan_test
cd japan_test

Then download, unzip, and delete the download file.

curl -OL http://nlp.ist.i.kyoto-u.ac.jp/nl-resource/JapaneseBertPretrainedModel/Japanese_L-12_H-768_A-12_E-30_BPE.zip
unzip Japanese_L-12_H-768_A-12_E-30_BPE.zip
rm -rf Japanese_L-12_H-768_A-12_E-30_BPE.zip

The model is now installed at / home / anaconda / bert / japan_test / apanese_L-12_H-768_A-12_E-30_BPE.

Exit from the directory you created.

cd ../..

Place the Python program file bert_test.py here for the BERT test in/ home / anaconda /.

bert_test.py


from transformers import BertTokenizer, BertForMaskedLM, BertConfig
import torch
import numpy as np

config = BertConfig.from_json_file('/home/anaconda/bert/japan_test/Japanese_L-12_H-768_A-12_E-30_BPE/bert_config.json')
model = BertForMaskedLM.from_pretrained('/home/anaconda/bert/japan_test/Japanese_L-12_H-768_A-12_E-30_BPE/pytorch_model.bin', config=config)
bert_tokenizer = BertTokenizer('/home/anaconda/bert/japan_test/Japanese_L-12_H-768_A-12_E-30_BPE/vocab.txt',
 do_lower_case=False, do_basic_tokenize=False)

tokenized_text = ['[CLS]', 'Once upon a time', 'Urashima', 'That', 'Fisherman', 'But', 'Live', 'Was there', '。'
, 'is there', 'Day', '、', '[MASK]', 'But', 'beach', 'To', 'Walking', 'When you are', '[SEP]']
masked_index = 12

tokens=bert_tokenizer.convert_tokens_to_ids(tokenized_text)
tokens_tensor=torch.tensor([tokens])

model.eval()
with torch.no_grad():
  outputs = model(tokens_tensor)
  predictions = outputs[0]
_,predicted_indexes=torch.topk(predictions[0,masked_index],k=10)
predicted_tokens = bert_tokenizer.convert_ids_to_tokens(predicted_indexes.tolist())
print(predicted_tokens)

This program

['[CLS]', 'Once upon a time', 'Urashima', 'That', 'Fisherman', 'But', 'Live', 'Was there', '。', 'is there', 'Day', '、', '[MASK]', 'But', 'beach', 'To', 'Walking', 'When you are', '[SEP]']

Predict the words that go into the [MASK] of.

python bert_test.py

If you run it with

['Fisherman', '[UNK]', 'Man', 'people', 'Man', 'Youth', 'woman', '役Man', 'boy', 'fish']

That's the result. You have succeeded in running BERT.

Other

I wanted to use this BERT analysis result in a web application, so I set up a local web server in Flask and made the BERT model daemonized (resident) in Flask. Then I set up a web server with Apache and published it as an API. In other words, when I access the Apache API from the outside, the API program accesses the BERT on Flask and obtains the analysis result of the BERT. that is,

BERT Word Guess Demo

It will be. Please try it if you like.

Recommended Posts

I tried running BERT with Sakura VPS (without GPU)
I tried running prolog with python 3.8.2.
I tried running faiss with python, Go, Rust
I tried running Deep Floor Plan with Python 3.6.10.
I tried running pymc
I tried running TensorFlow
I tried fp-growth with python
I tried scraping with Python
I tried Learning-to-Rank with Elasticsearch!
I tried clustering with PyCaret
I tried gRPC with Python
I tried scraping with python
I tried running Movidius NCS with python of Raspberry Pi3
I tried trimming efficiently with OpenCV
I tried summarizing sentences with summpy
I tried machine learning with liblinear
I tried web scraping with python.
I tried moving food with SinGAN
I tried implementing DeepPose with PyTorch
I tried running GAN in Colaboratory
I tried sending an email from the Sakura server with flask-mail
I tried Grumpy (Go running Python).
I tried face detection with MTCNN
I tried running the DNN part of OpenPose with Chainer CPU
I tried SMTP communication with Python
I tried sentence generation with GPT-2
I tried learning LightGBM with Yellowbrick
I tried running the TensorFlow tutorial with comments (_TensorFlow_2_0_Introduction for beginners)
I tried face recognition with OpenCV
I was addicted to running tensorflow on GPU with NVIDIA driver 440 + CUDA 10.2
I put Python 2.7 in Sakura VPS 1GB.
I tried multiple regression analysis with polynomial regression
I tried using Amazon SQS with django-celery
I tried to implement Autoencoder with TensorFlow
I tried linebot with flask (anaconda) + heroku
I tried to get started with Hy
I tried scraping Yahoo News with Python
I tried using Selenium with Headless chrome
I tried factor analysis with Titanic data!
I tried learning with Kaggle's Titanic (kaggle②)
I tried sending an email with python.
I tried non-photorealistic rendering with Python + opencv
I tried a functional language with Python
I tried batch normalization with PyTorch (+ note)
I tried recursion with Python ② (Fibonacci sequence)
I tried implementing DeepPose with PyTorch PartⅡ
I tried to implement CVAE with PyTorch
I tried playing with the image with Pillow
I tried to solve TSP with QAOA
I tried simple image recognition with Jupyter
I tried CNN fine tuning with Resnet
I tried natural language processing with transformers.
#I tried something like Vlookup with Python # 2
I tried Hello World with 64bit OS + C language without using the library
[Patent analysis] I tried to make a patent map with Python without spending money
I tried running the TensorFlow tutorial with comments (text classification of movie reviews)
I tried object detection with YOLO v3 (TensorFlow 2.1) on the GPU of windows!
I tried to predict next year with AI
I tried "smoothing" the image with Python + OpenCV
I tried hundreds of millions of SQLite with python
I tried to detect Mario with pytorch + yolov3