Create an environment of 64bit Windows + python 2.7 + MeCab 0.996

Correction history

2016/2/19 Code correction / required file update 2016/2/19 Addition of troubleshooting

Introduction

Why on Windows in the first place? I think there is a story, but there is one reason. This is because Windows has a powerful IDE, Visual Studio.

Code completion, snippets, debugging ... Although it is an IDE that has many merits, there are many difficulties in building an environment on Windows, and it seems that Python-related problems often cause build problems. There are various reasons, but Unix commands do not work, and the file path is written differently.

This time, we are going to overcome such a hurdle and install the Python library of MeCab, a purveyor to natural language processors.

__What is MeCab? __ [MeCab] [* 0] is a morphological analysis tool. A morpheme means something like the smallest block of meaningful words. For example, "separate writing with python" can be decomposed into "separate writing with python". Moreover, it is multifunctional, such as analyzing part of speech and remodeling it into a prototype. It's amazing. There are other morphological analysis tools such as [JUMAN] [* 1] and [KAKASI] [* 2] from Kyoto University.

Things necessary

things to do

Basically, refer to both articles [mecab-python -Python] [* 3] [Build MeCab for 64-bit Windows (using Visual Studio 2010) -iPentec] [* 4] on Windows 64bit. I will proceed. Thanks. .. .. ..

  1. Install MeCab with mecab-0.996.exe installer
  2. Partially change the source of mecab-0.996.tar.gz, build it, and replace the built one with the contents of the installed MeCab.
  3. Rewrite the script of mecab-python-0.996.tar.gz for Windows and execute build and setup.

It seems easy to write like this!

1. Install MeCab

Installation of MeCab itself. Just run the installer and write the environment variables.

  1. Download and execute mecab-0.996.exe. The character code of the dictionary seems to be versatile, so I made it utf-8. However, utf-8 is garbled on the command prompt, so don't be surprised.
  2. Add C: \ Mecab \ bin to the environment variable PATH.
  3. Create a new environment variable MECABRC and set C: \ MeCab \ etc \ mecabrc

Environment variables change depending on the installation destination, so please check by yourself.

2. Modify and build MeCab source

Modify and build the source file. Since it is for 32bit, fix it for 64bit. After that, I fixed something that seems to be a little mistake.

  1. Extract mecab-0.996.tar.gz
  2. Fixed some files in mecab-0.996 \ src folder. Make the installation folder your own environment. (Delete the line # and add after !>)

<! LDFLAGS = /nologo /OPT:REF /OPT:ICF /LTCG /NXCOMPAT /DYNAMICBASE /MACHINE:X86 ADVAPI32.LIB !>LDFLAGS = /nologo /OPT:REF /OPT:ICF /LTCG /NXCOMPAT /DYNAMICBASE /MACHINE:X64 ADVAPI32.LIB ```

<! -DDLL_EXPORT -DHAVE_GETENV -DHAVE_WINDOWS_H -DDIC_VERSION=@DIC_VERSION@
!>-DDLL_EXPORT -DHAVE_GETENV -DHAVE_WINDOWS_H -DDIC_VERSION=102 \ ```

<! -DVERSION=""@VERSION@"" -DPACKAGE=""mecab""
!>-DVERSION=""0.996"" -DPACKAGE=""mecab""
```

<! -DMECAB_DEFAULT_RC=""c:\Program Files\mecab\etc\mecabrc"" !>-DMECAB_DEFAULT_RC=""d:\Programs\mecab\etc\mecabrc"" ```

<! case 't': os_ << (size_t)path->rnode->char_type; break; !>case 't': os_ << (unsigned int)path->rnode->char_type; break; ```

<! case 'L': *os << lattice->size(); break; !>case 'L': *os << (unsigned int)lattice->size(); break; ```

<! #ifndef SIWG !>#ifndef SWIG ```

!>#include ```

  1. Build. Execute in the mecab-0.996 \ src folder. It seems that the command prompt should be started with administrator privileges.

call "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat" amd64 nmake -f Makefile.msvc.in ```

  1. libmecab.dll, mecab-cost-train.exe, mecab-dict-gen.exe, mecab-dict-index.exe, mecab-system in MeCab \ bin of Mecab body Overwrite copy of -eval.exe and mecab-test-gen.exe.
  2. Overwrite mecab.h and libmecab.lib in MeCab \ sdk of MeCab main body. In addition, copy the same file into the mecab-python-0.996 folder. (* Updated necessary steps)

This is the end of the MeCab build! It's quite a problem because different people say different things.

__Supplement __ ~~ 5. is sufficient but may not be the minimum required. I will re-verify it at a later date ~~ (Confirmed! See above.)

3. Setup of mecab for python

Finally Python! There is a script that works only on Unix, so let's rewrite it for Windows.

  1. Extract mecab-python-0.996.tar.gz

  2. Rewrite setup.py as follows. Please change the installation destination as appropriate.

#!/usr/bin/env python from distutils.core import setup,Extension,os setup(name = "mecab-python", version = "0.996", py_modules=["MeCab"], ext_modules = [ Extension("_MeCab", ["MeCab_wrap.cxx",], include_dirs=[r"C:\MeCab\sdk"], library_dirs=[r"C:\MeCab\sdk"], libraries=["libmecab"]) ]) ```

  1. Build. create a build folder

python setup.py build running build running build_py running build_ext building '_MeCab' extension C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\BIN\amd64\cl.exe /c /nolo go /Ox /MD /W3 /GS- /DNDEBUG -ID:\Programs\MeCab\sdk -IC:\Develop\python27\inclu de -IC:\Develop\python27\PC /TpMeCab_wrap.cxx /Fobuild\temp.win-amd64-2.7\Releas e\MeCab_wrap.obj MeCab_wrap.cxx MeCab_wrap.cxx(3747) : warning C4530: C++I'm using exception handling, but unwind Semantics are not enabled./Please specify EHsc. C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\BIN\amd64\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:D:\Programs\MeCab\sdk /LIBPATH:C:\Develop\python27\libs /LIBPATH:C:\Develop\python27\PCbuild\amd64 libmecab.lib /EXPORT:init_MeCab build\temp.win-amd64-2.7\Release\MeCab_wrap.obj /OUT:build\lib.win-amd64-2.7_MeCab.pyd /IMPLIB:build\temp.win-amd64-2.7\Release_MeCab.lib/MANIFESTFILE:build\temp.win-amd64-2.7\Release_MeCab.pyd.manifest MeCab_wrap.obj : warning LNK4197:export'init_MeCab'Is specified multiple times. Apply the very first specification. Library build\temp.win-amd64-2.7\Release_MeCab.lib and object build\t emp.win-amd64-2.7\Release_MeCab.Creating exp ```

  1. Installation. Various files are copied to Lib \ site-packages where python is installed

python setup.py install running install running build running build_py running build_ext running install_lib copying build\lib.win-amd64-2.7\MeCab.py -> C:\Develop\python27\Lib\site-packages copying build\lib.win-amd64-2.7_MeCab.pyd -> C:\Develop\python27\Lib\site-packages byte-compiling C:\Develop\python27\Lib\site-packages\MeCab.py to MeCab.pyc running install_egg_info Writing C:\Develop\python27\Lib\site-packages\mecab_python-0.996-py2.7.egg-info ```

This completes the python setup! As was the case with the previous procedure, due to the difference in Unix terminal and command prompt specifications, it is necessary to directly specify the version and installation destination.

trouble shooting

__1. In python setup.py build, ʻerror: Unable to find vcvarsall.bat` __ I can't seem to read the Visual Studio file. Refer to [stack overflow] [* 5].

SET VS90COMNTOOLS=%VS140COMNTOOLS%`

If you do something like that, it should work. VS14 for Visual Studio 2015. For Visual Studio 2014, it's VS13, confusing but be careful.

__2. I don't have vcvarsall.bat in the first place ... __ Let's fix and install Visual studio. Launch the Visual Studio installer. If you have already installed it, there should be an item called Change. So, go to Programming Language-> Visual C ++ and check it. Then press Update.

2016-02-19.png

This should be fine.

test

After all, you can finally use MeCab! Let's try it immediately

test.txt


Hello. I'm Big Hero 6. Protect your health.

First, prepare the above test file. And with python,

> python
Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)] on win
32
Type "help", "copyright", "credits" or "license" for more information.
>>> import MeCab
>>> import sys
>>> m = MeCab.Tagger("-Owakati")
>>> f = open('test.txt','r')
>>> text = f.read().decode('utf-8')
>>> f.close()
>>> f = open('test.txt','w')
>>> f.write(m.parse(text.encode('utf-8')))
>>> f.close()

like this!

Then, the test file I mentioned earlier

Hello . I'm Big Hero 6. Protect your health.

See you! It's written properly! However, because of MeCab's dictionary, Big Hero 6 is split. .. .. .. .. .. ..

__ (• ー •) <Do you really need a care robot that can correctly morphologically analyze? __

You should be able to do it! So next time, let's add a dictionary.

reference

MeCab: Yet Another Part-of-Speech and Morphological Analyzer [KAKASI --Kanji → Kana (Romaji) conversion program] [* 1] [Japanese morphological analysis system JUMAN] [* 2] [On Windows 64bit, mecab-python -Python] [* 3] [Build MeCab for 64-bit Windows (using Visual Studio 2010) -iPentec] [* 4] What got stuck with the introduction of MeCab Python Add Star -Beginning of data mining and machine learning stack overflow -error: Unable to find vcvarsall.bat

<!-Reference list->

Recommended Posts

Create an environment of 64bit Windows + python 2.7 + MeCab 0.996
Put MeCab in "Windows 10; Python3.5 (64bit)"
Install python2.7 on windows 32bit environment
[Django] Memo to create an environment of Django + MySQL + Vue.js [Python]
python windows environment
Create an OpenCV3 + python3 environment on OSX
Introduced binding of MeCab (Wakame seaweed) and Python to Windows 7 64bit (2016/08/18)
Unification of Python environment
python windows environment construction
[Python] Create an asynchronous task execution environment + monitoring environment
Create a Python environment
Create a Python virtual development environment on Windows
Create a comfortable Python 3 (Anaconda) development environment on windows
Save screenshot of [Python] [Windows] screen as an image
Create an OpenAI Gym environment with bash on Windows 10
Create a decent shell and python environment on Windows
Python environment construction (Windows10 + Emacs)
Build Python environment on Windows
Build python environment on windows
Python environment construction under Windows7 environment
Environment construction of python2 & 3 (OSX)
Create an exe file that works in a Windows environment without Python with PyInstaller
Let's get started with Python ~ Building an environment on Windows 10 ~
Overview of Python virtual environment and how to create it
Create an executable file (EXE) by PyInstaller in a hybrid environment (Nimporter) of Python + Nim
Environment construction of python and opencv
Python environment construction memo on Windows 10
Installing Kivy on Windows10 64bit Python3.5
Anaconda python environment construction on Windows 10
Studying Python Part.1 Creating an environment
Create a Python environment on Mac (2017/4)
Create a virtual environment with Python!
Create an Excel file with Python3
[Python3] Development environment construction << Windows edition >>
Windows10: Install MeCab library in python
Installing Kivy-Designer on Windows10 64bit Python3.5
Create a Linux environment on Windows 10
Build mlpy with python3.3 (64bit) (windows 64bit)
Mecab / Cabocha / KNP on Python + Windows
Create a python environment on centos
Installation of Python, SciPy, matplotlib (Windows)
Environment construction of python3.8 on mac
[2021 version] Python installation Windows 10 (64bit) edition
Install Python development environment on Windows 10
Create a Python environment for professionals in VS Code on Windows
Create a Python execution environment for Windows with VScode + Remote WSL
[Docker] Build an environment of python (Flask) + GraphQL (graphene) + MySQL (sqlalchemy)
Create another version of Python conda environment with one command line
I tried to create an environment of MkDocs on Amazon Linux
Example of building python development environment on windows (wsl2, vscode, pipenv)
Morphological analysis of sentences containing recent words in Windows10 64bit environment
Various ways to create an array of numbers from 1 to 10 in Python.
[Python] Building an environment with Anaconda [Mac]
[Definitive Edition] Building an environment for learning "machine learning" using Python on Windows
Change Python 64bit environment to 32bit environment with Anaconda
Build an environment for Blender built-in Python
Python 2.7, 3.4, 3.5 extension module build environment on Windows
Python project environment construction procedure (for windows)
Using venv in Windows + Docker environment [Python]
How to create an NVIDIA Docker environment
The story of making Python an exe