Python script written in PyTorch is converted to exe with PyInstaller

environment

・ Windows 10 pro 64bit ・ Python 3.7.9 ・ Anaconda 4.8.5 ・ PyTorch 1.6.0 + cpu ・ Pyinstaller 4.0

Introduction

When converting a Python script to an exe, there is a Pyinstaller that is often used. Here, we will describe how to deal with the error that occurred when the script that created the document classifier with PyTorch was converted to an exe.

For those who use Anaconda

When installing the library, make sure that pip and conda are not mixed. Also, when converting to an exe, it is recommended to create a virtual environment in which only the libraries required for the script to be converted are installed. How to create a virtual environment is as follows.

The PyTorch forum below says to install with pip, so I installed everything with pip.

[Create exe file - PyTorch Forums] (https://discuss.pytorch.org/t/create-exe-file/56626)

How to use Pyinstaller

Installation

$ pip install pyinstaller

Normally, in the directory containing the script you want to convert to an exe, you can create an exe file with one command below. (If you want to have one exe file, add --onefile after it)

$ pyinstaller sample.py

This time, I edited the spec file and couldn't execute it well with onefile, so I will make it an exe file by the following two steps.

First, create a spec file with the following command.

$ pyi-makespec sample.py --onedir

Next, create the following command exe file.

$ pyinstaller sample.spec --clean

--clean is an option to delete files created by pyinstaller for the same script. If there is no data in build, there is no need to add it. There should be directories named build and dist in the directory where you ran this command. An exe file has been created in dist.

PyTorch error

The following error occurred when executing the exe file.

File "torch\__init__.py", line 189, in <module>
ImportError: numpy.core.multiarray failed to import

I solved it by adding the following statement to the python script I want to exe and explicitly importing multiarray.

import numpy.core.multiarray

Error or warning that dll file not found

lib not found: 〇〇.dll

I think that a file like 〇〇.spec is created when exe is made with pyi-makespec or pyinstaller. Modify the pathex part of a = Analysis in that file as follows:

sample.spec


a = Analysis(['sample.py'],
             pathex=['Path of file to be exe', 'The path of the directory containing the dll file that was giving the error(add to)'],
             binaries=[],
             datas=[],
             hiddenimports=[],
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher,
             noarchive=False)

In my case, it was torch_python.dll, so the'path (additional)'of the directory containing the dll file that was giving the error would be something like the following. (If you are working in a virtual environment called release) 'C:\ProgramData\Anaconda3\envs\release\Lib\site-packages\torch\lib'

$ pyinstaller sample.spec --clean

Janome error

If you are using Janome as a morphological analyzer, you will get the following error at runtime.

FileNotFoundError: [Errno 2] No such file or directory: 
'C:\\Users\\username\\AppData\\Local\\Temp\\_MEI172322\\janome\\sysdic\\entries_compact0.py'

File paths vary slightly from person to person. If you have the onefile option, you should have a similar path.

It seems that the cause is that the dictionary data used by Janome is not recognized by pyinstaller and cannot be incorporated. Copy __janome to the directory where the exe is located. There is no problem if you delete anything other than sysdic in janome.

In my case, the janome folder was in the following path. C:\ProgramData\Anaconda3\envs\release\Lib\site-packages\janome

The cause of the error was described in the following article. The story of making a typo / notation shake detection tool to reduce the depression of PowerPoint engineers

Errors related to transformers

If you are using tokenizer of transformers of huggingface, you will get the following error at runtime.

FileNotFoundError: [Errno 2] No such file or directory: 
'C:\\Users\\username\\AppData\\Local\\Temp\\_MEI191522\\sacremoses\\data\\perluniprops\\IsN.txt'

This is also the same as the error related to Janome, because the dictionary data used by __tokenizer is not recognized by pyinstaller and cannot be incorporated __. The workaround is the same as janome, and the solution is to copy __sacremoses to the directory where the exe is located. It is okay to delete anything other than data \ perluniprops in sacremoses. The path of sacremoses in my case is as follows. C:\ProgramData\Anaconda3\envs\release\Lib\site-packages\sacremoses

Finally

When using Pyinstaller, I get an error that I don't understand. If you can't figure out the cause, one way is to deal with the warning text. I was doing that too: sweat: When I looked it up and it didn't come out, I was about to give up, but it's something I can do. The development of Pyinstaller itself is still ongoing, so it may be resolved in a future update.

reference

Create exe file - PyTorch Forums https://discuss.pytorch.org/t/create-exe-file/56626

The story of making a typo / notation shake detection tool to reduce the depression of PowerPoint engineers https://qiita.com/youwht/items/062c41c88829fcf25107

Recommended Posts

Python script written in PyTorch is converted to exe with PyInstaller
PyInstaller memorandum Convert Python [.py] to [.exe] with 2 lines
[Python] What to do when No module named'pyproj.datadir' appears when Exe is done with PyInstaller
[Python Kivy] How to create an exe file with pyinstaller
Convert files written in python etc. to pdf with syntax highlighting
Try logging in to qiita with Python
Stress Test with Locust written in Python
Automatic update method with python Pyinstaller exe
How to work with BigQuery in Python
[Python] Introduction to CNN with Pytorch MNIST
To work with timestamp stations in Python
How to use is and == in Python
Creating an exe file with Python PyInstaller: PC freezes in parallel processing
[REAPER] How to play with Reascript in Python
Introduction to effectiveness verification Chapter 3 written in Python
Convert PDFs to images in bulk with Python
Log in to Yahoo Business with Selenium Python
[Improved version] Script to monitor CPU with Python
Introduction to Effectiveness Verification Chapter 2 Written in Python
Python script to get note information with REAPER
Output timing is incorrect when standard (error) output is converted to a file in Python
Type notes to Python scripts for running PyTorch model in C ++ with libtorch
How to convert / restore a string with [] in python
Periodically execute python script with cron in venv environment
How to add help to HDA (with Python script bonus)
Explain in detail how to make sounds with python
Create an exe file that works in a Windows environment without Python with PyInstaller
How to run tests in bulk with Python unittest
A memorandum to run a python script in a bat file
Convert the image in .zip to PDF with Python
I was addicted to scraping with Selenium (+ Python) in 2020
Preparing to script control Rhino objects in Grasshopper / Python
I want to work with a robot in python.
It is easy to execute SQL with Python and output the result in Excel
Script to count and stop up to 5 seconds in Python in Blender
Assigned scaffolding macro in Python script file to F12 key
After all it is wrong to cat with python subprocess.
How to test that Exception is raised in python unittest
[Road to intermediate Python] Install packages in bulk with pip
How to extract any appointment in Google Calendar with Python
How to implement Python EXE for Windows in Docker container
Introduction to Effectiveness Verification Chapters 4 and 5 are written in Python
Things to keep in mind when using Python with AtCoder
Things to keep in mind when using cgi with python.
How to log in to AtCoder with Python and submit automatically
Until API made with Flask + MySQL is converted to Docker
Connect to BigQuery with Python
Scraping with selenium in Python
To flush stdout in Python
Working with LibreOffice in Python
Scraping with chromedriver in python
POST json with Python3 script
Debugging with pdb in Python
Login to website in Python
Post to slack with Python 3
Working with sounds in Python
Scraping with Selenium in Python
Scraping with Tor in Python
Tweet with image in Python
Combined with permutations in Python
Speech to speech in python [text to speech]