Make Scrapy an exe with Pyinstaller

Overview

I was able to convert it to an exe with Pyinstaller, but it crashed when I started the exe file. Now that I know how to make it work perfectly, make a note of it.

environment

OS:Windows 10 Language: Python 3.7 scrapy version: 1.7.3.

Premise

You have created a scrapy spider, and you have a script file that runs the scrapy file.

The following configuration demo_scrapy/ ├── scrapy.cfg ├── lauch_demo.py ← This is the script file that runs the scrapy file └── demo_scrapy     ├── init.py     ├── pycache     ├── items.py     ├── middlewares.py     ├── pipelines.py     ├── settings.py     └── spiders         ├── init.py         ├── demo_spider.py         └── pycache

Contents of lauch_demo.py import datetime import time from scrapy.spiderloader import SpiderLoader from scrapy.crawler import CrawlerProcess from scrapy.utils.project import get_project_settings

now = datetime.datetime.now() dt_now_date = now.strftime("%Y-%m-%d-%H%M%S") settings = get_project_settings()

settings.set('FEED_URI', 'demo' + '_' + dt_now_date + '.csv') process = CrawlerProcess(settings) process.crawl('demo_spider', domain='demo.com') process.start() # the script will block here until the crawling is finished

print ("completed")

problem

Here, when launch_demo.py is started as usual, it works normally, However, the exe file does not work even if it is converted to exe with Pyinstaller.

Perform the following method.

Solution

  1. Create a scrapy folder in any location for the VERSION and mime.types files and save it there. VERSION and mime.types C: \ Users \ username \ AppData \ Local \ Programs \ Python \ Python37-32 \ Lib \ site-packages \ scrapy It is stored in.

  2. Add hidden-import. At the command prompt pyinstaller --hidden-import scrapy.spiderloader --hidden-import scrapy.statscollectors --hidden-import scrapy.logformatter --hidden-import scrapy.extensions --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.telnet --hidden-import scrapy.extensions.memusage --hidden-import scrapy.extensions.memdebug --hidden-import scrapy.extensions.closespider --hidden-import scrapy.extensions.feedexport --hidden-import scrapy.extensions.logstats --hidden-import scrapy.extensions.spiderstate --hidden-import scrapy.extensions.throttle --hidden-import scrapy.core.scheduler --hidden-import scrapy.squeues --hidden-import queuelib --hidden-import scrapy.core.downloader --hidden-import scrapy.downloadermiddlewares --hidden-import scrapy.downloadermiddlewares.robotstxt --hidden-import scrapy.downloadermiddlewares.httpauth --hidden-import scrapy.downloadermiddlewares.downloadtimeout --hidden-import scrapy.downloadermiddlewares.defaultheaders --hidden-import scrapy.downloadermiddlewares.useragent --hidden-import scrapy.downloadermiddlewares.retry --hidden-import scrapy.downloadermiddlewares.ajaxcrawl --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.httpcompression --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.cookies --hidden-import scrapy.downloadermiddlewares.httpproxy --hidden-import scrapy.downloadermiddlewares.stats --hidden-import scrapy.downloadermiddlewares.httpcache --hidden-import scrapy.spidermiddlewares --hidden-import scrapy.spidermiddlewares.httperror --hidden-import scrapy.spidermiddlewares.offsite --hidden-import scrapy.spidermiddlewares.referer --hidden-import scrapy.spidermiddlewares.urllength --hidden-import scrapy.spidermiddlewares.depth --hidden-import scrapy.pipelines --hidden-import scrapy.dupefilters --hidden-import scrapy.core.downloader.handlers.datauri --hidden-import scrapy.core.downloader.handlers.file --hidden-import scrapy.core.downloader.handlers.http --hidden-import scrapy.core.downloader.handlers.s3 --hidden-import scrapy.core.downloader.handlers.ftp --hidden-import scrapy.core.downloader.webclient --hidden-import scrapy.core.downloader.contextfactory "C:/path/to/launch_demo.py"

To execute. If you get "module not found XXX", add --hidden-import XXX.

  1. Store the scrapy folder created in 1 in the created. \ Dist \ lauch_demo \ folder.

  2. Running the exe file works fine.

References

How to create a single executable file in windows 10 with scrapy and pyinstaller? https://stackoverflow.com/questions/55331478/how-to-create-a-single-executable-file-in-windows-10-with-scrapy-and-pyinstaller

Pyinstaller scrapy error: https://stackoverflow.com/questions/25557693/pyinstaller-scrapy-error

Finally

If you have any questions, please let us know. Thank you very much.

Recommended Posts

Make Scrapy an exe with Pyinstaller
Make an audio interface controller with pyusb (2)
Make an audio interface controller with pyusb (1)
Automatic update method with python Pyinstaller exe
When I get an error with PyInstaller
EXE the application created with PyQt5 with PyInstaller
Creating an exe file with Python PyInstaller: PC freezes in parallel processing
Restart with Scrapy
[Python] Make a game with Pyxel-Use an editor-
Make a monitoring device with an infrared sensor
Make Lambda Layers with Lambda
Make Yubaba with Discord.py
Scraping with scrapy shell
Reinforcement learning 37 Make an automatic start with Atari's wrapper
How to make an HTTPS server with Go / Gin
PyInstaller memorandum Convert Python [.py] to [.exe] with 2 lines
Make a CSV formatting tool with Python Pandas PyInstaller
Make an umbrella reminder with Raspberry Pi Zero W
Problems with installing Scrapy
I tried to make an OCR application with PySimpleGUI
I can't exe a project using PyWebView with PyInstaller
Make slides with iPython
Make an application using tkinter an executable file with cx_freeze
Precautions when inputting from CSV with Python and outputting to json to make it an exe
Make an air conditioner integrated PC "airpi" with Raspberry Pi 3!
Dynamically import modules at runtime with .exe hardened with pyinstaller --onefile
Create an environment with virtualenv
Make an MQTT Publisher box
Create an API with Django
Make sci-fi-like buttons with Kivy
Festive scraping with Python, scrapy
Creating GUI tools with pyinstaller
Let's make an Errbot plugin
Easy to make with syntax
Make Puyo Puyo AI with Python
Easy web scraping with Scrapy
Let's make dice with tkinter
Make a fortune with Python
Make a fire with kdeplot
Make an MQTT Subscriber box
Make Slack chatbot with Errbot
How to get the directory where the EXE built with Pyinstaller exists
When I run the exe file with pyinstaller, my PC crashes.
Make an effector for video conferencing with Spout + OpenCV + dlib (Part 1)
Python script written in PyTorch is converted to exe with PyInstaller
Solution when an error occurs when hiding the console screen with PyInstaller
Let's make an IoT shirt with Lambda, Kinesis, Raspberry Pi [Part 1]
I tried to make an image similarity function with Python + OpenCV