Install Scrapy on python3
When I was reading "Web scraping with Python", I decided to use scrapy for some, but I could not install it well, so I will briefly summarize the solution. (By the way, the above books are mainly scraped using Beautiful Soup)
Set environment
MacBook Air (13-inch, Mid 2011) Processor: 1.8 GHz Intel Core i7 Memory: 4 GB 1333 MHz DDR3 Version: 10.11.5 Python: 3.6.2
Try putting Scrapy
1, This time I will put it in the environment of Python3.
$ pip3 install scrapy
Then ... I got the following error. Apparently, pip should be upgraded.
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
2, Follow the instructions in the error statement to upgrade pip.
$ pip3 install --upgrade pip
Again there is an error ... This time it seems to be around xcode.
1 error generated.
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
Perhaps try: xcode-select --install
*********************************************************************************
error: command '/usr/bin/clang' failed with exit status 1
3, Next, just follow the instructions, so type the command.
$ xcode-select --install
xcode-select: note: install requested for command line developer tools
Then a dialog will be displayed, so agree and download the developer tools.
4, After downloading, execute the following again to complete the installation.
$ sudo pip3 install scrapy
(Omitted)
Installing collected packages: lxml, parsel, scrapy
Running setup.py install for lxml ... done
Successfully installed lxml-3.6.0 parsel-1.0.2 scrapy-1.1.0
Recommended Posts