What to do when there is no response due to Proxy setting in Python web scraping

I will leave a memorandum of what to do if you get stuck in an error when trying to create a program that automatically extracts the Web information required for scraping with Python.

I was trying to make a web scraping program with Python's urllib and Beautifulsoup. However, I got an error that I couldn't get a response from the first urllib.request.urlopen (...) (I get a message like the one below).

It seems that communication has not been established due to the existence of the Proxy server. Proxy was as follows in Internet Explorer.

-[Tools]-> [Internet Options]-> [Connections]-> [LAN Settings]

-[x] Use automatic configuration script

The automatic configuration script was http://proxy.-----.co.jp/proxy.pac. (----- is not the actual one, it is in hidden letters.)

Before urlopen, I solved it by preparing ProxyHandler for urllib.request, setting it in build_opener, and installing build_opener.

The sample code is below.

scrapetest.py


import urllib.request
proxies ={'http':'http://proxy.-----.co.jp/proxy.pac'}
proxy_handler = urllib.request.ProxyHandler(proxies)
opener = urllib.request.build_opener(proxy_handler)
urllib.request.install_opener(opener)
html = urllib.request.urelopen("http://wwww.pythonscraping.com/pages/page1.html")
print(html.read())

The development environment is Python 3.5.2 by Anaconda on Windows.

You will learn about web scraping programming in the book below. Web scraping with Python (O'Reilly)

Recommended Posts

What to do when there is no response due to Proxy setting in Python web scraping
What to do when ModuleNotFoundError: No module named'XXX' occurs in Python
What to do when the value type is ambiguous in Python?
What to do if there is a decimal in python json .dumps
What to do when [Errno 2] No such file or directory appears in Python
[Python] What to do when No module named'pyproj.datadir' appears when Exe is done with PyInstaller
What to do when "SSL: CERTIFICATE_VERIFY_FAILED _ssl.c: 1056" appears in Python
There is no switch in python
NameError: global name'dot_parser' is not defined and what to do when it comes up in python
What to do if No Python documentation found for ... appears in pydoc
[Python] What to do when an error related to SSL authentication is returned
What to do when the warning "The environment is in consistent ..." appears in the Anaconda environment
What to do when a warning message is displayed in pip list
[Python] What to do when PEP8 is violated in the process of importing from the directory added to sys.path
How to write what to do when an application is first displayed in Qt for Python with Designer
What to do to get google spreadsheet in python
What to do when a warning appears around Python integration in Neovim's CheckHealth
What to do if abort is displayed when inputting camera video in OpenCV
Python virtual environment in 2021 ~ There is no reason not to use venv now ~
What I do when imitating embedded go in python
[openpyxl] What to do when IllegalCharacterError appears in pandas.DataFrame.to_excel
curl: (60) What to do when Issuer certificate is invalid.
What to do when "cannot import name xxx" [Python]
pipenv shell is no longer available ... what to do?
I want to do something in Python when I finish
What to do when you can't bind CaboCha to Python
[Python] Type Error:'WebElement' object is not iterable What to do when an error occurs
What to do when is not in the sudoers file.This incident will be reported.
What to do if you get `No kernel for language python found` in Hydrogen
What to do when only the window is displayed and nothing is displayed in pygame Note
What to do if you get an error when importing matplotlib in Python (Mac)
What to do if the progress bar is not displayed in tqdm of python
What to do if Python IntelliSense is not displayed in VS Code on Windows
What to do when Python starts up in Anaconda does not come out unexpectedly
What to do if ʻarguments [0] .scrollIntoView ();` fails in python selenium
What to do when Japanese is not displayed on matplotlib
Web scraping notes in python3
What to do when a Missing artifact occurs in a jar that is not defined in pom.xml
What to do if you get Swagger-codegen in python and Import Error: No module named
What to do when "Invalid HTTP_HOST header" appears in Django
When "No changes detected" is displayed in python3 manage.py makemigrations
ImportError: No module What to do when you are told
What to do when no display name occurs when unittesting Python + Tkinter on Github Actions Memo
What to do when the value type is ambiguous in Python?
What to do if there is a decimal in python json .dumps
What is "mahjong" in the Python library? ??
What is wheezy in the Docker Python image?
What to do when is not in the sudoers file.This incident will be reported.
What to do if the progress bar is not displayed in tqdm of python
Solve the one-stroke writing (backtrack without recursion in Python)
What to do when there is no response due to Proxy setting in Python web scraping
What to do if an error message "There was a problem displaying this web page" is displayed in Chrome's Web Driver (Selenium)
What to do if you get `locale.Error: unsupported locale setting` when getting the day of the week from a date in Python
When merging via pull request, there is no committer information in the response from Github API
What to do if Insecure Platform Warning appears when running Python
What to do if "Unnamed: 0" is added in to_csv-> read_csv in pandas
What to do when UnicodeDecodeError occurs during read_csv in pandas (pd.read_table ())
What to do with PYTHON release?
[Python] Whiten the parts that turn black when there is no data in the Choropleth map of Folium.
[Mac OS] What to do when Python is not installed as a framework. Is displayed when import matplotlib is performed.
What to do if PyInstaller3.5 gives an error in Python3.8 (TypeError: an integer is required (got type bytes))
What to do in my case when pyenv install is not possible after upgrading to macOS Big Sur
[Python] What to do if you get a ModuleNotFoundError when importing pandas using Jupyter Notebook in Anaconda
What to do if you can't use scikit grid search in Python
How to write a string when there are multiple lines in python
When issuing an INSERT statement in Python, no such column is displayed
After all, what should I use to do type comparisons in Python?
What is the fastest way to create a reverse dictionary in python?