I tried scraping with Python3 before (Scraping books borrowed from the library using Python3 and BeautifulSoup --Qiita), but the environment changed and I tried again. It didn't work. After changing the settings, it worked, so leave a note at that time.
Google Chrome OS
Version 80.0.3987.89 (Official Build) beta (64-bit)
$ python3 --version
Python 3.5.3
$ sudo apt-get install python3-pip
$ pip3 --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.5)
$ sudo apt-get install python3-bs4
$ sudo apt-get install python3-requests
$ pip3 list
beautifulsoup4 (4.5.3)
requests (2.12.4)
Referenced site ↓ Supports requests from Python 3.4 or later --Qiita
I get an error when I run it normally.
$ python3 scrape.py
request
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 308, in ssl_wrap_socket
context.load_verify_locations(ca_certs, ca_cert_dir)
ssl.SSLError: unknown error (_ssl.c:3172)
(abridgement)
After searching, this site was a hit. Python requests library can't make HTTPS connections (was: Apache certbot error) - Help - Let's Encrypt Community Support
I tried the command written here in `Solved by:`.
$ cat /etc/ssl/certs/ca-certificates.crt
$ sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
done.
$ cat /etc/ssl/certs/ca-certificates.crt
-----BEGIN CERTIFICATE-----
(abridgement)
-----END CERTIFICATE-----
After this, I ran it again and it worked. It seems that the CA certificate was not included. Is it a public key? It seems that the default is not included when launching a virtual environment by yourself. I didn't like it. I'm not very good at this. For the time being, I'm glad I moved.
Reference site ↓ Ubuntu allows you to install CA certificates with the ca-certificates package-Hidamari sockets won't break What is a Certificate Authority (CA)? ｜ GMO GlobalSign [Official]
Recommended Posts