Automate simple tasks with Python Part1 Scraping

at first

The first of the first series is Python as much as possible ... (Does this series continue properly?) Also, this series is not like writing code that can automate work as it is, but it is a series that does ** automation of this, etc. ** **

Before scraping

** Please be sure to read this. ** ** First of all, scraping means that the computer does the work that humans do, so you can access it many times. However, doing so puts a burden on the server, so it is necessary to take measures such as once per second.

From here is the most important. You have to make sure that the site you are scraping ** allows scraping ** and so on.

Main story

I will write it for those who are old and scrape quickly without saying such a thing.

First of all, install the libraries required for scraping.

It's a library called Beautiful Soup.

If you have Anaconda installed, it is included from the beginning, but if you get an error, please execute this code.

conda install BeautifulSoup4 lxml

Eh? Isn't the conda command pip? It can't be helped. ~~ Gentle Faguri will write it. ~~

pip install BeautifulSoup4 lxml

Please run the.

How to use

code.py


from bs4 import BeautifulSoup
import requests

page_data = requests.get('https://ja.wikipedia.org/wiki/%E3%82%A6%E3%82%A7%E3%83%96%E3%82%B9%E3%82%AF%E3%83%AC%E3%82%A4%E3%83%94%E3%83%B3%E3%82%B0').text
page = BeautifulSoup(page_data, 'lxml')

for element in page.select("#mw-content-text > div > p:nth-child(1)"):
    print(element.text)

Run

Web scraping (English: Web scraping) is a computer software technology that extracts information from websites. Also known as a web crawler [1] or web spider [2]. Such software programs typically acquire WWW content by implementing low-level HTTP or by embedding a web browser.

However

For those who are just starting out, I think it's Nanikore, especially "for element in page.select (" # mw-content-text> div> p: nth-child (1) "):" I will. # mw-content-text> div> p: nth-child (1) "): How to find it (like a math explanation) Right click in Google Chrome> Verification> And![Verification.png](https: / /qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/551445/6751d074-acea-990a-04af-3e246bd654fb.png)

Click the area surrounded by the red frame, move the cursor to the area you want to scrape, right-click the area that is light blue, copy it with Copy> Copy selector, and paste it in that area. ~~ This will not complain even for beginners ~~

At the end

It depends on the person, but Python can be scraped by this alone. Take care of your body! (It doesn't matter at all)

Recommended Posts

Automate simple tasks with Python Part1 Scraping
Automate simple tasks with Python Part0
Scraping with Selenium + Python Part 1
Scraping with Selenium + Python Part 2
Scraping with Python
Scraping with Python
Python: Scraping Part 1
Python: Scraping Part 2
[Part1] Scraping with Python → Organize to csv!
Scraping with Python (preparation)
Try scraping with Python.
Scraping with Python + PhantomJS
Scraping with Selenium [Python]
Scraping with Python + PyQuery
Scraping RSS with Python
Automating simple tasks with Python Table of contents
Image processing with Python (Part 2)
Bordering images with python Part 1
Web scraping with python + JupyterLab
Scraping with selenium in Python
Scraping with chromedriver in python
Festive scraping with Python, scrapy
Automate python testing with CircleCI
Scraping with Selenium in Python
Studying Python with freeCodeCamp part2
Image processing with Python (Part 1)
Scraping with Tor in Python
Solving Sudoku with Python (Part 2)
Image processing with Python (Part 3)
Scraping weather forecast with python
I tried scraping with python
Automate sushi making with Python
Web scraping beginner with python
Try scraping with Python + Beautiful Soup
Playing handwritten numbers with python Part 1
Scraping with Node, Ruby and Python
Web scraping with Python ① (Scraping prior knowledge)
[Automation with python! ] Part 1: Setting file
Scraping with Selenium in Python (Basic)
Scraping with Python, Selenium and Chromedriver
Web scraping with Python First step
I tried web scraping with python.
Scraping with Python and Beautiful Soup
Let's do image scraping with Python
Get Qiita trends with Python scraping
[Automation with python! ] Part 2: File operation
"Scraping & machine learning with Python" Learning memo
Get weather information with Python & scraping
Excel aggregation with Python pandas Part 1
Freezing of tasks (2/2)
Administrative tasks
Automating simple tasks with Python Table of contents
[Scraping] Python scraping
[Python] Automate Pelican builds with Travis CI
Creating a simple PowerPoint file with Python
WEB scraping with Python (for personal notes)
Play handwritten numbers with python Part 2 (identify)
FM modulation and demodulation with Python Part 3
Process Pubmed .xml data with python [Part 2]
Simple Slack API client made with Python
Getting Started with Python Web Scraping Practice
I tried scraping Yahoo News with Python
Machine learning with python (2) Simple regression analysis