web scraping (prototype)

Introduction

This time, I tried to create scraping with python. I wanted to write it neatly to csv, but due to lack of technology, I just wrote it for the time being, so I want to make it easier to see as shown in the table. I created it in a fumbling state, so it works, but I would appreciate it if you could comment if there is something that should be changed.

environment

windows 10 python 3.8.0 Library urllib.request # Required to access the URL. csv #csv Required when working with files. BeautifulSoup #A module specializing in scraping required for parsing HTML and XML documents. datetime # Required to get the date.


code

scraping.py


import urllib.request
import csv
from bs4 import BeautifulSoup
import datetime

#URL of the site
url = "URL of the site you want to scrape"
#Open URL
html = urllib.request.urlopen("URL of the site you want to scrape")
#Open with Beautiful Soup
soup = BeautifulSoup(html, 'html.parser')

#Extraction of elements
site_title = soup.find_all("Elements you want to extract")
print(site_title)


#Include the date when creating the csv file
csv_date = datetime.datetime.today().strftime("%Y%m%d")

#file name
csv_file_name = "freedom" + csv_date + ".csv"

#open csv
csv_file = open(csv_file_name, "w")

#Write to csv
write_csv = csv.writer(csv_file, lineterminator='\n')
write_csv.writerows(site_title)

#close csv
f.close()

Recommended Posts

web scraping (prototype)
web scraping
Introduction to Web Scraping
Python web scraping selenium
Web scraping with python + JupyterLab
Web scraping notes in python3
Save images with web scraping
Web scraping technology and concerns
Trade-offs in web scraping & crawling
Easy web scraping with Scrapy
Image collection by web scraping
Web scraping using Selenium (Python)
Web scraping using AWS lambda
Web scraping beginner with python
Scraping 1
Algorithm-based web scraping library Scrapely
One-liner web scraping by tse
Web scraping with Python ① (Scraping prior knowledge)
Web scraping with BeautifulSoup4 (layered page)
Scraping Alexa's web rank with pyQuery
Web scraping with Python First step
I tried web scraping with python.
GAN: DCGAN Part1 --Scraping Web images
Beginners use Python for web scraping (1)
Web scraping for weather warning notifications.
Beginners use Python for web scraping (4) ―― 1
10 questions to check before web scraping
WEB scraping with Python (for personal notes)
Various scraping
Start scraping
Getting Started with Python Web Scraping Practice
[Personal note] Web page scraping with python3
Web scraping with Python ② (Actually scraping stock sites)
[Scraping] Python scraping
Horse Racing Site Web Scraping with Python
Getting Started with Python Web Scraping Practice
Scraping sample
Practice web scraping with Python and Selenium
Easy web scraping with Python and Ruby
Web scraping with BeautifulSoup4 (serial number page)
[For beginners] Try web scraping with Python
I tried web scraping to analyze the lyrics.
I tried web scraping using python and selenium
Pharmaceutical company researchers summarized web scraping using Python
[Python] Flow from web scraping to data analysis
Get boat race match information by web scraping
Let's do web scraping with Python (weather forecast)
Let's do web scraping with Python (stock price)