In the recent trend of big data, how much data should be collected is required.
As one of the technologies, web scraping may be adopted.
In this article, I would like to summarize the methods of web scraping and precautions regarding their use.
Web scraping (English: Web scraping) is a computer software technology that extracts information from websites. Also known as a web crawler [1] or web spider [2]. Such software programs typically acquire WWW content by implementing low-level HTTP or by embedding a web browser. (From Wikipedia)
3-1. IP blocking 3-2. Compliance with company ethics and compliance
From the conclusion, the load on the server due to scraping, if there is no corresponding description in the terms of use of the relevant site, ** if the purpose is to analyze the information **, the copyright is exceptional. It seems that the general view is that there is no problem in recording or adapting other companies' information obtained by scraping on a recording medium without obtaining the consent of the person. (As of 02/23/2020)
Article 47-5 Contribute to promoting the use of copyrighted works by creating new knowledge or information through information processing using computers. Persons who perform the acts listed in the following items (including those who perform a part of the act and limited to those who perform the act in accordance with the standards specified by Cabinet Order) are provided or presented to the public (enable transmission). (Hereinafter, the same shall apply in this Article) (hereinafter referred to as "publicly provided presentation work" in this Article and the following Article, Paragraph 2, Item 2) (Published work or transmission enablement) (Limited to copyrighted works), to the extent deemed necessary for the purpose of the acts listed in each item, regardless of which method is used in connection with the act (the copyrighted work presented to the public) Of these, the proportion of the part used for the use, the amount of the part used for the use, the accuracy of the display when the part is used, and other factors are limited to those that are minor. "Minor use") can be performed. However, if the provision or presentation of the work presented to the public to the public infringes the copyright (if the provision or presentation to the public made overseas is made domestically) If you make a minor use while knowing that it should be an infringement of copyright), it will unduly harm the interests of the copyright holder in light of the type and use of the copyrighted work presented to the public and the mode of the minor use. If this is the case, this does not apply.
(1) The title or author name of the work in which the information obtained by searching using a computer (hereinafter referred to as "search information" in this issue) is recorded, and the sender identification code related to the search information enabled for transmission. (A character, number, symbol or other code for identifying the source of automatic public transmission.) Searching for information regarding the identification or location of other search information, and providing the results.
(Ii) ** Perform information analysis by computer and provide the results. ** **
(Iii) In addition to the items listed in the preceding two items, it is an act of creating new knowledge or information by information processing by a computer and providing the result, and it is a Cabinet Order that contributes to the improvement of convenience of people's lives. What to define
2 ** Persons who prepare for the acts listed in each item of the preceding paragraph (limited to those who collect, organize, and provide information for the preparation of the acts in accordance with the standards specified by Cabinet Order) are copyrighted works presented to the public. Regarding, to the extent deemed necessary for preparation for minor use pursuant to the provisions of the same paragraph, duplication or public transmission (in the case of automatic public transmission, transmission enablement is included. Hereinafter, this paragraph and the following Article 2 The same shall apply in item 2), or a copy thereof may be distributed. ** However, this shall apply if it would unreasonably harm the interests of the copyright holder in light of the type and use of the copyrighted work presented to the public, the number of copies of the copy or distribution, and the mode of reproduction, public transmission or distribution. Not.
Ethics of scraping ① Overseas trouble cases
There are services that prohibit scraping in order to protect personal information and prevent vandalism.
For example, the matching app Pairs explicitly prohibits scraping and crawling in its terms of use.
The Company does not permit the use of posted content to other users or other third parties except the user himself, and the user acts infringing the rights of the posted content of other users. Must not be. In addition, the user shall not automatically collect and analyze the posted content by crawling or the like. (Terms of Service | Pairs)
Similarly, Twitter prohibits scraping in its terms of service.
Access or search Twitter by any other means (automatically or otherwise) without going through (and complying with its terms of use) our currently available public interface provided by Twitter. Do or try to access or search. However, this does not apply if a separate contract with Twitter specifically allows this to be done. Twitter crawling is permitted as required by the robots.txt file. However, scraping without the prior consent of Twitter is expressly prohibited. (Rules | Twitter)
--Human copy and paste --Full-text search and regular expression match --HTTP programming --Data mining algorithm --DOM analysis --HTML parser --Web scraping software --Vertical integration platform --Meaning annotation recognition
Open the specified Web page with the built-in browser of OctoParse, select the data you want to extract, and a crawler will be created. No programming knowledge required, anyone can use it easily. When you run the crawler, you can output various data on the website in the desired format.
As for support, it supports Japanese and responds quickly.
--Since it is a browser operation, it is easy to respond to sites that require infinite scrolling and login. --Difficult to get caught in bot judgment
--Processing is heavy due to browser operation. Not suitable for high-speed processing.
--Abundant frameworks and libraries --Processing is fast
--Customization is required to support browser operations (infinite scrolling, processing by JavaScript)
[Introduction to Python] Basics of scraping with Beautiful Soup 4 (1/2) Beautiful Soup in 10 minutes Practice / Python scraping style in the field
Summary of knowledge when web scraping with Python [Web scraping-Wikipedia](https://ja.wikipedia.org/wiki/%E3%82%A6%E3%82%A7%E3%83%96%E3%82%B9%E3%82%AF% E3% 83% AC% E3% 82% A4% E3% 83% 94% E3% 83% B3% E3% 82% B0) Is scraping illegal? Attorneys explain three legal issues and countermeasures in 5 minutes [Preserved version] Thorough explanation for beginners on how to scrape with Python![Sample code available]
Recommended Posts