Use twitterscraper-ruby to search tweets from several years ago by date

You can use the Twitter Search API to search for tweets on Twitter, but you can only search for the last 7 days **.

To get around this limitation, try using twitterscraper-ruby gem. twitterscraper-ruby scrapes Twitter directly, so you can search ** any tweet from the first day Twitter starts **.

Note 1

twitterscraper-ruby scrapes Twitter directly, but Twitter's Terms of Service do not allow scraping. Use it only for collecting a small amount of personal tweets.

Note 2

Twitter's search function does not cover all tweets. Furthermore, it seems that you may not be able to search for tweets in Japanese well. Therefore, it is not suitable for collecting all tweets.

Search for tweets by date

Let's compare the results when using twitterscraper-ruby and when using the Twitter Search API.

When using twitterscraper-ruby

I will try to get up to 1000 tweets on January 1, 2020.

Gemfile


gem 'twitterscraper-ruby'
require 'twitterscraper'

#There is a proxy function to reduce the possibility that the IP will be banned,
#Instead, the proxy is turned off because it slows down considerably.
client = Twitterscraper::Client.new(proxy: false)

tweets = client.query_tweets('Twitter', start_date: '2020-01-01', end_date: '2020-01-02', lang: 'ja', limit: 1000)

puts tweets.size
# => 1000

tweets.take(3).each { |t| puts t.created_at }
# => 2020-01-01 23:59:59 +0000
# => 2020-01-01 23:59:59 +0000
# => 2020-01-01 23:59:59 +0000

I got exactly 1000 items. I have tried other numbers, but it seems that they may exceed a little.

When using the Twitter Search API (twitter gem)

I also tried to get tweets with the same date using the Twitter Search API, but I couldn't get any tweets as expected. This is because the Twitter Search API can only retrieve tweets for the last 7 days.

# https://github.com/sferik/twitter
client = Twitter::REST::Client.new

tweets = client.search('Twitter since:2020-01-01 until:2020-01-02', count: 100)

puts tweets.size
# => 0

Reference link

Recommended Posts

Use twitterscraper-ruby to search tweets from several years ago by date
Use twitterscraper-ruby to search tweets from several years ago by date
Convert from ○ months to ○ years ○ months
From introduction to use of ActiveHash