[Ruby / Rails] Mechanism for retrying Thread Error

Overview

When multithreading was performed using Thread.new (), a resource shortage error occurred on Heroku.

can't create Thread: Resource temporarily unavailable (ThreadError)

We created a thread error handling and a retry mechanism that uses threads on a best effort basis.


By the way, the number of process threads that can be executed on Heroku is quite limited, so be careful about the difference from the local environment.

https://devcenter.heroku.com/articles/limits#processes-threads

As of 2020/08

free, hobby and standard-1x dynos support no more than 256
standard-2x and private-s dynos support no more than 512
performance-m and private-m dynos support no more than 16384
performance-l and private-l dynos support no more than 32768

If your local environment is MacOS, you can use the command sysctl kern.num_taskthreads to find out the maximum number of threads per process.

$ sysctl kern.num_taskthreads
kern.num_taskthreads: 4096

Caution

There is a high possibility that a design mistake is made in the process of running out of threads when the load is slightly higher than usual.

First consider how to reduce the number of threads that must be set up, such as whether processing can be delegated to another server or whether there is a way to do it with a small number of requests due to API specifications.

code

def retry_threads(times: 3)
  try = 0
  begin
    try += 1
    Thread.new { yield }
  rescue ThreadError
    sleep(1 * try)
    retry if try < times
    raise
  end
end

If no thread is available, wait a few seconds and retry.

The number of seconds to wait is variable, such as 1 second for the first retry and 2 seconds for the second retry, to prevent time loss.

There is also a way to manage retries in detail by setting the waiting time to microseconds.


Actual usage

def heavy_task(url)
  #Heavy processing
end

# urls = ["...","...",...]
threads = []
urls.all each do |url|
  threads << retry_threads{ heavy_task(url) }
end
threads.each(&:join)

benchmark

We investigated how much the tolerance for threading was actually increased by this retry mechanism.

conditions

--Processing that takes 10 seconds is subject to retry --Increase the retry waiting time by 1 second --Up to 3 retries --Run in local environment (Mac OS Catalina) --Maximum number of threads: 4096 --Set the number of tasks that can be processed to the benchmark value

Measurement code

def heavy_task
  sleep(10)
end

def retry_threads(times: 3)
  try = 0
  begin
    try += 1
    Thread.new { yield }
  rescue ThreadError
    sleep(1 * try)
    retry if try < times
    p $count
    raise
  end
end

def no_retry_threads()
  begin
    Thread.new { yield }
  rescue ThreadError
    p $count
    raise
  end
end

$count = 0

#No retry
loop do
  no_retry_threads{ heavy_task }
  $count += 1
end

#With retry
loop do
  retry_threads{ heavy_task }
  $count += 1
end

result

The number of tasks processed that took 10 seconds in the local environment (maximum number of threads: 4096).

No retries With retry
4094 212888

reference

Recommended Posts

[Ruby / Rails] Mechanism for retrying Thread Error
[Ruby on Rails] About bundler (for beginners)
Testing for Error Messages: Rails Tutorial Notes-Chapter 7
Tips for gRPC error handling in Ruby
Explanation of Ruby on rails for beginners ①
[Ruby on Rails] How to display error messages
[Ruby on Rails] Select2 introduction memo for Webpacker
[Rails] Procedure for linking databases with Ruby On Rails
[Ruby on Rails] Individual display of error messages
[Ruby On Rails] Causes and remedies for ActionView :: Template :: Error (ActiveStorage :: InvariableError) (hypothesis / verification)
Rails uninitialized constant A simple checklist for error resolution
Workaround for Bundler.require error when executing ruby with crontab
Rails / Ruby: How to get HTML text for Mail
Explanation of Ruby on rails for beginners ⑥ ~ Creation of validation ~
Explanation of Ruby on rails for beginners ② ~ Creating links ~
Explanation of Ruby on rails for beginners ⑦ ~ Flash implementation ~
[Note] Rails error list
About Ruby error messages
Catch Rails Routing Error
Chapter 4 Rails Flavored Ruby
abbreviation for ruby method
Ruby Thread # [] = method notes
[Rails error] unexpected tIDENTIFIER
Mac Rails Install Error
Ruby on Rails Elementary
rails error resolution summary
Ruby On Rails Association
Ruby Learning # 23 For Loops
Ruby, Rails finally installed
Scraping for beginners (Ruby)
Definitely useful! Debug code for development in Ruby on Rails
[Rails] Workaround for classes automatically generated by devise error messages
Ruby on Rails for beginners! !! Summary of new posting functions
[Procedure 1 for beginners] Ruby on Rails: Construction of development environment
From Ruby on Rails error message display to Japanese localization
Explanation of Ruby on rails for beginners ③ ~ Creating a database ~
[Rails] How to display error messages for comment function (for beginners)