[Ruby on Rails] Stop "looping until ..."

Why write

It's been a full month since I started working. In cooperation with synchronization, we released one function and succeeded in operating normally. However, in fact, I once had a problem that the process could not be completed in the production environment. I will output it as a memorandum to prevent the same thing from happening again.

Why it happened

Simply put, it's because you've fallen into an "infinite loop." The implementation was to monitor jobs running on Sidekiq to detect duplication or loss of jobs and alert Slack.

The first thing I hit was ・ Execute from schedule ・ Scheduled from running If a job is acquired and checked at the timing of both of these, it will often be detected as an error.

There was a lag of 1 second up to 5 seconds in the speed of coming and going (checked in the local environment). If the new function monitoring job moves at that timing, it will be detected as an error even if the job is running normally.

The measures I took against this ** Turn a loop when acquiring a job, and if all the data is the same, take 5 seconds of "sleep" while acquiring 10 times, and pass it to the next process ** It was that. This wasn't good ...

How it improved

In the production environment, it runs every 30 seconds and finishes the process in about 10 seconds, moving around the schedule and execution at a dizzying pace, which is a lively job.

It takes more than 50 seconds to get it while turning the loop (I was doing it on purpose), so the reason is that the data obtained in 10 acquisitions will not be the same forever. It was an infinite loop.

When making corrections, I stopped saying ** looping until all the data are the same **. But I knew that if I didn't screen the acquisition properly, I would end up messing up Slack's alerts.

Therefore, we decided the upper limit of the loop and changed it to the specification that passes the most data among the obtained data to the next process **.

monitoring_sidekiq.rb


#abridgement
def sidekiq_logs_arr_in
  consistency_check_sidekiq_logs = []
  #I used to use while statements and redo, but I set a clear upper limit.
  50.times do
    queues = []
    #Avoid using sleep due to variations in timing from appointment to execution and execution to appointment
    sleep 2
    #Get a job on Sidekiq
    Sidekiq::Workers.new.each { |_process_id, _thread_id, job| queues << job["queue"] }
    Sidekiq::ScheduledSet.new.each { |job| queues << job["queue"] }
    queues = queues.sort
    consistency_check_sidekiq_logs << queues
  end
  #Finally returns the most data in the loop by counting
  consistency_check_sidekiq_logs.max_by { |x| consistency_check_sidekiq_logs.count x }
end
#abridgement

By doing this, it is no longer necessary to throw an exception even during normal processing during the subsequent processing, and it is now possible to obtain a consistent log even if there is a healthy job. ..

What to do in the future

First of all, if a specific condition is not met, try not to perform the process of redoing in the loop process as much as possible. I was keenly aware that it is extremely important to set an upper limit and form a loop that can be escaped under any circumstances.

I'm really careful in the future because causing an infinite loop in a production environment can cause great damage to the company.

Recommended Posts

[Ruby on Rails] Stop "looping until ..."
Ruby on Rails Elementary
Ruby On Rails Association
Rails new in Ruby on Rails ~ Memorandum until deployment 2
Rails new in Ruby on Rails ~ Memorandum until deployment 1
Ruby on rails learning record -2020.10.03
Portfolio creation Ruby on Rails
Ruby on rails learning record -2020.10.04
[Ruby on Rails] Debug (binding.pry)
Ruby on rails learning record -2020.10.05
Ruby on rails learning record -2020.10.09
Ruby on Rails config configuration
Ruby on Rails basic learning ①
[Ruby on Rails] about has_secure_password
Ruby on rails learning record-2020.10.07 ②
Commentary on partial! --Ruby on Rails
Ruby on rails learning record-2020.10.07 ①
Cancel Ruby on Rails migration
Ruby on rails learning record -2020.10.06
Ruby on Rails validation summary
Ruby on Rails Basic Memorandum
Ruby on Rails Overview (Beginner Summary)
[Ruby on Rails] Read try (: [],: key)
[Ruby on Rails] yarn install --check-files
Ruby on Rails variable, constant summary
Installing Ruby + Rails on Ubuntu 18.04 (rbenv)
[Ruby on Rails] Introduced paging function
Basic knowledge of Ruby on Rails
Progate Ruby on Rails5 Looking Back
[Ruby on Rails] Add / Remove Columns
Ruby on Rails Japanese-English support i18n
(Ruby on Rails6) "Erase" posted content
[Ruby on Rails] CSV output function
[Ruby on Rails] What is Bcrypt?
[Ruby on Rails] Confirmation page creation
Ruby On Rails devise routing conflict
[Ruby on Rails] Comment function implementation
[Ruby on Rails] DM, chat function
[Ruby on Rails] Convenient helper method
I summarized the flow until implementing simple_calendar in Ruby on Rails.
[Ruby on Rails] Introduction of initial data
[Ruby on Rails] Search function (not selected)
Until ruby can be used on windows ...
[Rails] Addition of Ruby On Rails comment function
[Ruby on Rails] Creating an inquiry form
Ruby on Rails6 Practical Guide cp13 ~ cp15 [Memo]
[Ruby on Rails] How to use CarrierWave
[Ruby on Rails] Code check using Rubocop-airbnb
[Ruby on Rails] 1 model CRUD (Routing Main)
Ruby on Rails installation method [Mac edition]
[Ruby on Rails] model, controller terminal command
Let's summarize "MVC" of Ruby on Rails
Ruby on Rails model creation / deletion command
[Ruby on Rails] About bundler (for beginners)
part of the syntax of ruby ​​on rails
Ruby on Rails in Visual Studio Codespaces
Notes on using FCM with Ruby on Rails
[Ruby on Rails] Controller test with RSpec
Deploy to Heroku [Ruby on Rails] Beginner
[Ruby on Rails] Image slideshow using Skippr
Ruby on Rails controller create / delete command