[Ruby] Story that find_in_batches reduced memory consumption to 1/100

5 minute read

How to handle large amount of data with low memory consumption

Are you writing code in Rails with memory in mind?

Rails uses Ruby’s garbage collection*1, so you can write code without worrying about memory release.

*1 Ruby collects objects that are no longer used and releases the memory automatically.

Therefore, there is a case where the production server suddenly drops (due to a memory error) without noticing that even though the implementation consumes as much memory as you do before you know it.

The reason why I can say such a thing is that this phenomenon occurred at the site where I am currently working w

The person in charge of implementation correction made it myself, but since I learned a lot from that experience, I will leave a note so that I can not forget it.

Research of cause

First of all, you have to investigate where the memory error is.

I used ObjectSpace.memsize_of_all to investigate memory usage in Rails.

By using this method, you can check the memory usage that all living objects are consuming in bytes.

We set this method as a checkpoint at a place where execution processing is likely to drop, and we will steadily investigate where memory consumption is large.

■ Example of use to check memory usage

class Hoge
  def self.hoge
    puts'number of object memory before memory expansion by map'
    puts ObjectSpace.memsize_of_all <==== checkpoint
    array = ('a'..'z').to_a
    array.map do |item| <==== 
      puts "number of object memory of #{item}"
      puts ObjectSpace.memsize_of_all <==== checkpoint

■ Execution result

irb(main):001:0> Hoge.hoge
Number of object memory before memory expansion by map

Number of object memory in a

Number of object memory in b

Number of object memory in c

Number of object memories in d

Number of object memory of e


Number of object memory of x
number of object memories in y
Number of object memory of z
=> ["A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K", "L" , "M", "N", "O", "P", "Q", "R", "S", "T", "U", "V", "W", "X", " Y", "Z"]

From this execution result, you can see that the data passed by map is expanded in memory at once, and the memory consumption increased there. (1)

You can also see that the memory consumption increases with each loop.

If the process is simple like the sample code this time, there is no problem.

If there is a large amount of data to be passed and the implementation performed in the loop process is complicated, memory consumption will be pressured.

** It becomes a memory error (an error that occurs when memory processing cannot keep up). **

In this survey, I also checked the above procedure, and as a result, I came to the conclusion that a memory error occurred because I implemented a heavy process that sends a large amount of data and emits queries in map.


I understood the cause.

Next, let’s think about countermeasures.

The following three measures were first conceived.

1. Increase the memory with the power of gold
2. Make parallel processing with Thread
3. Batch processing

1. Increase the memory with the power of money

Honestly, this is the fastest, it’s only necessary to increase the memory specs of the server with the power of money, so let’s do this!

I thought.

There is no memory-intensive implementation other than this process, so I thought it would be foolish to spend money just for this part, so I stopped this plan.

2. Make parallel processing with Thread

I came up with parallel processing of Ruby as the next countermeasure, but when the bottleneck is the processing time (timeout), it is correct because it will be faster if you make multiple threads in parallel and calculate and merge, but this time it is correct. Since the bottleneck is memory pressure due to memory error, the amount of data to be handled by multiple threads does not change, so it is assumed that a memory error will result in the end.

3. Batch processing

The biggest cause of the memory error this time is a memory error that occurs when a large amount of data is expanded at once and the high load processing is repeated in a loop.

Therefore, I thought that it would be good if it could be implemented while dividing the large amount of data in batch processing into 1,000 units without expanding the memory all at once, because it would be possible to implement while saving memory.

Rails has a method called find_in_batches, and if you use it, you can process 1000 records by default.

Example) In case of 10,000 cases, divide into 1,000 cases and divide into 10 batch processes.
An image that uses less memory by limiting processing with find_in_batches.


Use find_in_batches for batch processing


If you know how to deal with it, you only have to implement it.

Let’s implement it. (I can not show the code of the company, so I will only list the image)

■ Mounting image

User.find_in_batches(batch_size: 1000) do |users|
  # Something

Even if 10,000 User data are acquired, if find_in_batches is used, each 1000 will be processed.

In other words, 10,000 / 1000 = image divided into 10 processes.


Memory consumption has been reduced to 1/100.

Ideas to improve

**However, the biggest disadvantage of this implementation is that it takes too much processing time. **

If you are using heroku etc., this implementation will result in a RequestTimeOut error*1.

*1 With heroku, processing that takes more than 30 seconds results in a RequestTimeOut error

Therefore, I think it is better to move this high-load implementation to background processing.

If you are using Rails, you can do it by using Sidekiq.

I think that it is good to work with the following procedure.

STEP1. Use find_in_batches to reduce memory consumption

STEP2. At the stage where STEP1 is completed, it should take time, but it should be in a state of operation without causing a memory error.
However, it takes time to process, so move it to the background.


At first, I thought it would be a painful task.

There was a lot of learning, and it was nice to implement it now.


https://techblog.lclco.com/entry/2019/07/31/180000 https://qiita.com/kinushu/items/a2ec4078410284b9856d