Batch Process A Large Activerecord Query Without Memory Bloat
Order.all.each loads every row into memory before iterating. At forty thousand rows that is slow; at four million it is an out-of-memory kill. Rails has three batching methods and they solve slightly different problems.
Description
find_each yields one record at a time while loading a batch (1000 by default) behind the scenes. find_in_batches yields the array so you can do bulk work per batch. in_batches yields a relation, which lets you call update_all or delete_all on it without instantiating any models at all.
All three order by primary key internally, so a custom order is ignored — and Rails warns about it. Use in_batches(order: :desc) if direction matters.
Add a sleep between batches on production data so replicas can catch up and vacuum can run.
Sample input:
Order.where(status: "pending").in_batches(of: 5_000) { |batch| ... }
Sample Output:
Constant memory regardless of table size
Answer
# One record at a time, constant memory
Order.where(status: "pending").find_each(batch_size: 1_000) do |order|
order.recalculate_total!
end
# An array per batch — for bulk work
Order.find_in_batches(batch_size: 500) do |orders|
Warehouse.push_bulk(orders.map(&:to_payload))
end
# A relation per batch — no model instantiation at all, much faster
Order.where(archived: false).in_batches(of: 5_000) do |relation|
relation.update_all(archived: true, archived_at: Time.current)
sleep 0.1
end
# Resume from a known point after a failure
Order.where("id > ?", checkpoint).find_each do |order|
process(order)
Checkpoint.update!(last_id: order.id) if order.id % 1_000 == 0
end
# Careful: a custom order is IGNORED and Rails warns
Order.order(created_at: :desc).find_each { } # ordered by id, not created_at
Order.in_batches(order: :desc) { |r| } # this direction IS respected
# Lazy enumerator when you want Enumerable methods
Order.find_each.lazy.select(&:overdue?).first(10)
Check viewARU - Brand Newsletter!
Newsletter to DEVs by DEVs - boost your Personal Brand & career! 🚀
