HostingRails.com

Hi there - I believe Mongrel will cache all that data so its ready to spit it out again - I don't think its a mySQL issue.  The line...

 Flight.find_all.each do |flight|

...is your 'problem' - is there anyway to avoid making a call that processes 19,000 records every time?

The routine verifies that data is correct.  Its only run very rarely (if at all) in 'real -life'. :)

Maybe if the routine is called multiple times in smaller chunks...

Yeah that would be a better way to go about it - let me know how it goes...

It works great!

I can change the amount of 'memory used' vs 'speed' with the buffer variable, here is the code:

def repair_flight
  changes=0
  zapcache=false
  if session['user'].is_in_role?(:Admin)
    noflights = Flight.count
    buffer = 100 # the larger this, the more memory...  but faster!
    count= (noflights / buffer) + 1
    offset=0
    count.times do
      flights=Flight.find(:all, :offset => offset, :limit => buffer)  
      offset += buffer
      for flight in flights
        changed=false

# i've deleted what goes in here - but the problem is fixed! :)

      end
    end
  end
  redirect_to :action => 'index'
end

This problem will probabily apply (in much smaller amounts) to other routines.

This is a very nice find.  Thanks for posting it.  What is the default buffer number?   (hopefully it is unlimited)

Cheers,

~William  

Its not a nice find, its a software buffer (see the code - its the buffer=100 bit)...

The side effect of making the buffer value higher is that more system memory is lost in to that class (it seems to be limited to each class).

Making the buffer value lower will use less system memory, but take more cpu time in the short term.

Does anyone know if there is a sleep or delay rails command - to release time to the system - thus reduce cpu time?

I'm having this same problem (memory usage goes above my limit during the request and won't go back down after) but there isn't anywhere in my code that I can significantly reduce the amount of data that I access for each request. Is there any way to have Mongrel not cache the requested data? I've tried initiating garbage collection as Ctc said with GC.start in a few different places but it only makes a minimal difference.

EDIT: I'm trying to fix this problem because I got a warning email this morning saying I was over my limit. Does the automated check look at all processes under my username or just my mongrel process? I know at the time my mongrel process was over the limit, but when I'm working on my app I'll often have two SSH windows and an sftp client going, and those in addition to my normal mongrel process easily go over 50mb (since the server starts an sshd/sftp-server for each connection).

EDIT2: Another site that I'm responsible for just got a memory usage warning. It seems like 50MB is a very tight memory usage limit for moderately sized apps that get a decent amount of traffic. I can probably put in a temporary fix by having it restart mongrel every 6 hours or so, but are any other people having these problems?

FYI: The automated script pulls data from "ps -aux |grep mongrel_rails" only -- so it only includes mongrel data at this point.  However, we've noticed clients sneaking in other static processes (or starting up things in a sneaky fashion) so we'll probably have to get much more strict about this - which is a shame because we've enjoyed being flexible with our clients thus far.

50MB is more than enough for a single mongrel for probably 90% of the apps we host.  Larger apps simply need to purchase more memory for their account.  For those in memory trouble who'd rather not pay for more memory, I would highly suggest following Antonio's tips here.  There are also ways of parsing down your Rails app so it doesn't load libs you don't use - we'll run some tests and see what we can come up with to help you guys out.