HostingRails.com

Hi Shoreside,

>>I can't use the mongrel_rails restart command as my >log/mongrel.pid gets stolen all the time and the restart >command is helpless without mongrel.pid -> why is the >mongrel.pid being deleted and when? Do you have any other >suggestions how to automatically restart mongrel over night?

Crashing of the mongrel instance will lead to the mongrel.pid value to become invalid. While restarting mongrel using this invalid mongrel.pid file, the command will fail. Please issue a command "ps" and verify if the PID value of the mongrel instance running and the PID value present inside the log/mongrel.pid file is same. If the PID value is different (or null) then it is reason for the "mongrel_rails restart" command to fail. You will get reason for mongrel instance to crash from the log/mongrel.log of your railsapp.  

Please Set cron job as shown below to restart the mongrel instance

59 23 * * * cd /home/username/railsapp/log && rm mongrel.pid -f

59 23 * * * cd /home/username/railsapp && /usr/local/bin/mongrel_rails start -e production -p 4*** -d

Every day The Cron job will run at 11:59 pm.

>>What is the best way to restart the FastCGI app: using >script/process/reaper --dispatcher=dispatch.fcgi as in the >capistrano task?

You don't need to restart FCGI process, the better way is to kill the FCGI instance, for that you can use the following command

 pkill -9 dispatch.fcgi -u <username>

After killing the FCGI instance, It will fire up again if any request come from the browser. Please make sure to change the username, railsapp and port according to your rails application.

Hi Vinayan,

My problem (and according to the forum not only mine) with the missing mongrel.pid is:

- it (log/mongrel.pid) is missing (gone, deleted) AND
- mongrel is running fine and didn't get restarted/crashed in between at all (verify through 'ps', contents of log/mongrel.log, working app)

In this case your suggestion doesn't work since it will merely produce an error like this:

** Starting Mongrel listening at 0.0.0.0:4***
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/../lib/mongrel/tcphack.rb:12:in `initialize_without_backlog': Address already in use - bind(2) (Errno::EADDRINUSE)

There must be a reason for the file 'mongrel.pid' being deleted from the app specific log file! Right after I kill mongrel and do a clean start (mongrel_rails start -e production -p 4*** -d) the mongrel.pid file was written and exists, when I come back the next day its gone and I don't see why? The existence of the mongrel.pid should be reliable:
- if its there -> mongrel IS running, can be REstarted
- if not -> mongrel is NOT running, can be STARTED

Any ideas? Thanks very much for your help!
Cheers!

Hi Shoreside,

Have you set the cronjob as shown below to restart the mongrel process daily?

==
59 23 * * * cd /home/username/railsapp/log && rm mongrel.pid -f

59 23 * * * cd /home/username/railsapp && /usr/local/bin/mongrel_rails start -e production -p 4*** -d

==

Please make sure to change the username, railsapp and port according to your rails application and let us if the issue persists after setting the cronjob to restart the mongrel daily.  I am not sure about why the mongrel.pid file is missing daily.

>>In this case your suggestion doesn't work since it will >merely produce an error like this:
>** Starting Mongrel listening at 0.0.0.0:4***
>/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/../lib/mongrel/tcphack.rb:12:in >`initialize_without_backlog': Address already in use - >bind(2) (Errno::EADDRINUSE)

Please throw more information about how you encountered an error like this. Normally this error will encounter when you try to start one mongrel process which is already running.

Hi Vinayan,

no, I didn't set a cronjob like you suggested because it would produce an error as mentioned before. You are right, the error

** Starting Mongrel listening at 0.0.0.0:4***
/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/../lib/mongrel/tcphack.rb:12:in `initialize_without_backlog': Address already in use - bind(2) (Errno::EADDRINUSE)

occurs if I try to start an already running mongrel. Thats exactly whats happening if I'd set the cronjob as suggested: removing of mongrel.pid would have no effect because there is none, next command tries to start the already running mongrel :(

The problem is the missing mongrel.pid I think or maybe I'm missing something?

The above mentioned error I found in the mongrel.log file. You can reproduce it if you watch the log with tail -f while trying to start a mongrel process thats already running. It should rather be restarted (or stopped and started).

Any help is appreciated, thanks very much!
Cheers!

Hi Shoreside,

I have written a script to restart the mongrel if the pid is missing. I also have set a cronjob to run this script every
day at 11:59 pm

======  shell-scrpt==========================================
cd /home/username/railsapp/log && rm mongrel.pid -f

cd /home/username/railsapp/log && pkill -9 mongrel_rails -u username

cd /home/username/railsapp && /usr/local/bin/mongrel_rails start -e production -p 4*** -d
============================================================

=====cron====
59 23 * * * /bin/sh /home/user/path_to_script
=============

If the issue is still persisting then please post a ticket to support team so that we can investigate the issue in a better way.

Hi Vinayan,

thanks for your help! Its working now (I didn't know about the pkill command), but I'd still be happier if the mysterious disappearance of the mongrel.pid could be unraveled.

Right now the mongrel.pid exists, maybe you guys can investigate who and when its being deleted? Maybe a restriction of access rights would help, but if its root that deletes it... :(

Thank you, have a great weekend!
Cheers!

Hi Shoreside,

Happy to hear that the cron job is working fine. But we couldn't find any explicit commands deleting your mongrel pid file. We will get back to you on this thread if we find any such processes in future.

I'm noticing that my mongrel.pid file is being deleted as well overnight.

I manually start my mongrel server using the following command:
mongrel_rails start -e production -p 4XXX -d

and verified that the mongrel.pid file exists in the /log/mongrel.pid

However, the next morning, the /log/mongrel.pid file is gone.

My crontab file is as follows:
# clear out pid file if it didn't get erased
@reboot cd /home/asicdesign/projects/app/current/log && rm mongrel.pid -f

# start up your Mongrel server
@reboot cd /home/asicdesign/projects/app/current && /usr/local/bin/mongrel_rails start -e production -p 4XXX -d

Hi Asicdesign,

I have made some tweaks in the Cron job. Please verify it and let us know if the issue is still persisting at your end.

Hi Vinayan,

Since your post 2008-05-31, I let the cronjob run for a few days and mongrel.pid file is still missing (assuming that it should be located at: ../app/current/log directory).

Not sure who or what's deleting the mongrel.pid file.

Hi there!

I think, I might know the reason for the mongrel.pid getting stolen every night: the @reboot entries in crontab. The first one deletes the mongrel.pid and the second one should start mongrel again which would lead to a new mongrel.pid being written into the log directory. I'm not 100% sure yet (will verify tonight and tell you tomorrow) but it seems that the order of execution of several @reboot entries is not determined or at least not according to the order they are written in the crontab.

I took two time stamps last night: the first one before the deletion and the second one after the new start of mongrel (both as separate @reboot entries in crontab) and the second one actually got executed BEFORE the first one (difference was 392599 nanoseconds). The mongrel.pid was gone this morning as usual and the mongrel.log contained the error mentioned a couple of post before (/usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/../lib/mongrel/tcphack.rb:12:in `initialize_without_backlog': Address already in use - bind(2) (Errno::EADDRINUSE)) at exactly the time the reboot happened: 02:36.

The solution would be quite easy: use only one @reboot entry to restart mongrel (e.g. calling a script to do so). Trying tonight, will let you know tomorrow!

Cheers!

Good morning!

Last night the two time stamps were taken as expected: the first one first, the second one last. I used a script to restart mongrel and now the mongrel.pid still exists. My conclusion would be that the order of execution of several @reboot entries in crontab is not determined. The use of a script to do the restart should prevent us from being deprived of mongrel.pid.

The content of my restart script is:

#!/bin/sh

# Workaround for a restart of the mongrel instance by Vinayan (hostingrails.com) :
# mongrel_rails restart ... doesnt work because the mongrel.pid file keeps getting stolen...

cd /path/to/your/app/log  && rm mongrel.pid -f
pkill -9 mongrel_rails -u <username>
cd /path/to/your/app && /usr/local/bin/mongrel_rails start -e production -p 4*** -d

It could of course be changed now to use mongrel_rails restart to restart your mongrel instance. But as Vinayan said using 'restart' could fail if the mongrel crashed and the content of mongrel.pid became invalid.

Oh, to prevent others from this problem, the instructions in the wiki http://www.hostingrails.com/forums/wiki_thread/19) should be adjusted as well, do you want me to do this?

Hope that helps!
Cheers!

That would be great if you could edit the wiki.  Thanks Shoreside.  mongrel_rails restart is notoriously problematic and we usually put a mongrel_rails stop and mongrel_rails start in clients' Capfiles for that reason.

Hi!

I edited the wiki. It now contains only one @reboot entry:

# restart your mongrel server
@reboot cd /home/username/railsapp && /usr/local/bin/mongrel_rails stop && /usr/local/bin/mongrel_rails start -e production -p 4*** -d

I assume a similar problem exists with a mongrel cluster, but due to the lack of testing opportunities I leave this edit to you.

Cheers!

Hi again!

I edited the wiki again since I had problems this morning as my mongrel was not running. It only got stopped at 2:36 reboot time but didn't get started up again. I'm using the script restart-mon.sh in my crontab now and changed the wiki accordingly.

Cheers!