Re: Strange quit behavior

[Eric Wong <normalperson@yhbt.net>]

Alex Sharp <ajsharp@gmail.com> wrote:
> On Fri, Aug 5, 2011 at 1:07 AM, Eric Wong <normalperson@yhbt.net> wrote:
> > Can I get more? (until the next select() call, at least).  I'd like to
> > see the timeout argument that gets passed to the next select().
> 
> Here's some more output. This is from the old master though, not a new
> master that is pegging the CPU. In this instance it's almost like the
> old master just ignores the signal.

Wait, weren't you trying to track down a problem with the master that's
pegging the CPU?  Isn't the CPU usage the problem here?  I was also
thinking the CPU usage in the new master was caused by constant
worker respawning because bundler wasn't loaded correctly somehow....

> select(4, [3], NULL, NULL, {8, 129984}) = 0 (Timeout)
<snip>
> select(4, [3], NULL, NULL, {8, 976047}

OK, so /this/ master is confirmed to be sleeping with a reasonable
timeout and not pegging the CPU...

If you want to actually track down whether or not a signal is being
delivered, use "strace -f -e '!futex'" since Ruby 1.9 has a dedicated
signal handling thread (at the OS level).

(You only need the -e '!futex' part to filter out the noise from the
 signal handling thread in <= 1.9.2, 1.9.3 is much quieter :)


Below is a proposed patch (to unicorn.git) which may help debug issues
in the signal -> handler master path (but only once it enters the Ruby
runtime).  I'm a hesitant to commit it since it worthless if the Ruby
process is stuck because of some bad C extension.  That's the most
common cause of stuck/unresponsive processes I've seen.

Subject: [PATCH] http_server: add debug statements for master sig handlers

This should help users track down what's going on as soon as
Ruby can process the signal.  It's still not going to be useful
if the Ruby process is stuck because of a bug in a C extension
or Ruby itself, though.

Unicorn always defers signals in the master process to serialize
signal handling since some of the important actions (e.g. HUP,
USR1, USR2) are NOT reentrant.
---
 lib/unicorn/http_server.rb |   13 +++++++++++--
 1 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index aa8212e..b45d6d6 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -126,8 +126,17 @@ class Unicorn::HttpServer
     # setup signal handlers before writing pid file in case people get
     # trigger happy and send signals as soon as the pid file exists.
     # Note that signals don't actually get handled until the #join method
-    QUEUE_SIGS.each { |sig| trap(sig) { SIG_QUEUE << sig; awaken_master } }
-    trap(:CHLD) { awaken_master }
+    QUEUE_SIGS.each do |sig|
+      trap(sig) do
+        @logger.debug("received SIG#{sig}")
+        SIG_QUEUE << sig
+        awaken_master
+      end
+    end
+    trap(:CHLD) do
+      @logger.debug("received SIGCHLD")
+      awaken_master
+    end
     self.pid = config[:pid]
 
     self.master_pid = $$
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying