Re: [PATCH] `kill -SIGTRAP <worker pid>`

[Eric Wong <normalperson@yhbt.net>]

Subject: [PATCH] `kill -SIGTRAP <worker pid>` to get a live ruby
 backtrace + generate backtrace when murdered worker due to timeout

Please keep Subject lines a reasonable length (git recommends the commit
message subject wrap at ~50 columns or so) and wrap code at <= 80
columns

Cedric Maion <cedric@maion.com> wrote:
> ---
>  lib/unicorn/http_server.rb |    3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
> index 14a6f9a..8507fe4 100644
> --- a/lib/unicorn/http_server.rb
> +++ b/lib/unicorn/http_server.rb
> @@ -457,6 +457,8 @@ class Unicorn::HttpServer
>        next_sleep = 0
>        logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \
>                     "(#{diff}s > #{@timeout}s), killing"
> +      kill_worker(:TRAP, wpid)
> +      sleep(0.5)
>        kill_worker(:KILL, wpid) # take no prisoners for timeout violations

SIGKILL timeout is only a last line of defense when the Ruby VM itself
is completely broken.  Handling SIGTRAP implies the worker can still
respond (and /can/ be rescued), so your SIGTRAP handler is worthless if
SIGKILL is required to kill a process.

See http://unicorn.bogomips.org/Application_Timeouts.html

Sleeping here is also unacceptable since it blocks the main loop,
making masters signal handlers non-responsive for too long.

> @@ -594,6 +596,7 @@ class Unicorn::HttpServer
>      # closing anything we IO.select on will raise EBADF
>      trap(:USR1) { nr = -65536; SELF_PIPE[0].close rescue nil }
>      trap(:QUIT) { worker = nil; LISTENERS.each { |s| s.close rescue nil }.clear }
> +    trap(:TRAP) { logger.info("worker=#{worker.nr} pid:#{$$} received TRAP signal, showing backtrace:\n#{caller.join("\n")}") }
>      logger.info "worker=#{worker.nr} ready"

Using the Logger class inside a signal handler can deadlock.  Logger
attempts to acquire a non-reentrant lock when called.  Unicorn doesn't
use threads itself, but the Rack app may use threads internally.

Thanks for your interest in unicorn!
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying