From: "Yeung, Jeffrey" <Jeffrey.Yeung@polycom.com>
To: Eric Wong <normalperson@yhbt.net>,
unicorn list <mongrel-unicorn@rubyforge.org>
Subject: RE: Unicorn_rails ignores USR2 signal
Date: Fri, 9 Mar 2012 17:07:47 -0800 [thread overview]
Message-ID: <E8D9E7CCC2621343A0A3BB45E8DEDFA91C6738D5F8@CRPMBOXPRD04.polycom.com> (raw)
In-Reply-To: <20120310000239.GA27195@dcvr.yhbt.net>
I have it configured for 4 workers. I just turned it down to 1 worker and tested both TTIN and TTOU. They both work, creating and killing workers, respectively. The following is the strace capture from an TTOU signal, followed by a USR2 signal. Yes, that is the only output from the USR2 signal. Behavior doesn't change if the master is freshly started, or has been running for a while. One other thing I did not mention earlier is that the <pid>.oldbin file never gets created on the USR2 signal, but that's probably obvious already.
[pid 14463] --- SIGTTOU (Stopped (tty output)) @ 0 (0) ---
[pid 14463] rt_sigreturn(0x16) = -1 EINTR (Interrupted system call)
[pid 14483] rt_sigprocmask(SIG_SETMASK, ~[SEGV VTALRM RTMIN RT_1], NULL, 8) = 0
[pid 14483] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 14483] tgkill(14456, 14456, SIGVTALRM) = 0
[pid 14456] <... select resumed> ) = ? ERESTARTNOHAND (To be restarted)
[pid 14456] --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
[pid 14456] rt_sigreturn(0x1a) = -1 EINTR (Interrupted system call)
[pid 14456] fcntl(6, F_GETFL) = 0x801 (flags O_WRONLY|O_NONBLOCK)
[pid 14456] write(6, ".", 1) = 1
[pid 14456] select(6, [5], NULL, NULL, {30, 755188}) = 1 (in [5], left {30, 755184})
[pid 14456] fcntl(5, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK)
[pid 14456] read(5, ".", 11) = 1
[pid 14456] wait4(-1, 0x7fffcb5560ec, WNOHANG, NULL) = 0
[pid 14456] wait4(-1, 0x7fffcb5560ec, WNOHANG, NULL) = 0
[pid 14456] kill(14482, SIGQUIT <unfinished ...>
[pid 14482] <... select resumed> ) = ? ERESTARTNOHAND (To be restarted)
[pid 14456] <... kill resumed> ) = 0
[pid 14482] --- SIGQUIT (Quit) @ 0 (0) ---
[pid 14456] select(6, [5], NULL, NULL, {83, 0} <unfinished ...>
[pid 14482] rt_sigreturn(0x3) = -1 EINTR (Interrupted system call)
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield() = 0
[pid 14482] sched_yield( <unfinished ...>
[pid 14485] rt_sigprocmask(SIG_SETMASK, ~[SEGV VTALRM RTMIN RT_1], NULL, 8) = 0
[pid 14485] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 14482] <... sched_yield resumed> ) = 0
[pid 14482] close(3) = 0
[pid 14482] select(6, [3], NULL, [4 5], {22, 883208}) = -1 EBADF (Bad file descriptor)
[pid 14482] rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT KILL USR1 SEGV USR2 TERM CHLD STOP TTIN TTOU VTALRM WINCH RTMIN RT_1], NULL, 8) = 0
[pid 14482] rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT KILL USR1 SEGV USR2 TERM CHLD STOP TTIN TTOU VTALRM WINCH RTMIN RT_1], NULL, 8) = 0
[pid 14482] rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT KILL USR1 SEGV USR2 TERM CHLD STOP TTIN TTOU VTALRM WINCH RTMIN RT_1], NULL, 8) = 0
[pid 14482] getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
[pid 14485] _exit(0) = ?
Process 14485 detached
[pid 14482] rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER|SA_SIGINFO, 0x7f10515428f0}, {0x48fc00, [], SA_RESTORER|SA_SIGINFO, 0x7f10515428f0}, 8) = 0
[pid 14482] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER|SA_SIGINFO, 0x7f10515428f0}, {SIG_IGN, [], SA_RESTORER|SA_SIGINFO, 0x7f10515428f0}, 8) = 0
[pid 14482] munmap(0x7f10517b4000, 4096) = 0
[pid 14482] close(4) = 0
[pid 14482] close(5) = 0
[pid 14482] close(7) = 0
[pid 14482] close(12) = 0
[pid 14482] unlink("/tmp/.java_pid14456") = -1 EPERM (Operation not permitted)
[pid 14482] exit_group(0) = ?
Process 14482 detached
[pid 14463] --- SIGCHLD (Child exited) @ 0 (0) ---
[pid 14463] rt_sigreturn(0x11) = -1 EINTR (Interrupted system call)
[pid 14483] rt_sigprocmask(SIG_SETMASK, ~[SEGV VTALRM RTMIN RT_1], NULL, 8) = 0
[pid 14483] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
[pid 14483] tgkill(14456, 14456, SIGVTALRM) = 0
[pid 14456] <... select resumed> ) = ? ERESTARTNOHAND (To be restarted)
[pid 14456] --- SIGVTALRM (Virtual timer expired) @ 0 (0) ---
[pid 14456] rt_sigreturn(0x1a) = -1 EINTR (Interrupted system call)
[pid 14456] fcntl(6, F_GETFL) = 0x801 (flags O_WRONLY|O_NONBLOCK)
[pid 14456] write(6, ".", 1) = 1
[pid 14456] select(6, [5], NULL, NULL, {82, 908414}) = 1 (in [5], left {82, 908410})
[pid 14456] fcntl(5, F_GETFL) = 0x800 (flags O_RDONLY|O_NONBLOCK)
[pid 14456] read(5, ".", 11) = 1
[pid 14456] wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 14482
[pid 14456] write(2, "reaped #<Process::Status: pid 14"..., 53) = 53
[pid 14456] wait4(-1, 0x7fffcb5560ec, WNOHANG, NULL) = 0
[pid 14456] select(6, [5], NULL, NULL, {91, 0}
<unfinished ...>
[pid 14463] --- SIGUSR2 (User defined signal 2) @ 0 (0) ---
[pid 14463] rt_sigreturn(0x20) = 202
-----Original Message-----
From: Eric Wong [mailto:normalperson@yhbt.net]
Sent: Friday, March 09, 2012 4:03 PM
To: unicorn list
Cc: Yeung, Jeffrey
Subject: Re: Unicorn_rails ignores USR2 signal
"Yeung, Jeffrey" <Jeffrey.Yeung@polycom.com> wrote:
> Thanks Eric. New strace capture as follows:
>
> $ sudo strace -f -e '!futex' -p 14255 Process 14255 attached with 12
> threads - interrupt to quit
>From that, you have 5 worker processes?
For debugging this, it can cut down on noise to only use one worker process, too. You can check if SIGTTOU works, too :)
Also, can you reproduce this on a freshly-started master? Or has the master been running and handling other signals for a while?
The most common cause of USR2 failures is due to an executable or library being moved/replaced/upgraded away (sometimes due to Capistrano), but that should definitely get logged and doesn't seem to be the case for you.
> [pid 14322] restart_syscall(<... resuming interrupted call ...>
> <unfinished ...> [pid 14271] restart_syscall(<... resuming
> interrupted call ...> <unfinished ...> [pid 14267] accept(12,
> <unfinished ...> [pid 14264] restart_syscall(<... resuming
> interrupted call ...> <unfinished ...> [pid 14255] select(6, [5],
> NULL, NULL, {42, 86504} <unfinished ...> [pid 14322] <...
> restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid
> 14271] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out) [pid 14264] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed out) [pid 14262] --- SIGUSR2 (User defined signal 2) @ 0 (0) ---
> [pid 14262] rt_sigreturn(0x20) = 202
>
> The last two lines do not say much for the event. :(
Anything more after that? What happens when you send a previously working signal (USR1, HUP) after sending a failed USR2 to that process?
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
next prev parent reply other threads:[~2012-03-10 1:29 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-03-09 21:48 Unicorn_rails ignores USR2 signal Yeung, Jeffrey
2012-03-09 22:24 ` Eric Wong
2012-03-09 22:39 ` Yeung, Jeffrey
2012-03-10 0:02 ` Eric Wong
2012-03-10 1:07 ` Yeung, Jeffrey [this message]
2012-03-10 1:30 ` Eric Wong
2012-03-12 21:21 ` Eric Wong
2012-03-12 22:39 ` Yeung, Jeffrey
2012-03-12 22:44 ` Eric Wong
2012-03-20 19:57 ` Eric Wong
2012-03-20 23:09 ` Yeung, Jeffrey
2012-03-21 2:27 ` Devin Ben-Hur
2012-03-30 22:16 ` Yeung, Jeffrey
2012-03-30 22:51 ` Alex Sharp
2013-05-08 18:15 ` Ryan Jones (RyanonRails)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://yhbt.net/unicorn/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=E8D9E7CCC2621343A0A3BB45E8DEDFA91C6738D5F8@CRPMBOXPRD04.polycom.com \
--to=jeffrey.yeung@polycom.com \
--cc=mongrel-unicorn@rubyforge.org \
--cc=normalperson@yhbt.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).