Re: Strange quit behavior

[Alex Sharp <ajsharp@gmail.com>]

On Fri, Aug 5, 2011 at 1:07 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Is this Unicorn 3.x?  Which 3.x version exactly?  Maybe give 4.0.1
> or the 4.0.2 git prerelease a try, too.

We're using version 3.6.2 with ruby 1.9.2-p180 on ubuntu 11.x. I know
the kernel on this version of ubuntu has a know signal trapping bug,
but I don't think that's what happening here. The processes respond
after I send the restart signal to them again (USR2 + QUIT).

> Can I get more? (until the next select() call, at least).  I'd like to
> see the timeout argument that gets passed to the next select().

Here's some more output. This is from the old master though, not a new
master that is pegging the CPU. In this instance it's almost like the
old master just ignores the signal.

select(4, [3], NULL, NULL, {8, 129984}) = 0 (Timeout)
wait4(-1, 0x7fff16b82e4c, WNOHANG, NULL) = 0
clock_gettime(CLOCK_REALTIME, {1313554942, 259408364}) = 0
fstat(12, {st_dev=makedev(202, 1), st_ino=20373, st_mode=S_IFREG,
st_nlink=0, st_uid=1001, st_gid=1001, st_blksize=4096, st_blocks=0,
st_size=0, st_atime=2011/08/12-23:14:20, st_mtime=2011/08/12-23:14:20,
st_ctime=2011/08/17-04:22:21}) = 0
clock_gettime(CLOCK_REALTIME, {1313554942, 259775504}) = 0
fstat(13, {st_dev=makedev(202, 1), st_ino=20381, st_mode=S_IFREG,
st_nlink=0, st_uid=1001, st_gid=1001, st_blksize=4096, st_blocks=0,
st_size=0, st_atime=2011/08/12-23:14:20, st_mtime=2011/08/12-23:14:20,
st_ctime=2011/08/17-04:22:17}) = 0
clock_gettime(CLOCK_REALTIME, {1313554942, 259936816}) = 0
fstat(14, {st_dev=makedev(202, 1), st_ino=20382, st_mode=S_IFREG,
st_nlink=0, st_uid=1001, st_gid=1001, st_blksize=4096, st_blocks=0,
st_size=0, st_atime=2011/08/12-23:14:21, st_mtime=2011/08/12-23:14:21,
st_ctime=2011/08/17-04:22:19}) = 0
clock_gettime(CLOCK_REALTIME, {1313554942, 260086380}) = 0
fstat(15, {st_dev=makedev(202, 1), st_ino=20185, st_mode=S_IFREG,
st_nlink=0, st_uid=1001, st_gid=1001, st_blksize=4096, st_blocks=0,
st_size=0, st_atime=2011/08/12-23:14:21, st_mtime=2011/08/12-23:14:21,
st_ctime=2011/08/17-04:22:17}) = 0
clock_gettime(CLOCK_REALTIME, {1313554942, 260235797}) = 0
fstat(16, {st_dev=makedev(202, 1), st_ino=20255, st_mode=S_IFREG,
st_nlink=0, st_uid=1001, st_gid=1001, st_blksize=4096, st_blocks=0,
st_size=0, st_atime=2011/08/12-23:14:21, st_mtime=2011/08/12-23:14:21,
st_ctime=2011/08/17-04:22:19}) = 0
clock_gettime(CLOCK_REALTIME, {1313554942, 260384849}) = 0
fstat(17, {st_dev=makedev(202, 1), st_ino=20383, st_mode=S_IFREG,
st_nlink=0, st_uid=1001, st_gid=1001, st_blksize=4096, st_blocks=0,
st_size=0, st_atime=2011/08/12-23:14:21, st_mtime=2011/08/12-23:14:21,
st_ctime=2011/08/17-04:22:19}) = 0
clock_gettime(CLOCK_REALTIME, {1313554942, 260534792}) = 0
fstat(18, {st_dev=makedev(202, 1), st_ino=20384, st_mode=S_IFREG,
st_nlink=0, st_uid=1001, st_gid=1001, st_blksize=4096, st_blocks=0,
st_size=0, st_atime=2011/08/12-23:14:21, st_mtime=2011/08/12-23:14:21,
st_ctime=2011/08/17-04:22:19}) = 0
clock_gettime(CLOCK_REALTIME, {1313554942, 260684278}) = 0
fstat(10, {st_dev=makedev(202, 1), st_ino=20196, st_mode=S_IFREG,
st_nlink=0, st_uid=1001, st_gid=1001, st_blksize=4096, st_blocks=0,
st_size=0, st_atime=2011/08/16-16:33:46, st_mtime=2011/08/16-16:33:46,
st_ctime=2011/08/17-04:22:19}) = 0
clock_gettime(CLOCK_REALTIME, {1313554942, 260833725}) = 0
select(4, [3], NULL, NULL, {8, 976047}

> If you see select() with very small timeout, use "strace -v" to get the
> st_ctime from fstat()s...
>
> I could have a math bug in murder_lazy_workers (please read/review the
> logic in that method, I haven't noticed issues myself).
>
> I made some tweaks to the master sleep strategy within the past year to
> reduce wakeups during idle hours.  This is intended to go with Ruby
> 1.9.3 which will have /much/ better thread wakeup strategy that reduces
> power consumption during idle times.
>
>> The first line was when the master was idle, and then I threw a few
>> requests at it.
>
> Are all workers responding as expected and not dying?

The old workers appear to be serving requests.

- Alex
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying