Re: Master wait time metric

[Eric Wong <e@80x24.org>]

Sam Saffron <sam.saffron@gmail.com> wrote:
> Hi Eric / everyone :)
> 
> I would like to start graphing how long our master process spends
> waiting for worker processes to be available.

Fwiw, the master doesn't wait for workers to become available for
processing requests.  But I think I know what you mean to ask :>

Rather, the connection request sits in the listen queue (a kernel
object) shared by all workers, and instrumenting this is always
kernel-dependent (because unicorn avoids doing stuff in userspace).

> This metric will allow us to quickly tell if a unicorn is being
> overloaded and allow us to quickly remediate.
> 
> Once a minute I want to ask the master process how long it spent
> waiting for child processes to become available.
> 
> How would I go about getting that metric?

Linux-only, but you can probably look at Raindrops::LastDataRecv

	https://bogomips.org/raindrops/Raindrops/LastDataRecv.html

Raindrops::Middleware can give you how big the listen queue is,
too.  Ideally, this should never exceed 1.

	https://bogomips.org/raindrops/Raindrops/Middleware.html


You can probably get the same metrics directly from the kernel
via systemtap, dtrace, or similar, too.