unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / Atom feed
* Master wait time metric
@ 2016-10-31 23:08 Sam Saffron
  2016-10-31 23:36 ` Eric Wong
  0 siblings, 1 reply; 3+ messages in thread
From: Sam Saffron @ 2016-10-31 23:08 UTC (permalink / raw)
  To: unicorn-public

Hi Eric / everyone :)

I would like to start graphing how long our master process spends
waiting for worker processes to be available.

This metric will allow us to quickly tell if a unicorn is being
overloaded and allow us to quickly remediate.

Once a minute I want to ask the master process how long it spent
waiting for child processes to become available.

How would I go about getting that metric?

Sam

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Master wait time metric
  2016-10-31 23:08 Master wait time metric Sam Saffron
@ 2016-10-31 23:36 ` Eric Wong
  2016-11-01  8:21   ` Sam Saffron
  0 siblings, 1 reply; 3+ messages in thread
From: Eric Wong @ 2016-10-31 23:36 UTC (permalink / raw)
  To: Sam Saffron; +Cc: unicorn-public

Sam Saffron <sam.saffron@gmail.com> wrote:
> Hi Eric / everyone :)
> 
> I would like to start graphing how long our master process spends
> waiting for worker processes to be available.

Fwiw, the master doesn't wait for workers to become available for
processing requests.  But I think I know what you mean to ask :>

Rather, the connection request sits in the listen queue (a kernel
object) shared by all workers, and instrumenting this is always
kernel-dependent (because unicorn avoids doing stuff in userspace).

> This metric will allow us to quickly tell if a unicorn is being
> overloaded and allow us to quickly remediate.
> 
> Once a minute I want to ask the master process how long it spent
> waiting for child processes to become available.
> 
> How would I go about getting that metric?

Linux-only, but you can probably look at Raindrops::LastDataRecv

	https://bogomips.org/raindrops/Raindrops/LastDataRecv.html

Raindrops::Middleware can give you how big the listen queue is,
too.  Ideally, this should never exceed 1.

	https://bogomips.org/raindrops/Raindrops/Middleware.html


You can probably get the same metrics directly from the kernel
via systemtap, dtrace, or similar, too.

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: Master wait time metric
  2016-10-31 23:36 ` Eric Wong
@ 2016-11-01  8:21   ` Sam Saffron
  0 siblings, 0 replies; 3+ messages in thread
From: Sam Saffron @ 2016-11-01  8:21 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public

Brilliant, this will make it super easy to write a Prometheus
exporter, will post here once I am done.

On Tue, Nov 1, 2016 at 10:36 AM, Eric Wong <e@80x24.org> wrote:
> Sam Saffron <sam.saffron@gmail.com> wrote:
>> Hi Eric / everyone :)
>>
>> I would like to start graphing how long our master process spends
>> waiting for worker processes to be available.
>
> Fwiw, the master doesn't wait for workers to become available for
> processing requests.  But I think I know what you mean to ask :>
>
> Rather, the connection request sits in the listen queue (a kernel
> object) shared by all workers, and instrumenting this is always
> kernel-dependent (because unicorn avoids doing stuff in userspace).
>
>> This metric will allow us to quickly tell if a unicorn is being
>> overloaded and allow us to quickly remediate.
>>
>> Once a minute I want to ask the master process how long it spent
>> waiting for child processes to become available.
>>
>> How would I go about getting that metric?
>
> Linux-only, but you can probably look at Raindrops::LastDataRecv
>
>         https://bogomips.org/raindrops/Raindrops/LastDataRecv.html
>
> Raindrops::Middleware can give you how big the listen queue is,
> too.  Ideally, this should never exceed 1.
>
>         https://bogomips.org/raindrops/Raindrops/Middleware.html
>
>
> You can probably get the same metrics directly from the kernel
> via systemtap, dtrace, or similar, too.

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, back to index

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-10-31 23:08 Master wait time metric Sam Saffron
2016-10-31 23:36 ` Eric Wong
2016-11-01  8:21   ` Sam Saffron

unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help

Archives are clonable:
	git clone --mirror https://bogomips.org/unicorn-public
	git clone --mirror http://ou63pmih66umazou.onion/unicorn-public

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
	nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox