Fwd: Queued vs. Active in tcp_stats_listener

Fwd: Queued vs. Active in tcp_stats_listener

[John Pignata]

Hi all,

I'm attempting to use Raindrops in order to start instrumenting our Unicorn
workloads on Heroku using statsd. I'm a little confused at the numbers it
reports under test.

My primary confusion is that our active number never climbs above the
number of workers we have running. Since we have a large backlog, I
expected to see the total number of accept()'ed clients counted in this
number too. Unfortunately, I only spikes up to the total number of workers
and queued never spikes above 0. I expected the latter as we shouldn't ever
be refusing connections with our large backlog.

If active represents the number of requests that are currently being
serviced and queued represents the number of connections that have yet to
be accept()'ed, is there a way to get out the middle state: requests that
have been accept()'ed but not yet started?

Thanks,
-jp

Re: Fwd: Queued vs. Active in tcp_stats_listener

[Eric Wong]

John Pignata <john@pignata.com> wrote:
> Hi all,
> 
> I'm attempting to use Raindrops in order to start instrumenting our Unicorn
> workloads on Heroku using statsd. I'm a little confused at the numbers it
> reports under test.

I'm not sure what Heroku does, but I think they have some fancy load
balancer/router.  Maybe that limits the number of connection it opens to
the (virtual?) machine unicorn runs on.

You should ask Heroku if they do this.

> My primary confusion is that our active number never climbs above the
> number of workers we have running. Since we have a large backlog, I

By backlog, this is the :backlog parameter in the unicorn "listen"
directive, correct?

This :backlog parameter is only a hint to the kernel to limit the maximum
queue size.  If the kernel never sees many connections in the first place,
the limit will never be hit.

> expected to see the total number of accept()'ed clients counted in this
> number too. Unfortunately, I only spikes up to the total number of workers
> and queued never spikes above 0. I expected the latter as we shouldn't ever
> be refusing connections with our large backlog.

The "queued" in raindrops is the connections the kernel TCP stack knows
about.  If your load balancer doesn't send requests, the TCP stack in
the kernel won't see them.

Can you get around the load balancer and throw traffic at the box
directly?  You should see queued counts go up.

> If active represents the number of requests that are currently being
> serviced and queued represents the number of connections that have yet to
> be accept()'ed, is there a way to get out the middle state: requests that
> have been accept()'ed but not yet started?

unicorn does not accept() connections unless a worker is idle[1], so
what you're seeing with active == unicorn worker_processes is correct.


[1] - this is the crux of unicorn: one client per worker process;
      there's never any queueing/head-of-queue blocking in userspace

Re: Fwd: Queued vs. Active in tcp_stats_listener

[John Pignata]

Hi Eric,

Thanks for your super-fast and thorough response.

On Sun, Mar 3, 2013 at 4:46 PM, Eric Wong <normalperson@yhbt.net> wrote:

> I'm not sure what Heroku does, but I think they have some fancy load
> balancer/router.  Maybe that limits the number of connection it opens to
> the (virtual?) machine unicorn runs on.
>
> You should ask Heroku if they do this.


Right. I've been talking to Heroku about their overall router behavior. My
current understanding is that (on the Cedar stack) the router does not do
any intelligent queuing and doesn't apply back-pressure. As requests come
in they are passed off to a dyno where they may or may not be enqueued by
the application server.
https://blog.heroku.com/archives/2013/2/16/routing_performance_update has
some good details for any future readers of this thread trying to tune on
Heroku.


> By backlog, this is the :backlog parameter in the unicorn "listen"
> directive, correct?
>

Yes.


> This :backlog parameter is only a hint to the kernel to limit the maximum
> queue size.  If the kernel never sees many connections in the first place,
> the limit will never be hit.


Understood. Though, I'd expect I'd be able to move the queued number with
Apache Bench considering the default backlog of 1,024.

The "queued" in raindrops is the connections the kernel TCP stack knows
> about.  If your load balancer doesn't send requests, the TCP stack in
> the kernel won't see them.
>

Understood.


> Can you get around the load balancer and throw traffic at the box
> directly?  You should see queued counts go up.


I cannot on Heroku but I did prop up a Rack test application on an EC2
system and raindrops is behaving the way you're explaining.


> unicorn does not accept() connections unless a worker is idle[1], so
> what you're seeing with active == unicorn worker_processes is correct.
>

Understood. So the queued number should represent connections that (under
TCP) have been responded to with an ACK but are enqueued waiting for a
worker.

I think my issue is with how I'm conducting my test. I may deploy our
change to production and monitor it there. We made a similar change to Thin
running on Heroku to dump a number to statsd every time a connection
occurred and we saw a significant non-zero backlog of connections. I expect
to see a smaller but non-zero number under Unicorn.

Thanks again for your time and thoughts.

-jp