From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS47066 71.19.144.0/20 X-Spam-Status: No, score=-1.0 required=3.0 tests=AWL,BAYES_00, MSGID_FROM_MTA_HEADER shortcircuit=no autolearn=unavailable version=3.3.2 Path: news.gmane.org!not-for-mail From: John Pignata Newsgroups: gmane.comp.lang.ruby.raindrops.general Subject: Re: Fwd: Queued vs. Active in tcp_stats_listener Date: Sun, 3 Mar 2013 17:26:38 -0500 Message-ID: References: <20130303214613.GA7200@dcvr.yhbt.net> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-Trace: ger.gmane.org 1362349609 18034 80.91.229.3 (3 Mar 2013 22:26:49 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 3 Mar 2013 22:26:49 +0000 (UTC) To: raindrops@librelist.org Original-X-From: raindrops@librelist.org Sun Mar 03 23:27:09 2013 Return-path: Envelope-to: gclrrg-raindrops@m.gmane.org List-Archive: List-Help: List-Id: List-Post: List-Subscribe: List-Unsubscribe: Precedence: list Original-Sender: raindrops@librelist.org Xref: news.gmane.org gmane.comp.lang.ruby.raindrops.general:103 Archived-At: Received: from zedshaw2.xen.prgmr.com ([71.19.156.177]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1UCHN5-0004Yk-It for gclrrg-raindrops@m.gmane.org; Sun, 03 Mar 2013 23:27:08 +0100 Received: from zedshaw2.xen.prgmr.com (unknown [IPv6:::1]) by zedshaw2.xen.prgmr.com (Postfix) with ESMTP id D18FF74E01 for ; Sun, 3 Mar 2013 22:27:26 +0000 (UTC) Content-Transfer-Encoding: 7bit X-Content-Filtered-By: PublicInbox::Filter 0.0.1 Hi Eric, Thanks for your super-fast and thorough response. On Sun, Mar 3, 2013 at 4:46 PM, Eric Wong wrote: > I'm not sure what Heroku does, but I think they have some fancy load > balancer/router. Maybe that limits the number of connection it opens to > the (virtual?) machine unicorn runs on. > > You should ask Heroku if they do this. Right. I've been talking to Heroku about their overall router behavior. My current understanding is that (on the Cedar stack) the router does not do any intelligent queuing and doesn't apply back-pressure. As requests come in they are passed off to a dyno where they may or may not be enqueued by the application server. https://blog.heroku.com/archives/2013/2/16/routing_performance_update has some good details for any future readers of this thread trying to tune on Heroku. > By backlog, this is the :backlog parameter in the unicorn "listen" > directive, correct? > Yes. > This :backlog parameter is only a hint to the kernel to limit the maximum > queue size. If the kernel never sees many connections in the first place, > the limit will never be hit. Understood. Though, I'd expect I'd be able to move the queued number with Apache Bench considering the default backlog of 1,024. The "queued" in raindrops is the connections the kernel TCP stack knows > about. If your load balancer doesn't send requests, the TCP stack in > the kernel won't see them. > Understood. > Can you get around the load balancer and throw traffic at the box > directly? You should see queued counts go up. I cannot on Heroku but I did prop up a Rack test application on an EC2 system and raindrops is behaving the way you're explaining. > unicorn does not accept() connections unless a worker is idle[1], so > what you're seeing with active == unicorn worker_processes is correct. > Understood. So the queued number should represent connections that (under TCP) have been responded to with an ACK but are enqueued waiting for a worker. I think my issue is with how I'm conducting my test. I may deploy our change to production and monitor it there. We made a similar change to Thin running on Heroku to dump a number to statsd every time a connection occurred and we saw a significant non-zero backlog of connections. I expect to see a smaller but non-zero number under Unicorn. Thanks again for your time and thoughts. -jp