From: Ryan King <ryan@twitter.com>
To: raindrops@librelist.com
Subject: Re: queued is always 0
Date: Wed, 30 Jun 2010 11:51:15 -0700 [thread overview]
Message-ID: <AANLkTikC4C6z3RoHNAYBcsa5iTx0-afh33imJJYM4cu8@mail.gmail.com> (raw)
In-Reply-To: 20100630024549.GA9169@dcvr.yhbt.net
On Tue, Jun 29, 2010 at 7:45 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Ryan King <ryan@twitter.com> wrote:
>> On Mon, Jun 28, 2010 at 5:50 PM, Eric Wong <normalperson@yhbt.net> wrote:
>> > Ryan King <ryan@twitter.com> wrote:
>> >> I'm trying to see if raindrops will help us instrument our stack, so
>> >> I'm doing some tests.
>> >>
>> >> I have 16 unicorns running our app with rainbows and am hitting it
>> >> with ab with a concurrency of 100. However, according to
>> >> linux-tcp-listener-stats.rb I have 0 queued requests. Maybe I'm
>> >> missing something, but shouldn't there be queued requests somewhere
>> >> here?
>> >
>> > Hi Ryan,
>> >
>> > The Unicorns could be accept()-ing connections fast enough and the
>> > queued connections aren't noticeable[1].
>
> <snip>
>
>> I did as you describe here and I got the expected results.
>>
>> I think I'm just confused at this point. We have 16 unicorns running,
>> with 100 clients, but no queued connections, which makes me assume
>> that we have accepted connections that we can't actually service yet.
>> Is this true?
>
> Not true with Unicorn. Unicorn won't accept connections it can't
> service, ever.
>
> It could be ab itself isn't able to keep up with the requests/responses.
>
> How fast are your response times?
Pretty slow, actually. Running with concurrency of 16 (with 16 workers):
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 1.0 0 12
Processing: 1698 2444 351.8 2403 5885
Waiting: 1698 2444 351.9 2403 5885
Total: 1698 2444 352.0 2403 5886
Percentage of the requests served within a certain time (ms)
50% 2403
66% 2533
75% 2616
80% 2670
90% 2900
95% 3085
98% 3285
99% 3478
100% 5886 (longest request)
> What happens when you run fewer Unicorn workers or increase concurrency
> with ab?
I bumped the concurrency up to 2000 (with 16 workers) and still no
queuing (according to raindrops).
Interestingly, when I ran this benchmark ab died because of timeouts.
I lowered the concurrency back to 16 and ran it again, which quickly
caused timeouts. When I looked at the unicorn processes it was obvious
that they were still processing a backlog processes from the first ab
run. I'm not sure if this is how it is supposed to work, but it was
surprising to me.
> How large are your responses?
About 43 KB.
> If you have small responses, Unicorn could've also finished writing the
> response to the socket buffers and accepted another connection by the
> time ab gets around to reading the socket.
>
> Under Linux, Unicorn (and Mongrel) also uses TCP_DEFER_ACCEPT with a
> value of 1 (second)[1], so it won't even register in the queue if
> ab opened the socket and didn't have a chance to write to it, yet...
>
> You could try commenting out the setsockopt call to set
> TCP_DEFER_ACCEPT in lib/unicorn/socket_helper.rb for testing
> and see if the queue jumps up, too..
We're not setting the tcp_nodelay option, so that code shouldn't be
getting run, right?
> I noticed that the usage string for linux-tcp-listener-stats.rb was
> wrong yesterday (and updated it in rainbows.git), but you can try
> running it in a tight delay loop with "-d <seconds>":
>
> ruby linux-tcp-listener-stats.rb -d 0.01 | awk '$3 != 0 { print $0}'
>
> The above will only print lines if there are queued connections. You
> can replace "-d 0.01" with a smaller number, but Raindrops (and the
> underlying tcpdiag kernel module) can only give a snapshot of the
> current queue size).
I had done basically the same thing in previous test runs. Never any
queued requests.
I'm really hoping that I just misunderstand something here, but it
seems that our unicorn processes are accepting connections long before
they can service them.
-ryan
>
>
> [1] - I should actually make the defer timeout configurable, especially
> for Rainbows! nginx sets this to 60s and it seems to work fine.
>
> --
> Eric Wong
>
next prev parent reply other threads:[~2010-06-30 18:51 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-06-29 0:26 queued is always 0 Ryan King
2010-06-29 0:50 ` Eric Wong
2010-06-30 1:12 ` Ryan King
2010-06-30 2:45 ` Eric Wong
2010-06-30 18:51 ` Ryan King [this message]
2010-06-30 23:18 ` Ryan King
2010-07-01 10:18 ` Eric Wong
2010-07-01 18:37 ` Ryan King
2010-07-02 2:51 ` Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://yhbt.net/raindrops/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=AANLkTikC4C6z3RoHNAYBcsa5iTx0-afh33imJJYM4cu8@mail.gmail.com \
--to=ryan@twitter.com \
--cc=raindrops@librelist.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhbt.net/raindrops.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).