Rainbows! Rack HTTP server user/dev discussion
 help / color / mirror / code / Atom feed
From: Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org>
To: Rainbows! list <rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org>
Subject: Re: Page request roundtrip time increases substantially after a bit of use
Date: Mon, 24 Jan 2011 13:54:40 -0800	[thread overview]
Message-ID: <20110124215440.GA25489@dcvr.yhbt.net> (raw)
In-Reply-To: <571697.98064.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>

chris mckenzie <kristopolous-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> wrote:
> Every now and then, however, it will be about 2.5 seconds.  This will then be 
> followed by a bunch of the snappy millisecond level transaction  times.  This is 

That is probably GC, but...

> Everything changes about 5-10 minutes into things. 
> 
> Then every transaction takes about 2-4 seconds.  Static files that are 10 bytes 
> in size, 2-4 seconds. Ruby code to emit "Hello World"? 2-4 seconds.  Every 
> request.  Still using just 1 browser.
> 
> After I exit all browsers and then do a netstat on the client machine to see 
> that the connections have closed,  I can then do a curl command for a static 
> file; again 2-4 seconds.
> 
> On the machine running rainbows if I do a netstat, I get this:
> 
> tcp        1      0 10.10.192.12:7788       10.10.131.165:17443     CLOSE_WAIT
> tcp        1      0  10.10.192.12:7788       10.10.131.165:17352     CLOSE_WAIT
> tcp        1    196 10.10.192.12:7788       10.10.131.165:17317     CLOSE_WAIT
> tcp        1    196 10.10.192.12:7788       10.10.131.165:17310     CLOSE_WAIT

             ^ Strange that Send-Q is 1 across all those connections..

Did you see the machine/connection that ran curl in there?  How does
hitting Rainbows! from localhost work?

Are you dropping packets?

> I think that somewhere in the ruby stack, the connections are not
> closing.

I would do an lsof on some of the worker processes to see if they
still think a connection is open.  Do you see the connection from
curl in netstat?

> increase my worker_process count and prolong the long poll, then yes, I'll 
> survive for 15 minutes instead of 5; but the problem will still eventually occur 
> 
> and I will hit the wall.

> I have yet to try to test unicorn or zbatery for this style of
> solution because I need the keep-alive; and although I know that
> unicorn put in the keep-alive support for rainbows, I haven't really
> taken the time necessary to know how to invoke it.  If you think this
> would be instructive, I'd be happy to do so.

The Unicorn parser supports keepalive for Rainbows!, but Unicorn itself
does not.  Rainbows! "use :Base" (the default) is basically the same
thing with Unicorn+keepalive.  Zbatery doesn't do anything differently
for managing client connections than Rainbows!

> As far as my ruby set-up, I'm using ruby1.8 and I have the ThreadSpawn
> model.  We are running on a modern  version of Ubuntu without any
> serious customization.

What's your Rainbows! keepalive_timeout set to?  The default is 5s.
What's your worker_connections setting?

> If you think that another configuration would do the trick or if you
> know how to squash this bug, it would be very helpful.   This problem
> has become of great concern for us.  We love rainbows and all that it
> is. Thanks for the project and keep up the good work.

Do you notice your Rainbows! worker processes growing in memory usage
over time?  Which Ruby 1.8 patch level?  Are you running any custom
GC?  If your app supports it, try Ruby 1.9.2-p136, too.

Do you always set Content-Length or "Transfer-Encoding: chunked" in your
app responses?  Missing/inaccurate values will throw off clients if you
have keepalive, otherwise I've never seen anything like the problem you
describe :<

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


  parent reply	other threads:[~2011-01-24 22:16 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-01-24 21:03 Page request roundtrip time increases substantially after a bit of use chris mckenzie
     [not found] ` <571697.98064.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-24 21:54   ` Eric Wong [this message]
     [not found]     ` <20110124215440.GA25489-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25  0:11       ` Eric Wong
     [not found]         ` <20110125001107.GA1921-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25  1:14           ` chris mckenzie
     [not found]             ` <443004.19531.qm-9Fhxc66Fx5iB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-25  2:02               ` Eric Wong
2011-01-25  2:55           ` chris mckenzie
     [not found]             ` <288407.18061.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org>
2011-01-25  3:50               ` Eric Wong
     [not found]                 ` <20110125035048.GA8124-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-01-25 21:38                   ` Eric Wong
     [not found]                     ` <20110125213835.GA9421-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-02-03  2:54                       ` Eric Wong
     [not found]                         ` <20110203025415.GA3812-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2011-02-03  4:44                           ` chris mckenzie

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/rainbows/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20110124215440.GA25489@dcvr.yhbt.net \
    --to=normalperson-rmlxzr9ms24@public.gmane.org \
    --cc=rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://yhbt.net/rainbows.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).