From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: * X-Spam-ASN: AS14383 205.234.109.0/24 X-Spam-Status: No, score=1.0 required=3.0 tests=AWL,HK_RANDOM_FROM, MSGID_FROM_MTA_HEADER shortcircuit=no autolearn=no version=3.3.2 Path: news.gmane.org!not-for-mail From: Eric Wong Newsgroups: gmane.comp.lang.ruby.rainbows.general Subject: Re: Page request roundtrip time increases substantially after a bit of use Date: Mon, 24 Jan 2011 13:54:40 -0800 Message-ID: <20110124215440.GA25489@dcvr.yhbt.net> References: <571697.98064.qm@web63303.mail.re1.yahoo.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Trace: dough.gmane.org 1295907366 9686 80.91.229.12 (24 Jan 2011 22:16:06 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Mon, 24 Jan 2011 22:16:06 +0000 (UTC) To: Rainbows! list Original-X-From: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Mon Jan 24 23:16:01 2011 Return-path: Envelope-to: gclrrg-rainbows-talk@m.gmane.org X-Original-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Delivered-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Content-Disposition: inline In-Reply-To: <571697.98064.qm-oNR6tK37MtiB9c0Qi4KiSlZ8N9CAUha/QQ4Iyu8u01E@public.gmane.org> User-Agent: Mutt/1.5.18 (2008-05-17) X-BeenThere: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Errors-To: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Xref: news.gmane.org gmane.comp.lang.ruby.rainbows.general:197 Archived-At: Received: from rubyforge.org ([205.234.109.19]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1PhUhd-0004zP-6Q for gclrrg-rainbows-talk@m.gmane.org; Mon, 24 Jan 2011 23:16:01 +0100 Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 8E9421858357; Mon, 24 Jan 2011 17:15:46 -0500 (EST) Received: from dcvr.yhbt.net (dcvr.yhbt.net [64.71.152.64]) by rubyforge.org (Postfix) with ESMTP id 2BF251858363 for ; Mon, 24 Jan 2011 16:54:41 -0500 (EST) Received: from localhost (unknown [127.0.2.5]) by dcvr.yhbt.net (Postfix) with ESMTP id 17D671F4F5; Mon, 24 Jan 2011 21:54:41 +0000 (UTC) chris mckenzie wrote: > Every now and then, however, it will be about 2.5 seconds. This will then be > followed by a bunch of the snappy millisecond level transaction times. This is That is probably GC, but... > Everything changes about 5-10 minutes into things. > > Then every transaction takes about 2-4 seconds. Static files that are 10 bytes > in size, 2-4 seconds. Ruby code to emit "Hello World"? 2-4 seconds. Every > request. Still using just 1 browser. > > After I exit all browsers and then do a netstat on the client machine to see > that the connections have closed, I can then do a curl command for a static > file; again 2-4 seconds. > > On the machine running rainbows if I do a netstat, I get this: > > tcp 1 0 10.10.192.12:7788 10.10.131.165:17443 CLOSE_WAIT > tcp 1 0 10.10.192.12:7788 10.10.131.165:17352 CLOSE_WAIT > tcp 1 196 10.10.192.12:7788 10.10.131.165:17317 CLOSE_WAIT > tcp 1 196 10.10.192.12:7788 10.10.131.165:17310 CLOSE_WAIT ^ Strange that Send-Q is 1 across all those connections.. Did you see the machine/connection that ran curl in there? How does hitting Rainbows! from localhost work? Are you dropping packets? > I think that somewhere in the ruby stack, the connections are not > closing. I would do an lsof on some of the worker processes to see if they still think a connection is open. Do you see the connection from curl in netstat? > increase my worker_process count and prolong the long poll, then yes, I'll > survive for 15 minutes instead of 5; but the problem will still eventually occur > > and I will hit the wall. > I have yet to try to test unicorn or zbatery for this style of > solution because I need the keep-alive; and although I know that > unicorn put in the keep-alive support for rainbows, I haven't really > taken the time necessary to know how to invoke it. If you think this > would be instructive, I'd be happy to do so. The Unicorn parser supports keepalive for Rainbows!, but Unicorn itself does not. Rainbows! "use :Base" (the default) is basically the same thing with Unicorn+keepalive. Zbatery doesn't do anything differently for managing client connections than Rainbows! > As far as my ruby set-up, I'm using ruby1.8 and I have the ThreadSpawn > model. We are running on a modern version of Ubuntu without any > serious customization. What's your Rainbows! keepalive_timeout set to? The default is 5s. What's your worker_connections setting? > If you think that another configuration would do the trick or if you > know how to squash this bug, it would be very helpful. This problem > has become of great concern for us. We love rainbows and all that it > is. Thanks for the project and keep up the good work. Do you notice your Rainbows! worker processes growing in memory usage over time? Which Ruby 1.8 patch level? Are you running any custom GC? If your app supports it, try Ruby 1.9.2-p136, too. Do you always set Content-Length or "Transfer-Encoding: chunked" in your app responses? Missing/inaccurate values will throw off clients if you have keepalive, otherwise I've never seen anything like the problem you describe :< -- Eric Wong _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying