From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS33070 50.56.128.0/17 X-Spam-Status: No, score=0.0 required=3.0 tests=AWL,MSGID_FROM_MTA_HEADER, TVD_RCVD_IP shortcircuit=no autolearn=unavailable version=3.3.2 Path: news.gmane.org!not-for-mail From: "Lin Jen-Shin (godfat)" Newsgroups: gmane.comp.lang.ruby.rainbows.general Subject: Re: negative timeout in Rainbows::Fiber::Base Date: Thu, 6 Sep 2012 04:06:04 +0800 Message-ID: References: <20120825024556.GA25977@dcvr.yhbt.net> <20120829211707.GA22726@dcvr.yhbt.net> <20120831013731.GA16613@dcvr.yhbt.net> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1346876154 14804 80.91.229.3 (5 Sep 2012 20:15:54 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Wed, 5 Sep 2012 20:15:54 +0000 (UTC) To: "Rainbows! list" Original-X-From: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Wed Sep 05 22:15:55 2012 Return-path: Envelope-to: gclrrg-rainbows-talk@m.gmane.org X-Original-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Delivered-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=godfat.org; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=CB4hYzbNjlOVN1p/YaZNu6zOl7CPw8GVs+Pl68D5H4M=; b=UFaDAkK//2BNb2PRbCM7Fe5/Euu4peCyfAQcgv/1z9k3n1wfIDl93ChgEXZx6oco3F vM8Lm4wocfCyr4ABw5ACylgzuZDQ8xWhZQ9g0bujmhDU+FHZrFdf4fvfkOF7SJ/ObaRC YH9fsH7eo+xSa8EJh0cDbwFbc0YrCxCgRkQqc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:x-gm-message-state; bh=CB4hYzbNjlOVN1p/YaZNu6zOl7CPw8GVs+Pl68D5H4M=; b=QPr2V8rc9gO0Rzl/rzetkLE3VE8hLudgx81xlUyHnTht5pwwPluEC4UR4309+PJu50 /L0SxhQQ8TCiAt5ZOrmcpPO1aqlQRgfIcY3PyPHtzk49aFGT0SeS/kNTlsc8UV+T2htQ 9PHUJBMrfatAPmNoEq0p7oReDoJ8EiBmgr9dqwIE4IhRNwAlOWeIn67p9hrUY0QO5xNp HYbd1VemO+sHCJCWzVlMlemYlyApno0e2q9nKUcMM+W/V+c7eTHgyNpxpBdmvj+Gzkyk A+lM7CchTVQzUb4lt9d3Bfx0OS9UlWLYaOZahXyrY49LGwwNCuswLBanKovCkLacS3f4 jYtw== In-Reply-To: <20120831013731.GA16613-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> X-Gm-Message-State: ALoCoQkPKCPEvJToQ61H5mB7sHYJugbXW3qRaJ22m+wQbboIIvoNlbsPjSpvRm3HIQassfXQMKew X-BeenThere: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Errors-To: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Xref: news.gmane.org gmane.comp.lang.ruby.rainbows.general:403 Archived-At: Received: from 50-56-192-79.static.cloud-ips.com ([50.56.192.79] helo=rubyforge.org) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1T9M0u-00007z-LU for gclrrg-rainbows-talk@m.gmane.org; Wed, 05 Sep 2012 22:15:52 +0200 Received: from localhost.localdomain (localhost [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 777082E076; Wed, 5 Sep 2012 20:15:49 +0000 (UTC) Received: from mail-lb0-f178.google.com (mail-lb0-f178.google.com [209.85.217.178]) by rubyforge.org (Postfix) with ESMTP id 8D2542E074 for ; Wed, 5 Sep 2012 20:06:37 +0000 (UTC) Received: by lbok6 with SMTP id k6so798696lbo.23 for ; Wed, 05 Sep 2012 13:06:36 -0700 (PDT) Received: by 10.112.51.174 with SMTP id l14mr66612lbo.24.1346875596291; Wed, 05 Sep 2012 13:06:36 -0700 (PDT) Received: by 10.114.19.73 with HTTP; Wed, 5 Sep 2012 13:06:04 -0700 (PDT) On Fri, Aug 31, 2012 at 9:37 AM, Eric Wong wrote: > I seem to recall problems with some of the more esoteric test cases in > Rainbows! a few years ago. > > Now that I think more about it, it might've been related to client > pipelining. If a client pipelines requests, I don't think using > EM.defer {} makes it easy to guarantee the servers responses are > returned in the correct order. > > This is made worse since (AFAIK) EM provides no easy way to > temporarily disable firing read callbacks for a socket, so > a client which pipelines aggressively becomes bad news. After some experiments, now I understood why it is hard. But I can't figure it out by some quick glimpses for how you did solve this problems for other concurrency model? One possible and simple way would be... just make piped requests sequential, but this would greatly reduce the concurrency ability, is it right? At least Puma server runs quite poorly whenever I am testing pipeline requests. My test script is: httperf --hog --server localhost --port 8080 --uri /cpu --num-calls 4 --burst-length 2 --num-conn 2 --rate 8 --print-reply But Zbatery runs quite smoothly with ThreadPool and ThreadSpawn. I assume it's because Zbatery would handle piped requests concurrently and collect responses and reply them with the correct order, though I cannot tell from the code, at least from some quick glimpses. At this point I am more confident to say that Unicorn family is the best Ruby application servers. :) > Thank you, your code makes it clear. I think your approach will work > with most HTTP clients. > > However, I think pipelined requests will hit the same problems as > EM.defer, too. Can you try with pipelining? Honestly I don't know about keep-alive and pipelined requests, and I just learned it from trying httperf, which seems to be a very good tool to make trials against web servers. Puma and Thin worked poorly in my tests with above httperf command, while Zbatery worked perfectly fine. (except with my hack for adding fibers/threads on top of EventMachine, which were raising errors) After pondering and reading the codes in Rainbows for a while, I got managed to make it work without errors, but I believe it is still suffering from the ordering issues. There's no promise for ordering. Here's the new code. It's for fibers but I think it's the same with EM.defer. https://github.com/godfat/ruby-server-exp/commit/41644fc63ffa0a505eab616c1d23ead4a564d349 def app_call input # [...] as before Fiber.new{ status, headers, body = catch(:async) { APP.call(@env.merge!(RACK_DEFAULTS)) } if nil == status || -1 == status @deferred = true else @deferred = nil # response is ready, no more @deferred ev_write_response(status, headers, body, @hp.next?) end }.resume @deferred = true # we're always deferring end To address ordering issue, I guess we can remember the index of a certain request, and if there's a request being processed which has a lower index, the response shouldn't be written back before the lower one has been written. Not sure if this is wroth the effort though... This must touch Rainbows' internal, and it cannot be easily handled by simply extending the client class. > Maybe disabling keepalive/persistent connections will make this work > correctly (but you obviously lose latency benefits, too). > > I also don't think it's possible to say "no pipelining" to a client if > we support persistent connections at all. I wonder if we always run Nginx or something similar in front of Rainbows, does it still matter? Nevertheless, I guess it's good enough for us right now. Many thanks for your review. On the other hand, I would still be very interested to see if this could be addressed. Last time I want to replicate what other concurrency models solved this, but failed to see how. > It's likely some corner case in your code. Do you generate potentially > large responses or read in large amounts of data? (e.g. SELECT > statements without a LIMIT, large files (uploads?)). > > A slow client which triggers large server responses (which EM may > buffer even if the Rack app streams it out) can hit this, too. > I don't think EM can be configured to buffer writes to the file > system (nginx will automatically do this, though). I see. Never thought of that EM might be buffering a lot of large responses in the memory. As for loading large amounts of data into memory, I guess I can't tell. As far as I know, no, but who knows :P This must be accidental if there's one... Anyway, we don't see that often nowadays. Or it could be that Ruby 1.9.3 had fixed some memory leak issues. Or some other 3rd party libraries we're using. > Ruby 1.9 sets stack sizes to 512K regardless of ulimit -s. At least on > Linux, memory defaults to being overcommited and is lazily allocated in > increments of PAGE_SIZE (4K on x86*). It's likely the actual RSS overhead > of a native thread stack is <64K. > > VMSize overhead becomes important on 32-bit with many native threads, > though. In comparison, Fibers use only 4K stack and has no extra > overhead in the kernel. I see, thanks for the explanation. I guess that does matter a bit, but only if we're using thousands of threads/fibers, and it should be quite rarely in a web app, I guess. Using fibers are also risking from system stack overflow, especially in a Rails app with a lot of plugins, I guess... Umm, but I also heard that fibers stack is increased a bit in newer Ruby? >> Though I really doubt if threads are really that heavy comparing to fibers. >> At least in some simple tests, threads are fine and efficient enough. > > I agree native threads are light enough for most cases (especially since > you're already running Ruby :). Speaking to this and green threads, I wonder if it's worth the effort to implement m:n threading for Ruby? Or we can just compile and link against a threading library which supports m:n threading? Goroutine? :P >> EventMachine is still a lot faster than regular sockets (net/http) though, >> so I'll still keep EventMachine for a while even if I switched to threads. > > I think part of that is the HTTP parser and I/O buffering being > implemented in C/C++ vs Ruby. Things like the net-http-persistent gem > should help with pure-Ruby performance, though (and performance is > likely to be better with upcoming Ruby releases). I haven't got a chance to try net-http-persistent, but it seems I should try it. (or try that on em-http-request, it seems it supports it https://github.com/igrigorik/em-http-request/wiki/Keep-Alive-and-HTTP-Pipelining ) Or if it's all about HTTP parsing, [http][] gem should help too. It is using [http_parser.rb] underneath, which is based on NodeJS' [http-parser][]. Sometimes I feel it's all about throwing away EventMachine... I've heard that EM is bad, but not bad enough to be rewritten... [http]: https://github.com/tarcieri/http [http_parser.rb]: https://github.com/tmm1/http_parser.rb [http-parser]: https://github.com/joyent/http-parser > I enjoy my near-anonymity and want as little reputation/recognition as > possible. I'm always happy if people talk about software, but I prefer > software stand on its own and not on the reputation of its authors. > > (The only reason I use my name is for potential GPL enforcement) I see. Thanks for explaining. I'll then avoid talking about authors :) _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying