* negative timeout in Rainbows::Fiber::Base @ 2012-08-23 20:36 Lin Jen-Shin (godfat) [not found] ` <CAA2_N1unOXb7Z4Jr8oKoSLu266O9Ko4o=oWzAcMA1w3=9X74KA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Lin Jen-Shin (godfat) @ 2012-08-23 20:36 UTC (permalink / raw) To: Rainbows! list Greetings Unicorns, I am writing some stupid benchmarks, trying FiberSpawn and FiberPool, and seeing this error: listen loop error: time interval must be positive (ArgumentError) Looking into the code, in this line: http://bogomips.org/rainbows.git/tree/lib/rainbows/fiber/base.rb#n55 Isn't that possible `max <= (now + 1)` and `now > max`? Whenever this is the case, then timeout passed to `select` could be negative. I am not sure how to fix this correctly, but a blind `abs` could resolve this error. Hope this could give some hints, thank you! commit b599374ef10c7847445f29f80e74cde3e4d3b940 Author: Lin Jen-Shin <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> Date: Fri Aug 24 04:33:38 2012 +0800 fiber/base.rb: make timeout never be negative. In the case where `max <= (now + 1)` and `now > max`, we need to make sure that timeout is positive. diff --git a/lib/rainbows/fiber/base.rb b/lib/rainbows/fiber/base.rb index 00af214..7bd43cb 100644 --- a/lib/rainbows/fiber/base.rb +++ b/lib/rainbows/fiber/base.rb @@ -52,7 +52,7 @@ module Rainbows::Fiber::Base } fibs.each { |fib| fib.resume } now = Time.now - max.nil? || max > (now + 1) ? 1 : max - now + max.nil? || max > (now + 1) ? 1 : (max - now).abs end def process(client) _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply related [flat|nested] 18+ messages in thread
[parent not found: <CAA2_N1unOXb7Z4Jr8oKoSLu266O9Ko4o=oWzAcMA1w3=9X74KA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <CAA2_N1unOXb7Z4Jr8oKoSLu266O9Ko4o=oWzAcMA1w3=9X74KA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2012-08-25 2:45 ` Eric Wong [not found] ` <20120825024556.GA25977-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Eric Wong @ 2012-08-25 2:45 UTC (permalink / raw) To: Rainbows! list; +Cc: Lin Jen-Shin (godfat) "Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote: > Greetings Unicorns, > > I am writing some stupid benchmarks, trying FiberSpawn and > FiberPool, and seeing this error: > > listen loop error: time interval must be positive (ArgumentError) > > Looking into the code, in this line: > http://bogomips.org/rainbows.git/tree/lib/rainbows/fiber/base.rb#n55 > Isn't that possible `max <= (now + 1)` and `now > max`? > Whenever this is the case, then timeout passed to `select` > could be negative. I am not sure how to fix this correctly, > but a blind `abs` could resolve this error. The original code looks wrong and confusing :x It's been a while since I've looked hard at the Fiber* stuff (not a fan of it personally). However, I don't think abs() is correct, either. > Hope this could give some hints, thank you! Perhaps this is less confusing and more correct: diff --git a/lib/rainbows/fiber/base.rb b/lib/rainbows/fiber/base.rb index 6828e1d..103c97a 100644 --- a/lib/rainbows/fiber/base.rb +++ b/lib/rainbows/fiber/base.rb @@ -51,12 +51,15 @@ module Rainbows::Fiber::Base end } fibs.each { |fib| fib.resume } - now = Time.now + + max_sleep = 1.0 if max - timeout = max - now - timeout < 0.0 ? 0 : timeout + max -= Time.now + return 0 if max < 0.0 + return max_sleep if max > max_sleep + max else - 1 + max_sleep end end --- Thoughts? _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply related [flat|nested] 18+ messages in thread
[parent not found: <20120825024556.GA25977-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <20120825024556.GA25977-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> @ 2012-08-26 0:12 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1uhfcHDbTvY+ke0Cid6=i7KEhFn8jvEirx+ptYVDacdvA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Lin Jen-Shin (godfat) @ 2012-08-26 0:12 UTC (permalink / raw) To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw On Sat, Aug 25, 2012 at 10:45 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > The original code looks wrong and confusing :x I guess I know your intension in the new code now :) I can't apply the new patch, so I'll assume the new code is as following: fibs.each { |fib| fib.resume } max_sleep = 1.0 if max max -= Time.now return 0 if max < 0.0 return max_sleep if max > max_sleep max else max_sleep end I once thought that if `max < Time.now`, it should return 0, because apparently we don't want to wait in this case. But I am not sure if passing timeout as 0 would mean to wait forever... thinking that maybe 0.0001 might be better? Not sure if this would be a concern, or 0 means don't wait at all, that would be ideal in this case. > It's been a while since I've looked hard at the Fiber* stuff (not a fan > of it personally). However, I don't think abs() is correct, either. I can smell from the code that those are not well polished, since it seems a lot of things are missing. For example, there is Rainbows::Fiber::IO::Socket, but no TCPSocket. I blindly tried to create a TCPSocket version by creating a new class extended from Kgio::TCPSocket and include Rainbows::Fiber::IO::Methods. Well, I think it works, if using correctly, for example it does not accept domain names as built-in TCPSocket, and not all of the methods are respecting fibers. But I like the idea to patch the socket, that way, we can turn existing library to run under certain network architecture. For instance, this HTTP client accepts passing different socket class: https://github.com/tarcieri/http/blob/master/lib/http/options.rb#L33-37 @default_socket_class = TCPSocket @default_ssl_socket_class = OpenSSL::SSL::SSLSocket class << self attr_accessor :default_socket_class, :default_ssl_socket_class I didn't try if passing Rainbows' fiber sockets would work or not, but I think in theory it should work. I think Tony did this for his: https://github.com/celluloid/celluloid-io Quoted from README: """ Replacement classes: Celluloid::IO includes replacements for the core TCPServer and TCPSocket classes which automatically use an evented mode inside of Celluloid::IO actors. They're named Celluloid::IO::TCPServer and Celluloid::IO::TCPSocket, so they're automatically available inside your class when you include Celluloid::IO. """ Not sure if this is the way to go, but an interesting idea. At least better than implementing every single client in both eventmachine and built-in sockets I guess... So speaking to celluloid-io, there's already Revactor and Coolio support, what about celluloid-io? Last time I tried Coolio, occasionally it would crash with internal assertion failures... I would love to see if there's other alternatives to eventmachine. Also, there is CoolioFiberSpawn, CoolioThreadPool and so on so forth, why not EventMachineFiberSpawn and EventMachineThreadPool? Actually we are running EventMachineFiberSpawn on production at the moment, as shown in: https://github.com/cardinalblue/rest-more/blob/master/example/rainbows.rb#L15-19 class REMFClient < Rainbows::EventMachine::Client def app_call input Fiber.new{ super }.resume end end (well, neverblock works too, but it doesn't seem to be maintained now and I don't understand why it's a gem with a lot of codes but not as simple as above. As this sense, I feel async_sinatra is so wrong, but not sure if I missed something here.) It works great for us at the moment, though I am thinking about switching to threads, too, since it's a lot more flexible than fibers. Sorry that I might be asking too many questions in a single thread. If you're interested in how I am testing them, I put my testing codes at: https://github.com/godfat/ruby-server-exp I am thinking about if I should be giving a talk on rubyconf.tw about ruby application servers, promoting unicorn family. It is one of the reasons to try out all of the options. Thanks for your time :D _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <CAA2_N1uhfcHDbTvY+ke0Cid6=i7KEhFn8jvEirx+ptYVDacdvA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <CAA2_N1uhfcHDbTvY+ke0Cid6=i7KEhFn8jvEirx+ptYVDacdvA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2012-08-26 1:15 ` Eric Wong 2012-08-29 16:00 ` Lin Jen-Shin (godfat) 1 sibling, 0 replies; 18+ messages in thread From: Eric Wong @ 2012-08-26 1:15 UTC (permalink / raw) To: Rainbows! list "Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote: > On Sat, Aug 25, 2012 at 10:45 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > > The original code looks wrong and confusing :x > > I guess I know your intension in the new code now :) > I can't apply the new patch, so I'll assume the new > code is as following: > > fibs.each { |fib| fib.resume } > > max_sleep = 1.0 > if max > max -= Time.now > return 0 if max < 0.0 > return max_sleep if max > max_sleep > max > else > max_sleep > end Yes. (Though it should've be easy to just pipe that part of my message to "git apply" or even "patch -p1") > I once thought that if `max < Time.now`, it should return 0, > because apparently we don't want to wait in this case. > But I am not sure if passing timeout as 0 would mean to > wait forever... thinking that maybe 0.0001 might be better? > Not sure if this would be a concern, or 0 means don't wait > at all, that would be ideal in this case. Aa timeout of 0 returns immediately for select (and poll/epoll/kevent), but still marks ready descriptors. > > It's been a while since I've looked hard at the Fiber* stuff (not a fan > > of it personally). However, I don't think abs() is correct, either. > > I can smell from the code that those are not well polished, > since it seems a lot of things are missing. For example, > there is Rainbows::Fiber::IO::Socket, but no TCPSocket. > I blindly tried to create a TCPSocket version by creating > a new class extended from Kgio::TCPSocket and include > Rainbows::Fiber::IO::Methods. Yeah, it was an experiment and I've become less-interested in Fibers over time. > Well, I think it works, if using correctly, for example it > does not accept domain names as built-in TCPSocket, > and not all of the methods are respecting fibers. DNS resolution needs to be handled asynchronously as well, so I'd rather leave it out than block on the portable DNS lookup functions. > But I like the idea to patch the socket, that way, we can > turn existing library to run under certain network architecture. > For instance, this HTTP client accepts passing different > socket class: > https://github.com/tarcieri/http/blob/master/lib/http/options.rb#L33-37 > > @default_socket_class = TCPSocket > @default_ssl_socket_class = OpenSSL::SSL::SSLSocket > > class << self > attr_accessor :default_socket_class, :default_ssl_socket_class > > I didn't try if passing Rainbows' fiber sockets would work or not, > but I think in theory it should work. I think Tony did this for his: > https://github.com/celluloid/celluloid-io > Quoted from README: > > """ > Replacement classes: Celluloid::IO includes replacements for the core > TCPServer and TCPSocket classes which automatically use an evented > mode inside of Celluloid::IO actors. They're named > Celluloid::IO::TCPServer and Celluloid::IO::TCPSocket, so they're > automatically available inside your class when you include > Celluloid::IO. > """ > > Not sure if this is the way to go, but an interesting idea. At least > better than implementing every single client in both eventmachine > and built-in sockets I guess... > > So speaking to celluloid-io, there's already Revactor and Coolio > support, what about celluloid-io? Last time I tried Coolio, > occasionally it would crash with internal assertion failures... > I would love to see if there's other alternatives to eventmachine. All the Thread-based concurrency models should already just work with Celluloid-based apps. http://news.gmane.org/find-root.php?message_id=%3cBANLkTimnNQYpk7TRkiCvhDikzGHVy%3dkOdw%40mail.gmail.com%3e > Also, there is CoolioFiberSpawn, CoolioThreadPool and so on > so forth, why not EventMachineFiberSpawn and EventMachineThreadPool? It's much easier to unwatch IO objects with Cool.io than with EM, so I haven't done much with EM + Fibers. There's also rack-fiber_pool... Rainbows::EventMachine::TryDefer already uses the EM-internal thread pool, so I think EventMachineThreadPool would be redundant. > Actually we are running EventMachineFiberSpawn on production > at the moment, as shown in: > > https://github.com/cardinalblue/rest-more/blob/master/example/rainbows.rb#L15-19 > class REMFClient < Rainbows::EventMachine::Client > def app_call input > Fiber.new{ super }.resume > end > end Does it actually show benefits compared to regular EM? I suppose the Rack application still needs to be made Fiber-aware to reap benefits of Fibers > (well, neverblock works too, but it doesn't seem to be > maintained now and I don't understand why it's a gem > with a lot of codes but not as simple as above. As this > sense, I feel async_sinatra is so wrong, but not sure > if I missed something here.) > > It works great for us at the moment, though I am thinking about > switching to threads, too, since it's a lot more flexible than fibers. Yes, threads are far more compatible with existing gems/libraries and I think that's the better way to go. Of course, just processes (with regular unicorn) should be most compatible, too. > Sorry that I might be asking too many questions in a single thread. No problem! I'm just happy that folks are showing interest > If you're interested in how I am testing them, I put my testing codes > at: https://github.com/godfat/ruby-server-exp > I am thinking about if I should be giving a talk on rubyconf.tw > about ruby application servers, promoting unicorn family. > It is one of the reasons to try out all of the options. Cool! _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <CAA2_N1uhfcHDbTvY+ke0Cid6=i7KEhFn8jvEirx+ptYVDacdvA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2012-08-26 1:15 ` Eric Wong @ 2012-08-29 16:00 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1thakAOVp7ibCNic+TjEVvXE0OGLgzXH3fJ1c2UTs68oQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 1 sibling, 1 reply; 18+ messages in thread From: Lin Jen-Shin (godfat) @ 2012-08-29 16:00 UTC (permalink / raw) To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw Sorry that I didn't subscribe the mailing list (now subscribed), so I didn't realize there's already a response. The quoted text is copied from archive. Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > "Lin Jen-Shin (godfat)" <godfat at godfat.org> wrote: > Yes. (Though it should've be easy to just pipe that part of my > message to "git apply" or even "patch -p1") I did that in the beginning, but it looks like your patch was not created from 12281e4cee86496588814cd1851e8764caea024c, so I cannot apply it? Anyway, not important :) > All the Thread-based concurrency models should already just work > with Celluloid-based apps. > http://news.gmane.org/find-root.php?message_id=%3cBANLkTimnNQYpk7TRkiCvhDikzGHVy%3dkOdw%40mail.gmail.com%3e And it's not worth the effort to wrap around celluloid-io for buffering requests? Since it's using nio4r[0] underneath and it's using libev for I/O, I thought that would be a replacement for eventmachine? [0]: https://github.com/tarcieri/nio4r > It's much easier to unwatch IO objects with Cool.io than with EM, so I > haven't done much with EM + Fibers. There's also rack-fiber_pool... True, working with Cool.io is much more pleasant. As for rack-fiber_pool, I am not sure if I am correct or not, but in my humble option, that's a totally wrong approach. I think fibers should be handled in server level, not application level. That way, we don't need the hack for throwing :async, and the middlewares do not need to be aware of fibers and :async, and we won't need async-rack, and the server doesn't need to be aware of :async, so on so forth. I am not sure if this approach would have any bad influence, but at least it works fine in our applications. That is, just wrap fibers around the calling the app. (i.e. Fiber.new{ app_call(env) }.resume) > Rainbows::EventMachine::TryDefer already uses the EM-internal thread > pool, so I think EventMachineThreadPool would be redundant. I didn't know this before, I guess it should work for me, though it might not be as direct as: https://github.com/godfat/ruby-server-exp/blob/9d44f8387739f5395bf97aff1689f17615cc4c7e/config/rainbows-em-thread-pool.rb#L21-25 class REMTPoolClient < Rainbows::EventMachine::Client def app_call input EM.defer{ super } end end > > https://github.com/cardinalblue/rest-more/blob/master/example/rainbows.rb#L15-19 > > class REMFClient < Rainbows::EventMachine::Client > > def app_call input > > Fiber.new{ super }.resume > > end > > end > Does it actually show benefits compared to regular EM? > I suppose the Rack application still needs to be made Fiber-aware > to reap benefits of Fibers The Rack application doesn't need to be fiber-aware, but if the libraries are fiber-aware, then it would be beneficial. The rest of program doesn't have to know the existence of fibers, because we don't throw :async. > Yes, threads are far more compatible with existing gems/libraries and I > think that's the better way to go. Of course, just processes (with > regular unicorn) should be most compatible, too. At first I thought fibers are better for I/O, but after implementing a thread based approach for the same mechanism, I started to doubt that. I don't have a conclusion yet though. (I am a bit nervous to switch to thread based approach on production to find out, since it's not yet tested intensively) On the other hand, all our apps are running on Heroku, and we need to reduce memory usage because they have a hard limit. That's why we're running Zbatery at the moment. > > Sorry that I might be asking too many questions in a single thread. > No problem! I'm just happy that folks are showing interest I really don't understand why Unicorn family didn't get much attention in Ruby community. And I always feel wrong when people are comparing Unicorn with Thin or even Passenger. (sigh) Many thanks for your work and help! _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <CAA2_N1thakAOVp7ibCNic+TjEVvXE0OGLgzXH3fJ1c2UTs68oQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <CAA2_N1thakAOVp7ibCNic+TjEVvXE0OGLgzXH3fJ1c2UTs68oQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2012-08-29 21:17 ` Eric Wong [not found] ` <20120829211707.GA22726-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Eric Wong @ 2012-08-29 21:17 UTC (permalink / raw) To: Rainbows! list "Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote: > Sorry that I didn't subscribe the mailing list (now subscribed), > so I didn't realize there's already a response. The quoted > text is copied from archive. Oops :x I encourage unsubscribed senders to remind repliers to Cc: them for this reason. Unfortunately ruby-talk has conditioned the Ruby community to favor subscriber-only lists and the Mailman config for rainbows-talk avoids Cc-by-default (even though we'll still accept unsubscribed senders). > Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > > "Lin Jen-Shin (godfat)" <godfat at godfat.org> wrote: > > Yes. (Though it should've be easy to just pipe that part of my > > message to "git apply" or even "patch -p1") > > I did that in the beginning, but it looks like your patch was not > created from 12281e4cee86496588814cd1851e8764caea024c, > so I cannot apply it? Anyway, not important :) Anyways, I've pushed my patch to master as commit e794a40049959a23ba311c572e518bb7c2861812 Will probably release v4.4.1 with that tonight unless there's something else... > > All the Thread-based concurrency models should already just work > > with Celluloid-based apps. > > http://news.gmane.org/find-root.php?message_id=%3cBANLkTimnNQYpk7TRkiCvhDikzGHVy%3dkOdw%40mail.gmail.com%3e > > And it's not worth the effort to wrap around celluloid-io for > buffering requests? Since it's using nio4r[0] underneath > and it's using libev for I/O, I thought that would be a replacement > for eventmachine? > > [0]: https://github.com/tarcieri/nio4r I haven't followed celluloid* closely since it was originally announced. Maybe it's worth it to offer celluloid-io as an option and I will accept patches for it. > > It's much easier to unwatch IO objects with Cool.io than with EM, so I > > haven't done much with EM + Fibers. There's also rack-fiber_pool... > > True, working with Cool.io is much more pleasant. As for rack-fiber_pool, > I am not sure if I am correct or not, but in my humble option, that's a > totally wrong approach. I think fibers should be handled in server level, > not application level. That way, we don't need the hack for throwing :async, > and the middlewares do not need to be aware of fibers and :async, > and we won't need async-rack, and the server doesn't need to be > aware of :async, so on so forth. > > I am not sure if this approach would have any bad influence, but at least > it works fine in our applications. That is, just wrap fibers around the calling > the app. (i.e. Fiber.new{ app_call(env) }.resume) Rainbows! tries not to favor/disfavor any approaches to concurrency. And rack-fiber_pool was easy-to-support, so I added support for it. > > Rainbows::EventMachine::TryDefer already uses the EM-internal thread > > pool, so I think EventMachineThreadPool would be redundant. > > I didn't know this before, I guess it should work for me, though > it might not be as direct as: > > https://github.com/godfat/ruby-server-exp/blob/9d44f8387739f5395bf97aff1689f17615cc4c7e/config/rainbows-em-thread-pool.rb#L21-25 > class REMTPoolClient < Rainbows::EventMachine::Client > def app_call input > EM.defer{ super } > end > end I seem to recall this didn't work well with some corner cases (large static files/socket-proxying in responses). > > > https://github.com/cardinalblue/rest-more/blob/master/example/rainbows.rb#L15-19 > > > class REMFClient < Rainbows::EventMachine::Client > > > def app_call input > > > Fiber.new{ super }.resume > > > end > > > end > > Does it actually show benefits compared to regular EM? > > I suppose the Rack application still needs to be made Fiber-aware > > to reap benefits of Fibers > > The Rack application doesn't need to be fiber-aware, but if > the libraries are fiber-aware, then it would be beneficial. > The rest of program doesn't have to know the existence of > fibers, because we don't throw :async. OK. But are clients served concurrently by Rainbows! in this case? I'm not sure if I'm following this correctly (haven't thought about Fibers in a long time), but control never goes back to Rainbows! itself in your above case, does it? > > Yes, threads are far more compatible with existing gems/libraries and I > > think that's the better way to go. Of course, just processes (with > > regular unicorn) should be most compatible, too. > > At first I thought fibers are better for I/O, but after implementing > a thread based approach for the same mechanism, I started to > doubt that. I don't have a conclusion yet though. > (I am a bit nervous to switch to thread based approach on > production to find out, since it's not yet tested intensively) > > On the other hand, all our apps are running on Heroku, and > we need to reduce memory usage because they have a hard > limit. That's why we're running Zbatery at the moment. Interesting. Are you on 32 or 64-bit and are you constrained by VMSize or RSS? If it's VMSize and you're not dealing with deep data structures _and_ your code doesn't use recursion much; lowering pthreads stack size to 64K in thread_pthread.c should help with memory usage. Not sure if you can supply your own Ruby builds for that hosting service, though. If you have time, you could also try making something like GNU pth or npth work with Ruby 1.9. I suspect Fibers will become unusable with *pth, though... > > > Sorry that I might be asking too many questions in a single thread. > > No problem! I'm just happy that folks are showing interest > > I really don't understand why Unicorn family didn't get much > attention in Ruby community. And I always feel wrong when > people are comparing Unicorn with Thin or even Passenger. > (sigh) I'm actually surprised unicorn got the attention it has. As project leader, my preference to maintain a low public profile and my refusal to endorse/associate with for-profit interests certainly hurts adoption. If somebody else (like you) wants to market these projects, then more power to them :) I'm certainly never going to speak at any conference. Rainbows! just overwhelms potential users with too many choices, so it's unlikely to ever be widely-adopted. > Many thanks for your work and help! No problem :) _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <20120829211707.GA22726-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <20120829211707.GA22726-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> @ 2012-08-30 21:33 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1tc=Xx8WHaM8H=EWshyzGEyX04PnkdBGj9Jdb7cSzmbRQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Lin Jen-Shin (godfat) @ 2012-08-30 21:33 UTC (permalink / raw) To: Rainbows! list On Thu, Aug 30, 2012 at 5:17 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > Oops :x I encourage unsubscribed senders to remind repliers > to Cc: them for this reason. > > Unfortunately ruby-talk has conditioned the Ruby community to favor > subscriber-only lists and the Mailman config for rainbows-talk avoids > Cc-by-default (even though we'll still accept unsubscribed senders). That's ok, since most of time I would check the archive regularly whenever I am expecting responses. I forgot this time because I was quite busy recently. And now I subscribed the list because I guess there won't have too many mails anyway and I would be definitely interested in discussing Rainbows! :) > Anyways, I've pushed my patch to master as > commit e794a40049959a23ba311c572e518bb7c2861812 > > Will probably release v4.4.1 with that tonight unless there's something > else... Yeah, I saw it, thank you. > I haven't followed celluloid* closely since it was originally announced. > Maybe it's worth it to offer celluloid-io as an option and I will accept > patches for it. Cool. Then I might want to try it some other time. I don't have confidence though. >> https://github.com/godfat/ruby-server-exp/blob/9d44f8387739f5395bf97aff1689f17615cc4c7e/config/rainbows-em-thread-pool.rb#L21-25 >> class REMTPoolClient < Rainbows::EventMachine::Client >> def app_call input >> EM.defer{ super } >> end >> end > > I seem to recall this didn't work well with some corner cases > (large static files/socket-proxying in responses). Sorry that I have no idea about those corner cases. (not sure if they are corner enough?) I am not surprised if this approach won't work for all cases. I would be interested to try out though. Not sure what's socket-proxying, but I'll try large static files, and see why it's not working properly. Hope it would be easy to reproduce. >> > > https://github.com/cardinalblue/rest-more/blob/master/example/rainbows.rb#L15-19 >> > > class REMFClient < Rainbows::EventMachine::Client >> > > def app_call input >> > > Fiber.new{ super }.resume >> > > end >> > > end >> > Does it actually show benefits compared to regular EM? >> > I suppose the Rack application still needs to be made Fiber-aware >> > to reap benefits of Fibers >> >> The Rack application doesn't need to be fiber-aware, but if >> the libraries are fiber-aware, then it would be beneficial. >> The rest of program doesn't have to know the existence of >> fibers, because we don't throw :async. > > OK. But are clients served concurrently by Rainbows! in this case? > > I'm not sure if I'm following this correctly (haven't thought about > Fibers in a long time), but control never goes back to Rainbows! itself > in your above case, does it? It does as far as I know. I am not good at explaining in natural languages, and English is not my first language. Let me show this concept in codes. It's something like this: ( Also in gist: https://gist.github.com/3540749 ) # Wrap a fiber around app_call Fiber.new{ # [...] # Below is modified from event_machine/client.rb, in def app_call input, # but let's forget about :async for now. # In APP.call, it would call Fiber.yield whenever we're waiting for data. status, headers, body = APP.call(@env.merge!(RACK_DEFAULTS)) # The response is fully ready at this point. ev_write_response(status, headers, body, @hp.next?) }.resume # * * * * * # Here's the app APP = lambda{ |env| # First we remember where we need to jump back f = Fiber.current # Wait for 5 seconds to emulate waiting for data EM.add_timer(5){ # Here then we have data available, let's resume back. # But we also wrap resuming in a next_tick block, # giving EM some chances to cleanup it's internal states. EM.next_tick{ f.resume('OK') } } # Here we immediately yield and give control back to Rainbows! # and Rainbows! would then go back to EventMachine's regular loop, # buffering requests, making requests, etc. body = Fiber.yield # So now body is referring the data resumed from above. # Rainbows! should handle the response at this point. [200, {}, [body]] } Not sure if this is clear enough, please let me know if I didn't make it clear, or there's anything wrong in my assumption or the over simplified codes. Thanks! > Interesting. Are you on 32 or 64-bit and are you constrained by VMSize > or RSS? I think it's an x86_64 VM. Not sure what's VMSize, but we're observing memory usage in RSS. They claim we have about 500M for one process, but sometimes the app cannot allocate more memory even below 500M. (Well, I don't quite understand how the app would grow to 500M? It's about 160M on average) > If it's VMSize and you're not dealing with deep data structures _and_ > your code doesn't use recursion much; lowering pthreads stack size to > 64K in thread_pthread.c should help with memory usage. Not sure if you > can supply your own Ruby builds for that hosting service, though. There's a chance to do that on Heroku, but I guess it's too much effort to do so. I don't feel there's a standard way to compile stuffs on it. How do I tell the current stack size for a thread in pthread? It seems I cannot run ulimit on Heroku. On the other hand, I am not sure if it's significant enough to reduce thread stack size. I guess we can also reduce the concurrent level (worker_connections?) to reduce memory usage, too? > If you have time, you could also try making something like GNU pth or > npth work with Ruby 1.9. I suspect Fibers will become unusable > with *pth, though... Cool, might be interesting. Could then threads be as lightweight as fibers? :P Though I really doubt if threads are really that heavy comparing to fibers. At least in some simple tests, threads are fine and efficient enough. EventMachine is still a lot faster than regular sockets (net/http) though, so I'll still keep EventMachine for a while even if I switched to threads. > I'm actually surprised unicorn got the attention it has. > > As project leader, my preference to maintain a low public profile and my > refusal to endorse/associate with for-profit interests certainly hurts > adoption. > > If somebody else (like you) wants to market these projects, then more > power to them :) I'm certainly never going to speak at any conference. > > Rainbows! just overwhelms potential users with too many choices, > so it's unlikely to ever be widely-adopted. Umm... I don't know the reason why you want to stay away from them, but please let me know if you don't want someone to market them. I'll then try to be as natural as possible when talking about Unicorns. I guess Github's post certainly attracted at lot of people's attention. I don't remember where I did see Unicorn in the beginning, but certainly I became more interested because of that post. I guess that's also one of the reasons that people didn't know about Rainbows! (they didn't mention it) Or you might be right, a lot of people just don't try Rainbows! because there are too many choices. Or, use it wrongly, I guess... _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <CAA2_N1tc=Xx8WHaM8H=EWshyzGEyX04PnkdBGj9Jdb7cSzmbRQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <CAA2_N1tc=Xx8WHaM8H=EWshyzGEyX04PnkdBGj9Jdb7cSzmbRQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2012-08-31 1:37 ` Eric Wong [not found] ` <20120831013731.GA16613-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Eric Wong @ 2012-08-31 1:37 UTC (permalink / raw) To: Rainbows! list "Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote: > On Thu, Aug 30, 2012 at 5:17 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > > Will probably release v4.4.1 with that tonight unless there's something > > else... > > Yeah, I saw it, thank you. Will release today before I forget again :> > > I haven't followed celluloid* closely since it was originally announced. > > Maybe it's worth it to offer celluloid-io as an option and I will accept > > patches for it. > > Cool. Then I might want to try it some other time. I don't have > confidence though. No problem, take your time :) > >> https://github.com/godfat/ruby-server-exp/blob/9d44f8387739f5395bf97aff1689f17615cc4c7e/config/rainbows-em-thread-pool.rb#L21-25 > >> class REMTPoolClient < Rainbows::EventMachine::Client > >> def app_call input > >> EM.defer{ super } > >> end > >> end > > > > I seem to recall this didn't work well with some corner cases > > (large static files/socket-proxying in responses). > > Sorry that I have no idea about those corner cases. (not sure > if they are corner enough?) I am not surprised if this approach > won't work for all cases. I would be interested to try out though. > Not sure what's socket-proxying, but I'll try large static files, > and see why it's not working properly. Hope it would be easy > to reproduce. I seem to recall problems with some of the more esoteric test cases in Rainbows! a few years ago. Now that I think more about it, it might've been related to client pipelining. If a client pipelines requests, I don't think using EM.defer {} makes it easy to guarantee the servers responses are returned in the correct order. This is made worse since (AFAIK) EM provides no easy way to temporarily disable firing read callbacks for a socket, so a client which pipelines aggressively becomes bad news. > >> > > https://github.com/cardinalblue/rest-more/blob/master/example/rainbows.rb#L15-19 > >> > > class REMFClient < Rainbows::EventMachine::Client > >> > > def app_call input > >> > > Fiber.new{ super }.resume > >> > > end > >> > > end > >> > Does it actually show benefits compared to regular EM? > >> > I suppose the Rack application still needs to be made Fiber-aware > >> > to reap benefits of Fibers > >> > >> The Rack application doesn't need to be fiber-aware, but if > >> the libraries are fiber-aware, then it would be beneficial. > >> The rest of program doesn't have to know the existence of > >> fibers, because we don't throw :async. > > > > OK. But are clients served concurrently by Rainbows! in this case? > > > > I'm not sure if I'm following this correctly (haven't thought about > > Fibers in a long time), but control never goes back to Rainbows! itself > > in your above case, does it? > > It does as far as I know. I am not good at explaining in natural languages, > and English is not my first language. Let me show this concept in codes. > It's something like this: > ( Also in gist: https://gist.github.com/3540749 ) > > # Wrap a fiber around app_call > Fiber.new{ > # [...] > # Below is modified from event_machine/client.rb, in def app_call input, > # but let's forget about :async for now. > # In APP.call, it would call Fiber.yield whenever we're waiting for data. > status, headers, body = APP.call(@env.merge!(RACK_DEFAULTS)) > > # The response is fully ready at this point. > ev_write_response(status, headers, body, @hp.next?) > }.resume > > # * * * * * > > # Here's the app > APP = lambda{ |env| > # First we remember where we need to jump back > f = Fiber.current > # Wait for 5 seconds to emulate waiting for data > EM.add_timer(5){ > # Here then we have data available, let's resume back. > # But we also wrap resuming in a next_tick block, > # giving EM some chances to cleanup it's internal states. > EM.next_tick{ > f.resume('OK') > } > } > # Here we immediately yield and give control back to Rainbows! > # and Rainbows! would then go back to EventMachine's regular loop, > # buffering requests, making requests, etc. > body = Fiber.yield > > # So now body is referring the data resumed from above. > # Rainbows! should handle the response at this point. > [200, {}, [body]] > } > > Not sure if this is clear enough, please let me know if I didn't > make it clear, or there's anything wrong in my assumption or > the over simplified codes. Thanks! Thank you, your code makes it clear. I think your approach will work with most HTTP clients. However, I think pipelined requests will hit the same problems as EM.defer, too. Can you try with pipelining? Maybe disabling keepalive/persistent connections will make this work correctly (but you obviously lose latency benefits, too). I also don't think it's possible to say "no pipelining" to a client if we support persistent connections at all. > > Interesting. Are you on 32 or 64-bit and are you constrained by VMSize > > or RSS? > > I think it's an x86_64 VM. Not sure what's VMSize, but we're observing > memory usage in RSS. They claim we have about 500M for one process, > but sometimes the app cannot allocate more memory even below 500M. > (Well, I don't quite understand how the app would grow to 500M? It's > about 160M on average) It's likely some corner case in your code. Do you generate potentially large responses or read in large amounts of data? (e.g. SELECT statements without a LIMIT, large files (uploads?)). A slow client which triggers large server responses (which EM may buffer even if the Rack app streams it out) can hit this, too. I don't think EM can be configured to buffer writes to the file system (nginx will automatically do this, though). > > If it's VMSize and you're not dealing with deep data structures _and_ > > your code doesn't use recursion much; lowering pthreads stack size to > > 64K in thread_pthread.c should help with memory usage. Not sure if you > > can supply your own Ruby builds for that hosting service, though. > > There's a chance to do that on Heroku, but I guess it's too much effort > to do so. I don't feel there's a standard way to compile stuffs on it. > > How do I tell the current stack size for a thread in pthread? It seems I > cannot run ulimit on Heroku. On the other hand, I am not sure if it's > significant enough to reduce thread stack size. Ruby 1.9 sets stack sizes to 512K regardless of ulimit -s. At least on Linux, memory defaults to being overcommited and is lazily allocated in increments of PAGE_SIZE (4K on x86*). It's likely the actual RSS overhead of a native thread stack is <64K. VMSize overhead becomes important on 32-bit with many native threads, though. In comparison, Fibers use only 4K stack and has no extra overhead in the kernel. > I guess we can also reduce the concurrent level (worker_connections?) > to reduce memory usage, too? > > > If you have time, you could also try making something like GNU pth or > > npth work with Ruby 1.9. I suspect Fibers will become unusable > > with *pth, though... > > Cool, might be interesting. > Could then threads be as lightweight as fibers? :P Yes. Fwiw, MRI 1.8 green threads and Neverblock are basically equivalent, too. > Though I really doubt if threads are really that heavy comparing to fibers. > At least in some simple tests, threads are fine and efficient enough. I agree native threads are light enough for most cases (especially since you're already running Ruby :). > EventMachine is still a lot faster than regular sockets (net/http) though, > so I'll still keep EventMachine for a while even if I switched to threads. I think part of that is the HTTP parser and I/O buffering being implemented in C/C++ vs Ruby. Things like the net-http-persistent gem should help with pure-Ruby performance, though (and performance is likely to be better with upcoming Ruby releases). > > I'm actually surprised unicorn got the attention it has. > > > > As project leader, my preference to maintain a low public profile and my > > refusal to endorse/associate with for-profit interests certainly hurts > > adoption. > > > > If somebody else (like you) wants to market these projects, then more > > power to them :) I'm certainly never going to speak at any conference. > > > > Rainbows! just overwhelms potential users with too many choices, > > so it's unlikely to ever be widely-adopted. > > Umm... I don't know the reason why you want to stay away from them, > but please let me know if you don't want someone to market them. > I'll then try to be as natural as possible when talking about Unicorns. I enjoy my near-anonymity and want as little reputation/recognition as possible. I'm always happy if people talk about software, but I prefer software stand on its own and not on the reputation of its authors. (The only reason I use my name is for potential GPL enforcement) _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <20120831013731.GA16613-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <20120831013731.GA16613-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> @ 2012-09-05 20:06 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1vfWXGw_CaaMWMijUSdMN2Pz882SYDtNEW2_6YWffgTKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Lin Jen-Shin (godfat) @ 2012-09-05 20:06 UTC (permalink / raw) To: Rainbows! list On Fri, Aug 31, 2012 at 9:37 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > I seem to recall problems with some of the more esoteric test cases in > Rainbows! a few years ago. > > Now that I think more about it, it might've been related to client > pipelining. If a client pipelines requests, I don't think using > EM.defer {} makes it easy to guarantee the servers responses are > returned in the correct order. > > This is made worse since (AFAIK) EM provides no easy way to > temporarily disable firing read callbacks for a socket, so > a client which pipelines aggressively becomes bad news. After some experiments, now I understood why it is hard. But I can't figure it out by some quick glimpses for how you did solve this problems for other concurrency model? One possible and simple way would be... just make piped requests sequential, but this would greatly reduce the concurrency ability, is it right? At least Puma server runs quite poorly whenever I am testing pipeline requests. My test script is: httperf --hog --server localhost --port 8080 --uri /cpu --num-calls 4 --burst-length 2 --num-conn 2 --rate 8 --print-reply But Zbatery runs quite smoothly with ThreadPool and ThreadSpawn. I assume it's because Zbatery would handle piped requests concurrently and collect responses and reply them with the correct order, though I cannot tell from the code, at least from some quick glimpses. At this point I am more confident to say that Unicorn family is the best Ruby application servers. :) > Thank you, your code makes it clear. I think your approach will work > with most HTTP clients. > > However, I think pipelined requests will hit the same problems as > EM.defer, too. Can you try with pipelining? Honestly I don't know about keep-alive and pipelined requests, and I just learned it from trying httperf, which seems to be a very good tool to make trials against web servers. Puma and Thin worked poorly in my tests with above httperf command, while Zbatery worked perfectly fine. (except with my hack for adding fibers/threads on top of EventMachine, which were raising errors) After pondering and reading the codes in Rainbows for a while, I got managed to make it work without errors, but I believe it is still suffering from the ordering issues. There's no promise for ordering. Here's the new code. It's for fibers but I think it's the same with EM.defer. https://github.com/godfat/ruby-server-exp/commit/41644fc63ffa0a505eab616c1d23ead4a564d349 def app_call input # [...] as before Fiber.new{ status, headers, body = catch(:async) { APP.call(@env.merge!(RACK_DEFAULTS)) } if nil == status || -1 == status @deferred = true else @deferred = nil # response is ready, no more @deferred ev_write_response(status, headers, body, @hp.next?) end }.resume @deferred = true # we're always deferring end To address ordering issue, I guess we can remember the index of a certain request, and if there's a request being processed which has a lower index, the response shouldn't be written back before the lower one has been written. Not sure if this is wroth the effort though... This must touch Rainbows' internal, and it cannot be easily handled by simply extending the client class. > Maybe disabling keepalive/persistent connections will make this work > correctly (but you obviously lose latency benefits, too). > > I also don't think it's possible to say "no pipelining" to a client if > we support persistent connections at all. I wonder if we always run Nginx or something similar in front of Rainbows, does it still matter? Nevertheless, I guess it's good enough for us right now. Many thanks for your review. On the other hand, I would still be very interested to see if this could be addressed. Last time I want to replicate what other concurrency models solved this, but failed to see how. > It's likely some corner case in your code. Do you generate potentially > large responses or read in large amounts of data? (e.g. SELECT > statements without a LIMIT, large files (uploads?)). > > A slow client which triggers large server responses (which EM may > buffer even if the Rack app streams it out) can hit this, too. > I don't think EM can be configured to buffer writes to the file > system (nginx will automatically do this, though). I see. Never thought of that EM might be buffering a lot of large responses in the memory. As for loading large amounts of data into memory, I guess I can't tell. As far as I know, no, but who knows :P This must be accidental if there's one... Anyway, we don't see that often nowadays. Or it could be that Ruby 1.9.3 had fixed some memory leak issues. Or some other 3rd party libraries we're using. > Ruby 1.9 sets stack sizes to 512K regardless of ulimit -s. At least on > Linux, memory defaults to being overcommited and is lazily allocated in > increments of PAGE_SIZE (4K on x86*). It's likely the actual RSS overhead > of a native thread stack is <64K. > > VMSize overhead becomes important on 32-bit with many native threads, > though. In comparison, Fibers use only 4K stack and has no extra > overhead in the kernel. I see, thanks for the explanation. I guess that does matter a bit, but only if we're using thousands of threads/fibers, and it should be quite rarely in a web app, I guess. Using fibers are also risking from system stack overflow, especially in a Rails app with a lot of plugins, I guess... Umm, but I also heard that fibers stack is increased a bit in newer Ruby? >> Though I really doubt if threads are really that heavy comparing to fibers. >> At least in some simple tests, threads are fine and efficient enough. > > I agree native threads are light enough for most cases (especially since > you're already running Ruby :). Speaking to this and green threads, I wonder if it's worth the effort to implement m:n threading for Ruby? Or we can just compile and link against a threading library which supports m:n threading? Goroutine? :P >> EventMachine is still a lot faster than regular sockets (net/http) though, >> so I'll still keep EventMachine for a while even if I switched to threads. > > I think part of that is the HTTP parser and I/O buffering being > implemented in C/C++ vs Ruby. Things like the net-http-persistent gem > should help with pure-Ruby performance, though (and performance is > likely to be better with upcoming Ruby releases). I haven't got a chance to try net-http-persistent, but it seems I should try it. (or try that on em-http-request, it seems it supports it https://github.com/igrigorik/em-http-request/wiki/Keep-Alive-and-HTTP-Pipelining ) Or if it's all about HTTP parsing, [http][] gem should help too. It is using [http_parser.rb] underneath, which is based on NodeJS' [http-parser][]. Sometimes I feel it's all about throwing away EventMachine... I've heard that EM is bad, but not bad enough to be rewritten... [http]: https://github.com/tarcieri/http [http_parser.rb]: https://github.com/tmm1/http_parser.rb [http-parser]: https://github.com/joyent/http-parser > I enjoy my near-anonymity and want as little reputation/recognition as > possible. I'm always happy if people talk about software, but I prefer > software stand on its own and not on the reputation of its authors. > > (The only reason I use my name is for potential GPL enforcement) I see. Thanks for explaining. I'll then avoid talking about authors :) _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <CAA2_N1vfWXGw_CaaMWMijUSdMN2Pz882SYDtNEW2_6YWffgTKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <CAA2_N1vfWXGw_CaaMWMijUSdMN2Pz882SYDtNEW2_6YWffgTKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2012-09-05 23:27 ` Eric Wong [not found] ` <20120905232739.GA25153-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Eric Wong @ 2012-09-05 23:27 UTC (permalink / raw) To: Rainbows! list "Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote: > On Fri, Aug 31, 2012 at 9:37 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > > I seem to recall problems with some of the more esoteric test cases in > > Rainbows! a few years ago. > > > > Now that I think more about it, it might've been related to client > > pipelining. If a client pipelines requests, I don't think using > > EM.defer {} makes it easy to guarantee the servers responses are > > returned in the correct order. > > > > This is made worse since (AFAIK) EM provides no easy way to > > temporarily disable firing read callbacks for a socket, so > > a client which pipelines aggressively becomes bad news. > > After some experiments, now I understood why it is hard. But I can't > figure it out by some quick glimpses for how you did solve this problems > for other concurrency model? Simple: we don't read from the socket at all while processing a request Extra data the client sends gets buffered in the kernel, eventually TCP backoff will kick in and the Internet stays usable :> With the inability to easily stop read callbacks via EM, the socket buffers constantly get drained so the clients are able to keep sending data. The only option I've found for Rainbows! + EM was to issue shutdown(SHUT_RD) if a client attempts to pipeline too much (which never happens with legitimate clients AFAIK). > One possible and simple way would be... just make piped requests > sequential, but this would greatly reduce the concurrency ability, > is it right? At least Puma server runs quite poorly whenever I am > testing pipeline requests. Yes, pipelining is handled sequentially. It's the easiest and safest way to implement since HTTP/1.1 requires the responses to be returned in the same order they were sent (as you mention below). > My test script is: > > httperf --hog --server localhost --port 8080 --uri /cpu --num-calls 4 > --burst-length 2 --num-conn 2 --rate 8 --print-reply > > But Zbatery runs quite smoothly with ThreadPool and ThreadSpawn. > I assume it's because Zbatery would handle piped requests concurrently > and collect responses and reply them with the correct order, though > I cannot tell from the code, at least from some quick glimpses. Rainbows!/Zbatery handles pipelined requests sequentially. They only have concurrency on a per-socket level. > At this point I am more confident to say that Unicorn family is the best > Ruby application servers. :) Good to hear :) <snip> > To address ordering issue, I guess we can remember the > index of a certain request, and if there's a request being > processed which has a lower index, the response shouldn't > be written back before the lower one has been written. > > Not sure if this is wroth the effort though... This must touch > Rainbows' internal, and it cannot be easily handled by > simply extending the client class. I don't think it is worth the effort. I'm not even sure how often pipelining is used in the real world, all I know is Rainbows! can handle it without falling over. > > Maybe disabling keepalive/persistent connections will make this work > > correctly (but you obviously lose latency benefits, too). > > > > I also don't think it's possible to say "no pipelining" to a client if > > we support persistent connections at all. > > I wonder if we always run Nginx or something similar in front of > Rainbows, does it still matter? It shouldn't matter for nginx, I don't think nginx will (ever) pipeline to a backend. Nowadays nginx can do persistent connections to backends, though I'm not sure how much of a benefit it is for local sockets. > I see. Never thought of that EM might be buffering a lot of large > responses in the memory. As for loading large amounts of data > into memory, I guess I can't tell. As far as I know, no, but who knows :P > This must be accidental if there's one... I think your comment is unfortunately representative of a lot of software development nowadays. Embrace pessimism and let it be your guide :) > > Ruby 1.9 sets stack sizes to 512K regardless of ulimit -s. At least on > > Linux, memory defaults to being overcommited and is lazily allocated in > > increments of PAGE_SIZE (4K on x86*). It's likely the actual RSS overhead > > of a native thread stack is <64K. > > > > VMSize overhead becomes important on 32-bit with many native threads, > > though. In comparison, Fibers use only 4K stack and has no extra > > overhead in the kernel. > > I see, thanks for the explanation. I guess that does matter a bit, but only > if we're using thousands of threads/fibers, and it should be quite rarely > in a web app, I guess. > > Using fibers are also risking from system stack overflow, especially in > a Rails app with a lot of plugins, I guess... Umm, but I also heard that > fibers stack is increased a bit in newer Ruby? You're right, fiber stacks got bigger. They're 64K on all 1.9.3 and 128K on 64-bit for 2.0.0dev. So there's even less benefit in using Fibers nowadays for memory concerns. > >> Though I really doubt if threads are really that heavy comparing to fibers. > >> At least in some simple tests, threads are fine and efficient enough. > > > > I agree native threads are light enough for most cases (especially since > > you're already running Ruby :). > > Speaking to this and green threads, I wonder if it's worth the effort to > implement m:n threading for Ruby? Or we can just compile and > link against a threading library which supports m:n threading? > Goroutine? :P *shrug* Not worth _my_ effort for m:n threads. _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <20120905232739.GA25153-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <20120905232739.GA25153-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> @ 2012-09-22 9:52 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1v460utbL31Qu-JbGuUxav1hY4X5+cEf=Mp2rOC5efzMw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Lin Jen-Shin (godfat) @ 2012-09-22 9:52 UTC (permalink / raw) To: Rainbows! list On Thu, Sep 6, 2012 at 7:27 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > Simple: we don't read from the socket at all while processing a request > > Extra data the client sends gets buffered in the kernel, eventually TCP > backoff will kick in and the Internet stays usable :> Understood, thanks! I guess this is simple enough that it's a nice trade off, since not every clients would do heavy pipelining. > With the inability to easily stop read callbacks via EM, the socket > buffers constantly get drained so the clients are able to keep sending > data. The only option I've found for Rainbows! + EM was to issue > shutdown(SHUT_RD) if a client attempts to pipeline too much (which > never happens with legitimate clients AFAIK). And the buffer size you set: 0x1c000, I guess that's large enough for most cases. > It shouldn't matter for nginx, I don't think nginx will (ever) pipeline > to a backend. Nowadays nginx can do persistent connections to backends, > though I'm not sure how much of a benefit it is for local sockets. Here's a problem troubles me for a long time, and I failed to figure out what's going on and cannot find a proper fix. I didn't mention this before, because I don't think it's Zbatery/Rainbows' issue, but it turns out that it might be related, and perhaps you would be interested to provide some hints, I would much appreciate it. So we're running all our apps on http://www.heroku.com It's a hosting service that you simply `git push` to deploy your app, worrying not all the complexity regarding deployment. There's a very weird issue running Zbatery/Rainbows on it. I don't remember if Unicorn would be suffering from that since I don't run Unicorn directly for a while now. The problem comes with Rails 3 and its "assets pipeline" feature. It's basically a feature that let your assets (e.g. images, javascripts, css, etc) no longer simple assets which could be served directly via nginx or some other static web server. Like, developers could name their coffeescripts files as: application.coffee.js And when a client is requesting "application.js", that "assets pipeline" thing would compile the coffeescript file into a javascript file on the fly, and return it to the client. We can also "precompile" those assets before deploying to production. Heroku would do this if it detects that the application is deploying is built with Rails 3. If this precompile thing is done successfully, then there's no problems, at least in some of the cases.... But what if precompile failed? In my cases, it would usually cause system stack overflow whenever it tries to compile it on the fly. Oh, suddenly I think maybe it's because I would wrap a fiber around each request in some cases... and it just used too many method calls. I am not sure about that... might check it next time I saw it. Somehow I feel this might be related to persistent connections or pipelined requests, because last time I didn't write the client class correctly to handle pipelined requests as you pointed out, it would also cause some issues regarding "assets pipeline" on Heroku. I believe Heroku is running nginx in front off Zbatery, and if I didn't get pipelined requests correctly, it would result timeout for many of assets the client is asking about. I feel Heroku is using pipelined requests if the client is using this. And the most weird thing I can't understand is, I am trying to convince one of my friend to try out Zbatery, but using ThreadSpawn model would usually cause serving assets timeout, just like in my case where I didn't get pipelined requests correct. Switching to EventMachine solved the problem! Huh? Moreover, once there are some assets timeout issues on EventMachine, too. When I tried to debug this, I put some traces into Rainbows, realizing that sometimes EventMachine didn't call `receive_data' when receiving some pipelined requests. Could it be an eventmachine bug!? Sorry that I wrote so much and it's unclear what's going on, since I don't understand so I can't remember correctly... If you have any idea for a specific case, I can try to provide all the information I could collect. I can't get the nginx configuration Heroku is using though :( > I think your comment is unfortunately representative of a lot of > software development nowadays. > > Embrace pessimism and let it be your guide :) I hope I could understand/realize this sooner :) Cheers, _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <CAA2_N1v460utbL31Qu-JbGuUxav1hY4X5+cEf=Mp2rOC5efzMw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <CAA2_N1v460utbL31Qu-JbGuUxav1hY4X5+cEf=Mp2rOC5efzMw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2012-09-22 19:42 ` Eric Wong [not found] ` <20120922194222.GA6839-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Eric Wong @ 2012-09-22 19:42 UTC (permalink / raw) To: Rainbows! list "Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote: > Here's a problem troubles me for a long time, and I failed to figure out > what's going on and cannot find a proper fix. I didn't mention this before, > because I don't think it's Zbatery/Rainbows' issue, but it turns out that > it might be related, and perhaps you would be interested to provide > some hints, I would much appreciate it. > > So we're running all our apps on http://www.heroku.com > It's a hosting service that you simply `git push` to deploy your app, > worrying not all the complexity regarding deployment. > > There's a very weird issue running Zbatery/Rainbows on it. > I don't remember if Unicorn would be suffering from that > since I don't run Unicorn directly for a while now. > > The problem comes with Rails 3 and its "assets pipeline" feature. > It's basically a feature that let your assets (e.g. images, javascripts, > css, etc) no longer simple assets which could be served directly > via nginx or some other static web server. > > Like, developers could name their coffeescripts files as: > > application.coffee.js > > And when a client is requesting "application.js", that "assets pipeline" > thing would compile the coffeescript file into a javascript file on the fly, > and return it to the client. > > We can also "precompile" those assets before deploying to production. > Heroku would do this if it detects that the application is deploying is > built with Rails 3. > > If this precompile thing is done successfully, then there's no problems, > at least in some of the cases.... But what if precompile failed? In my cases, > it would usually cause system stack overflow whenever it tries to compile > it on the fly. > > Oh, suddenly I think maybe it's because I would wrap a fiber around > each request in some cases... and it just used too many method calls. > I am not sure about that... might check it next time I saw it. > > Somehow I feel this might be related to persistent connections or > pipelined requests, because last time I didn't write the client class > correctly to handle pipelined requests as you pointed out, it would > also cause some issues regarding "assets pipeline" on Heroku. > > I believe Heroku is running nginx in front off Zbatery, and if I didn't > get pipelined requests correctly, it would result timeout for many of > assets the client is asking about. I feel Heroku is using pipelined > requests if the client is using this. > > And the most weird thing I can't understand is, I am trying to convince > one of my friend to try out Zbatery, but using ThreadSpawn model > would usually cause serving assets timeout, just like in my case > where I didn't get pipelined requests correct. Switching to EventMachine > solved the problem! Huh? > > Moreover, once there are some assets timeout issues on EventMachine, > too. When I tried to debug this, I put some traces into Rainbows, > realizing that sometimes EventMachine didn't call `receive_data' > when receiving some pipelined requests. Could it be an eventmachine bug!? It could be the front-end proxy (incorrectly) detected the Rainbows! instance was down and stopped sending traffic to it. Does this information get logged? > Sorry that I wrote so much and it's unclear what's going on, since > I don't understand so I can't remember correctly... Yeah, it's a bit much to understand. Can you reproduce it consistently? With the serving timeout for Zbatery+ThreadSpawn, can you ensure Content-Length/Transfer-Encoding:chunked is set in the response headers? Since you mentioned stack overflows in response generation, perhaps whatever proxy Heroku is using doesn't handle crashed servers during the response correctly... Can you get stderr logs from Heroku? > If you have any idea for a specific case, I can try to provide all the > information I could collect. I can't get the nginx configuration Heroku > is using though :( I highly doubt nginx will pipeline requests, but we're not sure if they're really using nginx, yet. With the problems you've described, it doesn't sound like they are, or they're using some broken version of it. _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <20120922194222.GA6839-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <20120922194222.GA6839-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> @ 2012-09-28 15:14 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1usHJVZgn5n7RaTyDCbK7eu6G4ocZAsvqsVeL6cPERskw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Lin Jen-Shin (godfat) @ 2012-09-28 15:14 UTC (permalink / raw) To: Rainbows! list On Sun, Sep 23, 2012 at 3:42 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: [...] >> Moreover, once there are some assets timeout issues on EventMachine, >> too. When I tried to debug this, I put some traces into Rainbows, >> realizing that sometimes EventMachine didn't call `receive_data' >> when receiving some pipelined requests. Could it be an eventmachine bug!? > > It could be the front-end proxy (incorrectly) detected the Rainbows! > instance was down and stopped sending traffic to it. Does this > information get logged? It would eventually show out as a timeout for 30 seconds. It's a log from front-end proxy (router in their terminology). I am not sure if this means it is sending traffic or not. If it's not, then I guess it explains... > Yeah, it's a bit much to understand. Can you reproduce it consistently? Yes, it could be reproduced consistently, but only in a certain environment (e.g. on Heroku) which might not always be found. > With the serving timeout for Zbatery+ThreadSpawn, can you ensure > Content-Length/Transfer-Encoding:chunked is set in the response headers? I just realized that this is not a timeout issue here. (doh, too many strange issues) I just tried to curl it, and it would immediately return an error. heroku[router]: Error H13 (Connection closed without response) -> GET /assets/application-4e83bff8c0e77de81926c169e1fcacf2.css dyno=web.1 queue= wait= service= status=503 bytes= There is Content-Length: 98794 and no Transfer-Encoding. I don't see if there's any difference between using EventMachine :( By the way, weirdly that it seems there's no problems at all if we're using Thin server. I guess they are testing against Thin, so Thin works correctly... still can't I tell what's the difference. > Since you mentioned stack overflows in response generation, perhaps > whatever proxy Heroku is using doesn't handle crashed servers during > the response correctly... Probably. And I think I am 80% sure where it causes stack overflows now. If I took out fibers, then it would be ok. So I guess that assets things are using too much stack. Too bad I can't switch to threads though :( If I switched to threads, ActiveRecord would be complaining it could not get a connection from the connection pool within 5 seconds. It's not convenient to increase the size of connection pool on Heroku, either, and making it too large would also cause other issues. (sigh) > Can you get stderr logs from Heroku? Yes, Heroku would redirect both stdout and stderr to a place where we could see. It's collapsed into one huge log though. > I highly doubt nginx will pipeline requests, but we're not sure if > they're really using nginx, yet. With the problems you've described, > it doesn't sound like they are, or they're using some broken version > of it. Umm... after reading the log, I think they are using another [router] (in their terminology) in front of [nginx]. So it might be an issue in their [router], I am not sure... I'll keep you posted if you're interested. Thanks for all your help. p.s. They might be using https://github.com/mochi/mochiweb too, since if there's an issue, it would return this header: Server: MochiWeb/1.0 (Any of you quaids got a smint?) p.s.2. Perhaps I could even give you access to one of the apps. I'll need to ask the owner though. Let me know if you're interested, otherwise you could simply ignore this. I know it's too much to ask. _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <CAA2_N1usHJVZgn5n7RaTyDCbK7eu6G4ocZAsvqsVeL6cPERskw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <CAA2_N1usHJVZgn5n7RaTyDCbK7eu6G4ocZAsvqsVeL6cPERskw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2012-09-28 19:11 ` Eric Wong [not found] ` <20120928191132.GA14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Eric Wong @ 2012-09-28 19:11 UTC (permalink / raw) To: Rainbows! list "Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote: > On Sun, Sep 23, 2012 at 3:42 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > [...] > >> Moreover, once there are some assets timeout issues on EventMachine, > >> too. When I tried to debug this, I put some traces into Rainbows, > >> realizing that sometimes EventMachine didn't call `receive_data' > >> when receiving some pipelined requests. Could it be an eventmachine bug!? > > > > It could be the front-end proxy (incorrectly) detected the Rainbows! > > instance was down and stopped sending traffic to it. Does this > > information get logged? > > It would eventually show out as a timeout for 30 seconds. It's a log > from front-end proxy (router in their terminology). I am not sure > if this means it is sending traffic or not. If it's not, then I guess it > explains... > > > Yeah, it's a bit much to understand. Can you reproduce it consistently? > > Yes, it could be reproduced consistently, but only in a certain environment > (e.g. on Heroku) which might not always be found. > > > With the serving timeout for Zbatery+ThreadSpawn, can you ensure > > Content-Length/Transfer-Encoding:chunked is set in the response headers? > > I just realized that this is not a timeout issue here. (doh, too many strange > issues) I just tried to curl it, and it would immediately return an error. > > heroku[router]: Error H13 (Connection closed without response) -> GET > /assets/application-4e83bff8c0e77de81926c169e1fcacf2.css dyno=web.1 > queue= wait= service= status=503 bytes= > > There is Content-Length: 98794 and no Transfer-Encoding. So the _actual_ Content-Length that's sent is zero? Rainbows! should drop a connection if an exception is raised while sending the response body, perhaps the heroku router is confused by that? > I don't see if there's any difference between using EventMachine :( > > By the way, weirdly that it seems there's no problems at all if we're > using Thin server. I guess they are testing against Thin, so Thin works > correctly... still can't I tell what's the difference. Are you allowed to "strace -f" Thin and Rainbows! in an isolated (single thread/process instance)? I'd love to see the difference between how the servers handle this case differently... Can you try setting "keepalive_timeout 0" to disable persistent connections in Rainbows! and see if this helps? I don't think persistent connections make a big difference for LAN, especially not with typical HTTP response sizes (your 98K is pretty big :) > > Since you mentioned stack overflows in response generation, perhaps > > whatever proxy Heroku is using doesn't handle crashed servers during > > the response correctly... > > Probably. And I think I am 80% sure where it causes stack overflows now. > If I took out fibers, then it would be ok. So I guess that assets things are > using too much stack. > > Too bad I can't switch to threads though :( If I switched to threads, > ActiveRecord would be complaining it could not get a connection from > the connection pool within 5 seconds. It's not convenient to increase the > size of connection pool on Heroku, either, and making it too large would > also cause other issues. > > (sigh) > > > Can you get stderr logs from Heroku? > > Yes, Heroku would redirect both stdout and stderr to a place where we > could see. It's collapsed into one huge log though. > > > I highly doubt nginx will pipeline requests, but we're not sure if > > they're really using nginx, yet. With the problems you've described, > > it doesn't sound like they are, or they're using some broken version > > of it. > > Umm... after reading the log, I think they are using another [router] > (in their terminology) in front of [nginx]. So it might be an issue in > their [router], I am not sure... > > I'll keep you posted if you're interested. Thanks for all your help. Yes, please. > p.s. They might be using https://github.com/mochi/mochiweb > too, since if there's an issue, it would return this header: > Server: MochiWeb/1.0 (Any of you quaids got a smint?) > > p.s.2. Perhaps I could even give you access to one of the apps. > I'll need to ask the owner though. Let me know if you're interested, > otherwise you could simply ignore this. I know it's too much to ask. I think I can just walk you through the troubleshooting process here. (I'd rather avoid the potential for personal liability if given access to a server :) _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <20120928191132.GA14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <20120928191132.GA14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> @ 2012-09-28 19:24 ` Eric Wong [not found] ` <20120928192449.GB14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 0 siblings, 1 reply; 18+ messages in thread From: Eric Wong @ 2012-09-28 19:24 UTC (permalink / raw) To: Rainbows! list Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > So the _actual_ Content-Length that's sent is zero? > > Rainbows! should drop a connection if an exception is raised while > sending the response body, perhaps the heroku router is confused > by that? Instead of trying "keepalive_timeout 0", you can also try using the following middleware. This only works for Content-Length responses (which seems to be your case) Of course, like all middleware, this makes your stack deeper, so maybe it's not good, either... Furthermore, this code is totally untested, it may not even compile, but I hope you get the idea and fix trivial errors :) class PadOnError def initialize(app) @app = app end def call(env) status, headers, body = @app.call(env) headers = Rack::Utils::HeaderHash.new(headers) clen = headers["Content-Length"] if clen [ status, headers, Padder.new(clen.to_i, body) ] else # TODO: write your own Transfer-Encoding: chunked padder :) # It would need to parse the chunked encoding and keep track # of each chunk length, pad any unfinished chunk, and ensure # termination with "0\r\n\r\n" [ status, headers, body ] end end class Padder def initialize(clen, body) @clen = clen @body = body @bytes_sent = 0 end def each @body.each do |chunk| @bytes_sent += chunk.bytesize # if this raises due to a socket error... we'll just end up # raising (again) below yield chunk end ensure # pad the response in case @body.each raised an error while # generating. left_to_send = clen - @bytes_sent # XXX assuming you're sending HTTP responses that fit comfortably in # memory, this can OOM if you're sending multi-mega/gigabyte files # like most of my applications :P if left_to_send > 0 padding = " " * left_to_send yield padding end end def close @body.close if @body.respond_to?(:close) end end end ----------------------- config.ru ------------------------ use PadOnError run YourApp.new --------------------------------------------------------- _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <20120928192449.GB14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <20120928192449.GB14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> @ 2012-10-31 0:14 ` Lin Jen-Shin (godfat) 2012-12-18 11:09 ` Lin Jen-Shin (godfat) 1 sibling, 0 replies; 18+ messages in thread From: Lin Jen-Shin (godfat) @ 2012-10-31 0:14 UTC (permalink / raw) To: Rainbows! list On Sat, Sep 29, 2012 at 3:24 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: Ouch, it's already been one month? Sorry that I guess I don't have time to investigate this further recently. The conference is coming soon, I need to prepare the talk and I also need to work on other things. I promise I would come back to give some conclusions after things are settled. (e.g. after the talk is done or if it is well prepared) One thing might worth some words though. I just realized that in Heroku's new architecture, the cedar stack, actually all requests won't be passed through Nginx! But there are exceptions. If we don't CNAME our domain to Heroku, but use A record to point to their IPs, then requests *would* be passed through both Nginx and Varnish. I guess that explains why sometimes it made me feel this way and that way, because I thought they were the same, and I don't remember which is which. The end result is that I was very confused. I'll review my posts and clues with this new clue next time when I do the investigation again... _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <20120928192449.GB14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 2012-10-31 0:14 ` Lin Jen-Shin (godfat) @ 2012-12-18 11:09 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1tcA-HK20C8Ok1Lv9KWwMD4fctCOPHTLeD9ayRJqWby1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 1 sibling, 1 reply; 18+ messages in thread From: Lin Jen-Shin (godfat) @ 2012-12-18 11:09 UTC (permalink / raw) To: Rainbows! list On Sat, Sep 29, 2012 at 3:24 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: > Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote: >> So the _actual_ Content-Length that's sent is zero? >> >> Rainbows! should drop a connection if an exception is raised while >> sending the response body, perhaps the heroku router is confused >> by that? > > Instead of trying "keepalive_timeout 0", you can also try using the > following middleware. This only works for Content-Length responses > (which seems to be your case) > > Of course, like all middleware, this makes your stack deeper, so > maybe it's not good, either... > > Furthermore, this code is totally untested, it may not even compile, > but I hope you get the idea and fix trivial errors :) [...] Thank you again for the helps. Since the conference (http://rubyconf.tw/2012) had ended, I got much more time to work with this now. I didn't try this middleware because now I feel it might be the problem in their application, given that it used a ton of gems which might have some thread-safety issues. On the other hand, the person who asked me about this telling me that it's fine to run Rainbows! with EventMachine, but not with ThreadPool. So it is very likely that it's their issues. As for the issue on my own last time, since it only happened on an application which we no longer run, and Rainbows! (Zbatery) runs totally fine on our true production side, I would guess it's a Heroku issue at that time and it might have been fixed already? As a result, I want to ignore this issue from now on, and move forward. I'll send a patch which contains the concurrency models we use on the production site later in a new thread. Also I might start trying to implement a new concurrency model based on celluloid-io which I believe the API would be similar to cool.io, thus might not be hard to implement. Hope we could get there soon :) _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <CAA2_N1tcA-HK20C8Ok1Lv9KWwMD4fctCOPHTLeD9ayRJqWby1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>]
* Re: negative timeout in Rainbows::Fiber::Base [not found] ` <CAA2_N1tcA-HK20C8Ok1Lv9KWwMD4fctCOPHTLeD9ayRJqWby1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> @ 2012-12-18 19:19 ` Eric Wong 0 siblings, 0 replies; 18+ messages in thread From: Eric Wong @ 2012-12-18 19:19 UTC (permalink / raw) To: Rainbows! list "Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote: > Thank you again for the helps. Since the conference (http://rubyconf.tw/2012) > had ended, I got much more time to work with this now. No problem! > I didn't try this middleware because now I feel it might be the problem > in their application, given that it used a ton of gems which might have > some thread-safety issues. On the other hand, the person who asked > me about this telling me that it's fine to run Rainbows! with EventMachine, > but not with ThreadPool. So it is very likely that it's their issues. They probably a thread-safety issue. I've used ThreadPool/Spawn heavily myself (much more than EM) with great success. (I always verify any gems I use are thread-safe) > As for the issue on my own last time, since it only happened on an > application which we no longer run, and Rainbows! (Zbatery) runs > totally fine on our true production side, I would guess it's a Heroku > issue at that time and it might have been fixed already? *shrug* Maybe check with Heroku changelogs/news releases if they publish those. > As a result, I want to ignore this issue from now on, and move forward. > > I'll send a patch which contains the concurrency models we use on the > production site later in a new thread. Also I might start trying to implement > a new concurrency model based on celluloid-io which I believe the API > would be similar to cool.io, thus might not be hard to implement. > Hope we could get there soon :) Alright, looking forward to it. I was hoping to look at celluloid-io myself last week, but got sidetracked by something else _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2012-12-18 19:20 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-08-23 20:36 negative timeout in Rainbows::Fiber::Base Lin Jen-Shin (godfat) [not found] ` <CAA2_N1unOXb7Z4Jr8oKoSLu266O9Ko4o=oWzAcMA1w3=9X74KA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2012-08-25 2:45 ` Eric Wong [not found] ` <20120825024556.GA25977-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 2012-08-26 0:12 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1uhfcHDbTvY+ke0Cid6=i7KEhFn8jvEirx+ptYVDacdvA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2012-08-26 1:15 ` Eric Wong 2012-08-29 16:00 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1thakAOVp7ibCNic+TjEVvXE0OGLgzXH3fJ1c2UTs68oQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2012-08-29 21:17 ` Eric Wong [not found] ` <20120829211707.GA22726-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 2012-08-30 21:33 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1tc=Xx8WHaM8H=EWshyzGEyX04PnkdBGj9Jdb7cSzmbRQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2012-08-31 1:37 ` Eric Wong [not found] ` <20120831013731.GA16613-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 2012-09-05 20:06 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1vfWXGw_CaaMWMijUSdMN2Pz882SYDtNEW2_6YWffgTKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2012-09-05 23:27 ` Eric Wong [not found] ` <20120905232739.GA25153-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 2012-09-22 9:52 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1v460utbL31Qu-JbGuUxav1hY4X5+cEf=Mp2rOC5efzMw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2012-09-22 19:42 ` Eric Wong [not found] ` <20120922194222.GA6839-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 2012-09-28 15:14 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1usHJVZgn5n7RaTyDCbK7eu6G4ocZAsvqsVeL6cPERskw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2012-09-28 19:11 ` Eric Wong [not found] ` <20120928191132.GA14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 2012-09-28 19:24 ` Eric Wong [not found] ` <20120928192449.GB14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> 2012-10-31 0:14 ` Lin Jen-Shin (godfat) 2012-12-18 11:09 ` Lin Jen-Shin (godfat) [not found] ` <CAA2_N1tcA-HK20C8Ok1Lv9KWwMD4fctCOPHTLeD9ayRJqWby1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2012-12-18 19:19 ` Eric Wong
Code repositories for project(s) associated with this public inbox https://yhbt.net/rainbows.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).