negative timeout in Rainbows::Fiber::Base

negative timeout in Rainbows::Fiber::Base

[Lin Jen-Shin (godfat)]

Greetings Unicorns,

I am writing some stupid benchmarks, trying FiberSpawn and
FiberPool, and seeing this error:

    listen loop error: time interval must be positive (ArgumentError)

Looking into the code, in this line:
http://bogomips.org/rainbows.git/tree/lib/rainbows/fiber/base.rb#n55
Isn't that possible `max <= (now + 1)` and `now > max`?
Whenever this is the case, then timeout passed to `select`
could be negative. I am not sure how to fix this correctly,
but a blind `abs` could resolve this error.

Hope this could give some hints, thank you!

commit b599374ef10c7847445f29f80e74cde3e4d3b940
Author: Lin Jen-Shin <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org>
Date:   Fri Aug 24 04:33:38 2012 +0800

    fiber/base.rb: make timeout never be negative.

    In the case where `max <= (now + 1)` and `now > max`,
    we need to make sure that timeout is positive.

diff --git a/lib/rainbows/fiber/base.rb b/lib/rainbows/fiber/base.rb
index 00af214..7bd43cb 100644
--- a/lib/rainbows/fiber/base.rb
+++ b/lib/rainbows/fiber/base.rb
@@ -52,7 +52,7 @@ module Rainbows::Fiber::Base
     }
     fibs.each { |fib| fib.resume }
     now = Time.now
-    max.nil? || max > (now + 1) ? 1 : max - now
+    max.nil? || max > (now + 1) ? 1 : (max - now).abs
   end

   def process(client)
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Eric Wong]

"Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
> Greetings Unicorns,
> 
> I am writing some stupid benchmarks, trying FiberSpawn and
> FiberPool, and seeing this error:
> 
>     listen loop error: time interval must be positive (ArgumentError)
> 
> Looking into the code, in this line:
> http://bogomips.org/rainbows.git/tree/lib/rainbows/fiber/base.rb#n55
> Isn't that possible `max <= (now + 1)` and `now > max`?
> Whenever this is the case, then timeout passed to `select`
> could be negative. I am not sure how to fix this correctly,
> but a blind `abs` could resolve this error.

The original code looks wrong and confusing :x

It's been a while since I've looked hard at the Fiber* stuff (not a fan
of it personally).  However, I don't think abs() is correct, either.

> Hope this could give some hints, thank you!

Perhaps this is less confusing and more correct:

diff --git a/lib/rainbows/fiber/base.rb b/lib/rainbows/fiber/base.rb
index 6828e1d..103c97a 100644
--- a/lib/rainbows/fiber/base.rb
+++ b/lib/rainbows/fiber/base.rb
@@ -51,12 +51,15 @@ module Rainbows::Fiber::Base
       end
     }
     fibs.each { |fib| fib.resume }
-    now = Time.now
+
+    max_sleep = 1.0
     if max
-      timeout = max - now
-      timeout < 0.0 ? 0 : timeout
+      max -= Time.now
+      return 0 if max < 0.0
+      return max_sleep if max > max_sleep
+      max
     else
-      1
+      max_sleep
     end
   end
 
---
Thoughts?
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Lin Jen-Shin (godfat)]

On Sat, Aug 25, 2012 at 10:45 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> The original code looks wrong and confusing :x

I guess I know your intension in the new code now :)
I can't apply the new patch, so I'll assume the new
code is as following:

    fibs.each { |fib| fib.resume }

    max_sleep = 1.0
    if max
      max -= Time.now
      return 0 if max < 0.0
      return max_sleep if max > max_sleep
      max
    else
      max_sleep
    end

I once thought that if `max < Time.now`, it should return 0,
because apparently we don't want to wait in this case.
But I am not sure if passing timeout as 0 would mean to
wait forever... thinking that maybe 0.0001 might be better?
Not sure if this would be a concern, or 0 means don't wait
at all, that would be ideal in this case.

> It's been a while since I've looked hard at the Fiber* stuff (not a fan
> of it personally).  However, I don't think abs() is correct, either.

I can smell from the code that those are not well polished,
since it seems a lot of things are missing. For example,
there is Rainbows::Fiber::IO::Socket, but no TCPSocket.
I blindly tried to create a TCPSocket version by creating
a new class extended from Kgio::TCPSocket and include
Rainbows::Fiber::IO::Methods.

Well, I think it works, if using correctly, for example it
does not accept domain names as built-in TCPSocket,
and not all of the methods are respecting fibers.

But I like the idea to patch the socket, that way, we can
turn existing library to run under certain network architecture.
For instance, this HTTP client accepts passing different
socket class:
https://github.com/tarcieri/http/blob/master/lib/http/options.rb#L33-37

    @default_socket_class = TCPSocket
    @default_ssl_socket_class = OpenSSL::SSL::SSLSocket

    class << self
      attr_accessor :default_socket_class, :default_ssl_socket_class

I didn't try if passing Rainbows' fiber sockets would work or not,
but I think in theory it should work. I think Tony did this for his:
https://github.com/celluloid/celluloid-io
Quoted from README:

"""
Replacement classes: Celluloid::IO includes replacements for the core
TCPServer and TCPSocket classes which automatically use an evented
mode inside of Celluloid::IO actors. They're named
Celluloid::IO::TCPServer and Celluloid::IO::TCPSocket, so they're
automatically available inside your class when you include
Celluloid::IO.
"""

Not sure if this is the way to go, but an interesting idea. At least
better than implementing every single client in both eventmachine
and built-in sockets I guess...

So speaking to celluloid-io, there's already Revactor and Coolio
support, what about celluloid-io? Last time I tried Coolio,
occasionally it would crash with internal assertion failures...
I would love to see if there's other alternatives to eventmachine.

Also, there is CoolioFiberSpawn, CoolioThreadPool and so on
so forth, why not EventMachineFiberSpawn and EventMachineThreadPool?
Actually we are running EventMachineFiberSpawn on production
at the moment, as shown in:

https://github.com/cardinalblue/rest-more/blob/master/example/rainbows.rb#L15-19
class REMFClient < Rainbows::EventMachine::Client
  def app_call input
    Fiber.new{ super }.resume
  end
end

(well, neverblock works too, but it doesn't seem to be
maintained now and I don't understand why it's a gem
with a lot of codes but not as simple as above. As this
sense, I feel async_sinatra is so wrong, but not sure
if I missed something here.)

It works great for us at the moment, though I am thinking about
switching to threads, too, since it's a lot more flexible than fibers.

Sorry that I might be asking too many questions in a single thread.
If you're interested in how I am testing them, I put my testing codes
at: https://github.com/godfat/ruby-server-exp
I am thinking about if I should be giving a talk on rubyconf.tw
about ruby application servers, promoting unicorn family.
It is one of the reasons to try out all of the options.

Thanks for your time :D
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Eric Wong]

"Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
> On Sat, Aug 25, 2012 at 10:45 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> > The original code looks wrong and confusing :x
> 
> I guess I know your intension in the new code now :)
> I can't apply the new patch, so I'll assume the new
> code is as following:
> 
>     fibs.each { |fib| fib.resume }
> 
>     max_sleep = 1.0
>     if max
>       max -= Time.now
>       return 0 if max < 0.0
>       return max_sleep if max > max_sleep
>       max
>     else
>       max_sleep
>     end

Yes.  (Though it should've be easy to just pipe that part of my
message to "git apply" or even "patch -p1")

> I once thought that if `max < Time.now`, it should return 0,
> because apparently we don't want to wait in this case.
> But I am not sure if passing timeout as 0 would mean to
> wait forever... thinking that maybe 0.0001 might be better?
> Not sure if this would be a concern, or 0 means don't wait
> at all, that would be ideal in this case.

Aa timeout of 0 returns immediately for select (and
poll/epoll/kevent), but still marks ready descriptors.

> > It's been a while since I've looked hard at the Fiber* stuff (not a fan
> > of it personally).  However, I don't think abs() is correct, either.
> 
> I can smell from the code that those are not well polished,
> since it seems a lot of things are missing. For example,
> there is Rainbows::Fiber::IO::Socket, but no TCPSocket.
> I blindly tried to create a TCPSocket version by creating
> a new class extended from Kgio::TCPSocket and include
> Rainbows::Fiber::IO::Methods.

Yeah, it was an experiment and I've become less-interested in
Fibers over time.

> Well, I think it works, if using correctly, for example it
> does not accept domain names as built-in TCPSocket,
> and not all of the methods are respecting fibers.

DNS resolution needs to be handled asynchronously as well,
so I'd rather leave it out than block on the portable DNS
lookup functions.

> But I like the idea to patch the socket, that way, we can
> turn existing library to run under certain network architecture.
> For instance, this HTTP client accepts passing different
> socket class:
> https://github.com/tarcieri/http/blob/master/lib/http/options.rb#L33-37
> 
>     @default_socket_class = TCPSocket
>     @default_ssl_socket_class = OpenSSL::SSL::SSLSocket
> 
>     class << self
>       attr_accessor :default_socket_class, :default_ssl_socket_class
> 
> I didn't try if passing Rainbows' fiber sockets would work or not,
> but I think in theory it should work. I think Tony did this for his:
> https://github.com/celluloid/celluloid-io
> Quoted from README:
> 
> """
> Replacement classes: Celluloid::IO includes replacements for the core
> TCPServer and TCPSocket classes which automatically use an evented
> mode inside of Celluloid::IO actors. They're named
> Celluloid::IO::TCPServer and Celluloid::IO::TCPSocket, so they're
> automatically available inside your class when you include
> Celluloid::IO.
> """
> 
> Not sure if this is the way to go, but an interesting idea. At least
> better than implementing every single client in both eventmachine
> and built-in sockets I guess...
> 
> So speaking to celluloid-io, there's already Revactor and Coolio
> support, what about celluloid-io? Last time I tried Coolio,
> occasionally it would crash with internal assertion failures...
> I would love to see if there's other alternatives to eventmachine.

All the Thread-based concurrency models should already just work
with Celluloid-based apps.

http://news.gmane.org/find-root.php?message_id=%3cBANLkTimnNQYpk7TRkiCvhDikzGHVy%3dkOdw%40mail.gmail.com%3e

> Also, there is CoolioFiberSpawn, CoolioThreadPool and so on
> so forth, why not EventMachineFiberSpawn and EventMachineThreadPool?

It's much easier to unwatch IO objects with Cool.io than with EM, so I
haven't done much with EM + Fibers.  There's also rack-fiber_pool...

Rainbows::EventMachine::TryDefer already uses the EM-internal thread
pool, so I think EventMachineThreadPool would be redundant.

> Actually we are running EventMachineFiberSpawn on production
> at the moment, as shown in:
> 
> https://github.com/cardinalblue/rest-more/blob/master/example/rainbows.rb#L15-19
> class REMFClient < Rainbows::EventMachine::Client
>   def app_call input
>     Fiber.new{ super }.resume
>   end
> end

Does it actually show benefits compared to regular EM?
I suppose the Rack application still needs to be made Fiber-aware
to reap benefits of Fibers

> (well, neverblock works too, but it doesn't seem to be
> maintained now and I don't understand why it's a gem
> with a lot of codes but not as simple as above. As this
> sense, I feel async_sinatra is so wrong, but not sure
> if I missed something here.)
> 
> It works great for us at the moment, though I am thinking about
> switching to threads, too, since it's a lot more flexible than fibers.

Yes, threads are far more compatible with existing gems/libraries and I
think that's the better way to go.  Of course, just processes (with
regular unicorn) should be most compatible, too.

> Sorry that I might be asking too many questions in a single thread.

No problem!  I'm just happy that folks are showing interest

> If you're interested in how I am testing them, I put my testing codes
> at: https://github.com/godfat/ruby-server-exp
> I am thinking about if I should be giving a talk on rubyconf.tw
> about ruby application servers, promoting unicorn family.
> It is one of the reasons to try out all of the options.

Cool!
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Lin Jen-Shin (godfat)]

Sorry that I didn't subscribe the mailing list (now subscribed),
so I didn't realize there's already a response. The quoted
text is copied from archive.

Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> "Lin Jen-Shin (godfat)" <godfat at godfat.org> wrote:
> Yes.  (Though it should've be easy to just pipe that part of my
> message to "git apply" or even "patch -p1")

I did that in the beginning, but it looks like your patch was not
created from 12281e4cee86496588814cd1851e8764caea024c,
so I cannot apply it? Anyway, not important :)

> All the Thread-based concurrency models should already just work
> with Celluloid-based apps.
> http://news.gmane.org/find-root.php?message_id=%3cBANLkTimnNQYpk7TRkiCvhDikzGHVy%3dkOdw%40mail.gmail.com%3e

And it's not worth the effort to wrap around celluloid-io for
buffering requests? Since it's using nio4r[0] underneath
and it's using libev for I/O, I thought that would be a replacement
for eventmachine?

[0]: https://github.com/tarcieri/nio4r

> It's much easier to unwatch IO objects with Cool.io than with EM, so I
> haven't done much with EM + Fibers.  There's also rack-fiber_pool...

True, working with Cool.io is much more pleasant. As for rack-fiber_pool,
I am not sure if I am correct or not, but in my humble option, that's a
totally wrong approach. I think fibers should be handled in server level,
not application level. That way, we don't need the hack for throwing :async,
and the middlewares do not need to be aware of fibers and :async,
and we won't need async-rack, and the server doesn't need to be
aware of :async, so on so forth.

I am not sure if this approach would have any bad influence, but at least
it works fine in our applications. That is, just wrap fibers around the calling
the app. (i.e. Fiber.new{ app_call(env) }.resume)

> Rainbows::EventMachine::TryDefer already uses the EM-internal thread
> pool, so I think EventMachineThreadPool would be redundant.

I didn't know this before, I guess it should work for me, though
it might not be as direct as:

https://github.com/godfat/ruby-server-exp/blob/9d44f8387739f5395bf97aff1689f17615cc4c7e/config/rainbows-em-thread-pool.rb#L21-25
  class REMTPoolClient < Rainbows::EventMachine::Client
    def app_call input
      EM.defer{ super }
    end
  end

> > https://github.com/cardinalblue/rest-more/blob/master/example/rainbows.rb#L15-19
> > class REMFClient < Rainbows::EventMachine::Client
> >   def app_call input
> >     Fiber.new{ super }.resume
> >   end
> > end
> Does it actually show benefits compared to regular EM?
> I suppose the Rack application still needs to be made Fiber-aware
> to reap benefits of Fibers

The Rack application doesn't need to be fiber-aware, but if
the libraries are fiber-aware, then it would be beneficial.
The rest of program doesn't have to know the existence of
fibers, because we don't throw :async.

> Yes, threads are far more compatible with existing gems/libraries and I
> think that's the better way to go.  Of course, just processes (with
> regular unicorn) should be most compatible, too.

At first I thought fibers are better for I/O, but after implementing
a thread based approach for the same mechanism, I started to
doubt that. I don't have a conclusion yet though.
(I am a bit nervous to switch to thread based approach on
production to find out, since it's not yet tested intensively)

On the other hand, all our apps are running on Heroku, and
we need to reduce memory usage because they have a hard
limit. That's why we're running Zbatery at the moment.

> > Sorry that I might be asking too many questions in a single thread.
> No problem!  I'm just happy that folks are showing interest

I really don't understand why Unicorn family didn't get much
attention in Ruby community. And I always feel wrong when
people are comparing Unicorn with Thin or even Passenger.
(sigh)

Many thanks for your work and help!
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Eric Wong]

"Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
> Sorry that I didn't subscribe the mailing list (now subscribed),
> so I didn't realize there's already a response. The quoted
> text is copied from archive.

Oops :x  I encourage unsubscribed senders to remind repliers
to Cc: them for this reason.

Unfortunately ruby-talk has conditioned the Ruby community to favor
subscriber-only lists and the Mailman config for rainbows-talk avoids
Cc-by-default (even though we'll still accept unsubscribed senders).

> Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> > "Lin Jen-Shin (godfat)" <godfat at godfat.org> wrote:
> > Yes.  (Though it should've be easy to just pipe that part of my
> > message to "git apply" or even "patch -p1")
> 
> I did that in the beginning, but it looks like your patch was not
> created from 12281e4cee86496588814cd1851e8764caea024c,
> so I cannot apply it? Anyway, not important :)

Anyways, I've pushed my patch to master as
commit e794a40049959a23ba311c572e518bb7c2861812

Will probably release v4.4.1 with that tonight unless there's something
else...

> > All the Thread-based concurrency models should already just work
> > with Celluloid-based apps.
> > http://news.gmane.org/find-root.php?message_id=%3cBANLkTimnNQYpk7TRkiCvhDikzGHVy%3dkOdw%40mail.gmail.com%3e
> 
> And it's not worth the effort to wrap around celluloid-io for
> buffering requests? Since it's using nio4r[0] underneath
> and it's using libev for I/O, I thought that would be a replacement
> for eventmachine?
> 
> [0]: https://github.com/tarcieri/nio4r

I haven't followed celluloid* closely since it was originally announced.
Maybe it's worth it to offer celluloid-io as an option and I will accept
patches for it.

> > It's much easier to unwatch IO objects with Cool.io than with EM, so I
> > haven't done much with EM + Fibers.  There's also rack-fiber_pool...
> 
> True, working with Cool.io is much more pleasant. As for rack-fiber_pool,
> I am not sure if I am correct or not, but in my humble option, that's a
> totally wrong approach. I think fibers should be handled in server level,
> not application level. That way, we don't need the hack for throwing :async,
> and the middlewares do not need to be aware of fibers and :async,
> and we won't need async-rack, and the server doesn't need to be
> aware of :async, so on so forth.
> 
> I am not sure if this approach would have any bad influence, but at least
> it works fine in our applications. That is, just wrap fibers around the calling
> the app. (i.e. Fiber.new{ app_call(env) }.resume)

Rainbows! tries not to favor/disfavor any approaches to concurrency.
And rack-fiber_pool was easy-to-support, so I added support for it.

> > Rainbows::EventMachine::TryDefer already uses the EM-internal thread
> > pool, so I think EventMachineThreadPool would be redundant.
> 
> I didn't know this before, I guess it should work for me, though
> it might not be as direct as:
> 
> https://github.com/godfat/ruby-server-exp/blob/9d44f8387739f5395bf97aff1689f17615cc4c7e/config/rainbows-em-thread-pool.rb#L21-25
>   class REMTPoolClient < Rainbows::EventMachine::Client
>     def app_call input
>       EM.defer{ super }
>     end
>   end

I seem to recall this didn't work well with some corner cases
(large static files/socket-proxying in responses).

> > > https://github.com/cardinalblue/rest-more/blob/master/example/rainbows.rb#L15-19
> > > class REMFClient < Rainbows::EventMachine::Client
> > >   def app_call input
> > >     Fiber.new{ super }.resume
> > >   end
> > > end
> > Does it actually show benefits compared to regular EM?
> > I suppose the Rack application still needs to be made Fiber-aware
> > to reap benefits of Fibers
> 
> The Rack application doesn't need to be fiber-aware, but if
> the libraries are fiber-aware, then it would be beneficial.
> The rest of program doesn't have to know the existence of
> fibers, because we don't throw :async.

OK.  But are clients served concurrently by Rainbows! in this case?

I'm not sure if I'm following this correctly (haven't thought about
Fibers in a long time), but control never goes back to Rainbows! itself
in your above case, does it?

> > Yes, threads are far more compatible with existing gems/libraries and I
> > think that's the better way to go.  Of course, just processes (with
> > regular unicorn) should be most compatible, too.
> 
> At first I thought fibers are better for I/O, but after implementing
> a thread based approach for the same mechanism, I started to
> doubt that. I don't have a conclusion yet though.
> (I am a bit nervous to switch to thread based approach on
> production to find out, since it's not yet tested intensively)
> 
> On the other hand, all our apps are running on Heroku, and
> we need to reduce memory usage because they have a hard
> limit. That's why we're running Zbatery at the moment.

Interesting.  Are you on 32 or 64-bit and are you constrained by VMSize
or RSS?

If it's VMSize and you're not dealing with deep data structures _and_
your code doesn't use recursion much; lowering pthreads stack size to
64K in thread_pthread.c should help with memory usage.  Not sure if you
can supply your own Ruby builds for that hosting service, though.

If you have time, you could also try making something like GNU pth or
npth work with Ruby 1.9.  I suspect Fibers will become unusable
with *pth, though...

> > > Sorry that I might be asking too many questions in a single thread.
> > No problem!  I'm just happy that folks are showing interest
> 
> I really don't understand why Unicorn family didn't get much
> attention in Ruby community. And I always feel wrong when
> people are comparing Unicorn with Thin or even Passenger.
> (sigh)

I'm actually surprised unicorn got the attention it has.

As project leader, my preference to maintain a low public profile and my
refusal to endorse/associate with for-profit interests certainly hurts
adoption.

If somebody else (like you) wants to market these projects, then more
power to them :)  I'm certainly never going to speak at any conference.

Rainbows! just overwhelms potential users with too many choices,
so it's unlikely to ever be widely-adopted.

> Many thanks for your work and help!

No problem :)
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Lin Jen-Shin (godfat)]

On Thu, Aug 30, 2012 at 5:17 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Oops :x  I encourage unsubscribed senders to remind repliers
> to Cc: them for this reason.
>
> Unfortunately ruby-talk has conditioned the Ruby community to favor
> subscriber-only lists and the Mailman config for rainbows-talk avoids
> Cc-by-default (even though we'll still accept unsubscribed senders).

That's ok, since most of time I would check the archive regularly
whenever I am expecting responses. I forgot this time because I was
quite busy recently. And now I subscribed the list because I guess
there won't have too many mails anyway and I would be definitely
interested in discussing Rainbows! :)

> Anyways, I've pushed my patch to master as
> commit e794a40049959a23ba311c572e518bb7c2861812
>
> Will probably release v4.4.1 with that tonight unless there's something
> else...

Yeah, I saw it, thank you.

> I haven't followed celluloid* closely since it was originally announced.
> Maybe it's worth it to offer celluloid-io as an option and I will accept
> patches for it.

Cool. Then I might want to try it some other time. I don't have
confidence though.

>> https://github.com/godfat/ruby-server-exp/blob/9d44f8387739f5395bf97aff1689f17615cc4c7e/config/rainbows-em-thread-pool.rb#L21-25
>>   class REMTPoolClient < Rainbows::EventMachine::Client
>>     def app_call input
>>       EM.defer{ super }
>>     end
>>   end
>
> I seem to recall this didn't work well with some corner cases
> (large static files/socket-proxying in responses).

Sorry that I have no idea about those corner cases. (not sure
if they are corner enough?) I am not surprised if this approach
won't work for all cases. I would be interested to try out though.
Not sure what's socket-proxying, but I'll try large static files,
and see why it's not working properly. Hope it would be easy
to reproduce.

>> > > https://github.com/cardinalblue/rest-more/blob/master/example/rainbows.rb#L15-19
>> > > class REMFClient < Rainbows::EventMachine::Client
>> > >   def app_call input
>> > >     Fiber.new{ super }.resume
>> > >   end
>> > > end
>> > Does it actually show benefits compared to regular EM?
>> > I suppose the Rack application still needs to be made Fiber-aware
>> > to reap benefits of Fibers
>>
>> The Rack application doesn't need to be fiber-aware, but if
>> the libraries are fiber-aware, then it would be beneficial.
>> The rest of program doesn't have to know the existence of
>> fibers, because we don't throw :async.
>
> OK.  But are clients served concurrently by Rainbows! in this case?
>
> I'm not sure if I'm following this correctly (haven't thought about
> Fibers in a long time), but control never goes back to Rainbows! itself
> in your above case, does it?

It does as far as I know. I am not good at explaining in natural languages,
and English is not my first language. Let me show this concept in codes.
It's something like this:
( Also in gist: https://gist.github.com/3540749 )

# Wrap a fiber around app_call
Fiber.new{
  # [...]
  # Below is modified from event_machine/client.rb, in def app_call input,
  # but let's forget about :async for now.
  # In APP.call, it would call Fiber.yield whenever we're waiting for data.
  status, headers, body = APP.call(@env.merge!(RACK_DEFAULTS))

  # The response is fully ready at this point.
  ev_write_response(status, headers, body, @hp.next?)
}.resume

# * * * * *

# Here's the app
APP = lambda{ |env|
  # First we remember where we need to jump back
  f = Fiber.current
  # Wait for 5 seconds to emulate waiting for data
  EM.add_timer(5){
    # Here then we have data available, let's resume back.
    # But we also wrap resuming in a next_tick block,
    # giving EM some chances to cleanup it's internal states.
    EM.next_tick{
      f.resume('OK')
    }
  }
  # Here we immediately yield and give control back to Rainbows!
  # and Rainbows! would then go back to EventMachine's regular loop,
  # buffering requests, making requests, etc.
  body = Fiber.yield

  # So now body is referring the data resumed from above.
  # Rainbows! should handle the response at this point.
  [200, {}, [body]]
}

Not sure if this is clear enough, please let me know if I didn't
make it clear, or there's anything wrong in my assumption or
the over simplified codes. Thanks!

> Interesting.  Are you on 32 or 64-bit and are you constrained by VMSize
> or RSS?

I think it's an x86_64 VM. Not sure what's VMSize, but we're observing
memory usage in RSS. They claim we have about 500M for one process,
but sometimes the app cannot allocate more memory even below 500M.
(Well, I don't quite understand how the app would grow to 500M? It's
about 160M on average)

> If it's VMSize and you're not dealing with deep data structures _and_
> your code doesn't use recursion much; lowering pthreads stack size to
> 64K in thread_pthread.c should help with memory usage.  Not sure if you
> can supply your own Ruby builds for that hosting service, though.

There's a chance to do that on Heroku, but I guess it's too much effort
to do so. I don't feel there's a standard way to compile stuffs on it.

How do I tell the current stack size for a thread in pthread? It seems I
cannot run ulimit on Heroku. On the other hand, I am not sure if it's
significant enough to reduce thread stack size.

I guess we can also reduce the concurrent level (worker_connections?)
to reduce memory usage, too?

> If you have time, you could also try making something like GNU pth or
> npth work with Ruby 1.9.  I suspect Fibers will become unusable
> with *pth, though...

Cool, might be interesting.
Could then threads be as lightweight as fibers? :P
Though I really doubt if threads are really that heavy comparing to fibers.
At least in some simple tests, threads are fine and efficient enough.

EventMachine is still a lot faster than regular sockets (net/http) though,
so I'll still keep EventMachine for a while even if I switched to threads.

> I'm actually surprised unicorn got the attention it has.
>
> As project leader, my preference to maintain a low public profile and my
> refusal to endorse/associate with for-profit interests certainly hurts
> adoption.
>
> If somebody else (like you) wants to market these projects, then more
> power to them :)  I'm certainly never going to speak at any conference.
>
> Rainbows! just overwhelms potential users with too many choices,
> so it's unlikely to ever be widely-adopted.

Umm... I don't know the reason why you want to stay away from them,
but please let me know if you don't want someone to market them.
I'll then try to be as natural as possible when talking about Unicorns.

I guess Github's post certainly attracted at lot of people's attention.
I don't remember where I did see Unicorn in the beginning, but
certainly I became more interested because of that post.
I guess that's also one of the reasons that people didn't know
about Rainbows! (they didn't mention it) Or you might be right,
a lot of people just don't try Rainbows! because there are too
many choices. Or, use it wrongly, I guess...
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Eric Wong]

"Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
> On Thu, Aug 30, 2012 at 5:17 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> > Will probably release v4.4.1 with that tonight unless there's something
> > else...
> 
> Yeah, I saw it, thank you.

Will release today before I forget again :>

> > I haven't followed celluloid* closely since it was originally announced.
> > Maybe it's worth it to offer celluloid-io as an option and I will accept
> > patches for it.
> 
> Cool. Then I might want to try it some other time. I don't have
> confidence though.

No problem, take your time :)

> >> https://github.com/godfat/ruby-server-exp/blob/9d44f8387739f5395bf97aff1689f17615cc4c7e/config/rainbows-em-thread-pool.rb#L21-25
> >>   class REMTPoolClient < Rainbows::EventMachine::Client
> >>     def app_call input
> >>       EM.defer{ super }
> >>     end
> >>   end
> >
> > I seem to recall this didn't work well with some corner cases
> > (large static files/socket-proxying in responses).
> 
> Sorry that I have no idea about those corner cases. (not sure
> if they are corner enough?) I am not surprised if this approach
> won't work for all cases. I would be interested to try out though.
> Not sure what's socket-proxying, but I'll try large static files,
> and see why it's not working properly. Hope it would be easy
> to reproduce.

I seem to recall problems with some of the more esoteric test cases in
Rainbows! a few years ago.

Now that I think more about it, it might've been related to client
pipelining.  If a client pipelines requests, I don't think using
EM.defer {} makes it easy to guarantee the servers responses are
returned in the correct order.

This is made worse since (AFAIK) EM provides no easy way to
temporarily disable firing read callbacks for a socket, so
a client which pipelines aggressively becomes bad news.

> >> > > https://github.com/cardinalblue/rest-more/blob/master/example/rainbows.rb#L15-19
> >> > > class REMFClient < Rainbows::EventMachine::Client
> >> > >   def app_call input
> >> > >     Fiber.new{ super }.resume
> >> > >   end
> >> > > end
> >> > Does it actually show benefits compared to regular EM?
> >> > I suppose the Rack application still needs to be made Fiber-aware
> >> > to reap benefits of Fibers
> >>
> >> The Rack application doesn't need to be fiber-aware, but if
> >> the libraries are fiber-aware, then it would be beneficial.
> >> The rest of program doesn't have to know the existence of
> >> fibers, because we don't throw :async.
> >
> > OK.  But are clients served concurrently by Rainbows! in this case?
> >
> > I'm not sure if I'm following this correctly (haven't thought about
> > Fibers in a long time), but control never goes back to Rainbows! itself
> > in your above case, does it?
> 
> It does as far as I know. I am not good at explaining in natural languages,
> and English is not my first language. Let me show this concept in codes.
> It's something like this:
> ( Also in gist: https://gist.github.com/3540749 )
> 
> # Wrap a fiber around app_call
> Fiber.new{
>   # [...]
>   # Below is modified from event_machine/client.rb, in def app_call input,
>   # but let's forget about :async for now.
>   # In APP.call, it would call Fiber.yield whenever we're waiting for data.
>   status, headers, body = APP.call(@env.merge!(RACK_DEFAULTS))
> 
>   # The response is fully ready at this point.
>   ev_write_response(status, headers, body, @hp.next?)
> }.resume
> 
> # * * * * *
> 
> # Here's the app
> APP = lambda{ |env|
>   # First we remember where we need to jump back
>   f = Fiber.current
>   # Wait for 5 seconds to emulate waiting for data
>   EM.add_timer(5){
>     # Here then we have data available, let's resume back.
>     # But we also wrap resuming in a next_tick block,
>     # giving EM some chances to cleanup it's internal states.
>     EM.next_tick{
>       f.resume('OK')
>     }
>   }
>   # Here we immediately yield and give control back to Rainbows!
>   # and Rainbows! would then go back to EventMachine's regular loop,
>   # buffering requests, making requests, etc.
>   body = Fiber.yield
> 
>   # So now body is referring the data resumed from above.
>   # Rainbows! should handle the response at this point.
>   [200, {}, [body]]
> }
> 
> Not sure if this is clear enough, please let me know if I didn't
> make it clear, or there's anything wrong in my assumption or
> the over simplified codes. Thanks!

Thank you, your code makes it clear.  I think your approach will work
with most HTTP clients.

However, I think pipelined requests will hit the same problems as
EM.defer, too.  Can you try with pipelining?

Maybe disabling keepalive/persistent connections will make this work
correctly (but you obviously lose latency benefits, too).

I also don't think it's possible to say "no pipelining" to a client if
we support persistent connections at all.

> > Interesting.  Are you on 32 or 64-bit and are you constrained by VMSize
> > or RSS?
> 
> I think it's an x86_64 VM. Not sure what's VMSize, but we're observing
> memory usage in RSS. They claim we have about 500M for one process,
> but sometimes the app cannot allocate more memory even below 500M.
> (Well, I don't quite understand how the app would grow to 500M? It's
> about 160M on average)

It's likely some corner case in your code.  Do you generate potentially
large responses or read in large amounts of data?  (e.g. SELECT
statements without a LIMIT, large files (uploads?)).

A slow client which triggers large server responses (which EM may
buffer even if the Rack app streams it out) can hit this, too.
I don't think EM can be configured to buffer writes to the file
system (nginx will automatically do this, though).

> > If it's VMSize and you're not dealing with deep data structures _and_
> > your code doesn't use recursion much; lowering pthreads stack size to
> > 64K in thread_pthread.c should help with memory usage.  Not sure if you
> > can supply your own Ruby builds for that hosting service, though.
> 
> There's a chance to do that on Heroku, but I guess it's too much effort
> to do so. I don't feel there's a standard way to compile stuffs on it.
> 
> How do I tell the current stack size for a thread in pthread? It seems I
> cannot run ulimit on Heroku. On the other hand, I am not sure if it's
> significant enough to reduce thread stack size.

Ruby 1.9 sets stack sizes to 512K regardless of ulimit -s.  At least on
Linux, memory defaults to being overcommited and is lazily allocated in
increments of PAGE_SIZE (4K on x86*).  It's likely the actual RSS overhead
of a native thread stack is <64K.

VMSize overhead becomes important on 32-bit with many native threads,
though.  In comparison, Fibers use only 4K stack and has no extra
overhead in the kernel.

> I guess we can also reduce the concurrent level (worker_connections?)
> to reduce memory usage, too?
> 
> > If you have time, you could also try making something like GNU pth or
> > npth work with Ruby 1.9.  I suspect Fibers will become unusable
> > with *pth, though...
> 
> Cool, might be interesting.
> Could then threads be as lightweight as fibers? :P

Yes.  Fwiw, MRI 1.8 green threads and Neverblock are basically
equivalent, too.

> Though I really doubt if threads are really that heavy comparing to fibers.
> At least in some simple tests, threads are fine and efficient enough.

I agree native threads are light enough for most cases (especially since
you're already running Ruby :).

> EventMachine is still a lot faster than regular sockets (net/http) though,
> so I'll still keep EventMachine for a while even if I switched to threads.

I think part of that is the HTTP parser and I/O buffering being
implemented in C/C++ vs Ruby.  Things like the net-http-persistent gem
should help with pure-Ruby performance, though (and performance is
likely to be better with upcoming Ruby releases).

> > I'm actually surprised unicorn got the attention it has.
> >
> > As project leader, my preference to maintain a low public profile and my
> > refusal to endorse/associate with for-profit interests certainly hurts
> > adoption.
> >
> > If somebody else (like you) wants to market these projects, then more
> > power to them :)  I'm certainly never going to speak at any conference.
> >
> > Rainbows! just overwhelms potential users with too many choices,
> > so it's unlikely to ever be widely-adopted.
> 
> Umm... I don't know the reason why you want to stay away from them,
> but please let me know if you don't want someone to market them.
> I'll then try to be as natural as possible when talking about Unicorns.

I enjoy my near-anonymity and want as little reputation/recognition as
possible.  I'm always happy if people talk about software, but I prefer
software stand on its own and not on the reputation of its authors.

(The only reason I use my name is for potential GPL enforcement)
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Lin Jen-Shin (godfat)]

On Fri, Aug 31, 2012 at 9:37 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> I seem to recall problems with some of the more esoteric test cases in
> Rainbows! a few years ago.
>
> Now that I think more about it, it might've been related to client
> pipelining.  If a client pipelines requests, I don't think using
> EM.defer {} makes it easy to guarantee the servers responses are
> returned in the correct order.
>
> This is made worse since (AFAIK) EM provides no easy way to
> temporarily disable firing read callbacks for a socket, so
> a client which pipelines aggressively becomes bad news.

After some experiments, now I understood why it is hard. But I can't
figure it out by some quick glimpses for how you did solve this problems
for other concurrency model?

One possible and simple way would be... just make piped requests
sequential, but this would greatly reduce the concurrency ability,
is it right? At least Puma server runs quite poorly whenever I am
testing pipeline requests.

My test script  is:

httperf --hog --server localhost --port 8080 --uri /cpu --num-calls 4
--burst-length 2 --num-conn 2 --rate 8 --print-reply

But Zbatery runs quite smoothly with ThreadPool and ThreadSpawn.
I assume it's because Zbatery would handle piped requests concurrently
and collect responses and reply them with the correct order, though
I cannot tell from the code, at least from some quick glimpses.

At this point I am more confident to say that Unicorn family is the best
Ruby application servers. :)

> Thank you, your code makes it clear.  I think your approach will work
> with most HTTP clients.
>
> However, I think pipelined requests will hit the same problems as
> EM.defer, too.  Can you try with pipelining?

Honestly I don't know about keep-alive and pipelined requests, and
I just learned it from trying httperf, which seems to be a very good tool
to make trials against web servers. Puma and Thin worked poorly
in my tests with above httperf command, while Zbatery worked
perfectly fine. (except with my hack for adding fibers/threads on top
of EventMachine, which were raising errors)

After pondering and reading the codes in Rainbows for a while, I got
managed to make it work without errors, but I believe it is still suffering
from the ordering issues. There's no promise for ordering.

Here's the new code. It's for fibers but I think it's the same with EM.defer.
https://github.com/godfat/ruby-server-exp/commit/41644fc63ffa0a505eab616c1d23ead4a564d349
def app_call input
  # [...] as before
  Fiber.new{
    status, headers, body = catch(:async) {
      APP.call(@env.merge!(RACK_DEFAULTS))
    }
    if nil == status || -1 == status
      @deferred = true
    else
      @deferred = nil # response is ready, no more @deferred
      ev_write_response(status, headers, body, @hp.next?)
    end
  }.resume
  @deferred = true # we're always deferring
end

To address ordering issue, I guess we can remember the
index of a certain request, and if there's a request being
processed which has a lower index, the response shouldn't
be written back before the lower one has been written.

Not sure if this is wroth the effort though... This must touch
Rainbows' internal, and it cannot be easily handled by
simply extending the client class.

> Maybe disabling keepalive/persistent connections will make this work
> correctly (but you obviously lose latency benefits, too).
>
> I also don't think it's possible to say "no pipelining" to a client if
> we support persistent connections at all.

I wonder if we always run Nginx or something similar in front of
Rainbows, does it still matter?

Nevertheless, I guess it's good enough for us right now.
Many thanks for your review. On the other hand, I would still be
very interested to see if this could be addressed. Last time I want
to replicate what other concurrency models solved this, but failed
to see how.

> It's likely some corner case in your code.  Do you generate potentially
> large responses or read in large amounts of data?  (e.g. SELECT
> statements without a LIMIT, large files (uploads?)).
>
> A slow client which triggers large server responses (which EM may
> buffer even if the Rack app streams it out) can hit this, too.
> I don't think EM can be configured to buffer writes to the file
> system (nginx will automatically do this, though).

I see. Never thought of that EM might be buffering a lot of large
responses in the memory. As for loading large amounts of data
into memory, I guess I can't tell. As far as I know, no, but who knows :P
This must be accidental if there's one...

Anyway, we don't see that often nowadays. Or it could be that
Ruby 1.9.3 had fixed some memory leak issues. Or some other
3rd party libraries we're using.

> Ruby 1.9 sets stack sizes to 512K regardless of ulimit -s.  At least on
> Linux, memory defaults to being overcommited and is lazily allocated in
> increments of PAGE_SIZE (4K on x86*).  It's likely the actual RSS overhead
> of a native thread stack is <64K.
>
> VMSize overhead becomes important on 32-bit with many native threads,
> though.  In comparison, Fibers use only 4K stack and has no extra
> overhead in the kernel.

I see, thanks for the explanation. I guess that does matter a bit, but only
if we're using thousands of threads/fibers, and it should be quite rarely
in a web app, I guess.

Using fibers are also risking from system stack overflow, especially in
a Rails app with a lot of plugins, I guess... Umm, but I also heard that
fibers stack is increased a bit in newer Ruby?

>> Though I really doubt if threads are really that heavy comparing to fibers.
>> At least in some simple tests, threads are fine and efficient enough.
>
> I agree native threads are light enough for most cases (especially since
> you're already running Ruby :).

Speaking to this and green threads, I wonder if it's worth the effort to
implement m:n threading for Ruby? Or we can just compile and
link against a threading library which supports m:n threading?
Goroutine? :P

>> EventMachine is still a lot faster than regular sockets (net/http) though,
>> so I'll still keep EventMachine for a while even if I switched to threads.
>
> I think part of that is the HTTP parser and I/O buffering being
> implemented in C/C++ vs Ruby.  Things like the net-http-persistent gem
> should help with pure-Ruby performance, though (and performance is
> likely to be better with upcoming Ruby releases).

I haven't got a chance to try net-http-persistent, but it seems I should try it.
(or try that on em-http-request, it seems it supports it
https://github.com/igrigorik/em-http-request/wiki/Keep-Alive-and-HTTP-Pipelining
)

Or if it's all about HTTP parsing, [http][] gem should help too.
It is using [http_parser.rb] underneath, which is based on NodeJS'
[http-parser][].

Sometimes I feel it's all about throwing away EventMachine...
I've heard that EM is bad, but not bad enough to be rewritten...

[http]: https://github.com/tarcieri/http
[http_parser.rb]: https://github.com/tmm1/http_parser.rb
[http-parser]: https://github.com/joyent/http-parser

> I enjoy my near-anonymity and want as little reputation/recognition as
> possible.  I'm always happy if people talk about software, but I prefer
> software stand on its own and not on the reputation of its authors.
>
> (The only reason I use my name is for potential GPL enforcement)

I see. Thanks for explaining. I'll then avoid talking about authors :)
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Eric Wong]

"Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
> On Fri, Aug 31, 2012 at 9:37 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> > I seem to recall problems with some of the more esoteric test cases in
> > Rainbows! a few years ago.
> >
> > Now that I think more about it, it might've been related to client
> > pipelining.  If a client pipelines requests, I don't think using
> > EM.defer {} makes it easy to guarantee the servers responses are
> > returned in the correct order.
> >
> > This is made worse since (AFAIK) EM provides no easy way to
> > temporarily disable firing read callbacks for a socket, so
> > a client which pipelines aggressively becomes bad news.
> 
> After some experiments, now I understood why it is hard. But I can't
> figure it out by some quick glimpses for how you did solve this problems
> for other concurrency model?

Simple: we don't read from the socket at all while processing a request

Extra data the client sends gets buffered in the kernel, eventually TCP
backoff will kick in and the Internet stays usable :>

With the inability to easily stop read callbacks via EM, the socket
buffers constantly get drained so the clients are able to keep sending
data.  The only option I've found for Rainbows! + EM was to issue
shutdown(SHUT_RD) if a client attempts to pipeline too much (which
never happens with legitimate clients AFAIK).

> One possible and simple way would be... just make piped requests
> sequential, but this would greatly reduce the concurrency ability,
> is it right? At least Puma server runs quite poorly whenever I am
> testing pipeline requests.

Yes, pipelining is handled sequentially.  It's the easiest and safest
way to implement since HTTP/1.1 requires the responses to be returned in
the same order they were sent (as you mention below).

> My test script  is:
> 
> httperf --hog --server localhost --port 8080 --uri /cpu --num-calls 4
> --burst-length 2 --num-conn 2 --rate 8 --print-reply
> 
> But Zbatery runs quite smoothly with ThreadPool and ThreadSpawn.
> I assume it's because Zbatery would handle piped requests concurrently
> and collect responses and reply them with the correct order, though
> I cannot tell from the code, at least from some quick glimpses.

Rainbows!/Zbatery handles pipelined requests sequentially.  They only
have concurrency on a per-socket level.

> At this point I am more confident to say that Unicorn family is the best
> Ruby application servers. :)

Good to hear :)

<snip>

> To address ordering issue, I guess we can remember the
> index of a certain request, and if there's a request being
> processed which has a lower index, the response shouldn't
> be written back before the lower one has been written.
> 
> Not sure if this is wroth the effort though... This must touch
> Rainbows' internal, and it cannot be easily handled by
> simply extending the client class.

I don't think it is worth the effort.  I'm not even sure how often
pipelining is used in the real world, all I know is Rainbows! can
handle it without falling over.

> > Maybe disabling keepalive/persistent connections will make this work
> > correctly (but you obviously lose latency benefits, too).
> >
> > I also don't think it's possible to say "no pipelining" to a client if
> > we support persistent connections at all.
> 
> I wonder if we always run Nginx or something similar in front of
> Rainbows, does it still matter?

It shouldn't matter for nginx, I don't think nginx will (ever) pipeline
to a backend.  Nowadays nginx can do persistent connections to backends,
though I'm not sure how much of a benefit it is for local sockets.

> I see. Never thought of that EM might be buffering a lot of large
> responses in the memory. As for loading large amounts of data
> into memory, I guess I can't tell. As far as I know, no, but who knows :P
> This must be accidental if there's one...

I think your comment is unfortunately representative of a lot of
software development nowadays.

Embrace pessimism and let it be your guide :)

> > Ruby 1.9 sets stack sizes to 512K regardless of ulimit -s.  At least on
> > Linux, memory defaults to being overcommited and is lazily allocated in
> > increments of PAGE_SIZE (4K on x86*).  It's likely the actual RSS overhead
> > of a native thread stack is <64K.
> >
> > VMSize overhead becomes important on 32-bit with many native threads,
> > though.  In comparison, Fibers use only 4K stack and has no extra
> > overhead in the kernel.
> 
> I see, thanks for the explanation. I guess that does matter a bit, but only
> if we're using thousands of threads/fibers, and it should be quite rarely
> in a web app, I guess.
> 
> Using fibers are also risking from system stack overflow, especially in
> a Rails app with a lot of plugins, I guess... Umm, but I also heard that
> fibers stack is increased a bit in newer Ruby?

You're right, fiber stacks got bigger.  They're 64K on all 1.9.3 and
128K on 64-bit for 2.0.0dev.  So there's even less benefit in using
Fibers nowadays for memory concerns.

> >> Though I really doubt if threads are really that heavy comparing to fibers.
> >> At least in some simple tests, threads are fine and efficient enough.
> >
> > I agree native threads are light enough for most cases (especially since
> > you're already running Ruby :).
> 
> Speaking to this and green threads, I wonder if it's worth the effort to
> implement m:n threading for Ruby? Or we can just compile and
> link against a threading library which supports m:n threading?
> Goroutine? :P

*shrug*  Not worth _my_ effort for m:n threads.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Lin Jen-Shin (godfat)]

On Thu, Sep 6, 2012 at 7:27 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Simple: we don't read from the socket at all while processing a request
>
> Extra data the client sends gets buffered in the kernel, eventually TCP
> backoff will kick in and the Internet stays usable :>

Understood, thanks! I guess this is simple enough that it's a nice
trade off, since not every clients would do heavy pipelining.

> With the inability to easily stop read callbacks via EM, the socket
> buffers constantly get drained so the clients are able to keep sending
> data.  The only option I've found for Rainbows! + EM was to issue
> shutdown(SHUT_RD) if a client attempts to pipeline too much (which
> never happens with legitimate clients AFAIK).

And the buffer size you set: 0x1c000, I guess that's large enough
for most cases.

> It shouldn't matter for nginx, I don't think nginx will (ever) pipeline
> to a backend.  Nowadays nginx can do persistent connections to backends,
> though I'm not sure how much of a benefit it is for local sockets.

Here's a problem troubles me for a long time, and I failed to figure out
what's going on and cannot find a proper fix. I didn't mention this before,
because I don't think it's Zbatery/Rainbows' issue, but it turns out that
it might be related, and perhaps you would be interested to provide
some hints, I would much appreciate it.

So we're running all our apps on http://www.heroku.com
It's a hosting service that you simply `git push` to deploy your app,
worrying not all the complexity regarding deployment.

There's a very weird issue running Zbatery/Rainbows on it.
I don't remember if Unicorn would be suffering from that
since I don't run Unicorn directly for a while now.

The problem comes with Rails 3 and its "assets pipeline" feature.
It's basically a feature that let your assets (e.g. images, javascripts,
css, etc) no longer simple assets which could be served directly
via nginx or some other static web server.

Like, developers could name their coffeescripts files as:

application.coffee.js

And when a client is requesting "application.js", that "assets pipeline"
thing would compile the coffeescript file into a javascript file on the fly,
and return it to the client.

We can also "precompile" those assets before deploying to production.
Heroku would do this if it detects that the application is deploying is
built with Rails 3.

If this precompile thing is done successfully, then there's no problems,
at least in some of the cases.... But what if precompile failed? In my cases,
it would usually cause system stack overflow whenever it tries to compile
it on the fly.

Oh, suddenly I think maybe it's because I would wrap a fiber around
each request in some cases... and it just used too many method calls.
I am not sure about that... might check it next time I saw it.

Somehow I feel this might be related to persistent connections or
pipelined requests, because last time I didn't write the client class
correctly to handle pipelined requests as you pointed out, it would
also cause some issues regarding "assets pipeline" on Heroku.

I believe Heroku is running nginx in front off Zbatery, and if I didn't
get pipelined requests correctly, it would result timeout for many of
assets the client is asking about. I feel Heroku is using pipelined
requests if the client is using this.

And the most weird thing I can't understand is, I am trying to convince
one of my friend to try out Zbatery, but using ThreadSpawn model
would usually cause serving assets timeout, just like in my case
where I didn't get pipelined requests correct. Switching to EventMachine
solved the problem! Huh?

Moreover, once there are some assets timeout issues on EventMachine,
too. When I tried to debug this, I put some traces into Rainbows,
realizing that sometimes EventMachine didn't call `receive_data'
when receiving some pipelined requests. Could it be an eventmachine bug!?

Sorry that I wrote so much and it's unclear what's going on, since
I don't understand so I can't remember correctly...

If you have any idea for a specific case, I can try to provide all the
information I could collect. I can't get the nginx configuration Heroku
is using though :(

> I think your comment is unfortunately representative of a lot of
> software development nowadays.
>
> Embrace pessimism and let it be your guide :)

I hope I could understand/realize this sooner :)

Cheers,
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Eric Wong]

"Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
> Here's a problem troubles me for a long time, and I failed to figure out
> what's going on and cannot find a proper fix. I didn't mention this before,
> because I don't think it's Zbatery/Rainbows' issue, but it turns out that
> it might be related, and perhaps you would be interested to provide
> some hints, I would much appreciate it.
> 
> So we're running all our apps on http://www.heroku.com
> It's a hosting service that you simply `git push` to deploy your app,
> worrying not all the complexity regarding deployment.
> 
> There's a very weird issue running Zbatery/Rainbows on it.
> I don't remember if Unicorn would be suffering from that
> since I don't run Unicorn directly for a while now.
> 
> The problem comes with Rails 3 and its "assets pipeline" feature.
> It's basically a feature that let your assets (e.g. images, javascripts,
> css, etc) no longer simple assets which could be served directly
> via nginx or some other static web server.
> 
> Like, developers could name their coffeescripts files as:
> 
> application.coffee.js
> 
> And when a client is requesting "application.js", that "assets pipeline"
> thing would compile the coffeescript file into a javascript file on the fly,
> and return it to the client.
> 
> We can also "precompile" those assets before deploying to production.
> Heroku would do this if it detects that the application is deploying is
> built with Rails 3.
> 
> If this precompile thing is done successfully, then there's no problems,
> at least in some of the cases.... But what if precompile failed? In my cases,
> it would usually cause system stack overflow whenever it tries to compile
> it on the fly.
> 
> Oh, suddenly I think maybe it's because I would wrap a fiber around
> each request in some cases... and it just used too many method calls.
> I am not sure about that... might check it next time I saw it.
> 
> Somehow I feel this might be related to persistent connections or
> pipelined requests, because last time I didn't write the client class
> correctly to handle pipelined requests as you pointed out, it would
> also cause some issues regarding "assets pipeline" on Heroku.
> 
> I believe Heroku is running nginx in front off Zbatery, and if I didn't
> get pipelined requests correctly, it would result timeout for many of
> assets the client is asking about. I feel Heroku is using pipelined
> requests if the client is using this.
> 
> And the most weird thing I can't understand is, I am trying to convince
> one of my friend to try out Zbatery, but using ThreadSpawn model
> would usually cause serving assets timeout, just like in my case
> where I didn't get pipelined requests correct. Switching to EventMachine
> solved the problem! Huh?
> 
> Moreover, once there are some assets timeout issues on EventMachine,
> too. When I tried to debug this, I put some traces into Rainbows,
> realizing that sometimes EventMachine didn't call `receive_data'
> when receiving some pipelined requests. Could it be an eventmachine bug!?

It could be the front-end proxy (incorrectly) detected the Rainbows!
instance was down and stopped sending traffic to it.  Does this
information get logged?

> Sorry that I wrote so much and it's unclear what's going on, since
> I don't understand so I can't remember correctly...

Yeah, it's a bit much to understand.  Can you reproduce it consistently?

With the serving timeout for Zbatery+ThreadSpawn, can you ensure
Content-Length/Transfer-Encoding:chunked is set in the response headers?

Since you mentioned stack overflows in response generation, perhaps
whatever proxy Heroku is using doesn't handle crashed servers during
the response correctly...

Can you get stderr logs from Heroku?

> If you have any idea for a specific case, I can try to provide all the
> information I could collect. I can't get the nginx configuration Heroku
> is using though :(

I highly doubt nginx will pipeline requests, but we're not sure if
they're really using nginx, yet.  With the problems you've described,
it doesn't sound like they are, or they're using some broken version
of it.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Lin Jen-Shin (godfat)]

On Sun, Sep 23, 2012 at 3:42 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
[...]
>> Moreover, once there are some assets timeout issues on EventMachine,
>> too. When I tried to debug this, I put some traces into Rainbows,
>> realizing that sometimes EventMachine didn't call `receive_data'
>> when receiving some pipelined requests. Could it be an eventmachine bug!?
>
> It could be the front-end proxy (incorrectly) detected the Rainbows!
> instance was down and stopped sending traffic to it.  Does this
> information get logged?

It would eventually show out as a timeout for 30 seconds. It's a log
from front-end proxy (router in their terminology). I am not sure
if this means it is sending traffic or not. If it's not, then I guess it
explains...

> Yeah, it's a bit much to understand.  Can you reproduce it consistently?

Yes, it could be reproduced consistently, but only in a certain environment
(e.g. on Heroku) which might not always be found.

> With the serving timeout for Zbatery+ThreadSpawn, can you ensure
> Content-Length/Transfer-Encoding:chunked is set in the response headers?

I just realized that this is not a timeout issue here. (doh, too many strange
issues) I just tried to curl it, and it would immediately return an error.

heroku[router]: Error H13 (Connection closed without response) -> GET
/assets/application-4e83bff8c0e77de81926c169e1fcacf2.css dyno=web.1
queue= wait= service= status=503 bytes=

There is Content-Length: 98794 and no Transfer-Encoding.

I don't see if there's any difference between using EventMachine :(

By the way, weirdly that it seems there's no problems at all if we're
using Thin server. I guess they are testing against Thin, so Thin works
correctly... still can't I tell what's the difference.

> Since you mentioned stack overflows in response generation, perhaps
> whatever proxy Heroku is using doesn't handle crashed servers during
> the response correctly...

Probably. And I think I am 80% sure where it causes stack overflows now.
If I took out fibers, then it would be ok. So I guess that assets things are
using too much stack.

Too bad I can't switch to threads though :( If I switched to threads,
ActiveRecord would be complaining it could not get a connection from
the connection pool within 5 seconds. It's not convenient to increase the
size of connection pool on Heroku, either, and making it too large would
also cause other issues.

(sigh)

> Can you get stderr logs from Heroku?

Yes, Heroku would redirect both stdout and stderr to a place where we
could see. It's collapsed into one huge log though.

> I highly doubt nginx will pipeline requests, but we're not sure if
> they're really using nginx, yet.  With the problems you've described,
> it doesn't sound like they are, or they're using some broken version
> of it.

Umm... after reading the log, I think they are using another [router]
(in their terminology) in front of [nginx]. So it might be an issue in
their [router], I am not sure...

I'll keep you posted if you're interested. Thanks for all your help.

p.s. They might be using https://github.com/mochi/mochiweb
too, since if there's an issue, it would return this header:
Server: MochiWeb/1.0 (Any of you quaids got a smint?)

p.s.2. Perhaps I could even give you access to one of the apps.
I'll need to ask the owner though. Let me know if you're interested,
otherwise you could simply ignore this. I know it's too much to ask.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Eric Wong]

"Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
> On Sun, Sep 23, 2012 at 3:42 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> [...]
> >> Moreover, once there are some assets timeout issues on EventMachine,
> >> too. When I tried to debug this, I put some traces into Rainbows,
> >> realizing that sometimes EventMachine didn't call `receive_data'
> >> when receiving some pipelined requests. Could it be an eventmachine bug!?
> >
> > It could be the front-end proxy (incorrectly) detected the Rainbows!
> > instance was down and stopped sending traffic to it.  Does this
> > information get logged?
> 
> It would eventually show out as a timeout for 30 seconds. It's a log
> from front-end proxy (router in their terminology). I am not sure
> if this means it is sending traffic or not. If it's not, then I guess it
> explains...
> 
> > Yeah, it's a bit much to understand.  Can you reproduce it consistently?
> 
> Yes, it could be reproduced consistently, but only in a certain environment
> (e.g. on Heroku) which might not always be found.
> 
> > With the serving timeout for Zbatery+ThreadSpawn, can you ensure
> > Content-Length/Transfer-Encoding:chunked is set in the response headers?
> 
> I just realized that this is not a timeout issue here. (doh, too many strange
> issues) I just tried to curl it, and it would immediately return an error.
> 
> heroku[router]: Error H13 (Connection closed without response) -> GET
> /assets/application-4e83bff8c0e77de81926c169e1fcacf2.css dyno=web.1
> queue= wait= service= status=503 bytes=
> 
> There is Content-Length: 98794 and no Transfer-Encoding.

So the _actual_ Content-Length that's sent is zero?

Rainbows! should drop a connection if an exception is raised while
sending the response body, perhaps the heroku router is confused
by that?

> I don't see if there's any difference between using EventMachine :(
> 
> By the way, weirdly that it seems there's no problems at all if we're
> using Thin server. I guess they are testing against Thin, so Thin works
> correctly... still can't I tell what's the difference.

Are you allowed to "strace -f" Thin and Rainbows! in an isolated (single
thread/process instance)?  I'd love to see the difference between how
the servers handle this case differently...

Can you try setting "keepalive_timeout 0" to disable persistent
connections in Rainbows! and see if this helps?  I don't think
persistent connections make a big difference for LAN, especially
not with typical HTTP response sizes (your 98K is pretty big :)

> > Since you mentioned stack overflows in response generation, perhaps
> > whatever proxy Heroku is using doesn't handle crashed servers during
> > the response correctly...
> 
> Probably. And I think I am 80% sure where it causes stack overflows now.
> If I took out fibers, then it would be ok. So I guess that assets things are
> using too much stack.
> 
> Too bad I can't switch to threads though :( If I switched to threads,
> ActiveRecord would be complaining it could not get a connection from
> the connection pool within 5 seconds. It's not convenient to increase the
> size of connection pool on Heroku, either, and making it too large would
> also cause other issues.
> 
> (sigh)
> 
> > Can you get stderr logs from Heroku?
> 
> Yes, Heroku would redirect both stdout and stderr to a place where we
> could see. It's collapsed into one huge log though.
> 
> > I highly doubt nginx will pipeline requests, but we're not sure if
> > they're really using nginx, yet.  With the problems you've described,
> > it doesn't sound like they are, or they're using some broken version
> > of it.
> 
> Umm... after reading the log, I think they are using another [router]
> (in their terminology) in front of [nginx]. So it might be an issue in
> their [router], I am not sure...
> 
> I'll keep you posted if you're interested. Thanks for all your help.

Yes, please.

> p.s. They might be using https://github.com/mochi/mochiweb
> too, since if there's an issue, it would return this header:
> Server: MochiWeb/1.0 (Any of you quaids got a smint?)
> 
> p.s.2. Perhaps I could even give you access to one of the apps.
> I'll need to ask the owner though. Let me know if you're interested,
> otherwise you could simply ignore this. I know it's too much to ask.

I think I can just walk you through the troubleshooting process
here.  (I'd rather avoid the potential for personal liability if
given access to a server :)
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Eric Wong]

Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> So the _actual_ Content-Length that's sent is zero?
> 
> Rainbows! should drop a connection if an exception is raised while
> sending the response body, perhaps the heroku router is confused
> by that?

Instead of trying "keepalive_timeout 0", you can also try using the
following middleware.  This only works for Content-Length responses
(which seems to be your case)

Of course, like all middleware, this makes your stack deeper, so
maybe it's not good, either...

Furthermore, this code is totally untested, it may not even compile,
but I hope you get the idea and fix trivial errors :)

class PadOnError
  def initialize(app)
    @app = app
  end

  def call(env)
    status, headers, body = @app.call(env)
    headers = Rack::Utils::HeaderHash.new(headers)
    clen = headers["Content-Length"]
    if clen
      [ status, headers, Padder.new(clen.to_i, body) ]
    else
      # TODO: write your own Transfer-Encoding: chunked padder :)
      # It would need to parse the chunked encoding and keep track
      # of each chunk length, pad any unfinished chunk, and ensure
      # termination with "0\r\n\r\n"
      [ status, headers, body ]
    end
  end

  class Padder
    def initialize(clen, body)
      @clen = clen
      @body = body
      @bytes_sent = 0
    end

    def each
      @body.each do |chunk|
        @bytes_sent += chunk.bytesize

        # if this raises due to a socket error... we'll just end up
        # raising (again) below
        yield chunk
      end
    ensure
      # pad the response in case @body.each raised an error while
      # generating.
      left_to_send = clen - @bytes_sent

      # XXX assuming you're sending HTTP responses that fit comfortably in
      # memory, this can OOM if you're sending multi-mega/gigabyte files
      # like most of my applications :P
      if left_to_send > 0
        padding = " " * left_to_send
        yield padding
      end
    end

    def close
      @body.close if @body.respond_to?(:close)
    end
  end
end

----------------------- config.ru ------------------------
use PadOnError
run YourApp.new
---------------------------------------------------------
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Lin Jen-Shin (godfat)]

On Sat, Sep 29, 2012 at 3:24 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:

Ouch, it's already been one month? Sorry that I guess I don't have time to
investigate this further recently. The conference is coming soon, I need to
prepare the talk and I also need to work on other things. I promise I would
come back to give some conclusions after things are settled. (e.g. after
the talk is done or if it is well prepared)

One thing might worth some words though. I just realized that in Heroku's
new architecture, the cedar stack, actually all requests won't be passed
through Nginx! But there are exceptions. If we don't CNAME our domain
to Heroku, but use A record to point to their IPs, then requests *would*
be passed through both Nginx and Varnish.

I guess that explains why sometimes it made me feel this way and that
way, because I thought they were the same, and I don't remember which
is which. The end result is that I was very confused. I'll review my posts
and clues with this new clue next time when I do the investigation again...
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Lin Jen-Shin (godfat)]

On Sat, Sep 29, 2012 at 3:24 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
>> So the _actual_ Content-Length that's sent is zero?
>>
>> Rainbows! should drop a connection if an exception is raised while
>> sending the response body, perhaps the heroku router is confused
>> by that?
>
> Instead of trying "keepalive_timeout 0", you can also try using the
> following middleware.  This only works for Content-Length responses
> (which seems to be your case)
>
> Of course, like all middleware, this makes your stack deeper, so
> maybe it's not good, either...
>
> Furthermore, this code is totally untested, it may not even compile,
> but I hope you get the idea and fix trivial errors :)
[...]

Thank you again for the helps.  Since the conference (http://rubyconf.tw/2012)
had ended, I got much more time to work with this now.

I didn't try this middleware because now I feel it might be the problem
in their application, given that it used a ton of gems which might have
some thread-safety issues.  On the other hand, the person who asked
me about this telling me that it's fine to run Rainbows! with EventMachine,
but not with ThreadPool.  So it is very likely that it's their issues.

As for the issue on my own last time, since it only happened on an
application which we no longer run, and Rainbows! (Zbatery) runs
totally fine on our true production side, I would guess it's a Heroku
issue at that time and it might have been fixed already?

As a result, I want to ignore this issue from now on, and move forward.

I'll send a patch which contains the concurrency models we use on the
production site later in a new thread.  Also I might start trying to implement
a new concurrency model based on celluloid-io which I believe the API
would be similar to cool.io, thus might not be hard to implement.
Hope we could get there soon :)
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: negative timeout in Rainbows::Fiber::Base

[Eric Wong]

"Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
> Thank you again for the helps.  Since the conference (http://rubyconf.tw/2012)
> had ended, I got much more time to work with this now.

No problem!

> I didn't try this middleware because now I feel it might be the problem
> in their application, given that it used a ton of gems which might have
> some thread-safety issues.  On the other hand, the person who asked
> me about this telling me that it's fine to run Rainbows! with EventMachine,
> but not with ThreadPool.  So it is very likely that it's their issues.

They probably a thread-safety issue.  I've used ThreadPool/Spawn heavily
myself (much more than EM) with great success.  (I always verify any
gems I use are thread-safe)

> As for the issue on my own last time, since it only happened on an
> application which we no longer run, and Rainbows! (Zbatery) runs
> totally fine on our true production side, I would guess it's a Heroku
> issue at that time and it might have been fixed already?

*shrug*  Maybe check with Heroku changelogs/news releases if they
publish those.

> As a result, I want to ignore this issue from now on, and move forward.
> 
> I'll send a patch which contains the concurrency models we use on the
> production site later in a new thread.  Also I might start trying to implement
> a new concurrency model based on celluloid-io which I believe the API
> would be similar to cool.io, thus might not be hard to implement.
> Hope we could get there soon :)

Alright, looking forward to it.  I was hoping to look at celluloid-io
myself last week, but got sidetracked by something else
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying