c10k paradigm

c10k paradigm

[Alexandre Riveira]

What is A better paradigm for C10K paradigm


 From what I'm seeing are the best:

EventMachine (less memory and cpu usage)
XEpollThreadPool (more memory and cpu usage but there are problems when 
blocking io, for example database

I appreciate the comments from everyone.
Tanks


Alexandre Riveira
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm

[Eric Wong]

Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> What is A better paradigm for C10K paradigm
> 
> 
> From what I'm seeing are the best:
> 
> EventMachine (less memory and cpu usage)

Not all DB adapters are fully non-blocking.  If you want to do uncached
filesystem I/O, you'll either have to use threads or block, too.

> XEpollThreadPool (more memory and cpu usage but there are problems
> when blocking io, for example database

Maybe not CPU usage, but probably more memory.  But this is probably the
most compatible with existing apps/gems and Ruby stdlib (assuming you
use Linux).

Fwiw, I usually go with this one (or something close[1]) when uncached
disk I/O is a requirement.

If you're not on Linux, (nginx + (portable)ThreadPool) should get you
close to XEpollThreadPool.

Likewise for (nginx + (portable)ThreadSpawn) and XEpollThreadSpawn


[1] http://bogomips.org/cmogstored/queues.txt
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm

[Eric Wong]

Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> Em 15-04-2013 18:30, Eric Wong escreveu:
> >Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> >>What is A better paradigm for C10K paradigm
> >>
> >>
> >> From what I'm seeing are the best:
> >>
> >>EventMachine (less memory and cpu usage)
> >Not all DB adapters are fully non-blocking.  If you want to do uncached
> >filesystem I/O, you'll either have to use threads or block, too.
> 
> would be possible to combine being EventMachine and Threads,
> If the thread is blocked it would to anotherexample:
> 
>  use :EventMachine, :pool_size => 50

(top-posting corrected, Cc: to list re-added)

Yes, you would use EventMachine.threadpool_size = 50, though, with
Rainbows::EventMachine::TryDefer

http://rainbows.rubyforge.org/Rainbows/EventMachine/TryDefer.html

I forgot about this, not sure if it's used much, but "app.deferred?" is
an ad-hoc extension which Thin also supports
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm

[Alexandre Riveira]

Em 15-04-2013 19:34, Eric Wong escreveu:
Hi Eric,


* EventMachine (less memory and cpu usage)
I did a test  using a  core  with  ab-n  3000-c  1000 
http://127.0.0.1:3000/manager and  rainbows  with 3  workers,  
EventMachine  then 25%  cpu free  while  XEpollThreadPool  left only  
5%  free.

*EventMachine.threadpool_size = 50
I found  this code  and run  perfectly.  But  he has  a kind of lock  
slowness  as if using  XEpollThreadPool  but memory consumption  was 
lower.  Wonder  you can put  some  of the  rails Controllers  work  by  
EventMachine  without using  threads  while others  controlles  using  
threads.  Explain  the need.

The application  that  runs on  rainbows  +  rails  is an Enterprise 
Resource Planning  ERP  to  support  e-commerce.  That same  ERP and  
e-commerce  rotate in the same  rainbows.  But  the conclusion  that 
ERP  is  best  run  with  XEpollThreadPool  (or EventMachine + pool 
threads),while the  site was  high competition  C10K  is  best  run  
with  EventMachine.A detail about e-commerce  is that  as much as 
possible  the pages  are cached.


Could it be that  the controllers  of the  rails  would run without  
party  pool  threads  only with  EventMachine  (C10K) while others  
would use the  controller  with  EventMachine thread pool  (erp)?
I appreciate the help!


Alexandre Riveira

Em 15-04-2013 19:34, Eric Wong escreveu:
> Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
>> Em 15-04-2013 18:30, Eric Wong escreveu:
>>> Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
>>>> What is A better paradigm for C10K paradigm
>>>>
>>>>
>>>>  From what I'm seeing are the best:
>>>>
>>>> EventMachine (less memory and cpu usage)
>>> Not all DB adapters are fully non-blocking.  If you want to do uncached
>>> filesystem I/O, you'll either have to use threads or block, too.
>> would be possible to combine being EventMachine and Threads,
>> If the thread is blocked it would to anotherexample:
>>
>>   use :EventMachine, :pool_size => 50
> (top-posting corrected, Cc: to list re-added)
>
> Yes, you would use EventMachine.threadpool_size = 50, though, with
> Rainbows::EventMachine::TryDefer
>
> http://rainbows.rubyforge.org/Rainbows/EventMachine/TryDefer.html
>
> I forgot about this, not sure if it's used much, but "app.deferred?" is
> an ad-hoc extension which Thin also supports
>

_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm

[Alexandre Riveira]

* EventMachine (less memory and cpu usage)
I did a test  using a  core  with  ab-n  3000-c  1000 
http://127.0.0.1:3000/manager and  rainbows  with 3  workers,  
EventMachine  then 25%  cpu free while  XEpollThreadPool  left only  5%  
free.

*EventMachine.threadpool_size = 50
I found  this code  and run  perfectly.  But  he has  a kind of lock  
slowness  as if using  XEpollThreadPool  but memory consumption  was 
lower.  Wonder  you can put  some  of the  rails Controllers  work  by  
EventMachine  without using  threads  while others  controlles  using  
threads.  Explain  the need.

The application  that  runs on  rainbows  +  rails  is an Enterprise 
Resource Planning  ERP  to  support  e-commerce.  That same  ERP and  
e-commerce  rotate in the same  rainbows.  But  the conclusion  that 
ERP  is  best  run  with  XEpollThreadPool  (or EventMachine + pool 
threads),while the  site was  high competition  C10K  is  best  run  
with  EventMachine.A detail about e-commerce  is that  as much as 
possible  the pages  are cached.


Could it be that  the controllers  of the  rails  would run without  
party  pool  threads  only with  EventMachine  (C10K) while others  
would use the  controller  with  EventMachine thread pool  (erp)?
I appreciate the help!


Alexandre Riveira

Em 15-04-2013 19:34, Eric Wong escreveu:
> Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
>> Em 15-04-2013 18:30, Eric Wong escreveu:
>>> Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
>>>> What is A better paradigm for C10K paradigm
>>>>
>>>>
>>>>  From what I'm seeing are the best:
>>>>
>>>> EventMachine (less memory and cpu usage)
>>> Not all DB adapters are fully non-blocking.  If you want to do uncached
>>> filesystem I/O, you'll either have to use threads or block, too.
>> would be possible to combine being EventMachine and Threads,
>> If the thread is blocked it would to anotherexample:
>>
>>   use :EventMachine, :pool_size => 50
> (top-posting corrected, Cc: to list re-added)
>
> Yes, you would use EventMachine.threadpool_size = 50, though, with
> Rainbows::EventMachine::TryDefer
>
> http://rainbows.rubyforge.org/Rainbows/EventMachine/TryDefer.html
>
> I forgot about this, not sure if it's used much, but "app.deferred?" is
> an ad-hoc extension which Thin also supports
>

_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm

[Alexandre Riveira]

> "lock slowness" - which version of Ruby is this?
ruby 1.9.3p385 (2013-02-06 revision 39114) [i686-linux]
>
> Did you modify your app to use app.deferred? + TryDefer as I pointed
> you to in the other message?
>
>> Could it be that  the controllers  of the  rails  would run without
>> party  pool  threads  only with  EventMachine  (C10K) while others
>> would use the  controller  with  EventMachine thread pool  (erp)?
> That should allow some rails controllers to use threads
> (app.deferred? => true) while others do not use threads
> (app.deferred? => false).  Keep in mind this is not a very common
> configuration, so not many people have experience with it in
> production.

Your suggestion seems to be better, but I'm sorry I did not understand 
where I put this method, the controller would be as below?

class HomeController < ApplicationController
   def deferred?
     true
   end
end

Tanks


Alexandre Riveira
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm

[Alexandre Riveira]

This is valid for ruby on rails 2.3?

Em 16-04-2013 18:19, Mark J. Titorenko escreveu:
> On 16 Apr 2013, at 18:24, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
>> Did you modify your app to use app.deferred? + TryDefer as I pointed
>> you to in the other message?
>>
>>> Could it be that  the controllers  of the  rails  would run without
>>> party  pool  threads  only with  EventMachine  (C10K) while others
>>> would use the  controller  with  EventMachine thread pool  (erp)?
>> That should allow some rails controllers to use threads
>> (app.deferred? => true) while others do not use threads
>> (app.deferred? => false).  Keep in mind this is not a very common
>> configuration, so not many people have experience with it in
>> production.
> In case it's of any use I am using #deferred? and TryDefer in my Rails app.  I have it set up such that all controller requests are deferred (ie. use threads) apart from those which are, by their nature, asynchronous - eg. I am using Faye mounted at '/ext/bayeux' within my application which already uses the [-1, {}, []] technique to provide an async response.  I have #deferred set up by including a module in my Rails::Application instance.
>
> In application.rb:
>
> module App::Application < Rails::Application
>    require 'extensions/deferred_application'
>    include Extensions::DeferredApplication
> end
>
> In lib/extensions/deferred_application.rb:
>
> module Extensions
>    module DeferredApplication
>      BAYEUX_REGEX = %r(^/ext/bayeux(/.*)?$).freeze
>
>      def deferred?(env)
>        !(env['REQUEST_PATH'] =~ BAYEUX_REGEX)
>      end
>    end
> end
>
> HTH!
>
> Cheers,
>
> Mark.
>
> _______________________________________________
> Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
> http://rubyforge.org/mailman/listinfo/rainbows-talk
> Do not quote signatures (like this one) or top post when replying
>

_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm

[Alexandre Riveira]

In file ems/rainbows-4.5.0/lib/rainbows/event_machine.rb line 74: 
server.app.respond_to?(:deferred?)
server.app is a Rails::Rack::LogTailer class.

should not be ActionController::Dispatcher.new or like ???


Tanks

Alexandre Riveira


Em 16-04-2013 18:35, Eric Wong escreveu:
> Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
>>> "lock slowness" - which version of Ruby is this?
>> ruby 1.9.3p385 (2013-02-06 revision 39114) [i686-linux]
> OK, I think you were just unable to take advantage of threading in EM
> via app.deferred?
>
> I think you also need to enable threading in Rails, too.
>
>>> Did you modify your app to use app.deferred? + TryDefer as I pointed
>>> you to in the other message?
>>>
>>>> Could it be that  the controllers  of the  rails  would run without
>>>> party  pool  threads  only with  EventMachine  (C10K) while others
>>>> would use the  controller  with  EventMachine thread pool  (erp)?
>>> That should allow some rails controllers to use threads
>>> (app.deferred? => true) while others do not use threads
>>> (app.deferred? => false).  Keep in mind this is not a very common
>>> configuration, so not many people have experience with it in
>>> production.
>> Your suggestion seems to be better, but I'm sorry I did not
>> understand where I put this method, the controller would be as
>> below?
>>
>> class HomeController < ApplicationController
>>    def deferred?
>>      true
>>    end
>> end
> Probably not, this needs to be visible at the Rack layer, not Rails.
>
> See what Mark just said.  Also, t/app_deferred.ru in the rainbows
> source[1].  There's also a link to Ezra's old post about it in the
> TryDefer documentation.
>
> [1] - git clone git://bogomips.org/rainbows && $EDITOR t/app_deferred.ru
> _______________________________________________
> Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
> http://rubyforge.org/mailman/listinfo/rainbows-talk
> Do not quote signatures (like this one) or top post when replying
>

_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm

[Eric Wong]

Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> Em 15-04-2013 19:34, Eric Wong escreveu:
> >Yes, you would use EventMachine.threadpool_size = 50, though, with
> >Rainbows::EventMachine::TryDefer
> >
> >http://rainbows.rubyforge.org/Rainbows/EventMachine/TryDefer.html
> >
> >I forgot about this, not sure if it's used much, but "app.deferred?" is
> >an ad-hoc extension which Thin also supports
> 
> * EventMachine (less memory and cpu usage)
> I did a test  using a  core  with  ab-n  3000-c  1000
> http://127.0.0.1:3000/manager and  rainbows  with 3  workers,
> EventMachine  then 25%  cpu free while  XEpollThreadPool  left only
> 5%  free.

With Matz Ruby, XEpollThreadPool is only a win if much of your
code/extensions releases the GVL (or if you don't feel like porting
your code to take advantage of EM).

> *EventMachine.threadpool_size = 50
> I found  this code  and run  perfectly.  But  he has  a kind of lock
> slowness  as if using  XEpollThreadPool  but memory consumption  was
> lower.  Wonder  you can put  some  of the  rails Controllers  work
> by  EventMachine  without using  threads  while others  controlles
> using  threads.  Explain  the need.

"lock slowness" - which version of Ruby is this?

Did you modify your app to use app.deferred? + TryDefer as I pointed
you to in the other message?

> Could it be that  the controllers  of the  rails  would run without
> party  pool  threads  only with  EventMachine  (C10K) while others
> would use the  controller  with  EventMachine thread pool  (erp)?

That should allow some rails controllers to use threads
(app.deferred? => true) while others do not use threads
(app.deferred? => false).  Keep in mind this is not a very common
configuration, so not many people have experience with it in
production.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm

[Mark J. Titorenko]

On 16 Apr 2013, at 18:24, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> Did you modify your app to use app.deferred? + TryDefer as I pointed
> you to in the other message?
> 
>> Could it be that  the controllers  of the  rails  would run without
>> party  pool  threads  only with  EventMachine  (C10K) while others
>> would use the  controller  with  EventMachine thread pool  (erp)?
> 
> That should allow some rails controllers to use threads
> (app.deferred? => true) while others do not use threads
> (app.deferred? => false).  Keep in mind this is not a very common
> configuration, so not many people have experience with it in
> production.

In case it's of any use I am using #deferred? and TryDefer in my Rails app.  I have it set up such that all controller requests are deferred (ie. use threads) apart from those which are, by their nature, asynchronous - eg. I am using Faye mounted at '/ext/bayeux' within my application which already uses the [-1, {}, []] technique to provide an async response.  I have #deferred set up by including a module in my Rails::Application instance.

In application.rb:

module App::Application < Rails::Application
  require 'extensions/deferred_application'
  include Extensions::DeferredApplication
end

In lib/extensions/deferred_application.rb:

module Extensions
  module DeferredApplication
    BAYEUX_REGEX = %r(^/ext/bayeux(/.*)?$).freeze

    def deferred?(env)
      !(env['REQUEST_PATH'] =~ BAYEUX_REGEX)
    end
  end
end

HTH!

Cheers,

Mark.

_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm

[Eric Wong]

Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> >"lock slowness" - which version of Ruby is this?
> ruby 1.9.3p385 (2013-02-06 revision 39114) [i686-linux]

OK, I think you were just unable to take advantage of threading in EM
via app.deferred?

I think you also need to enable threading in Rails, too.

> >Did you modify your app to use app.deferred? + TryDefer as I pointed
> >you to in the other message?
> >
> >>Could it be that  the controllers  of the  rails  would run without
> >>party  pool  threads  only with  EventMachine  (C10K) while others
> >>would use the  controller  with  EventMachine thread pool  (erp)?
> >That should allow some rails controllers to use threads
> >(app.deferred? => true) while others do not use threads
> >(app.deferred? => false).  Keep in mind this is not a very common
> >configuration, so not many people have experience with it in
> >production.
> 
> Your suggestion seems to be better, but I'm sorry I did not
> understand where I put this method, the controller would be as
> below?
> 
> class HomeController < ApplicationController
>   def deferred?
>     true
>   end
> end

Probably not, this needs to be visible at the Rack layer, not Rails.

See what Mark just said.  Also, t/app_deferred.ru in the rainbows
source[1].  There's also a link to Ezra's old post about it in the
TryDefer documentation.

[1] - git clone git://bogomips.org/rainbows && $EDITOR t/app_deferred.ru
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm

[Mark J. Titorenko]

On 16 Apr 2013, at 16:45, Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> This is valid for ruby on rails 2.3?

I'm afraid I don't know if it works under Rails 2.3.x - YMMV.  I am using Rails 3.2.13.

Cheers,

Mark.

_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm [SOLVED]

[Alexandre Riveira]

Tanks for all helpers. Erick, this solutions CheckDefered is pefect !And 
run in production !

1) rack middleware
class CheckDeferred

   def initialize(app)
     @app = app
   end

   def call(env)
     @app.call(env)
   end

   def deferred?(env)
     case env["PATH_INFO"]
       when /\/ecommer.*/
         false
       else
         true # erp run by thread
     end
   end
end


2) edit your config.ru
require File.expand_path "config/environment"

use CheckDeferred #first if not function
use Rails::Rack::LogTailer
use Rails::Rack::Static
run ActionController::Dispatcher.new


Alexandre Riveira



Em 16-04-2013 20:48, Eric Wong escreveu:
> Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
>> Em 16-04-2013 18:35, Eric Wong escreveu:
>>> See what Mark just said.  Also, t/app_deferred.ru in the rainbows
>>> source[1].  There's also a link to Ezra's old post about it in the
>>> TryDefer documentation.
>>>
>>> [1] - git clone git://bogomips.org/rainbows && $EDITOR t/app_deferred.ru
>> In file ems/rainbows-4.5.0/lib/rainbows/event_machine.rb line 74:
>> server.app.respond_to?(:deferred?)
>> server.app is a Rails::Rack::LogTailer class.
>>
>> should not be ActionController::Dispatcher.new or like ???
> You should probably write a new middleware around/outside of your
> normal Rack layer.  Something like this:
>
> class CheckDeferred < Struct.new(:app)
>    def deferred?(env)
>      case env["PATH_INFO"]
>      when %r{/manager/},
>           %r{/foo/bar},
> 	 ... # whatever other paths you want multithreaded
>        true
>      else
>        false
>      end
>    end
> end
>
> ----------- config.ru --------------
> use CheckDeferred # needs to be the outermost layer of middleware
>
> use ...
> run Application.new
> ------------------------------------
> _______________________________________________
> Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
> http://rubyforge.org/mailman/listinfo/rainbows-talk
> Do not quote signatures (like this one) or top post when replying
>

_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: c10k paradigm

[Eric Wong]

Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> Em 16-04-2013 18:35, Eric Wong escreveu:
> >See what Mark just said.  Also, t/app_deferred.ru in the rainbows
> >source[1].  There's also a link to Ezra's old post about it in the
> >TryDefer documentation.
> >
> >[1] - git clone git://bogomips.org/rainbows && $EDITOR t/app_deferred.ru
> 
> In file ems/rainbows-4.5.0/lib/rainbows/event_machine.rb line 74:
> server.app.respond_to?(:deferred?)
> server.app is a Rails::Rack::LogTailer class.
> 
> should not be ActionController::Dispatcher.new or like ???

You should probably write a new middleware around/outside of your
normal Rack layer.  Something like this:

class CheckDeferred < Struct.new(:app)
  def deferred?(env)
    case env["PATH_INFO"]
    when %r{/manager/},
         %r{/foo/bar},
	 ... # whatever other paths you want multithreaded
      true
    else
      false
    end
  end
end

----------- config.ru --------------
use CheckDeferred # needs to be the outermost layer of middleware

use ...
run Application.new
------------------------------------
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying