notes for streaming responses with Rails 3.1

notes for streaming responses with Rails 3.1

[Eric Wong]

(I'll probably turn this into RDoc and put it on the website somewhere
after some editing, comments greatly appreciated as always)

Since there's increased interest with streaming responses with Rails 3.1
around the corner and Rainbows! was always designed with streaming
responses in mind, I'll suggest things I believe to be useful for use
with Rails 3.1 streaming.

Rails 3.1 streaming is primarily to reduce latency to clients even with
fast responses.   You want to be sending data to clients as quickly as
possible and buffer as little as possible on the server side.

This focuses on Ruby 1.9 since it's required for Rails 3.1 streaming
and Linux/NPTL since that's what I have the most experience on.

== Thread-safety

First off, enable thread-safety in Rails, most options here involve
threading in some way and Ruby threading (especially in 1.9.2+) isn't
nearly as bad as some people think[1].

== Rainbows! concurrency options

=== ThreadPool/ThreadSpawn

Both of these work pretty well in a standalone configuration (without
nginx in front).  Your entire app and all its dependencies *must* be
thread-safe.

=== WriterThreadSpawn/WriterThreadPool

These /may/ offer more consistent performance if your application (but
not rendering) is CPU-intensive as they can better balance requests with
multiple cores and more worker_processes.

Both of these require nginx in front to be effective against slow client
requests.  However, nginx users should use "proxy_buffering off" in their
nginx config file or "X-Accel-Buffering: no" in their Rack responses to
stream responses.  This allows parts of the response to get to the
client as quickly as possible.

All code accessed in your Rails views (helpers, possibly
models/controllers) must be thread-safe, but some controller-only code
does not require thread-safety.  If your middleware wraps responses,
those must be thread-safe, too.  It might just be as easy to make your
entire application thread-safe and worry less later on :)


== Hardware/Environment

=== Memory

64-bit is recommended as you can run more native threads with Ruby 1.9.
All you need is addressable (not physical) memory to run more threads
with NPTL in Linux.  Ruby 1.9.2 requires 512KB of addressable stack per
thread, but hopefully 1.9.3 can will have a smaller stack size[3].
Most threads are likely to need <= 64KB of physical stack memory.

When choosing hardware, look for a fast memory/bus and large CPU caches.

=== Context-switching costs

I suspect this is negligible compared to the overhead of a high-level
language like Ruby in the first place.  Feedback/measurements with
actual Ruby/Rails application processes/threads would be greatly
appreciated.

Maybe play around with `schedtool` or `taskset` in Linux if you feel
it's a problem.  Perhaps pinning all native threads within a process to
the same CPU core is the best option to reduce contention for the GVL,
but the kernel scheduler may already be smart enough to do that.

More analysis is definitely needed in this area....

=== worker_connections

This is typically the maximum number of threads you'll spawn.

worker_connections * worker_processes is the total number of connections
you can have on Rainbows!


=== worker_processes

It's safe to increase worker_processes as long as you have *physical*
RAM for them.  Having more worker_processes will amortize the cost of
running GC and increase the amount of work multiple CPUs can do given
the GVL.

Whatever you do, make sure processes don't get swapped out to disk.


More coming later...



[1] - if you use any Free/Open Source C extensions[2] that don't work
properly with 1.9 native threads, feel welcome to ask for my help
publically or privately via email.

[2] - I'll only help if the *entire* stack for the extensions is
Free/OSS.  It cannot be a binding to a non-Free/OSS library which I
cannot look at/hack the source for.

[3] http://redmine.ruby-lang.org/issues/4614

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: notes for streaming responses with Rails 3.1

[Eric Wong]

Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> == Rainbows! concurrency options
> 
> === ThreadPool/ThreadSpawn

==== Linux-only

I also added XEpollThreadSpawn to rainbows.git which basically uses
epoll to maintain idle keepalive connections but is otherwise the same
as ThreadSpawn for "hot" connections (including streaming "rack.input"
and also streaming respond_body#each with no enforced userspace
buffering).

==== Ruby 1.9 + Linux-only

The key advantage of the XEpoll* relying on epoll (and not more portable
things) is we can have one (native) thread stuck in epoll_wait() can be
woken up by another (native) thread adding to the epoll set with
epoll_ctl().  We combine this with the ability to do real blocking
accept() calls under 1.9 with "wake one" behavior[1] in the Linux
kernel, we can expect less contention and fairer load balancing across
multiple acceptors/processes.

The behavior I'm describing with multi-threading epoll isn't possible
with poll() or select().  I'm unsure about kqueue, nor do I know if
*BSDs implement "wake one" behavior with blocking accept() in the
kernel...

[1] http://www.citi.umich.edu/projects/linux-scalability/reports/accept.html

-- 
Eric Wong
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying