Re: rainbows for sleepy/lethargic apps

Re: rainbows for sleepy/lethargic apps

[Eric Wong]

Dylan Stamat <dstamat@elctech.com> wrote:
> I'm working on a project that has very bad application performance,
> specifically, tons of long running requests due to view and
> (specifically) database contention.
> 
> While I haven't tested our Unicorn setup under load yet, tweak backlog
> counts, etc... I'm wondering if I'd be better off giving Rainbows! a
> shot?  In the Rainbows! AppPool diagram, I'm having a hard time
> understanding the N:P relationship.  When the P threshold is met, what
> happens to N (the client)?

Hi Dylan,

With AppPool, request headers are fully parsed and end up getting queued
up in userspace.  No request bodies are read (since AppPool is only
compatible with the Thread-based concurrency models at the moment).

Rainbows is designed for apps that are intentionally/unavoidably sleepy.
Since I imagine the database is within your control, I'd optimize that
first and fall back to using Unicorn with a higher :backlog.

Since you're hitting database contention, throwing more concurrency at
the database with Rainbows! is probably the wrong thing to do... If your
database is just on a high latency network for whatever reason, then
maybe Rainbows! with Neverblock drivers can help.  Otherwise if your
database is bogged down because of memory/CPU/disk, then you probably
want fewer things hitting it at once.  Can you shard, cache or otherwise
offload DB requests?

With slow views, Unicorn is pretty effective at using all available
cores already with multiple worker processes.  Even with a Ruby VM with
non-crippled native threads (Rubinius maybe, not MRI 1.9.1) I doubt
it'll help with CPU/memory-bound rendering performance.

> Also, with both Unicorn/Rainbows!, is there a explicit timeout number
> set on the clients (and how does that differ between
> Unicorn/Rainbows!)?  For instance, in our Nginx config, I have
> proxy_buffering turned on, with proxy_read_timeout and
> proxy_send_timeout's, set pretty high.

Based on your nginx config, the easiest thing would probably be to set a
high :backlog for now and then work on tuning/optimizing/avoiding your
database ASAP.

Views are harder to optimize, Unicorn itself always uses as much
resources as the OS can give it to render.  A lot of slow views I've
seen hit a lot of memory allocation, so check out tcmalloc (included
with REE) or use 1.9 (possibly with tcmalloc, too).  1.9 has some good
optimizations for allocating shorter strings (should be more effective
on 64-bit) and small hashes/arrays, too.

My personal style is also to use destructive methods as much as possible
(concat/<< vs +/+=, map! vs map, gsub! vs gsub, tr! vs tr, etc...).
Avoid creating lots of OpenStruct, too, excessive use of
define_method/metadef allocates a lot of short-lived objects and
thrashes memory.


The timeout in Unicorn affects the entire request (including all I/O).

The timeout in Rainbows only kicks in when the process is completely
deadlocked.

Since apparently lots of "normal" HTTP clients have ridiculous keepalive
times, Rainbows! will be getting a separate keepalive_timeout directive
soon that only affects reading HTTP headers.  I don't plan enforcing a
timeout when processing request bodies or app responses (those can
probably be done on a more controlled manner in the app/middleware).

> Lastly... ¡You rock Eric!

Thanks, but much of the credit goes to random folks all over who
have helped with this (and projects leading up to this), too :>

-- 
Eric Wong