Fwd: Support for Soft Timeout in Unicorn

Fwd: Support for Soft Timeout in Unicorn

[Eric Wong]

Hi,

HTML attachments are wasteful and thus rejected from the mailing list.
On the other hand, it actually helps to include the patch itself
(inline) so it's readable without a (human) context switch :)

----- Forwarded message from Pierre Baillet <oct@fotonauts.com> -----

Subject: Fwd: Support for Soft Timeout in Unicorn
From: Pierre Baillet <oct@fotonauts.com>
To: unicorn@bogomips.org

Hi,

Just tried to send that through the ml, but it seems something went wrong...

Cheers,
-- 
Pierre.

---------- Forwarded message ----------
From: <mongrel-unicorn-owner@rubyforge.org>
Date: Thu, Jun 3, 2010 at 2:40 PM
Subject: Support for Soft Timeout in Unicorn
To: oct@fotonauts.com


The message's content type was not explicitly allowed



---------- Forwarded message ----------
From: Pierre Baillet <oct@fotonauts.com>
To: mongrel-unicorn@rubyforge.org
Date: Thu, 3 Jun 2010 14:16:01 +0200
Subject: Support for Soft Timeout in Unicorn
Hi,

We use Unicorn at fotopedia since yesterday in production. We switched from
Passenger due to an issue in the way Passenger was handling some error in
our main application. Things run very well on Unicorn.

We have also modified Unicorn to handle a soft timeout for its workers. The
Unicorn timeout was killing the workers without any chance for us to catch
the Rails stack trace and identify the issue. I've forked and branched
Unicorn from github and added a soft_timeout configuration value that is
used for long running workers.

The workers now handle SIGABRT and will raise an exception. This will crash
the Rails application if it can be crashed and force the framework to dump
the stack trace in the logs. Let me know if this might be useful for other
people and, why not, integrate that in the main Unicorn code !

http://github.com/octplane/unicorn/tree/SOFT_TIMEOUT_SUPPORT

Cheers,
-- 
Pierre Baillet <oct@fotonauts.com>
http://www.fotopedia.com/




-- 
Pierre Baillet <oct@fotonauts.com>
http://www.fotopedia.com/

----- End forwarded message -----
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Support for Soft Timeout in Unicorn

[Pierre Baillet]

Hello,

On Thu, Jun 3, 2010 at 7:37 PM, Eric Wong <normalperson@yhbt.net> wrote:
>
> Hi,
>
> HTML attachments are wasteful and thus rejected from the mailing list.
> On the other hand, it actually helps to include the patch itself
> (inline) so it's readable without a (human) context switch :)

Indeed,
sorry for the HTML attachment, I have no idea where it comes from. As
for the patch, here you are. This is really just a way to handle
SIGABRT in a specific way in the worker and allow proper termination
of the application. Note the FIXME comment I added in the
murder_lazy_workers method. If any worker blocks while all the others
are idle for a _timeout_ period of time, they will all be killed
anyway. The consequence of that is that Unicorn will restart all its
workers if traffic is very low on the server.

diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index a363014..855f26a 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -84,7 +84,7 @@ module Unicorn
   # Listener sockets are started in the master process and shared with
   # forked worker children.

-  class HttpServer < Struct.new(:app, :timeout, :worker_processes,
+  class HttpServer < Struct.new(:app, :soft_timeout, :timeout,
:worker_processes,
                                 :before_fork, :after_fork, :before_exec,
                                 :logger, :pid, :listener_opts, :preload_app,
                                 :reexec_pid, :orig_app, :init_listeners,
@@ -393,7 +393,7 @@ module Unicorn
           when nil
             # avoid murdering workers after our master process (or the
             # machine) comes out of suspend/hibernation
-            if (last_check + timeout) >= (last_check = Time.now)
+            if (last_check + soft_timeout) >= (last_check = Time.now)
               murder_lazy_workers
             else
               # wait for workers to wakeup on suspend
@@ -581,10 +581,20 @@ module Unicorn
         stat = worker.tmp.stat
         # skip workers that disable fchmod or have never fchmod-ed
         stat.mode == 0100600 and next
-        (diff = (Time.now - stat.ctime)) <= timeout and next
-        logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \
+        # FIXME: if the worker has not been working for soft_timeout,
it will be
+        # killed even if it is not blocking
+        (diff = (Time.now - stat.ctime)) <= soft_timeout and
+          diff <= timeout and next
+        # lazy since less than timeout, attempt soft kill
+        if diff < timeout
+          logger.error "worker=#{worker.nr} PID:#{wpid} soft timeout " \
+                     "(#{diff}s > #{soft_timeout}s), killing softly"
+          kill_worker(:ABRT, wpid)
+        else
+         logger.error "worker=#{worker.nr} PID:#{wpid} hard timeout " \
:
diff --git a/lib/unicorn.rb b/lib/unicorn.rb
index a363014..855f26a 100644
--- a/lib/unicorn.rb
+++ b/lib/unicorn.rb
@@ -84,7 +84,7 @@ module Unicorn
   # Listener sockets are started in the master process and shared with
   # forked worker children.

-  class HttpServer < Struct.new(:app, :timeout, :worker_processes,
+  class HttpServer < Struct.new(:app, :soft_timeout, :timeout,
:worker_processes,
                                 :before_fork, :after_fork, :before_exec,
                                 :logger, :pid, :listener_opts, :preload_app,
                                 :reexec_pid, :orig_app, :init_listeners,
@@ -393,7 +393,7 @@ module Unicorn
           when nil
             # avoid murdering workers after our master process (or the
             # machine) comes out of suspend/hibernation
-            if (last_check + timeout) >= (last_check = Time.now)
+            if (last_check + soft_timeout) >= (last_check = Time.now)
               murder_lazy_workers
             else
               # wait for workers to wakeup on suspend
@@ -581,10 +581,20 @@ module Unicorn
         stat = worker.tmp.stat
         # skip workers that disable fchmod or have never fchmod-ed
         stat.mode == 0100600 and next
-        (diff = (Time.now - stat.ctime)) <= timeout and next
-        logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \
+        # FIXME: if the worker has not been working for soft_timeout,
it will be
+        # killed even if it is not blocking
+        (diff = (Time.now - stat.ctime)) <= soft_timeout and
+          diff <= timeout and next
+        # lazy since less than timeout, attempt soft kill
+        if diff < timeout
+          logger.error "worker=#{worker.nr} PID:#{wpid} soft timeout " \
+                     "(#{diff}s > #{soft_timeout}s), killing softly"
+          kill_worker(:ABRT, wpid)
+        else
+         logger.error "worker=#{worker.nr} PID:#{wpid} hard timeout " \
                      "(#{diff}s > #{timeout}s), killing"
-        kill_worker(:KILL, wpid) # take no prisoners for timeout violations
+          kill_worker(:KILL, wpid) # take no prisoners for timeout violations
+        end
       end
     end

@@ -657,6 +667,12 @@ module Unicorn
       proc_name "worker[#{worker.nr}]"
       START_CTX.clear
       init_self_pipe!
+
+      # try to handle SIGABRT correctly
+      trap('ABRT') do
+        raise SignalException, "SIGABRT"
+      end
+
       WORKERS.values.each { |other| other.tmp.close rescue nil }
       WORKERS.clear
       LISTENERS.each { |sock| sock.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) }
diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb
index 64a25e3..6efb0c5 100644
--- a/lib/unicorn/configurator.rb
+++ b/lib/unicorn/configurator.rb
@@ -14,6 +14,8 @@ module Unicorn

     # Default settings for Unicorn
     DEFAULTS = {
+      # Backward compatibility soft timeout (disabled in default configuration)
+      :soft_timeout => 60,
       :timeout => 60,
       :logger => Logger.new($stderr),
       :worker_processes => 1,
@@ -129,6 +131,23 @@ module Unicorn

     # sets the timeout of worker processes to +seconds+.  Workers
     # handling the request/app.call/response cycle taking longer than
+    # this time period will be softly killed (via SIGABRT).  This
+    # timeout is enforced by the master process itself and not subject
+    # to the scheduling limitations by the worker process.  Due the
+    # low-complexity, low-overhead implementation, timeouts of less
+    # than 3.0 seconds can be considered inaccurate and unsafe.
+    # ABORT is handled by the worker and raise an exception, offering a
+    # way to log the stack trace in your rails application.
+
+    def soft_timeout(seconds)
+      Numeric === seconds or raise ArgumentError,
+                                  "not numeric: timeout=#{seconds.inspect}"
+      seconds >= 3 or raise ArgumentError,
+                                  "too low: timeout=#{seconds.inspect}"
+      set[:soft_timeout] = seconds
+    end
+    # sets the timeout of worker processes to +seconds+.  Workers
+    # handling the request/app.call/response cycle taking longer than
     # this time period will be forcibly killed (via SIGKILL).  This
     # timeout is enforced by the master process itself and not subject
     # to the scheduling limitations by the worker process.  Due the
@@ -159,6 +178,7 @@ module Unicorn
       set[:timeout] = seconds
     end

+
     # sets the current number of worker_processes to +nr+.  Each worker
     # process will serve exactly one client at a time.  You can
     # increment or decrement this value at runtime by sending SIGTTIN


Cheers,
--
Pierre Baillet <oct@fotonauts.com>
http://www.fotopedia.com/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Fwd: Support for Soft Timeout in Unicorn

[Eric Wong]

Pierre Baillet wrote:
> We use Unicorn at fotopedia since yesterday in production. We switched from
> Passenger due to an issue in the way Passenger was handling some error in
> our main application. Things run very well on Unicorn.
> 
> We have also modified Unicorn to handle a soft timeout for its workers. The
> Unicorn timeout was killing the workers without any chance for us to catch
> the Rails stack trace and identify the issue. I've forked and branched
> Unicorn from github and added a soft_timeout configuration value that is
> used for long running workers.
> 
> The workers now handle SIGABRT and will raise an exception. This will crash
> the Rails application if it can be crashed and force the framework to dump
> the stack trace in the logs. Let me know if this might be useful for other
> people and, why not, integrate that in the main Unicorn code !

Hi Pierre,

I'm thinking there's a better way to do this without involving the
master process.

The current timeout implementation[1] is really the last resort,
point-of-no-return situations when the workers are completely
stuck/blocked and cannot respond to other signals[2].  If the worker can
respond to SIGABRT (especially going through the interpreter and raising
an exception), then that means it could technically respond to
Thread#raise, too...

Unfortunately, the current core Ruby timeout implementation is extremely
naive and inefficient.  It shouldn't be hard to write a better timeout
implementation that reuses a single timer Thread which can be rearmed
with every request.

This would be doable as middleware, too, and if done carefully, even
safely reusable in multi-threaded web servers.  This would be a good
addition to rack-contrib, even.  I might consider doing it myself if I
had time.

[1] - I'm not completely happy with "needing" the current timeout
      implementation in the first place.  I will at least redo it
      at some point (after Unicorn 1.x) to gain some
      scalability/performance (and perhaps lose some portability).

[2] - SIGKILL and SIGSTOP are special, userspace has no mechanism
      to block/catch/ignore those signals, so we rely on the master
      process to deliver them.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Fwd: Support for Soft Timeout in Unicorn

[Pierre Baillet]

Eric,

On Thu, Jun 3, 2010 at 8:22 PM, Eric Wong <normalperson@yhbt.net> wrote:
> This would be doable as middleware, too, and if done carefully, even
> safely reusable in multi-threaded web servers.  This would be a good
> addition to rack-contrib, even.  I might consider doing it myself if I
> had time.

I also think that doing this as a Rack middleware is probably the
right way to do that kind of worker management. However the
application we use at fotopedia is still Rails 2.1 based (hence not
Rack compatible AFAIK). This probably means we'll have keep on
maintaining this slightly hacked version on our own then.

Thank you for Unicorn ! :)
-- 
Pierre Baillet <oct@fotonauts.com>
http://www.fotopedia.com/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Fwd: Support for Soft Timeout in Unicorn

[Eric Wong]

Pierre Baillet <oct@fotonauts.com> wrote:
> Eric,
> 
> On Thu, Jun 3, 2010 at 8:22 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > This would be doable as middleware, too, and if done carefully, even
> > safely reusable in multi-threaded web servers.  This would be a good
> > addition to rack-contrib, even.  I might consider doing it myself if I
> > had time.
> 
> I also think that doing this as a Rack middleware is probably the
> right way to do that kind of worker management. However the
> application we use at fotopedia is still Rails 2.1 based (hence not
> Rack compatible AFAIK). This probably means we'll have keep on
> maintaining this slightly hacked version on our own then.

Actually, internally, Unicorn only knows about Rack and wraps older
CGI-based Rails using the Unicorn::App::OldRails application (via
Unicorn::CGIWrapper).

"unicorn_rails" basically wraps up the following config for you,
but you could achieve the same effect using "unicorn" and an
explicitly written config.ru:

------------ config.ru -------------
ENV["RAILS_ENV"] ||= ENV["RACK_ENV"]
require 'config/boot'
require 'config/environment'
require 'unicorn/app/old_rails'
require 'unwritten_timeout_middleware'
use UnwrittenTimeoutMiddleware # :)
use Unicorn::App::OldRails::Static # optional
run Unicorn::App::OldRails.new
------------------------------------

> Thank you for Unicorn ! :)

No problem :)

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Fwd: Support for Soft Timeout in Unicorn

[Chris Wanstrath]

On Thu, Jun 3, 2010 at 11:47 AM, Eric Wong <normalperson@yhbt.net> wrote:

> Actually, internally, Unicorn only knows about Rack and wraps older
> CGI-based Rails using the Unicorn::App::OldRails application (via
> Unicorn::CGIWrapper).
>
> "unicorn_rails" basically wraps up the following config for you,
> but you could achieve the same effect using "unicorn" and an
> explicitly written config.ru:

That's what we do at GitHub. We're running Rails 2.2.2 and have a
custom config.ru, thanks to Unicorn:

http://gist.github.com/424352
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Fwd: Support for Soft Timeout in Unicorn

[Pierre Baillet]

Ohai,

On Thu, Jun 3, 2010 at 9:38 PM, Chris Wanstrath <chris@ozmm.org> wrote:
> On Thu, Jun 3, 2010 at 11:47 AM, Eric Wong <normalperson@yhbt.net> wrote:
>
>> Actually, internally, Unicorn only knows about Rack and wraps older
>> CGI-based Rails using the Unicorn::App::OldRails application (via
>> Unicorn::CGIWrapper).
>>
>> "unicorn_rails" basically wraps up the following config for you,
>> but you could achieve the same effect using "unicorn" and an
>> explicitly written config.ru:
>
> That's what we do at GitHub. We're running Rails 2.2.2 and have a
> custom config.ru, thanks to Unicorn:
>

That's great, thanks for the Idea, I'll look into rack middleware
then. I'll let you know if I manage to have something clean enough.

Cheers,
-- 
Pierre Baillet <oct@fotonauts.com>
http://www.fotopedia.com/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Fwd: Support for Soft Timeout in Unicorn

[Eric Wong]

Chris Wanstrath <chris@ozmm.org> wrote:
> That's what we do at GitHub. We're running Rails 2.2.2 and have a
> custom config.ru, thanks to Unicorn:
> 
> http://gist.github.com/424352

By the way, how's the OobGC middleware working for you guys?

Luke: did you also get a chance to try this in place of my original
monkey patch?

Thanks in advance for any info you can share.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Fwd: Support for Soft Timeout in Unicorn

[Pierre Baillet]

Hello Unicorns,

I've manage to create a simple middleware that replaces the soft
timeout feature. You can have a look at it at
http://gist.github.com/431451

Note that some weird Ruby interpreter behavior breaks at least the
first level of the generated stacktrace (it indicates the actual
method where the raise happened but the wrong line number and file).

Thanks for your assistance,
Cheers,
-- 
Pierre.


On Thu, Jun 3, 2010 at 9:40 PM, Pierre Baillet <oct@fotonauts.com> wrote:
> Ohai,
>
> On Thu, Jun 3, 2010 at 9:38 PM, Chris Wanstrath <chris@ozmm.org> wrote:
>> On Thu, Jun 3, 2010 at 11:47 AM, Eric Wong <normalperson@yhbt.net> wrote:
>>
>>> Actually, internally, Unicorn only knows about Rack and wraps older
>>> CGI-based Rails using the Unicorn::App::OldRails application (via
>>> Unicorn::CGIWrapper).
>>>
>>> "unicorn_rails" basically wraps up the following config for you,
>>> but you could achieve the same effect using "unicorn" and an
>>> explicitly written config.ru:
>>
>> That's what we do at GitHub. We're running Rails 2.2.2 and have a
>> custom config.ru, thanks to Unicorn:
>>
>
> That's great, thanks for the Idea, I'll look into rack middleware
> then. I'll let you know if I manage to have something clean enough.
>
> Cheers,
> --
> Pierre Baillet <oct@fotonauts.com>
> http://www.fotopedia.com/
>



-- 
Pierre Baillet <oct@fotonauts.com>
http://www.fotopedia.com/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Fwd: Support for Soft Timeout in Unicorn

[Eric Wong]

Pierre Baillet <oct@fotonauts.com> wrote:
> Hello Unicorns,
> 
> I've manage to create a simple middleware that replaces the soft
> timeout feature. You can have a look at it at
> http://gist.github.com/431451
> 
> Note that some weird Ruby interpreter behavior breaks at least the
> first level of the generated stacktrace (it indicates the actual
> method where the raise happened but the wrong line number and file).

You should be able to avoid spawning a new thread for every request, as
that can get very expensive on some systems.  Maybe a global hash
guarded by a mutex that tells the worker Thread which thread to raise
on.  The worker thread could sleep until the next timeout registered.

But then again, take care to only spawn new threads in workers (with
preload_app=true), as threads never get carried across fork.  And
threads may leave mutexes and such in a bad state when they vanish
in the child.

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Fwd: Support for Soft Timeout in Unicorn

[Ryan Tomayko]

On Fri, Jun 4, 2010 at 1:59 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Chris Wanstrath <chris@ozmm.org> wrote:
>> That's what we do at GitHub. We're running Rails 2.2.2 and have a
>> custom config.ru, thanks to Unicorn:
>>
>> http://gist.github.com/424352
>
> By the way, how's the OobGC middleware working for you guys?

We rolled out the OobGC middleware along with a basic RailsBench GC
config (RUBY_HEAP_MIN_SLOTS, etc.). Combined, they knocked about 20ms
(~15%) off the average response time across the site (real traffic).
The impact was significantly more for requests that allocate a lot of
objects -- as much as 50% decreases in response time for the worst
offenders. We saw no noticeable increase in CPU with OobGC set to run
every 10 requests, and a fair increase in CPU with OobGC set to run
every 5 requests.

Because I rolled this stuff out somewhat non-scientifically, I've
always wondered how much OobGC contributed to the overall savings vs.
the RailsBench GC config. Disabling the OobGC middleware but leaving
the RailsBench GC config in place, I get the following graph:

http://img.skitch.com/20100618-kihdc1cq6pjhq9rqftf8miuf6y.png

So we're spending ~1ms request time in GC with OobGC, and ~10ms
request time in GC without it.

Here's some system load graphs for the same time period just to show
that OobGC has no adverse effect when set to GC every 10 requests:

http://img.skitch.com/20100618-qp7p8f6i2agbqbdnjbpigik1d9.png

I assume the RailsBench GC patches improve the effect of OobGC
considerably by increasing the number of objects that can be allocated
between GC runs, allowing more of the GC work to be deferred to
in-between-requests time. Here's the RailsBench config we're using
today, for the record:

    RUBY_HEAP_MIN_SLOTS=800000
    RUBY_HEAP_FREE_MIN=100000
    RUBY_HEAP_SLOTS_INCREMENT=300000
    RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
    RUBY_GC_MALLOC_LIMIT=79000000

This is only barely tuned for us. I stole most of the numbers from the
Twitter and 37signals examples.

I've also experimented with tuning the GC specifically to take
advantage of OobGC:

https://gist.github.com/87d574a19372c6043c5f

# The following GC settings have been tuned for GitHub application web requests.
# Most settings are significantly higher than the example configs published by
# Twitter and 37signals. There's a couple reasons for this. First, the GitHub
# app has a memory footprint that's 3x-4x larger than the standard Rails app
# (roughly 200MB after first request compared to ~40MB-50MB). Second, because
# Unicorn is such an exceptional piece of software, we're able to schedule GC
# to run outside the context of requests so as not to effect response times.
# As such, we try to allocate enough memory to service 5 requests
without needing
# GC and then run GC manually immediately after each fifth request has been
# served but before the process starts accepting the next connection. The result
# is higher memory use (~300MB per Unicorn worker process on average) and a
# slight increase in CPU due to forced manual GC, but better response times.
# ...

Unfortunately, the bigger heap seems to cause a largish increase in
the time needed for each GC, so the unicorn workers were spending too
much time between requests. CPU and RES increases were even more than
I'd expected. It also didn't eliminate in-request GC entirely as I'd
hoped.

I eventually abandoned the idea -- even if I could get it to behave,
it's hardly worth the 1ms it would save. I mention it here because the
general approach might work in situations where the base heap size is
a bit smaller (say < 80MB) or perhaps I'm mistuning one of the
parameters.

Thanks,
Ryan

> Luke: did you also get a chance to try this in place of my original
> monkey patch?
>
> Thanks in advance for any info you can share.
>
> --
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Fwd: Support for Soft Timeout in Unicorn

[Eric Wong]

Ryan Tomayko <r@tomayko.com> wrote:
> On Fri, Jun 4, 2010 at 1:59 PM, Eric Wong <normalperson@yhbt.net> wrote:
> > Chris Wanstrath <chris@ozmm.org> wrote:
> >> That's what we do at GitHub. We're running Rails 2.2.2 and have a
> >> custom config.ru, thanks to Unicorn:
> >>
> >> http://gist.github.com/424352
> >
> > By the way, how's the OobGC middleware working for you guys?
> 
> We rolled out the OobGC middleware along with a basic RailsBench GC
> config (RUBY_HEAP_MIN_SLOTS, etc.). Combined, they knocked about 20ms
> (~15%) off the average response time across the site (real traffic).
> The impact was significantly more for requests that allocate a lot of
> objects -- as much as 50% decreases in response time for the worst
> offenders. We saw no noticeable increase in CPU with OobGC set to run
> every 10 requests, and a fair increase in CPU with OobGC set to run
> every 5 requests.

Cool.  Am I correct to assume the increased CPU usage at every 5
requests wasn't worth any performance gains you might have had?

> Because I rolled this stuff out somewhat non-scientifically, I've
> always wondered how much OobGC contributed to the overall savings vs.
> the RailsBench GC config. Disabling the OobGC middleware but leaving
> the RailsBench GC config in place, I get the following graph:
> 
> http://img.skitch.com/20100618-kihdc1cq6pjhq9rqftf8miuf6y.png
> 
> So we're spending ~1ms request time in GC with OobGC, and ~10ms
> request time in GC without it.

Awesome.

> Here's some system load graphs for the same time period just to show
> that OobGC has no adverse effect when set to GC every 10 requests:
> 
> http://img.skitch.com/20100618-qp7p8f6i2agbqbdnjbpigik1d9.png
> 
> I assume the RailsBench GC patches improve the effect of OobGC
> considerably by increasing the number of objects that can be allocated
> between GC runs, allowing more of the GC work to be deferred to
> in-between-requests time. Here's the RailsBench config we're using
> today, for the record:
> 
>     RUBY_HEAP_MIN_SLOTS=800000
>     RUBY_HEAP_FREE_MIN=100000
>     RUBY_HEAP_SLOTS_INCREMENT=300000
>     RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
>     RUBY_GC_MALLOC_LIMIT=79000000
> 
> This is only barely tuned for us. I stole most of the numbers from the
> Twitter and 37signals examples.
> 
> I've also experimented with tuning the GC specifically to take
> advantage of OobGC:
> 
> https://gist.github.com/87d574a19372c6043c5f
> 
> # The following GC settings have been tuned for GitHub application web requests.
> # Most settings are significantly higher than the example configs published by
> # Twitter and 37signals. There's a couple reasons for this. First, the GitHub
> # app has a memory footprint that's 3x-4x larger than the standard Rails app
> # (roughly 200MB after first request compared to ~40MB-50MB). Second, because

Yikes, 200MB after one request is a lot.  If you can easily find ways to
cut that down, it should be more of a gain than the monster heap you've
tried.

> # Unicorn is such an exceptional piece of software, we're able to schedule GC
> # to run outside the context of requests so as not to effect response times.
> # As such, we try to allocate enough memory to service 5 requests
> without needing
> # GC and then run GC manually immediately after each fifth request has been
> # served but before the process starts accepting the next connection. The result
> # is higher memory use (~300MB per Unicorn worker process on average) and a
> # slight increase in CPU due to forced manual GC, but better response times.
> # ...
> 
> Unfortunately, the bigger heap seems to cause a largish increase in
> the time needed for each GC, so the unicorn workers were spending too
> much time between requests. CPU and RES increases were even more than
> I'd expected. It also didn't eliminate in-request GC entirely as I'd
> hoped.
> 
> I eventually abandoned the idea -- even if I could get it to behave,
> it's hardly worth the 1ms it would save. I mention it here because the
> general approach might work in situations where the base heap size is
> a bit smaller (say < 80MB) or perhaps I'm mistuning one of the
> parameters.

So in conclusion, OobGC itself works, but too large of a heap isn't
worth it even for a memory hungry app.

I suppose having too big of a heap means it can fragment more badly.
Making GC run more often on a smaller heap can and give similar (or
maybe better) performance.  At best you'll get diminishing returns as
you seem to have concluded.

I have no doubt the Railsbench GC patches help.  Even with small apps,
being able being able to set a linear growth factor on long-running
servers is awesome.

Thanks for sharing this!

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Fwd: Support for Soft Timeout in Unicorn

[Ryan Tomayko]

On Fri, Jun 18, 2010 at 2:48 PM, Eric Wong <normalperson@yhbt.net> wrote:
> Ryan Tomayko <r@tomayko.com> wrote:
>> On Fri, Jun 4, 2010 at 1:59 PM, Eric Wong <normalperson@yhbt.net> wrote:
>> > Chris Wanstrath <chris@ozmm.org> wrote:
>> >> That's what we do at GitHub. We're running Rails 2.2.2 and have a
>> >> custom config.ru, thanks to Unicorn:
>> >>
>> >> http://gist.github.com/424352
>> >
>> > By the way, how's the OobGC middleware working for you guys?
>>
>> We rolled out the OobGC middleware along with a basic RailsBench GC
>> config (RUBY_HEAP_MIN_SLOTS, etc.). Combined, they knocked about 20ms
>> (~15%) off the average response time across the site (real traffic).
>> The impact was significantly more for requests that allocate a lot of
>> objects -- as much as 50% decreases in response time for the worst
>> offenders. We saw no noticeable increase in CPU with OobGC set to run
>> every 10 requests, and a fair increase in CPU with OobGC set to run
>> every 5 requests.
>
> Cool.  Am I correct to assume the increased CPU usage at every 5
> requests wasn't worth any performance gains you might have had?

Yes. I don't have exact numbers on how much more CPU was being
utilized but it was definitely noticeable on my graphs and had very
little, if any, effect on overall response time beyond the savings at
the req/10 interval.

>> Because I rolled this stuff out somewhat non-scientifically, I've
>> always wondered how much OobGC contributed to the overall savings vs.
>> the RailsBench GC config. Disabling the OobGC middleware but leaving
>> the RailsBench GC config in place, I get the following graph:
>>
>> http://img.skitch.com/20100618-kihdc1cq6pjhq9rqftf8miuf6y.png
>>
>> So we're spending ~1ms request time in GC with OobGC, and ~10ms
>> request time in GC without it.
>
> Awesome.
>
>> Here's some system load graphs for the same time period just to show
>> that OobGC has no adverse effect when set to GC every 10 requests:
>>
>> http://img.skitch.com/20100618-qp7p8f6i2agbqbdnjbpigik1d9.png
>>
>> I assume the RailsBench GC patches improve the effect of OobGC
>> considerably by increasing the number of objects that can be allocated
>> between GC runs, allowing more of the GC work to be deferred to
>> in-between-requests time. Here's the RailsBench config we're using
>> today, for the record:
>>
>>     RUBY_HEAP_MIN_SLOTS=800000
>>     RUBY_HEAP_FREE_MIN=100000
>>     RUBY_HEAP_SLOTS_INCREMENT=300000
>>     RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
>>     RUBY_GC_MALLOC_LIMIT=79000000
>>
>> This is only barely tuned for us. I stole most of the numbers from the
>> Twitter and 37signals examples.
>>
>> I've also experimented with tuning the GC specifically to take
>> advantage of OobGC:
>>
>> https://gist.github.com/87d574a19372c6043c5f
>>
>> # The following GC settings have been tuned for GitHub application web requests.
>> # Most settings are significantly higher than the example configs published by
>> # Twitter and 37signals. There's a couple reasons for this. First, the GitHub
>> # app has a memory footprint that's 3x-4x larger than the standard Rails app
>> # (roughly 200MB after first request compared to ~40MB-50MB). Second, because
>
> Yikes, 200MB after one request is a lot.  If you can easily find ways to
> cut that down, it should be more of a gain than the monster heap you've
> tried.

Indeed. We bring in *a lot* of libraries, some of which add
embarassingly large amounts of code and data at require time (aws-s3,
googlecharts - I'm looking at you).

>> # Unicorn is such an exceptional piece of software, we're able to schedule GC
>> # to run outside the context of requests so as not to effect response times.
>> # As such, we try to allocate enough memory to service 5 requests
>> without needing
>> # GC and then run GC manually immediately after each fifth request has been
>> # served but before the process starts accepting the next connection. The result
>> # is higher memory use (~300MB per Unicorn worker process on average) and a
>> # slight increase in CPU due to forced manual GC, but better response times.
>> # ...
>>
>> Unfortunately, the bigger heap seems to cause a largish increase in
>> the time needed for each GC, so the unicorn workers were spending too
>> much time between requests. CPU and RES increases were even more than
>> I'd expected. It also didn't eliminate in-request GC entirely as I'd
>> hoped.
>>
>> I eventually abandoned the idea -- even if I could get it to behave,
>> it's hardly worth the 1ms it would save. I mention it here because the
>> general approach might work in situations where the base heap size is
>> a bit smaller (say < 80MB) or perhaps I'm mistuning one of the
>> parameters.
>
> So in conclusion, OobGC itself works, but too large of a heap isn't
> worth it even for a memory hungry app.
>
> I suppose having too big of a heap means it can fragment more badly.
> Making GC run more often on a smaller heap can and give similar (or
> maybe better) performance.  At best you'll get diminishing returns as
> you seem to have concluded.

That sounds like an extremely plausible explanation.

> I have no doubt the Railsbench GC patches help.  Even with small apps,
> being able being able to set a linear growth factor on long-running
> servers is awesome.

Sure is.

Thanks,
Ryan

> Thanks for sharing this!
>
> --
> Eric Wong
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
>
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying