Garbage collection outside of request cycle?

Garbage collection outside of request cycle?

[Luke Melia]

I've been analyzing our Unicorn-powered Rails app's performance, and have found that garbage collection is a big factor in slow requests.

In the interest of avoiding those performance hits while handling requests, would it be possible to have a unicorn worker run garbage collection after handling a request and before waiting for the next one? Would this be a good idea?

Cheers,
Luke
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Garbage collection outside of request cycle?

[Eric Wong]

Luke Melia <luke@lukemelia.com> wrote:
> I've been analyzing our Unicorn-powered Rails app's performance, and
> have found that garbage collection is a big factor in slow requests.
> 
> In the interest of avoiding those performance hits while handling
> requests, would it be possible to have a unicorn worker run garbage
> collection after handling a request and before waiting for the next
> one? Would this be a good idea?

Hi Luke,

I made this for one heavyweight app a while back.

I guess I should throw this into the examples section, but it won't be
the default since it hurts simpler applications that don't generate
much garbage.

==> big_app_gc.rb <==
# This shouldn't hurt overall performance as long as the server cluster
# is at <=50% CPU capacity, and improves the performance of most memory
# intensive requests.  This serves to improve _client-visible_
# performance (possibly at the cost of overall performance).
#
# We'll call GC after each request is been written out to the socket, so
# the client never sees the extra GC hit it. It's ideal to call the GC
# inside the HTTP server (vs middleware or hooks) since the stack is
# smaller at this point, so the GC will both be faster and more
# effective at releasing unused memory.
#
# This monkey patch is _only_ effective for applications that use a lot
# of memory, and will hurt simpler apps/endpoints that can process
# multiple requests before incurring GC.

class Unicorn::HttpServer
  REQ = Unicorn::HttpRequest::REQ
  alias _process_client process_client
  undef_method :process_client
  def process_client(client)
    _process_client(client)
    REQ.clear
    GC.start
  end
end if defined?(Unicorn)
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Garbage collection outside of request cycle?

[Eric Wong]

Eric Wong <normalperson@yhbt.net> wrote:
> I guess I should throw this into the examples section

Added a few more comments, but the code is still the same:

http://unicorn.bogomips.org/examples/big_app_gc.rb
http://git.bogomips.org/cgit/unicorn.git/commit/?id=510a48dafc5f7e2cb618d785885395c79570821c

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Garbage collection outside of request cycle?

[Luke Melia]

On May 6, 2010, at 4:57 PM, Eric Wong wrote:

> I made this for one heavyweight app a while back.
> 
> ==> big_app_gc.rb <==
> # This shouldn't hurt overall performance as long as the server cluster
> # is at <=50% CPU capacity, and improves the performance of most memory
> # intensive requests.  This serves to improve _client-visible_
> # performance (possibly at the cost of overall performance).

This is exactly the tradeoff I'm looking for. Our app has a large footprint.

Thanks for the quick response. I'll report back when I have some data.

Cheers,
Luke

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Garbage collection outside of request cycle?

[Luke Melia]

> On May 6, 2010, at 4:57 PM, Eric Wong wrote:
> 
>> ==> big_app_gc.rb <==
>> # This shouldn't hurt overall performance as long as the server cluster
>> # is at <=50% CPU capacity, and improves the performance of most memory
>> # intensive requests.  This serves to improve _client-visible_
>> # performance (possibly at the cost of overall performance).

I thought the list might be interested in how this worked for us. I applied the patch to execute GC between each request. I'm using NewRelic to measure the app. Prior to the patch our, we spent about 25% of our aggregate time serving a request in GC and our application was running at around 20-30% CPU load. Our running app shows up as using ~330MB of memory.

Applying the patch cut the time spent in GC time to nearly zero and as predicted CPU spiked. Client-perceived responsiveness increased as well. Unfortunately, during our busiest time of the day, CPU load got so high that nginx locked up, so we rolled back the patch.

I made a simple change to execute GC once every 5 requests and applied it again. Aggregate time spent in GC reduced to out 10% of total request time. This resulted in a bout a 25% overall improvement in client response time. Big win! CPU maxes out at about 80% with this configuration..

One other thing I did was force GC to execute before_fork, on the theory that with COW, we would want to fork in the tidiest state possible. I have not measured this on it's own to evaluate it's impact.

Thanks again for the help and code on this, Eric.

Considering how useful this is, perhaps unicorn should have an after_request hook, to avoid the need to monkey-patch?

Cheers,
Luke
--
http://www.lukemelia.com/
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Garbage collection outside of request cycle?

[Eric Wong]

Luke Melia <luke@lukemelia.com> wrote:
> > On May 6, 2010, at 4:57 PM, Eric Wong wrote:
> > 
> >> ==> big_app_gc.rb <== # This shouldn't hurt overall performance as
> >> long as the server cluster # is at <=50% CPU capacity, and improves
> >> the performance of most memory # intensive requests.  This serves
> >> to improve _client-visible_ # performance (possibly at the cost of
> >> overall performance).
> 
> I thought the list might be interested in how this worked for us. I
> applied the patch to execute GC between each request. I'm using
> NewRelic to measure the app. Prior to the patch our, we spent about
> 25% of our aggregate time serving a request in GC and our application
> was running at around 20-30% CPU load. Our running app shows up as
> using ~330MB of memory.

Thanks for the feedback, Luke.

Was the original 30% CPU load during peak traffic or normal traffic?

> Applying the patch cut the time spent in GC time to nearly zero and as
> predicted CPU spiked. Client-perceived responsiveness increased as
> well. Unfortunately, during our busiest time of the day, CPU load got
> so high that nginx locked up, so we rolled back the patch.

Yikes, nginx locking up is rare.  It's worth investigating and fixing
that from the nginx side if you can reproduce it.

> I made a simple change to execute GC once every 5 requests and applied
> it again. Aggregate time spent in GC reduced to out 10% of total
> request time. This resulted in a bout a 25% overall improvement in
> client response time. Big win! CPU maxes out at about 80% with this
> configuration..

How many Unicorn workers do you have per-core?  I forgot to mention that
you might want to run more workers to "hide" GC costs, something like a
poor man's concurrent GC.

> One other thing I did was force GC to execute before_fork, on the
> theory that with COW, we would want to fork in the tidiest state
> possible. I have not measured this on it's own to evaluate it's
> impact.

I doubt it'd help for anything other than the first <=5 requests that
hit the worker process.

> Thanks again for the help and code on this, Eric.

Thank _you_ for actually being willing to run and report back on
crazy experiments I come up with :)

> Considering how useful this is, perhaps unicorn should have an
> after_request hook, to avoid the need to monkey-patch?

Given configurability requirements and the ability of such a feature to
penalize some apps, I think making the monkey patch into middleware is a
nice compromise.

The GC.start in the below middleware runs at a slightly deeper stack
level, meaning GC will scan more and reap less than the original monkey
patch, but I doubt there'll be real difference if your app is already
spending enough time in GC to be a problem.

I've also pushed the following up to git://git.bogomips.org/unicorn.git
Let me know if you get a chance to test it on your app in place of
the monkey patch.

>From 95b75a5043b34f39ece4f52befb4b3f884dfdd20 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Fri, 14 May 2010 18:27:35 +0000
Subject: [PATCH] add Unicorn::OobGC middleware

This middleware allows configurable out-of-band garbage
collection outside of the normal request/response cycle.

It offers configurable paths (to only GC on expensive actions)
and intervals to limit GC frequency.

It is only expected to work well with Unicorn, as it would
hurt performance on single-threaded servers if they
have keepalive enabled.  Obviously this does not work well
for multi-threaded or evented servers that serve multiple
clients at once.
---
 lib/unicorn/oob_gc.rb |   58 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 58 insertions(+), 0 deletions(-)
 create mode 100644 lib/unicorn/oob_gc.rb

diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
new file mode 100644
index 0000000..8dc4dcf
--- /dev/null
+++ b/lib/unicorn/oob_gc.rb
@@ -0,0 +1,58 @@
+# -*- encoding: binary -*-
+module Unicorn
+
+  # Run GC after every request, after closing the client socket and
+  # before attempting to accept more connections.
+  #
+  # This shouldn't hurt overall performance as long as the server cluster
+  # is at <50% CPU capacity, and improves the performance of most memory
+  # intensive requests.  This serves to improve _client-visible_
+  # performance (possibly at the cost of overall performance).
+  #
+  # We'll call GC after each request is been written out to the socket, so
+  # the client never sees the extra GC hit it.
+  #
+  # This middleware is _only_ effective for applications that use a lot
+  # of memory, and will hurt simpler apps/endpoints that can process
+  # multiple requests before incurring GC.
+  #
+  # This middleware is only designed to work with Unicorn, as it harms
+  # keepalive performance.
+  #
+  # Example (in config.ru):
+  #
+  #     require 'unicorn/oob_gc'
+  #
+  #     # GC ever two requests that hit /expensive/foo or /more_expensive/foo
+  #     # in your app.  By default, this will GC once every 5 requests
+  #     # for all endpoints in your app
+  #     use Unicorn::OobGC, 2, %r{\A/(?:expensive/foo|more_expensive/foo)}
+  class OobGC < Struct.new(:app, :interval, :path, :nr, :env, :body)
+
+    def initialize(app, interval = 5, path = %r{\A/})
+      super(app, interval, path, interval)
+    end
+
+    def call(env)
+      status, headers, self.body = app.call(self.env = env)
+      [ status, headers, self ]
+    end
+
+    def each(&block)
+      body.each(&block)
+    end
+
+    # in Unicorn, this is closed _after_ the client socket
+    def close
+      body.close if body.respond_to?(:close)
+
+      if path =~ env['PATH_INFO'] && ((self.nr -= 1) <= 0)
+        self.nr = interval
+        self.body = nil
+        env.clear
+        GC.start
+      end
+    end
+
+  end
+end
-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying