* Fwd: Support for Soft Timeout in Unicorn @ 2010-06-03 17:37 Eric Wong 2010-06-03 18:06 ` Pierre Baillet 2010-06-03 18:22 ` Fwd: " Eric Wong 0 siblings, 2 replies; 13+ messages in thread From: Eric Wong @ 2010-06-03 17:37 UTC (permalink / raw) To: mongrel-unicorn; +Cc: Pierre Baillet Hi, HTML attachments are wasteful and thus rejected from the mailing list. On the other hand, it actually helps to include the patch itself (inline) so it's readable without a (human) context switch :) ----- Forwarded message from Pierre Baillet <oct@fotonauts.com> ----- Subject: Fwd: Support for Soft Timeout in Unicorn From: Pierre Baillet <oct@fotonauts.com> To: unicorn@bogomips.org Hi, Just tried to send that through the ml, but it seems something went wrong... Cheers, -- Pierre. ---------- Forwarded message ---------- From: <mongrel-unicorn-owner@rubyforge.org> Date: Thu, Jun 3, 2010 at 2:40 PM Subject: Support for Soft Timeout in Unicorn To: oct@fotonauts.com The message's content type was not explicitly allowed ---------- Forwarded message ---------- From: Pierre Baillet <oct@fotonauts.com> To: mongrel-unicorn@rubyforge.org Date: Thu, 3 Jun 2010 14:16:01 +0200 Subject: Support for Soft Timeout in Unicorn Hi, We use Unicorn at fotopedia since yesterday in production. We switched from Passenger due to an issue in the way Passenger was handling some error in our main application. Things run very well on Unicorn. We have also modified Unicorn to handle a soft timeout for its workers. The Unicorn timeout was killing the workers without any chance for us to catch the Rails stack trace and identify the issue. I've forked and branched Unicorn from github and added a soft_timeout configuration value that is used for long running workers. The workers now handle SIGABRT and will raise an exception. This will crash the Rails application if it can be crashed and force the framework to dump the stack trace in the logs. Let me know if this might be useful for other people and, why not, integrate that in the main Unicorn code ! http://github.com/octplane/unicorn/tree/SOFT_TIMEOUT_SUPPORT Cheers, -- Pierre Baillet <oct@fotonauts.com> http://www.fotopedia.com/ -- Pierre Baillet <oct@fotonauts.com> http://www.fotopedia.com/ ----- End forwarded message ----- _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Support for Soft Timeout in Unicorn 2010-06-03 17:37 Fwd: Support for Soft Timeout in Unicorn Eric Wong @ 2010-06-03 18:06 ` Pierre Baillet 2010-06-03 18:22 ` Fwd: " Eric Wong 1 sibling, 0 replies; 13+ messages in thread From: Pierre Baillet @ 2010-06-03 18:06 UTC (permalink / raw) To: Eric Wong; +Cc: mongrel-unicorn Hello, On Thu, Jun 3, 2010 at 7:37 PM, Eric Wong <normalperson@yhbt.net> wrote: > > Hi, > > HTML attachments are wasteful and thus rejected from the mailing list. > On the other hand, it actually helps to include the patch itself > (inline) so it's readable without a (human) context switch :) Indeed, sorry for the HTML attachment, I have no idea where it comes from. As for the patch, here you are. This is really just a way to handle SIGABRT in a specific way in the worker and allow proper termination of the application. Note the FIXME comment I added in the murder_lazy_workers method. If any worker blocks while all the others are idle for a _timeout_ period of time, they will all be killed anyway. The consequence of that is that Unicorn will restart all its workers if traffic is very low on the server. diff --git a/lib/unicorn.rb b/lib/unicorn.rb index a363014..855f26a 100644 --- a/lib/unicorn.rb +++ b/lib/unicorn.rb @@ -84,7 +84,7 @@ module Unicorn # Listener sockets are started in the master process and shared with # forked worker children. - class HttpServer < Struct.new(:app, :timeout, :worker_processes, + class HttpServer < Struct.new(:app, :soft_timeout, :timeout, :worker_processes, :before_fork, :after_fork, :before_exec, :logger, :pid, :listener_opts, :preload_app, :reexec_pid, :orig_app, :init_listeners, @@ -393,7 +393,7 @@ module Unicorn when nil # avoid murdering workers after our master process (or the # machine) comes out of suspend/hibernation - if (last_check + timeout) >= (last_check = Time.now) + if (last_check + soft_timeout) >= (last_check = Time.now) murder_lazy_workers else # wait for workers to wakeup on suspend @@ -581,10 +581,20 @@ module Unicorn stat = worker.tmp.stat # skip workers that disable fchmod or have never fchmod-ed stat.mode == 0100600 and next - (diff = (Time.now - stat.ctime)) <= timeout and next - logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \ + # FIXME: if the worker has not been working for soft_timeout, it will be + # killed even if it is not blocking + (diff = (Time.now - stat.ctime)) <= soft_timeout and + diff <= timeout and next + # lazy since less than timeout, attempt soft kill + if diff < timeout + logger.error "worker=#{worker.nr} PID:#{wpid} soft timeout " \ + "(#{diff}s > #{soft_timeout}s), killing softly" + kill_worker(:ABRT, wpid) + else + logger.error "worker=#{worker.nr} PID:#{wpid} hard timeout " \ : diff --git a/lib/unicorn.rb b/lib/unicorn.rb index a363014..855f26a 100644 --- a/lib/unicorn.rb +++ b/lib/unicorn.rb @@ -84,7 +84,7 @@ module Unicorn # Listener sockets are started in the master process and shared with # forked worker children. - class HttpServer < Struct.new(:app, :timeout, :worker_processes, + class HttpServer < Struct.new(:app, :soft_timeout, :timeout, :worker_processes, :before_fork, :after_fork, :before_exec, :logger, :pid, :listener_opts, :preload_app, :reexec_pid, :orig_app, :init_listeners, @@ -393,7 +393,7 @@ module Unicorn when nil # avoid murdering workers after our master process (or the # machine) comes out of suspend/hibernation - if (last_check + timeout) >= (last_check = Time.now) + if (last_check + soft_timeout) >= (last_check = Time.now) murder_lazy_workers else # wait for workers to wakeup on suspend @@ -581,10 +581,20 @@ module Unicorn stat = worker.tmp.stat # skip workers that disable fchmod or have never fchmod-ed stat.mode == 0100600 and next - (diff = (Time.now - stat.ctime)) <= timeout and next - logger.error "worker=#{worker.nr} PID:#{wpid} timeout " \ + # FIXME: if the worker has not been working for soft_timeout, it will be + # killed even if it is not blocking + (diff = (Time.now - stat.ctime)) <= soft_timeout and + diff <= timeout and next + # lazy since less than timeout, attempt soft kill + if diff < timeout + logger.error "worker=#{worker.nr} PID:#{wpid} soft timeout " \ + "(#{diff}s > #{soft_timeout}s), killing softly" + kill_worker(:ABRT, wpid) + else + logger.error "worker=#{worker.nr} PID:#{wpid} hard timeout " \ "(#{diff}s > #{timeout}s), killing" - kill_worker(:KILL, wpid) # take no prisoners for timeout violations + kill_worker(:KILL, wpid) # take no prisoners for timeout violations + end end end @@ -657,6 +667,12 @@ module Unicorn proc_name "worker[#{worker.nr}]" START_CTX.clear init_self_pipe! + + # try to handle SIGABRT correctly + trap('ABRT') do + raise SignalException, "SIGABRT" + end + WORKERS.values.each { |other| other.tmp.close rescue nil } WORKERS.clear LISTENERS.each { |sock| sock.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) } diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb index 64a25e3..6efb0c5 100644 --- a/lib/unicorn/configurator.rb +++ b/lib/unicorn/configurator.rb @@ -14,6 +14,8 @@ module Unicorn # Default settings for Unicorn DEFAULTS = { + # Backward compatibility soft timeout (disabled in default configuration) + :soft_timeout => 60, :timeout => 60, :logger => Logger.new($stderr), :worker_processes => 1, @@ -129,6 +131,23 @@ module Unicorn # sets the timeout of worker processes to +seconds+. Workers # handling the request/app.call/response cycle taking longer than + # this time period will be softly killed (via SIGABRT). This + # timeout is enforced by the master process itself and not subject + # to the scheduling limitations by the worker process. Due the + # low-complexity, low-overhead implementation, timeouts of less + # than 3.0 seconds can be considered inaccurate and unsafe. + # ABORT is handled by the worker and raise an exception, offering a + # way to log the stack trace in your rails application. + + def soft_timeout(seconds) + Numeric === seconds or raise ArgumentError, + "not numeric: timeout=#{seconds.inspect}" + seconds >= 3 or raise ArgumentError, + "too low: timeout=#{seconds.inspect}" + set[:soft_timeout] = seconds + end + # sets the timeout of worker processes to +seconds+. Workers + # handling the request/app.call/response cycle taking longer than # this time period will be forcibly killed (via SIGKILL). This # timeout is enforced by the master process itself and not subject # to the scheduling limitations by the worker process. Due the @@ -159,6 +178,7 @@ module Unicorn set[:timeout] = seconds end + # sets the current number of worker_processes to +nr+. Each worker # process will serve exactly one client at a time. You can # increment or decrement this value at runtime by sending SIGTTIN Cheers, -- Pierre Baillet <oct@fotonauts.com> http://www.fotopedia.com/ _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: Fwd: Support for Soft Timeout in Unicorn 2010-06-03 17:37 Fwd: Support for Soft Timeout in Unicorn Eric Wong 2010-06-03 18:06 ` Pierre Baillet @ 2010-06-03 18:22 ` Eric Wong 2010-06-03 18:32 ` Pierre Baillet 1 sibling, 1 reply; 13+ messages in thread From: Eric Wong @ 2010-06-03 18:22 UTC (permalink / raw) To: mongrel-unicorn; +Cc: Pierre Baillet Pierre Baillet wrote: > We use Unicorn at fotopedia since yesterday in production. We switched from > Passenger due to an issue in the way Passenger was handling some error in > our main application. Things run very well on Unicorn. > > We have also modified Unicorn to handle a soft timeout for its workers. The > Unicorn timeout was killing the workers without any chance for us to catch > the Rails stack trace and identify the issue. I've forked and branched > Unicorn from github and added a soft_timeout configuration value that is > used for long running workers. > > The workers now handle SIGABRT and will raise an exception. This will crash > the Rails application if it can be crashed and force the framework to dump > the stack trace in the logs. Let me know if this might be useful for other > people and, why not, integrate that in the main Unicorn code ! Hi Pierre, I'm thinking there's a better way to do this without involving the master process. The current timeout implementation[1] is really the last resort, point-of-no-return situations when the workers are completely stuck/blocked and cannot respond to other signals[2]. If the worker can respond to SIGABRT (especially going through the interpreter and raising an exception), then that means it could technically respond to Thread#raise, too... Unfortunately, the current core Ruby timeout implementation is extremely naive and inefficient. It shouldn't be hard to write a better timeout implementation that reuses a single timer Thread which can be rearmed with every request. This would be doable as middleware, too, and if done carefully, even safely reusable in multi-threaded web servers. This would be a good addition to rack-contrib, even. I might consider doing it myself if I had time. [1] - I'm not completely happy with "needing" the current timeout implementation in the first place. I will at least redo it at some point (after Unicorn 1.x) to gain some scalability/performance (and perhaps lose some portability). [2] - SIGKILL and SIGSTOP are special, userspace has no mechanism to block/catch/ignore those signals, so we rely on the master process to deliver them. -- Eric Wong _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fwd: Support for Soft Timeout in Unicorn 2010-06-03 18:22 ` Fwd: " Eric Wong @ 2010-06-03 18:32 ` Pierre Baillet 2010-06-03 18:47 ` Eric Wong 0 siblings, 1 reply; 13+ messages in thread From: Pierre Baillet @ 2010-06-03 18:32 UTC (permalink / raw) To: Eric Wong; +Cc: mongrel-unicorn Eric, On Thu, Jun 3, 2010 at 8:22 PM, Eric Wong <normalperson@yhbt.net> wrote: > This would be doable as middleware, too, and if done carefully, even > safely reusable in multi-threaded web servers. This would be a good > addition to rack-contrib, even. I might consider doing it myself if I > had time. I also think that doing this as a Rack middleware is probably the right way to do that kind of worker management. However the application we use at fotopedia is still Rails 2.1 based (hence not Rack compatible AFAIK). This probably means we'll have keep on maintaining this slightly hacked version on our own then. Thank you for Unicorn ! :) -- Pierre Baillet <oct@fotonauts.com> http://www.fotopedia.com/ _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fwd: Support for Soft Timeout in Unicorn 2010-06-03 18:32 ` Pierre Baillet @ 2010-06-03 18:47 ` Eric Wong 2010-06-03 19:38 ` Chris Wanstrath 0 siblings, 1 reply; 13+ messages in thread From: Eric Wong @ 2010-06-03 18:47 UTC (permalink / raw) To: Pierre Baillet; +Cc: mongrel-unicorn Pierre Baillet <oct@fotonauts.com> wrote: > Eric, > > On Thu, Jun 3, 2010 at 8:22 PM, Eric Wong <normalperson@yhbt.net> wrote: > > This would be doable as middleware, too, and if done carefully, even > > safely reusable in multi-threaded web servers. This would be a good > > addition to rack-contrib, even. I might consider doing it myself if I > > had time. > > I also think that doing this as a Rack middleware is probably the > right way to do that kind of worker management. However the > application we use at fotopedia is still Rails 2.1 based (hence not > Rack compatible AFAIK). This probably means we'll have keep on > maintaining this slightly hacked version on our own then. Actually, internally, Unicorn only knows about Rack and wraps older CGI-based Rails using the Unicorn::App::OldRails application (via Unicorn::CGIWrapper). "unicorn_rails" basically wraps up the following config for you, but you could achieve the same effect using "unicorn" and an explicitly written config.ru: ------------ config.ru ------------- ENV["RAILS_ENV"] ||= ENV["RACK_ENV"] require 'config/boot' require 'config/environment' require 'unicorn/app/old_rails' require 'unwritten_timeout_middleware' use UnwrittenTimeoutMiddleware # :) use Unicorn::App::OldRails::Static # optional run Unicorn::App::OldRails.new ------------------------------------ > Thank you for Unicorn ! :) No problem :) -- Eric Wong _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fwd: Support for Soft Timeout in Unicorn 2010-06-03 18:47 ` Eric Wong @ 2010-06-03 19:38 ` Chris Wanstrath 2010-06-03 19:40 ` Pierre Baillet 2010-06-04 20:59 ` Eric Wong 0 siblings, 2 replies; 13+ messages in thread From: Chris Wanstrath @ 2010-06-03 19:38 UTC (permalink / raw) To: unicorn list; +Cc: Pierre Baillet On Thu, Jun 3, 2010 at 11:47 AM, Eric Wong <normalperson@yhbt.net> wrote: > Actually, internally, Unicorn only knows about Rack and wraps older > CGI-based Rails using the Unicorn::App::OldRails application (via > Unicorn::CGIWrapper). > > "unicorn_rails" basically wraps up the following config for you, > but you could achieve the same effect using "unicorn" and an > explicitly written config.ru: That's what we do at GitHub. We're running Rails 2.2.2 and have a custom config.ru, thanks to Unicorn: http://gist.github.com/424352 _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fwd: Support for Soft Timeout in Unicorn 2010-06-03 19:38 ` Chris Wanstrath @ 2010-06-03 19:40 ` Pierre Baillet 2010-06-09 13:17 ` Pierre Baillet 2010-06-04 20:59 ` Eric Wong 1 sibling, 1 reply; 13+ messages in thread From: Pierre Baillet @ 2010-06-03 19:40 UTC (permalink / raw) To: Chris Wanstrath; +Cc: unicorn list Ohai, On Thu, Jun 3, 2010 at 9:38 PM, Chris Wanstrath <chris@ozmm.org> wrote: > On Thu, Jun 3, 2010 at 11:47 AM, Eric Wong <normalperson@yhbt.net> wrote: > >> Actually, internally, Unicorn only knows about Rack and wraps older >> CGI-based Rails using the Unicorn::App::OldRails application (via >> Unicorn::CGIWrapper). >> >> "unicorn_rails" basically wraps up the following config for you, >> but you could achieve the same effect using "unicorn" and an >> explicitly written config.ru: > > That's what we do at GitHub. We're running Rails 2.2.2 and have a > custom config.ru, thanks to Unicorn: > That's great, thanks for the Idea, I'll look into rack middleware then. I'll let you know if I manage to have something clean enough. Cheers, -- Pierre Baillet <oct@fotonauts.com> http://www.fotopedia.com/ _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fwd: Support for Soft Timeout in Unicorn 2010-06-03 19:40 ` Pierre Baillet @ 2010-06-09 13:17 ` Pierre Baillet 2010-06-11 1:56 ` Eric Wong 0 siblings, 1 reply; 13+ messages in thread From: Pierre Baillet @ 2010-06-09 13:17 UTC (permalink / raw) To: Chris Wanstrath; +Cc: unicorn list Hello Unicorns, I've manage to create a simple middleware that replaces the soft timeout feature. You can have a look at it at http://gist.github.com/431451 Note that some weird Ruby interpreter behavior breaks at least the first level of the generated stacktrace (it indicates the actual method where the raise happened but the wrong line number and file). Thanks for your assistance, Cheers, -- Pierre. On Thu, Jun 3, 2010 at 9:40 PM, Pierre Baillet <oct@fotonauts.com> wrote: > Ohai, > > On Thu, Jun 3, 2010 at 9:38 PM, Chris Wanstrath <chris@ozmm.org> wrote: >> On Thu, Jun 3, 2010 at 11:47 AM, Eric Wong <normalperson@yhbt.net> wrote: >> >>> Actually, internally, Unicorn only knows about Rack and wraps older >>> CGI-based Rails using the Unicorn::App::OldRails application (via >>> Unicorn::CGIWrapper). >>> >>> "unicorn_rails" basically wraps up the following config for you, >>> but you could achieve the same effect using "unicorn" and an >>> explicitly written config.ru: >> >> That's what we do at GitHub. We're running Rails 2.2.2 and have a >> custom config.ru, thanks to Unicorn: >> > > That's great, thanks for the Idea, I'll look into rack middleware > then. I'll let you know if I manage to have something clean enough. > > Cheers, > -- > Pierre Baillet <oct@fotonauts.com> > http://www.fotopedia.com/ > -- Pierre Baillet <oct@fotonauts.com> http://www.fotopedia.com/ _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fwd: Support for Soft Timeout in Unicorn 2010-06-09 13:17 ` Pierre Baillet @ 2010-06-11 1:56 ` Eric Wong 0 siblings, 0 replies; 13+ messages in thread From: Eric Wong @ 2010-06-11 1:56 UTC (permalink / raw) To: unicorn list Pierre Baillet <oct@fotonauts.com> wrote: > Hello Unicorns, > > I've manage to create a simple middleware that replaces the soft > timeout feature. You can have a look at it at > http://gist.github.com/431451 > > Note that some weird Ruby interpreter behavior breaks at least the > first level of the generated stacktrace (it indicates the actual > method where the raise happened but the wrong line number and file). You should be able to avoid spawning a new thread for every request, as that can get very expensive on some systems. Maybe a global hash guarded by a mutex that tells the worker Thread which thread to raise on. The worker thread could sleep until the next timeout registered. But then again, take care to only spawn new threads in workers (with preload_app=true), as threads never get carried across fork. And threads may leave mutexes and such in a bad state when they vanish in the child. -- Eric Wong _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fwd: Support for Soft Timeout in Unicorn 2010-06-03 19:38 ` Chris Wanstrath 2010-06-03 19:40 ` Pierre Baillet @ 2010-06-04 20:59 ` Eric Wong 2010-06-18 20:13 ` Ryan Tomayko 1 sibling, 1 reply; 13+ messages in thread From: Eric Wong @ 2010-06-04 20:59 UTC (permalink / raw) To: unicorn list; +Cc: Luke Melia Chris Wanstrath <chris@ozmm.org> wrote: > That's what we do at GitHub. We're running Rails 2.2.2 and have a > custom config.ru, thanks to Unicorn: > > http://gist.github.com/424352 By the way, how's the OobGC middleware working for you guys? Luke: did you also get a chance to try this in place of my original monkey patch? Thanks in advance for any info you can share. -- Eric Wong _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fwd: Support for Soft Timeout in Unicorn 2010-06-04 20:59 ` Eric Wong @ 2010-06-18 20:13 ` Ryan Tomayko 2010-06-18 21:48 ` Eric Wong 0 siblings, 1 reply; 13+ messages in thread From: Ryan Tomayko @ 2010-06-18 20:13 UTC (permalink / raw) To: unicorn list; +Cc: Luke Melia On Fri, Jun 4, 2010 at 1:59 PM, Eric Wong <normalperson@yhbt.net> wrote: > Chris Wanstrath <chris@ozmm.org> wrote: >> That's what we do at GitHub. We're running Rails 2.2.2 and have a >> custom config.ru, thanks to Unicorn: >> >> http://gist.github.com/424352 > > By the way, how's the OobGC middleware working for you guys? We rolled out the OobGC middleware along with a basic RailsBench GC config (RUBY_HEAP_MIN_SLOTS, etc.). Combined, they knocked about 20ms (~15%) off the average response time across the site (real traffic). The impact was significantly more for requests that allocate a lot of objects -- as much as 50% decreases in response time for the worst offenders. We saw no noticeable increase in CPU with OobGC set to run every 10 requests, and a fair increase in CPU with OobGC set to run every 5 requests. Because I rolled this stuff out somewhat non-scientifically, I've always wondered how much OobGC contributed to the overall savings vs. the RailsBench GC config. Disabling the OobGC middleware but leaving the RailsBench GC config in place, I get the following graph: http://img.skitch.com/20100618-kihdc1cq6pjhq9rqftf8miuf6y.png So we're spending ~1ms request time in GC with OobGC, and ~10ms request time in GC without it. Here's some system load graphs for the same time period just to show that OobGC has no adverse effect when set to GC every 10 requests: http://img.skitch.com/20100618-qp7p8f6i2agbqbdnjbpigik1d9.png I assume the RailsBench GC patches improve the effect of OobGC considerably by increasing the number of objects that can be allocated between GC runs, allowing more of the GC work to be deferred to in-between-requests time. Here's the RailsBench config we're using today, for the record: RUBY_HEAP_MIN_SLOTS=800000 RUBY_HEAP_FREE_MIN=100000 RUBY_HEAP_SLOTS_INCREMENT=300000 RUBY_HEAP_SLOTS_GROWTH_FACTOR=1 RUBY_GC_MALLOC_LIMIT=79000000 This is only barely tuned for us. I stole most of the numbers from the Twitter and 37signals examples. I've also experimented with tuning the GC specifically to take advantage of OobGC: https://gist.github.com/87d574a19372c6043c5f # The following GC settings have been tuned for GitHub application web requests. # Most settings are significantly higher than the example configs published by # Twitter and 37signals. There's a couple reasons for this. First, the GitHub # app has a memory footprint that's 3x-4x larger than the standard Rails app # (roughly 200MB after first request compared to ~40MB-50MB). Second, because # Unicorn is such an exceptional piece of software, we're able to schedule GC # to run outside the context of requests so as not to effect response times. # As such, we try to allocate enough memory to service 5 requests without needing # GC and then run GC manually immediately after each fifth request has been # served but before the process starts accepting the next connection. The result # is higher memory use (~300MB per Unicorn worker process on average) and a # slight increase in CPU due to forced manual GC, but better response times. # ... Unfortunately, the bigger heap seems to cause a largish increase in the time needed for each GC, so the unicorn workers were spending too much time between requests. CPU and RES increases were even more than I'd expected. It also didn't eliminate in-request GC entirely as I'd hoped. I eventually abandoned the idea -- even if I could get it to behave, it's hardly worth the 1ms it would save. I mention it here because the general approach might work in situations where the base heap size is a bit smaller (say < 80MB) or perhaps I'm mistuning one of the parameters. Thanks, Ryan > Luke: did you also get a chance to try this in place of my original > monkey patch? > > Thanks in advance for any info you can share. > > -- > Eric Wong > _______________________________________________ > Unicorn mailing list - mongrel-unicorn@rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying > _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fwd: Support for Soft Timeout in Unicorn 2010-06-18 20:13 ` Ryan Tomayko @ 2010-06-18 21:48 ` Eric Wong 2010-06-21 19:03 ` Ryan Tomayko 0 siblings, 1 reply; 13+ messages in thread From: Eric Wong @ 2010-06-18 21:48 UTC (permalink / raw) To: unicorn list; +Cc: Luke Melia Ryan Tomayko <r@tomayko.com> wrote: > On Fri, Jun 4, 2010 at 1:59 PM, Eric Wong <normalperson@yhbt.net> wrote: > > Chris Wanstrath <chris@ozmm.org> wrote: > >> That's what we do at GitHub. We're running Rails 2.2.2 and have a > >> custom config.ru, thanks to Unicorn: > >> > >> http://gist.github.com/424352 > > > > By the way, how's the OobGC middleware working for you guys? > > We rolled out the OobGC middleware along with a basic RailsBench GC > config (RUBY_HEAP_MIN_SLOTS, etc.). Combined, they knocked about 20ms > (~15%) off the average response time across the site (real traffic). > The impact was significantly more for requests that allocate a lot of > objects -- as much as 50% decreases in response time for the worst > offenders. We saw no noticeable increase in CPU with OobGC set to run > every 10 requests, and a fair increase in CPU with OobGC set to run > every 5 requests. Cool. Am I correct to assume the increased CPU usage at every 5 requests wasn't worth any performance gains you might have had? > Because I rolled this stuff out somewhat non-scientifically, I've > always wondered how much OobGC contributed to the overall savings vs. > the RailsBench GC config. Disabling the OobGC middleware but leaving > the RailsBench GC config in place, I get the following graph: > > http://img.skitch.com/20100618-kihdc1cq6pjhq9rqftf8miuf6y.png > > So we're spending ~1ms request time in GC with OobGC, and ~10ms > request time in GC without it. Awesome. > Here's some system load graphs for the same time period just to show > that OobGC has no adverse effect when set to GC every 10 requests: > > http://img.skitch.com/20100618-qp7p8f6i2agbqbdnjbpigik1d9.png > > I assume the RailsBench GC patches improve the effect of OobGC > considerably by increasing the number of objects that can be allocated > between GC runs, allowing more of the GC work to be deferred to > in-between-requests time. Here's the RailsBench config we're using > today, for the record: > > RUBY_HEAP_MIN_SLOTS=800000 > RUBY_HEAP_FREE_MIN=100000 > RUBY_HEAP_SLOTS_INCREMENT=300000 > RUBY_HEAP_SLOTS_GROWTH_FACTOR=1 > RUBY_GC_MALLOC_LIMIT=79000000 > > This is only barely tuned for us. I stole most of the numbers from the > Twitter and 37signals examples. > > I've also experimented with tuning the GC specifically to take > advantage of OobGC: > > https://gist.github.com/87d574a19372c6043c5f > > # The following GC settings have been tuned for GitHub application web requests. > # Most settings are significantly higher than the example configs published by > # Twitter and 37signals. There's a couple reasons for this. First, the GitHub > # app has a memory footprint that's 3x-4x larger than the standard Rails app > # (roughly 200MB after first request compared to ~40MB-50MB). Second, because Yikes, 200MB after one request is a lot. If you can easily find ways to cut that down, it should be more of a gain than the monster heap you've tried. > # Unicorn is such an exceptional piece of software, we're able to schedule GC > # to run outside the context of requests so as not to effect response times. > # As such, we try to allocate enough memory to service 5 requests > without needing > # GC and then run GC manually immediately after each fifth request has been > # served but before the process starts accepting the next connection. The result > # is higher memory use (~300MB per Unicorn worker process on average) and a > # slight increase in CPU due to forced manual GC, but better response times. > # ... > > Unfortunately, the bigger heap seems to cause a largish increase in > the time needed for each GC, so the unicorn workers were spending too > much time between requests. CPU and RES increases were even more than > I'd expected. It also didn't eliminate in-request GC entirely as I'd > hoped. > > I eventually abandoned the idea -- even if I could get it to behave, > it's hardly worth the 1ms it would save. I mention it here because the > general approach might work in situations where the base heap size is > a bit smaller (say < 80MB) or perhaps I'm mistuning one of the > parameters. So in conclusion, OobGC itself works, but too large of a heap isn't worth it even for a memory hungry app. I suppose having too big of a heap means it can fragment more badly. Making GC run more often on a smaller heap can and give similar (or maybe better) performance. At best you'll get diminishing returns as you seem to have concluded. I have no doubt the Railsbench GC patches help. Even with small apps, being able being able to set a linear growth factor on long-running servers is awesome. Thanks for sharing this! -- Eric Wong _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Fwd: Support for Soft Timeout in Unicorn 2010-06-18 21:48 ` Eric Wong @ 2010-06-21 19:03 ` Ryan Tomayko 0 siblings, 0 replies; 13+ messages in thread From: Ryan Tomayko @ 2010-06-21 19:03 UTC (permalink / raw) To: unicorn list; +Cc: Luke Melia On Fri, Jun 18, 2010 at 2:48 PM, Eric Wong <normalperson@yhbt.net> wrote: > Ryan Tomayko <r@tomayko.com> wrote: >> On Fri, Jun 4, 2010 at 1:59 PM, Eric Wong <normalperson@yhbt.net> wrote: >> > Chris Wanstrath <chris@ozmm.org> wrote: >> >> That's what we do at GitHub. We're running Rails 2.2.2 and have a >> >> custom config.ru, thanks to Unicorn: >> >> >> >> http://gist.github.com/424352 >> > >> > By the way, how's the OobGC middleware working for you guys? >> >> We rolled out the OobGC middleware along with a basic RailsBench GC >> config (RUBY_HEAP_MIN_SLOTS, etc.). Combined, they knocked about 20ms >> (~15%) off the average response time across the site (real traffic). >> The impact was significantly more for requests that allocate a lot of >> objects -- as much as 50% decreases in response time for the worst >> offenders. We saw no noticeable increase in CPU with OobGC set to run >> every 10 requests, and a fair increase in CPU with OobGC set to run >> every 5 requests. > > Cool. Am I correct to assume the increased CPU usage at every 5 > requests wasn't worth any performance gains you might have had? Yes. I don't have exact numbers on how much more CPU was being utilized but it was definitely noticeable on my graphs and had very little, if any, effect on overall response time beyond the savings at the req/10 interval. >> Because I rolled this stuff out somewhat non-scientifically, I've >> always wondered how much OobGC contributed to the overall savings vs. >> the RailsBench GC config. Disabling the OobGC middleware but leaving >> the RailsBench GC config in place, I get the following graph: >> >> http://img.skitch.com/20100618-kihdc1cq6pjhq9rqftf8miuf6y.png >> >> So we're spending ~1ms request time in GC with OobGC, and ~10ms >> request time in GC without it. > > Awesome. > >> Here's some system load graphs for the same time period just to show >> that OobGC has no adverse effect when set to GC every 10 requests: >> >> http://img.skitch.com/20100618-qp7p8f6i2agbqbdnjbpigik1d9.png >> >> I assume the RailsBench GC patches improve the effect of OobGC >> considerably by increasing the number of objects that can be allocated >> between GC runs, allowing more of the GC work to be deferred to >> in-between-requests time. Here's the RailsBench config we're using >> today, for the record: >> >> RUBY_HEAP_MIN_SLOTS=800000 >> RUBY_HEAP_FREE_MIN=100000 >> RUBY_HEAP_SLOTS_INCREMENT=300000 >> RUBY_HEAP_SLOTS_GROWTH_FACTOR=1 >> RUBY_GC_MALLOC_LIMIT=79000000 >> >> This is only barely tuned for us. I stole most of the numbers from the >> Twitter and 37signals examples. >> >> I've also experimented with tuning the GC specifically to take >> advantage of OobGC: >> >> https://gist.github.com/87d574a19372c6043c5f >> >> # The following GC settings have been tuned for GitHub application web requests. >> # Most settings are significantly higher than the example configs published by >> # Twitter and 37signals. There's a couple reasons for this. First, the GitHub >> # app has a memory footprint that's 3x-4x larger than the standard Rails app >> # (roughly 200MB after first request compared to ~40MB-50MB). Second, because > > Yikes, 200MB after one request is a lot. If you can easily find ways to > cut that down, it should be more of a gain than the monster heap you've > tried. Indeed. We bring in *a lot* of libraries, some of which add embarassingly large amounts of code and data at require time (aws-s3, googlecharts - I'm looking at you). >> # Unicorn is such an exceptional piece of software, we're able to schedule GC >> # to run outside the context of requests so as not to effect response times. >> # As such, we try to allocate enough memory to service 5 requests >> without needing >> # GC and then run GC manually immediately after each fifth request has been >> # served but before the process starts accepting the next connection. The result >> # is higher memory use (~300MB per Unicorn worker process on average) and a >> # slight increase in CPU due to forced manual GC, but better response times. >> # ... >> >> Unfortunately, the bigger heap seems to cause a largish increase in >> the time needed for each GC, so the unicorn workers were spending too >> much time between requests. CPU and RES increases were even more than >> I'd expected. It also didn't eliminate in-request GC entirely as I'd >> hoped. >> >> I eventually abandoned the idea -- even if I could get it to behave, >> it's hardly worth the 1ms it would save. I mention it here because the >> general approach might work in situations where the base heap size is >> a bit smaller (say < 80MB) or perhaps I'm mistuning one of the >> parameters. > > So in conclusion, OobGC itself works, but too large of a heap isn't > worth it even for a memory hungry app. > > I suppose having too big of a heap means it can fragment more badly. > Making GC run more often on a smaller heap can and give similar (or > maybe better) performance. At best you'll get diminishing returns as > you seem to have concluded. That sounds like an extremely plausible explanation. > I have no doubt the Railsbench GC patches help. Even with small apps, > being able being able to set a linear growth factor on long-running > servers is awesome. Sure is. Thanks, Ryan > Thanks for sharing this! > > -- > Eric Wong > _______________________________________________ > Unicorn mailing list - mongrel-unicorn@rubyforge.org > http://rubyforge.org/mailman/listinfo/mongrel-unicorn > Do not quote signatures (like this one) or top post when replying > _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2010-06-21 19:21 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2010-06-03 17:37 Fwd: Support for Soft Timeout in Unicorn Eric Wong 2010-06-03 18:06 ` Pierre Baillet 2010-06-03 18:22 ` Fwd: " Eric Wong 2010-06-03 18:32 ` Pierre Baillet 2010-06-03 18:47 ` Eric Wong 2010-06-03 19:38 ` Chris Wanstrath 2010-06-03 19:40 ` Pierre Baillet 2010-06-09 13:17 ` Pierre Baillet 2010-06-11 1:56 ` Eric Wong 2010-06-04 20:59 ` Eric Wong 2010-06-18 20:13 ` Ryan Tomayko 2010-06-18 21:48 ` Eric Wong 2010-06-21 19:03 ` Ryan Tomayko
Code repositories for project(s) associated with this public inbox https://yhbt.net/unicorn.git/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).
