From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS14383 205.234.109.0/24 X-Spam-Status: No, score=-0.7 required=5.0 tests=AWL,MSGID_FROM_MTA_HEADER, RP_MATCHES_RCVD shortcircuit=no autolearn=unavailable version=3.3.2 Path: news.gmane.org!not-for-mail From: Pierre Baillet Newsgroups: gmane.comp.lang.ruby.unicorn.general Subject: Re: Support for Soft Timeout in Unicorn Date: Thu, 3 Jun 2010 20:06:02 +0200 Message-ID: References: <20100603173749.GA19649@dcvr.yhbt.net> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Trace: dough.gmane.org 1275589044 5811 80.91.229.12 (3 Jun 2010 18:17:24 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Thu, 3 Jun 2010 18:17:24 +0000 (UTC) Cc: mongrel-unicorn@rubyforge.org To: Eric Wong Original-X-From: mongrel-unicorn-bounces@rubyforge.org Thu Jun 03 20:17:13 2010 Return-path: Envelope-to: gclrug-mongrel-unicorn@m.gmane.org X-Original-To: mongrel-unicorn@rubyforge.org Delivered-To: mongrel-unicorn@rubyforge.org In-Reply-To: <20100603173749.GA19649@dcvr.yhbt.net> X-BeenThere: mongrel-unicorn@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: mongrel-unicorn-bounces@rubyforge.org Errors-To: mongrel-unicorn-bounces@rubyforge.org Xref: news.gmane.org gmane.comp.lang.ruby.unicorn.general:522 Archived-At: Received: from rubyforge.org ([205.234.109.19]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1OKEye-0001Ds-IC for gclrug-mongrel-unicorn@m.gmane.org; Thu, 03 Jun 2010 20:17:13 +0200 Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id E0EB91858335; Thu, 3 Jun 2010 14:17:11 -0400 (EDT) Received: from exprod7og116.obsmtp.com (exprod7og116.obsmtp.com [64.18.2.219]) by rubyforge.org (Postfix) with SMTP id 47984185834B for ; Thu, 3 Jun 2010 14:06:05 -0400 (EDT) Received: from source ([72.14.220.155]) by exprod7ob116.postini.com ([64.18.6.12]) with SMTP ID DSNKTAfvDIZ4xMJ9IOjbZRyn93vMWvMC17Wh@postini.com; Thu, 03 Jun 2010 11:06:05 PDT Received: by fg-out-1718.google.com with SMTP id 22so418699fge.5 for ; Thu, 03 Jun 2010 11:06:04 -0700 (PDT) Received: by 10.239.185.76 with SMTP id b12mr721405hbh.78.1275588362369; Thu, 03 Jun 2010 11:06:02 -0700 (PDT) Received: by 10.239.187.68 with HTTP; Thu, 3 Jun 2010 11:06:02 -0700 (PDT) Hello, On Thu, Jun 3, 2010 at 7:37 PM, Eric Wong=A0=A0wrote: > > Hi, > > HTML attachments are wasteful and thus rejected from the mailing list. > On the other hand, it actually helps to include the patch itself > (inline) so it's readable without a (human) context switch :) Indeed, sorry for the HTML attachment, I have no idea where it comes from. As for the patch, here you are. This is really just a way to handle SIGABRT in a specific way in the worker and allow proper termination of the application. Note the FIXME comment I added in the murder_lazy_workers method. If any worker blocks while all the others are idle for a _timeout_ period of time, they will all be killed anyway. The consequence of that is that Unicorn will restart all its workers if traffic is very low on the server. diff --git a/lib/unicorn.rb b/lib/unicorn.rb index a363014..855f26a 100644 --- a/lib/unicorn.rb +++ b/lib/unicorn.rb @@ -84,7 +84,7 @@ module Unicorn # Listener sockets are started in the master process and shared with # forked worker children. - class HttpServer < Struct.new(:app, :timeout, :worker_processes, + class HttpServer < Struct.new(:app, :soft_timeout, :timeout, :worker_processes, :before_fork, :after_fork, :before_exec, :logger, :pid, :listener_opts, :preload_ap= p, :reexec_pid, :orig_app, :init_listeners, @@ -393,7 +393,7 @@ module Unicorn when nil # avoid murdering workers after our master process (or the # machine) comes out of suspend/hibernation - if (last_check + timeout) >=3D (last_check =3D Time.now) + if (last_check + soft_timeout) >=3D (last_check =3D Time.now) murder_lazy_workers else # wait for workers to wakeup on suspend @@ -581,10 +581,20 @@ module Unicorn stat =3D worker.tmp.stat # skip workers that disable fchmod or have never fchmod-ed stat.mode =3D=3D 0100600 and next - (diff =3D (Time.now - stat.ctime)) <=3D timeout and next - logger.error "worker=3D#{worker.nr} PID:#{wpid} timeout " \ + # FIXME: if the worker has not been working for soft_timeout, it will be + # killed even if it is not blocking + (diff =3D (Time.now - stat.ctime)) <=3D soft_timeout and + diff <=3D timeout and next + # lazy since less than timeout, attempt soft kill + if diff < timeout + logger.error "worker=3D#{worker.nr} PID:#{wpid} soft timeout " \ + "(#{diff}s > #{soft_timeout}s), killing softly" + kill_worker(:ABRT, wpid) + else + logger.error "worker=3D#{worker.nr} PID:#{wpid} hard timeout " \ : diff --git a/lib/unicorn.rb b/lib/unicorn.rb index a363014..855f26a 100644 --- a/lib/unicorn.rb +++ b/lib/unicorn.rb @@ -84,7 +84,7 @@ module Unicorn # Listener sockets are started in the master process and shared with # forked worker children. - class HttpServer < Struct.new(:app, :timeout, :worker_processes, + class HttpServer < Struct.new(:app, :soft_timeout, :timeout, :worker_processes, :before_fork, :after_fork, :before_exec, :logger, :pid, :listener_opts, :preload_ap= p, :reexec_pid, :orig_app, :init_listeners, @@ -393,7 +393,7 @@ module Unicorn when nil # avoid murdering workers after our master process (or the # machine) comes out of suspend/hibernation - if (last_check + timeout) >=3D (last_check =3D Time.now) + if (last_check + soft_timeout) >=3D (last_check =3D Time.now) murder_lazy_workers else # wait for workers to wakeup on suspend @@ -581,10 +581,20 @@ module Unicorn stat =3D worker.tmp.stat # skip workers that disable fchmod or have never fchmod-ed stat.mode =3D=3D 0100600 and next - (diff =3D (Time.now - stat.ctime)) <=3D timeout and next - logger.error "worker=3D#{worker.nr} PID:#{wpid} timeout " \ + # FIXME: if the worker has not been working for soft_timeout, it will be + # killed even if it is not blocking + (diff =3D (Time.now - stat.ctime)) <=3D soft_timeout and + diff <=3D timeout and next + # lazy since less than timeout, attempt soft kill + if diff < timeout + logger.error "worker=3D#{worker.nr} PID:#{wpid} soft timeout " \ + "(#{diff}s > #{soft_timeout}s), killing softly" + kill_worker(:ABRT, wpid) + else + logger.error "worker=3D#{worker.nr} PID:#{wpid} hard timeout " \ "(#{diff}s > #{timeout}s), killing" - kill_worker(:KILL, wpid) # take no prisoners for timeout violations + kill_worker(:KILL, wpid) # take no prisoners for timeout violati= ons + end end end @@ -657,6 +667,12 @@ module Unicorn proc_name "worker[#{worker.nr}]" START_CTX.clear init_self_pipe! + + # try to handle SIGABRT correctly + trap('ABRT') do + raise SignalException, "SIGABRT" + end + WORKERS.values.each { |other| other.tmp.close rescue nil } WORKERS.clear LISTENERS.each { |sock| sock.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC= ) } diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb index 64a25e3..6efb0c5 100644 --- a/lib/unicorn/configurator.rb +++ b/lib/unicorn/configurator.rb @@ -14,6 +14,8 @@ module Unicorn # Default settings for Unicorn DEFAULTS =3D { + # Backward compatibility soft timeout (disabled in default configura= tion) + :soft_timeout =3D> 60, :timeout =3D> 60, :logger =3D> Logger.new($stderr), :worker_processes =3D> 1, @@ -129,6 +131,23 @@ module Unicorn # sets the timeout of worker processes to +seconds+. Workers # handling the request/app.call/response cycle taking longer than + # this time period will be softly killed (via SIGABRT). This + # timeout is enforced by the master process itself and not subject + # to the scheduling limitations by the worker process. Due the + # low-complexity, low-overhead implementation, timeouts of less + # than 3.0 seconds can be considered inaccurate and unsafe. + # ABORT is handled by the worker and raise an exception, offering a + # way to log the stack trace in your rails application. + + def soft_timeout(seconds) + Numeric =3D=3D=3D seconds or raise ArgumentError, + "not numeric: timeout=3D#{seconds.inspec= t}" + seconds >=3D 3 or raise ArgumentError, + "too low: timeout=3D#{seconds.inspect}" + set[:soft_timeout] =3D seconds + end + # sets the timeout of worker processes to +seconds+. Workers + # handling the request/app.call/response cycle taking longer than # this time period will be forcibly killed (via SIGKILL). This # timeout is enforced by the master process itself and not subject # to the scheduling limitations by the worker process. Due the @@ -159,6 +178,7 @@ module Unicorn set[:timeout] =3D seconds end + # sets the current number of worker_processes to +nr+. Each worker # process will serve exactly one client at a time. You can # increment or decrement this value at runtime by sending SIGTTIN Cheers, -- Pierre Baillet http://www.fotopedia.com/ _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying