From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS14383 205.234.109.0/24 X-Spam-Status: No, score=-0.6 required=5.0 tests=AWL,MSGID_FROM_MTA_HEADER, RP_MATCHES_RCVD,T_DKIM_INVALID shortcircuit=no autolearn=unavailable version=3.3.2 Path: news.gmane.org!not-for-mail From: Jamie Wilkinson Newsgroups: gmane.comp.lang.ruby.unicorn.general Subject: funky process tree + stillborn masters Date: Thu, 8 Apr 2010 14:21:48 -0700 Message-ID: <18DCBEFE-BC6D-41F7-996C-ACACA629ED24@tramchase.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Trace: dough.gmane.org 1270762171 26061 80.91.229.12 (8 Apr 2010 21:29:31 GMT) X-Complaints-To: usenet@dough.gmane.org NNTP-Posting-Date: Thu, 8 Apr 2010 21:29:31 +0000 (UTC) To: mongrel-unicorn@rubyforge.org Original-X-From: mongrel-unicorn-bounces@rubyforge.org Thu Apr 08 23:29:25 2010 Return-path: Envelope-to: gclrug-mongrel-unicorn@m.gmane.org X-Original-To: mongrel-unicorn@rubyforge.org Delivered-To: mongrel-unicorn@rubyforge.org X-Greylist: delayed 437 seconds by postgrey-1.31 at rubyforge.org; Thu, 08 Apr 2010 17:29:13 EDT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:from:content-type :content-transfer-encoding:subject:date:message-id:to:mime-version :x-mailer; bh=r24hoE9iL5AMT48stUdZzBCVL72jMCefEouKStAUNqs=; b=fJuiqUbou1oe78s0hDjziRxXO8v5ZMga6Y5HnY7XZVfhlg9IjKpsFuR3ZxmtsRsFEe Gf96wbnueptrdQ7ls5LUvFa5glcxkJZwwEV4xyar04SwU72NLsKLM88VXXJuHg66R6cO bWnKSoHHkBLSiNQBGhV8nABYXqPxVtWnshMEk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:from:content-type:content-transfer-encoding:subject:date :message-id:to:mime-version:x-mailer; b=TqWh5RaOiVx4zS+dXCs2d+S0f2lI7k2as67JjWkebUvqNIoenIajupnmmX5OF41A1f l53SGTE97oBx17UI3zRtIv4HHKZW0LaCwp9twOu+tuvHt6GUsKerNp4JWjGMFoeT86mw OiIiTXBAIJYXQuTkoMCWy+PvNF1WbviFhYN6Q= X-Mailer: Apple Mail (2.1078) X-BeenThere: mongrel-unicorn@rubyforge.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: mongrel-unicorn-bounces@rubyforge.org Errors-To: mongrel-unicorn-bounces@rubyforge.org Xref: news.gmane.org gmane.comp.lang.ruby.unicorn.general:460 Archived-At: Received: from rubyforge.org ([205.234.109.19]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1NzzHt-0000Dj-5Z for gclrug-mongrel-unicorn@m.gmane.org; Thu, 08 Apr 2010 23:29:21 +0200 Received: from rubyforge.org (rubyforge.org [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 73AC518582F4; Thu, 8 Apr 2010 17:29:20 -0400 (EDT) Received: from mail-bw0-f225.google.com (mail-bw0-f225.google.com [209.85.218.225]) by rubyforge.org (Postfix) with ESMTP id A1B0318582E3 for ; Thu, 8 Apr 2010 17:29:13 -0400 (EDT) Received: by bwz25 with SMTP id 25so2270637bwz.8 for ; Thu, 08 Apr 2010 14:29:13 -0700 (PDT) Received: by 10.204.142.207 with SMTP id r15mr766659bku.134.1270761715261; Thu, 08 Apr 2010 14:21:55 -0700 (PDT) Received: from [10.0.1.31] (173-13-169-18-sfba.hfc.comcastbusiness.net [173.13.169.18]) by mx.google.com with ESMTPS id 24sm3480489bkr.18.2010.04.08.14.21.52 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 08 Apr 2010 14:21:54 -0700 (PDT) Since upgrading bundler (but applying the RUBYOPT path fixes) we've started experiencing intermittent, difficult-to-isolate USR2 restart failures. After a USR2 signal our process tree winds up looking like this, with several master-esque processes listed as children (but without the "worker[N]" label): app 14402 4.4 0.8 199612 70264 ? S 14:07 0:04 unicorn_rails master -c config/unicorn.rb -E production -D app 14433 0.0 0.8 204540 68504 ? Sl 14:07 0:00 \_ unicorn_rails worker[0] -c config/unicorn.rb -E production -D app 14435 0.0 0.8 204540 68508 ? Sl 14:07 0:00 \_ unicorn_rails worker[1] -c config/unicorn.rb -E production -D app 14438 0.0 0.8 199748 65840 ? S 14:07 0:00 \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D app 14440 0.0 0.8 204540 68508 ? Sl 14:07 0:00 \_ unicorn_rails worker[3] -c config/unicorn.rb -E production -D app 14442 0.0 0.8 204540 68508 ? Sl 14:07 0:00 \_ unicorn_rails worker[4] -c config/unicorn.rb -E production -D app 14445 0.0 0.8 199760 65840 ? S 14:07 0:00 \_ /usr/bin/ruby1.8 /usr/bin/unicorn_rails -c config/unicorn.rb -E production -D app 14447 0.0 0.8 204540 68508 ? Sl 14:07 0:00 \_ unicorn_rails worker[6] -c config/unicorn.rb -E production -D app 14449 0.0 0.8 204780 69272 ? Sl 14:07 0:00 \_ unicorn_rails worker[7] -c config/unicorn.rb -E production -D Sending another USR2 signal will bring a new master into the mix as a child, spins up a single child worker of its own (which also resembles the "/usr/bin/ruby1.8" master-esque processes), and then fails to continue. Further USR2 restarts will obviously do nothing, and we're forced to either kill -9 the stillborn master or cold-restart all of the unicorns. Nothing out of the ordinary is dumped to stderr or stdout Starting unicorns fresh produces a nice process list where every child is listed cleanly as "unicorn_rails worker[N]" We only have this issue in one of our applications, on a machine that has 1 Rails app & 2 Sinatra apps, all powered by nginx+unicorn. We've also only noticed this since upgrading to bundler from bundler08 Are the goofy worker processes in the process tree a real problem, or just a red herring? -jamie _______________________________________________ Unicorn mailing list - mongrel-unicorn@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying