unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: Alberto De Gaspari <a.degaspari@18months.it>
Cc: unicorn-public@bogomips.org
Subject: Re: Reaping process with unknown worker
Date: Thu, 19 Oct 2017 19:37:53 +0000	[thread overview]
Message-ID: <20171019193753.GA587@starla> (raw)
In-Reply-To: <f6bc3edb-6ff4-4d59-f235-3eaeb277cf41@18months.it>

Alberto De Gaspari <a.degaspari@18months.it> wrote:
> >> with a line like this in the log:
> >> INFO -- : reaped #<Process::Status: pid 16101 exit 0> worker=unknown
> >> if i run
> >> # ps axf |grep 16101
> >> i only get the ps line:
> >> 10277 pts/4    S+     0:00          \_ grep 16101
> >>
> >> consider that i have loads of this line in the log, like at least 1
> >> every minute.
> >>
> >> what could cause those reaps?
> > 
> > Reaping happens after a process exits, so it won't show up
> > in "ps" once it's reaped.
> > 
> ok, got it:
> but if at t0 i run ps axf and save the result, when at t1 i find one of
> the lines in the stderr_log and try to grep the stated pid in the saved
> result of t0 i don't find anything.

I guess this is because the process is too short-lived.  Can you
audit your code for when you spawn applications and check that?

Actually, what is curious is your master process is reaping
these processes (not workers reaping); yet your workers are not
dying and getting reaped.

Are you using preload_app?

If so, do you spawn any background processes at load time?
Or are there any background threads in the master process?

Does the problem go away if you disable preload_app?

Because, in normal applications, workers may spawn background
processes but its rare for the master to spawn anything but
workers themselves.

> how can i detect what it's reaping? do you consider this a normal
> behavior for a rails app which actually only receives simple get
> requests and save the result in a pg db?(that's what the small instance
> of unicorn does, the other is more complex, but shows the same amount
> and type of info messages in errors)

AFAIK, you cannot detect what it's reaping portably...

You can try the following hack to look for defunct (zombie)
processes before unicorn calls waitpid2.  However, keep in mind
it is subject to race conditions so not 100% reliable:

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index f33aa25..f57271e 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -396,6 +396,7 @@ def awaken_master
   # reaps all unreaped workers
   def reap_all_workers
     begin
+      system('ps | grep defunct')
       wpid, status = Process.waitpid2(-1, Process::WNOHANG)
       wpid or return
       if @reexec_pid == wpid



One other thing you try do (on Linux) is strace the master:

	strace -p $PID_OF_MASTER -f -e execve,clone

And see what the master is spawning.  I suppose other OS has
similar functions (truss/ktruss/...)

      reply	other threads:[~2017-10-19 19:37 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-10-19 17:42 Reaping process with unknown worker Alberto De Gaspari
2017-10-19 18:20 ` Eric Wong
2017-10-19 18:46   ` Alberto De Gaspari
2017-10-19 19:05     ` Eric Wong
2017-10-19 19:23       ` Alberto De Gaspari
2017-10-19 19:37         ` Eric Wong [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20171019193753.GA587@starla \
    --to=e@80x24.org \
    --cc=a.degaspari@18months.it \
    --cc=unicorn-public@bogomips.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).