unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: Tony Devlin <tonydevlin@gmail.com>
To: Eric Wong <e@80x24.org>
Cc: Daniel Condomitti <daniel@condomitti.com>, unicorn-public@bogomips.org
Subject: Re: Weird Unicorn Timeout Issues (Hibernation problem?)
Date: Mon, 4 Aug 2014 16:24:56 -0400	[thread overview]
Message-ID: <CAKM1sPMMi+2Xs1EZfSmJbqojx2zD_EPGmze72BAsmW-WwH+umQ@mail.gmail.com> (raw)
In-Reply-To: <20140804193445.GA31366@dcvr.yhbt.net>

Yep, it occurs after 30 minutes of inactivity.  Down to the minute; I hit
the site at 3:40 and tried at 4:10 and sure enough:

E, [2014-08-04T16:10:52.143541 #2596] ERROR -- : worker=3D0 PID:2599 timeou=
t
(21s > 20s), killing
E, [2014-08-04T16:10:52.158459 #2596] ERROR -- : reaped #<Process::Status:
pid 2599 SIGKILL (signal 9)> worker=3D0
I, [2014-08-04T16:10:52.181648 #3086]  INFO -- : worker=3D0 ready

2014/08/04 16:10:52 [error] 1684#0: *13 upstream prematurely closed
connection while reading response header from upstream, client: *.*.*.*,
server: ***.org, request: "GET /outages HTTP/1.1", upstream:
"http://unix:/var/www/sites/***/shared/sockets/.unicorn.sock.0:/outages",
host: "***.org", referrer: "http://***.org/outages"

=E2=80=8B=E2=80=8B=3D=3D=3D

This occurs on both instances of unicorn workers that we have opened.  I'm
going to reduce that to one instance, per Eric, to continue troubleshooting
in the smallest possible way.

1) It does not appear to be an nginx persistent connection issue, because
once the worker is reaped and restarted, nginx serves the content with no
problems.
2) No NFS mounts, no file locks, no FIFO issues.  (note: one of the apps
does write to files, aside from logs, but problem exists in both apps).

It's also important to note that once the worker is reaped the site is
blazingly fast, sub second responses (2s most time spent to show the
biggest page).  Until 30 minutes of inactivity, in which case timeout issue
and worker is reaped (rinse and repeat).

For the database portion, the DBA says inactivity is killed after 3 hours.
 Far greater time span than this issue is occurring.

Have any other ideas of places I can look?  It's too consistent, it has to
be some specific setting or functionality that does this.

I checked my TCP Timeout settings just in case, but the timeout is set to
2hrs.


On Mon, Aug 4, 2014 at 3:34 PM, Eric Wong <e@80x24.org> wrote:

> Daniel Condomitti <daniel@condomitti.com> wrote:
> > It could also be that your TCP keepalive interval is higher than your
> > database server=E2=80=99s connection timeout. I=E2=80=99ve run into tha=
t in the past.
>
> That kicks in at around 2 hours by default on Linux systems.
> I'm not sure it would matter for Tony's case since he hit it
> after ~30 minutes of idle (unless he tuned the knobs himself).
>
> ref: tcp_keep* knobs in
>
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Docu=
mentation/networking/ip-sysctl.txt
>
> unicorn itself has no timers outside of the configurable timeout.
>


  reply	other threads:[~2014-08-04 20:24 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2014-08-04 18:12 Weird Unicorn Timeout Issues (Hibernation problem?) Tony Devlin
2014-08-04 18:39 ` Eric Wong
2014-08-04 18:41   ` Kapil Israni
2014-08-04 18:48     ` Eric Wong
2014-08-04 18:45   ` Tony Devlin
2014-08-04 19:11     ` Eric Wong
2014-08-04 19:21       ` Tony Devlin
2014-08-04 19:45         ` Eric Wong
2014-08-04 20:06           ` Michael Fischer
2014-08-04 18:55   ` Daniel Condomitti
2014-08-04 19:34     ` Eric Wong
2014-08-04 20:24       ` Tony Devlin [this message]
2014-08-04 20:44         ` Eric Wong
2014-08-04 20:46           ` Eric Wong
2014-08-05 14:46             ` Tony Devlin
2014-08-06  9:45               ` Eric Wong
2014-08-06 14:05                 ` Tony Devlin
2014-08-06 18:26                   ` Daniel Condomitti

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAKM1sPMMi+2Xs1EZfSmJbqojx2zD_EPGmze72BAsmW-WwH+umQ@mail.gmail.com \
    --to=tonydevlin@gmail.com \
    --cc=daniel@condomitti.com \
    --cc=e@80x24.org \
    --cc=unicorn-public@bogomips.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).