From: Daniel Condomitti <daniel@condomitti.com>
To: Tony Devlin <tonydevlin@gmail.com>
Cc: Eric Wong <e@80x24.org>, unicorn-public@bogomips.org
Subject: Re: Weird Unicorn Timeout Issues (Hibernation problem?)
Date: Wed, 6 Aug 2014 11:26:28 -0700 [thread overview]
Message-ID: <8E584028BCA3426ABB655EC529613E22@condomitti.com> (raw)
In-Reply-To: <CAKM1sPPsb24aNA-puN=E6aYn=RqAynMaNMBiGvWqHtyaFe4cew@mail.gmail.com>
This is exactly what happened to us and I should have been clearer. I wasn’t referring to the default Linux kernel settings causing the killing the connection; it was a network device between our application servers and the database server. It only affected certain applications as some were hit hundreds of times per second and would never be disconnected and the ones that would disconnect were only hit a few times per hour. I -believe- we just dropped the keepalive interval on both sides of the firewall below its idle timeout.
On Wednesday, August 6, 2014 at 7:05 AM, Tony Devlin wrote:
> Eric,
>
> The problem is a firewall that sits between the servers and the database. It is an idle session timeout of 30 minutes, so it is silently killing the connection. I have reached out to our Network Engineering department but they are saying they can not change that idle session timeout, nor create a special rule to allow this connection to bypass that rule.
>
> Currently, I setup a polling device that calls the applications URL every 20 minutes. This causes the connection between the server and DB to refresh it's idle timeout. This is obviously a very hacky way to handle it, so I am trying to look into AR and Oracle_Enhanced to see if they have some sort of keepalive option for the database. I thought it would work with the reaping_frequency, but apparently that does not work out as I had expected when you are not running in pools or a thread. So I'm still on the lookout for something to handle that.
>
>
>
>
> On Wed, Aug 6, 2014 at 5:45 AM, Eric Wong <e@80x24.org (mailto:e@80x24.org)> wrote:
> >
> > Any update? It looks like your DB driver is not using/respecting any
> > timeout at all[1]. It is bad to not have a timeout there. There should
> > be a way to set a timeout so you can at least tell the user the DB
> > connection dropped or maybe get your app to disconnect+retry once.
> >
> > A better looking strace would be something like:
> >
> > write(fd, ...); => success
> > (poll|select|ppoll) syscall ...
> > read(fd, ...); /* only if (poll|select|ppoll) was successful[2] */
> >
> > This goes for configuring all connections/services for any app.
> >
> > [1] or if it's relying on SO_RCVTIMEO socket option(rare), that's set
> > way too high. Any timeout set for any external connection should
> > be lower than the unicorn (last-resort) timeout feature.
> >
> > [2] any read() syscall after (poll|select|ppoll) should be non-blocking,
> > because (poll|select|ppoll) may spuriously wakeup.
prev parent reply other threads:[~2014-08-06 18:56 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-08-04 18:12 Weird Unicorn Timeout Issues (Hibernation problem?) Tony Devlin
2014-08-04 18:39 ` Eric Wong
2014-08-04 18:41 ` Kapil Israni
2014-08-04 18:48 ` Eric Wong
2014-08-04 18:45 ` Tony Devlin
2014-08-04 19:11 ` Eric Wong
2014-08-04 19:21 ` Tony Devlin
2014-08-04 19:45 ` Eric Wong
2014-08-04 20:06 ` Michael Fischer
2014-08-04 18:55 ` Daniel Condomitti
2014-08-04 19:34 ` Eric Wong
2014-08-04 20:24 ` Tony Devlin
2014-08-04 20:44 ` Eric Wong
2014-08-04 20:46 ` Eric Wong
2014-08-05 14:46 ` Tony Devlin
2014-08-06 9:45 ` Eric Wong
2014-08-06 14:05 ` Tony Devlin
2014-08-06 18:26 ` Daniel Condomitti [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://yhbt.net/unicorn/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8E584028BCA3426ABB655EC529613E22@condomitti.com \
--to=daniel@condomitti.com \
--cc=e@80x24.org \
--cc=tonydevlin@gmail.com \
--cc=unicorn-public@bogomips.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).