From: Eric Wong <normalperson@yhbt.net>
To: unicorn list <mongrel-unicorn@rubyforge.org>
Subject: Re: A barrage of unexplained timeouts
Date: Tue, 20 Aug 2013 21:32:57 +0000 [thread overview]
Message-ID: <20130820213257.GA26442@dcvr.yhbt.net> (raw)
In-Reply-To: <1377033545.89791642@apps.rackspace.com>
nick@auger.net wrote:
> "Eric Wong" <normalperson@yhbt.net> said:
> > nick@auger.net wrote:
> >> "Eric Wong" <normalperson@yhbt.net> said:
> > I'm stumped :<
>
> I was afraid you'd say that :(.
Actually, another potential issue is DNS lookups timing out. But they
shouldn't take *that* long...
> > Do you have any background threads running that could be hanging the
> > workers? This is Ruby 1.8, after all, so there's more likely to be
> > some blocking call hanging the entire process. AFAIK, some monitoring
> > software runs a background thread in the unicorn worker and maybe the
> > OpenSSL extension doesn't work as well if it encountered network
> > problems under Ruby 1.8
>
> We don't explicitly create any threads in our rails code. We do
> communicate with backgroundrb worker processes, although, none of the
> strangeness today involved any routes that would hit backgroundrb
> workers.
I proactively audit every piece of code (including external
libraries/gems) loaded by an app for potentially blocking calls (hits to
the filesystem, socket calls w/o timeout/blocking). I use strace to
help me find that sometimes...
> Is there any instrumentation that I could add that might help
> debugging in the future? ($request_time and $upstream_response_time
> are now in my nginx logs.) We have noticed these "unexplainable
> timeouts" before, but typically for a single worker. If there's some
> debugging that could be added I might be able to track it down during
> these one-off events.
As an experiment, can you replay traffic a few minutes leading up to and
including that 7m period in a test setup with only one straced worker?
Run "strace -T -f -o $FILE -p $PID_OF_WORKER" and see if there's any
unexpected/surprising dependencies (connect() to unrecognized addresses,
open() to networked filesystems, fcntl locks, etc...).
You can play around with some other strace options (-v/-s SIZE/-e filters)
Maybe you'll find something, there.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
next prev parent reply other threads:[~2013-08-20 21:34 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-08-20 14:47 A barrage of unexplained timeouts nick
2013-08-20 16:37 ` Eric Wong
2013-08-20 17:27 ` nick
2013-08-20 17:40 ` Eric Wong
2013-08-20 18:11 ` nick
2013-08-20 18:49 ` Eric Wong
2013-08-20 20:03 ` nick
2013-08-20 20:42 ` Eric Wong
2013-08-20 21:19 ` nick
2013-08-20 21:32 ` Eric Wong [this message]
2013-08-21 13:33 ` nick
2013-08-22 2:32 ` Jimmy Soho
2013-08-22 4:09 ` Eric Wong
2013-08-22 13:10 ` nick
2013-08-22 17:05 ` Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://yhbt.net/unicorn/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130820213257.GA26442@dcvr.yhbt.net \
--to=normalperson@yhbt.net \
--cc=mongrel-unicorn@rubyforge.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://yhbt.net/unicorn.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).