unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / Atom feed
From: Eric Wong <e@80x24.org>
To: Simon Eskildsen <simon.eskildsen@shopify.com>
Cc: unicorn-public@bogomips.org, Jeremy Evans <code@jeremyevans.net>
Subject: Re: after_worker_exit on murder
Date: Wed, 5 Apr 2017 18:33:39 +0000
Message-ID: <20170405183339.GA22772@dcvr> (raw)
In-Reply-To: <CAO3HKM5sLtUGFpJ9RC7i6KYCRzGr8gLtUrhJZueToh2rH8PBfw@mail.gmail.com>

Simon Eskildsen <simon.eskildsen@shopify.com> wrote:

Thank you for your reply.  It is a good reminder of how far away
I am from the rest of the web development world :>

<snip>

> It becomes difficult, because sometimes you have legitimate requests
> that take 10-20s, because the merchant's data set is so large that it
> exposes anomalies. Again, with the size of our code-base, we need this
> wiggle room in the global timeout to not just error on users. You can
> have endpoints that do 4 HTTP requests, 5 RPC requests, 4 MySQL
> queries, and 30 calls to Memcached. In that case, your worst case is
> the timeout of all of those actions, which easily exceeds the Unicorn
> timeout.

Wow, that is frightening... People will actually wait for a web
page to load in that case?  I guess that's why ccc exists :>

> We've debated having "budgets" and "shitlisting"
> (http://sirupsen.com/shitlists/) paths that obviously take longer than
> the budget for a single resource. The probability of more than one
> resource being very slow at once, is quite low (and if it is, again,
> we rely on the Unicorn timeout).

Interesting.  I guess for now, you can use nginx or similar to
route to differently-configured unicorns with different timeouts
(or even other servers)?

Anyways, I'm coming around to reconciling the two mindsets of
"typical" code running on unicorn ("it's alright to crash")
and the "no room for error: nuclear war starts if you screw up"
mindset I adopt for other projects.

> Some of these bugs are even deep in Ruby, Jean B, one of my co-workers
> submitted a bug about there being no write_timeout in Net::HTTP (you
> even replied!): https://bugs.ruby-lang.org/issues/13396

Yeah, that got me thinking of improving core timeouts again...

One big problem is the lack of a portable standard asynchronous
name resolution mechanism in the C standard library.  I'm not
sure how well resolv.rb/resolv-replace.rb holds up in real-world
usage, nor if pulling in something like ares2 would be an
acceptable dependency for ruby-core...

> BTW we deployed 5.3.0 and replaced our `before_murder` hook with
> `after_worker_exit`. Everything works perfectly and we finally are not
> using a forked version of Unicorn anymore. Thanks for the release!

Cool, good to know.

      reply index

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-04-04 14:08 Simon Eskildsen
2017-04-04 14:32 ` Jeremy Evans
2017-04-04 14:36   ` Simon Eskildsen
2017-04-05  1:19 ` Eric Wong
2017-04-05 10:55   ` Simon Eskildsen
2017-04-05 18:33     ` Eric Wong [this message]

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://bogomips.org/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20170405183339.GA22772@dcvr \
    --to=e@80x24.org \
    --cc=code@jeremyevans.net \
    --cc=simon.eskildsen@shopify.com \
    --cc=unicorn-public@bogomips.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help

Archives are clonable:
	git clone --mirror http://bogomips.org/unicorn-public
	git clone --mirror http://ou63pmih66umazou.onion/unicorn-public

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
	nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox