unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / Atom feed
From: Tom Burns <tom.burns@jadedpixel.com>
To: mongrel-unicorn@rubyforge.org
Subject: Combating nginx 499 HTTP responses during flash traffic scenario
Date: Mon, 29 Oct 2012 13:44:27 -0400
Message-ID: <CAK4qKG32dGgNbzTzCb6NoH5hu_DrHMOFaFhk-6Xvy-T86++ukA@mail.gmail.com> (raw)

Hi,

We're dealing with an issue with our large-scale deployment of unicorn
& nginx.  The problem occurs during "flash" scenarios where we receive
an order magnitude more traffic for up to an hour.  Much of this
traffic cannot be cached and it's typical for some of our rails
responses to take a few seconds to generate.  Our preference is to
queue incoming requests instead of returning 502 if we can respond
within a reasonable amount of time.

On each of our servers the stack is nginx -> unicorn.

The main connection queue in front of Rails is the unicorn connection
queue.  Our problem is that when the traffic hits, the unicorn queue
grows.  When users begin hitting refresh, their abandoned requests in
the unicorn queue are still passed to the rails application and
rendered.  In this case we see a 200 HTTP response in our Rails log
and a 499 in the nginx access log.  Once this starts happening the
problem can compound: app workers are busy rendering pages for clients
who have already detached so response time grows and more users hit
refresh, etc.

Our nginx config:
6 nginx workers, 1024 worker connections.

Our unicorn config:
70 unicorn workers, 2048 connection backlog.

Our goal is to not have our app process requests for clients that have
already disconnected while their connection is still queued in
unicorn.  We also would prefer not to shrink our queue such that we
begin to return 502 when our queue is a few seconds deep.

We're looking at potential solutions to this problem, including:
- modifying unicorn to select() on the client connection after reading
the request, to see if it's been closed upstream, and avoid calling
the app.
- Replacing nginx with haproxy and queuing connections there.  This
goes against the nginx recommendation at
http://unicorn.bogomips.org/PHILOSOPHY.html

Any input would be appreciated.

Thanks,
Tom
Developer @ Shopify
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

             reply index

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-10-29 17:44 Tom Burns [this message]
2012-10-29 18:45 ` Eric Wong
2012-10-29 19:27   ` Hongli Lai
2012-10-29 19:41     ` Eric Wong
2012-10-29 21:06       ` Hongli Lai
2012-10-29 21:53   ` Eric Wong
2012-10-29 22:21     ` Tom Burns
2012-10-30 20:40     ` Tom Burns
2012-10-30 21:37       ` Eric Wong
2012-11-02 17:59         ` Tom Burns
2012-11-02 19:38           ` Eric Wong
2012-11-03 22:45             ` Tom Burns
2012-11-05 11:48               ` Eric Wong
2012-11-06  3:16                 ` Tom Burns
2012-11-06 21:23                   ` Eric Wong
2012-11-29 15:52                     ` Tom Burns
2012-11-29 20:30                       ` Lawrence Pit
2012-11-29 20:57                         ` Tom Burns
2012-11-29 21:30                         ` Eric Wong
2012-11-30 23:47                           ` Eric Wong
2012-11-29 20:41                       ` Eric Wong
2012-12-04  3:00                         ` Eric Wong
2012-11-29 21:19                       ` Eric Wong
2012-11-29 21:55                         ` [RFC/PATCH] check_client_connection: document local-only requirement Eric Wong
2012-11-29 23:47                           ` Tom Burns

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://bogomips.org/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAK4qKG32dGgNbzTzCb6NoH5hu_DrHMOFaFhk-6Xvy-T86++ukA@mail.gmail.com \
    --to=tom.burns@jadedpixel.com \
    --cc=mongrel-unicorn@rubyforge.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help

Archives are clonable:
	git clone --mirror https://bogomips.org/unicorn-public
	git clone --mirror http://ou63pmih66umazou.onion/unicorn-public

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
	nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox