Rainbows! Rack HTTP server user/dev discussion
 help / Atom feed
From: Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org>
To: Rainbows! list <rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org>
Subject: Re: negative timeout in Rainbows::Fiber::Base
Date: Sat, 22 Sep 2012 19:42:22 +0000
Message-ID: <20120922194222.GA6839@dcvr.yhbt.net> (raw)
In-Reply-To: <CAA2_N1v460utbL31Qu-JbGuUxav1hY4X5+cEf=Mp2rOC5efzMw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>

"Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
> Here's a problem troubles me for a long time, and I failed to figure out
> what's going on and cannot find a proper fix. I didn't mention this before,
> because I don't think it's Zbatery/Rainbows' issue, but it turns out that
> it might be related, and perhaps you would be interested to provide
> some hints, I would much appreciate it.
> 
> So we're running all our apps on http://www.heroku.com
> It's a hosting service that you simply `git push` to deploy your app,
> worrying not all the complexity regarding deployment.
> 
> There's a very weird issue running Zbatery/Rainbows on it.
> I don't remember if Unicorn would be suffering from that
> since I don't run Unicorn directly for a while now.
> 
> The problem comes with Rails 3 and its "assets pipeline" feature.
> It's basically a feature that let your assets (e.g. images, javascripts,
> css, etc) no longer simple assets which could be served directly
> via nginx or some other static web server.
> 
> Like, developers could name their coffeescripts files as:
> 
> application.coffee.js
> 
> And when a client is requesting "application.js", that "assets pipeline"
> thing would compile the coffeescript file into a javascript file on the fly,
> and return it to the client.
> 
> We can also "precompile" those assets before deploying to production.
> Heroku would do this if it detects that the application is deploying is
> built with Rails 3.
> 
> If this precompile thing is done successfully, then there's no problems,
> at least in some of the cases.... But what if precompile failed? In my cases,
> it would usually cause system stack overflow whenever it tries to compile
> it on the fly.
> 
> Oh, suddenly I think maybe it's because I would wrap a fiber around
> each request in some cases... and it just used too many method calls.
> I am not sure about that... might check it next time I saw it.
> 
> Somehow I feel this might be related to persistent connections or
> pipelined requests, because last time I didn't write the client class
> correctly to handle pipelined requests as you pointed out, it would
> also cause some issues regarding "assets pipeline" on Heroku.
> 
> I believe Heroku is running nginx in front off Zbatery, and if I didn't
> get pipelined requests correctly, it would result timeout for many of
> assets the client is asking about. I feel Heroku is using pipelined
> requests if the client is using this.
> 
> And the most weird thing I can't understand is, I am trying to convince
> one of my friend to try out Zbatery, but using ThreadSpawn model
> would usually cause serving assets timeout, just like in my case
> where I didn't get pipelined requests correct. Switching to EventMachine
> solved the problem! Huh?
> 
> Moreover, once there are some assets timeout issues on EventMachine,
> too. When I tried to debug this, I put some traces into Rainbows,
> realizing that sometimes EventMachine didn't call `receive_data'
> when receiving some pipelined requests. Could it be an eventmachine bug!?

It could be the front-end proxy (incorrectly) detected the Rainbows!
instance was down and stopped sending traffic to it.  Does this
information get logged?

> Sorry that I wrote so much and it's unclear what's going on, since
> I don't understand so I can't remember correctly...

Yeah, it's a bit much to understand.  Can you reproduce it consistently?

With the serving timeout for Zbatery+ThreadSpawn, can you ensure
Content-Length/Transfer-Encoding:chunked is set in the response headers?

Since you mentioned stack overflows in response generation, perhaps
whatever proxy Heroku is using doesn't handle crashed servers during
the response correctly...

Can you get stderr logs from Heroku?

> If you have any idea for a specific case, I can try to provide all the
> information I could collect. I can't get the nginx configuration Heroku
> is using though :(

I highly doubt nginx will pipeline requests, but we're not sure if
they're really using nginx, yet.  With the problems you've described,
it doesn't sound like they are, or they're using some broken version
of it.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


  parent reply index

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-08-23 20:36 Lin Jen-Shin (godfat)
     [not found] ` <CAA2_N1unOXb7Z4Jr8oKoSLu266O9Ko4o=oWzAcMA1w3=9X74KA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-08-25  2:45   ` Eric Wong
     [not found]     ` <20120825024556.GA25977-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-08-26  0:12       ` Lin Jen-Shin (godfat)
     [not found]         ` <CAA2_N1uhfcHDbTvY+ke0Cid6=i7KEhFn8jvEirx+ptYVDacdvA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-08-26  1:15           ` Eric Wong
2012-08-29 16:00           ` Lin Jen-Shin (godfat)
     [not found]             ` <CAA2_N1thakAOVp7ibCNic+TjEVvXE0OGLgzXH3fJ1c2UTs68oQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-08-29 21:17               ` Eric Wong
     [not found]                 ` <20120829211707.GA22726-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-08-30 21:33                   ` Lin Jen-Shin (godfat)
     [not found]                     ` <CAA2_N1tc=Xx8WHaM8H=EWshyzGEyX04PnkdBGj9Jdb7cSzmbRQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-08-31  1:37                       ` Eric Wong
     [not found]                         ` <20120831013731.GA16613-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-09-05 20:06                           ` Lin Jen-Shin (godfat)
     [not found]                             ` <CAA2_N1vfWXGw_CaaMWMijUSdMN2Pz882SYDtNEW2_6YWffgTKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-09-05 23:27                               ` Eric Wong
     [not found]                                 ` <20120905232739.GA25153-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-09-22  9:52                                   ` Lin Jen-Shin (godfat)
     [not found]                                     ` <CAA2_N1v460utbL31Qu-JbGuUxav1hY4X5+cEf=Mp2rOC5efzMw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-09-22 19:42                                       ` Eric Wong [this message]
     [not found]                                         ` <20120922194222.GA6839-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-09-28 15:14                                           ` Lin Jen-Shin (godfat)
     [not found]                                             ` <CAA2_N1usHJVZgn5n7RaTyDCbK7eu6G4ocZAsvqsVeL6cPERskw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-09-28 19:11                                               ` Eric Wong
     [not found]                                                 ` <20120928191132.GA14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-09-28 19:24                                                   ` Eric Wong
     [not found]                                                     ` <20120928192449.GB14292-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2012-10-31  0:14                                                       ` Lin Jen-Shin (godfat)
2012-12-18 11:09                                                       ` Lin Jen-Shin (godfat)
     [not found]                                                         ` <CAA2_N1tcA-HK20C8Ok1Lv9KWwMD4fctCOPHTLeD9ayRJqWby1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2012-12-18 19:19                                                           ` Eric Wong

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://bogomips.org/rainbows/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20120922194222.GA6839@dcvr.yhbt.net \
    --to=normalperson-rmlxzr9ms24@public.gmane.org \
    --cc=rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Rainbows! Rack HTTP server user/dev discussion

Archives are clonable:
	git clone --mirror https://bogomips.org/rainbows-public
	git clone --mirror http://ou63pmih66umazou.onion/rainbows-public

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.lang.ruby.rainbows
	nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.rainbows

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox