Re: negative timeout in Rainbows::Fiber::Base

[Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org>]

"Lin Jen-Shin (godfat)" <godfat-hOE/xeEBYYIdnm+yROfE0A@public.gmane.org> wrote:
> On Sun, Sep 23, 2012 at 3:42 AM, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> wrote:
> [...]
> >> Moreover, once there are some assets timeout issues on EventMachine,
> >> too. When I tried to debug this, I put some traces into Rainbows,
> >> realizing that sometimes EventMachine didn't call `receive_data'
> >> when receiving some pipelined requests. Could it be an eventmachine bug!?
> >
> > It could be the front-end proxy (incorrectly) detected the Rainbows!
> > instance was down and stopped sending traffic to it.  Does this
> > information get logged?
> 
> It would eventually show out as a timeout for 30 seconds. It's a log
> from front-end proxy (router in their terminology). I am not sure
> if this means it is sending traffic or not. If it's not, then I guess it
> explains...
> 
> > Yeah, it's a bit much to understand.  Can you reproduce it consistently?
> 
> Yes, it could be reproduced consistently, but only in a certain environment
> (e.g. on Heroku) which might not always be found.
> 
> > With the serving timeout for Zbatery+ThreadSpawn, can you ensure
> > Content-Length/Transfer-Encoding:chunked is set in the response headers?
> 
> I just realized that this is not a timeout issue here. (doh, too many strange
> issues) I just tried to curl it, and it would immediately return an error.
> 
> heroku[router]: Error H13 (Connection closed without response) -> GET
> /assets/application-4e83bff8c0e77de81926c169e1fcacf2.css dyno=web.1
> queue= wait= service= status=503 bytes=
> 
> There is Content-Length: 98794 and no Transfer-Encoding.

So the _actual_ Content-Length that's sent is zero?

Rainbows! should drop a connection if an exception is raised while
sending the response body, perhaps the heroku router is confused
by that?

> I don't see if there's any difference between using EventMachine :(
> 
> By the way, weirdly that it seems there's no problems at all if we're
> using Thin server. I guess they are testing against Thin, so Thin works
> correctly... still can't I tell what's the difference.

Are you allowed to "strace -f" Thin and Rainbows! in an isolated (single
thread/process instance)?  I'd love to see the difference between how
the servers handle this case differently...

Can you try setting "keepalive_timeout 0" to disable persistent
connections in Rainbows! and see if this helps?  I don't think
persistent connections make a big difference for LAN, especially
not with typical HTTP response sizes (your 98K is pretty big :)

> > Since you mentioned stack overflows in response generation, perhaps
> > whatever proxy Heroku is using doesn't handle crashed servers during
> > the response correctly...
> 
> Probably. And I think I am 80% sure where it causes stack overflows now.
> If I took out fibers, then it would be ok. So I guess that assets things are
> using too much stack.
> 
> Too bad I can't switch to threads though :( If I switched to threads,
> ActiveRecord would be complaining it could not get a connection from
> the connection pool within 5 seconds. It's not convenient to increase the
> size of connection pool on Heroku, either, and making it too large would
> also cause other issues.
> 
> (sigh)
> 
> > Can you get stderr logs from Heroku?
> 
> Yes, Heroku would redirect both stdout and stderr to a place where we
> could see. It's collapsed into one huge log though.
> 
> > I highly doubt nginx will pipeline requests, but we're not sure if
> > they're really using nginx, yet.  With the problems you've described,
> > it doesn't sound like they are, or they're using some broken version
> > of it.
> 
> Umm... after reading the log, I think they are using another [router]
> (in their terminology) in front of [nginx]. So it might be an issue in
> their [router], I am not sure...
> 
> I'll keep you posted if you're interested. Thanks for all your help.

Yes, please.

> p.s. They might be using https://github.com/mochi/mochiweb
> too, since if there's an issue, it would return this header:
> Server: MochiWeb/1.0 (Any of you quaids got a smint?)
> 
> p.s.2. Perhaps I could even give you access to one of the apps.
> I'll need to ask the owner though. Let me know if you're interested,
> otherwise you could simply ignore this. I know it's too much to ask.

I think I can just walk you through the troubleshooting process
here.  (I'd rather avoid the potential for personal liability if
given access to a server :)
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying