From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS33070 50.56.128.0/17 X-Spam-Status: No, score=0.0 required=3.0 tests=AWL,MSGID_FROM_MTA_HEADER, TVD_RCVD_IP shortcircuit=no autolearn=unavailable version=3.3.2 Path: news.gmane.org!not-for-mail From: "Lin Jen-Shin (godfat)" Newsgroups: gmane.comp.lang.ruby.rainbows.general Subject: Re: negative timeout in Rainbows::Fiber::Base Date: Fri, 28 Sep 2012 23:14:33 +0800 Message-ID: References: <20120825024556.GA25977@dcvr.yhbt.net> <20120829211707.GA22726@dcvr.yhbt.net> <20120831013731.GA16613@dcvr.yhbt.net> <20120905232739.GA25153@dcvr.yhbt.net> <20120922194222.GA6839@dcvr.yhbt.net> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1348845983 14498 80.91.229.3 (28 Sep 2012 15:26:23 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Fri, 28 Sep 2012 15:26:23 +0000 (UTC) To: "Rainbows! list" Original-X-From: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Fri Sep 28 17:26:28 2012 Return-path: Envelope-to: gclrrg-rainbows-talk@m.gmane.org X-Original-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Delivered-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org X-Greylist: delayed 403 seconds by postgrey-1.31 at rubyforge; Fri, 28 Sep 2012 15:21:47 UTC DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=godfat.org; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=H7muec+Toib9cxKKfEjQtKYqhv63Z+/+UY/bjTOyaNU=; b=nbPDAK780xHWvaqXAD9LvOce320voI/WYnG8NQyUWFliZvet6vab4SNzLAXsTDfVS3 T9L1eEjkcVjraly6XHQwaZWTiTwFu9xEBzgEHMOtUKIjc9TP5X++2Y9yt/mBlHstCmCi WDI/HkJyrw992Gsf3/HHjLGi3OnJw1YDZkE+w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:x-gm-message-state; bh=H7muec+Toib9cxKKfEjQtKYqhv63Z+/+UY/bjTOyaNU=; b=DfNRNFc6/z83LczpBkuY7MlYJOe93YAu65W+1ipYtGlaQDTvwLwFX+DI/jVE00NwOE Aztmm8XG01Oul1op3CsNWFkNRXmAhoJvZvxzzlIwxqnb0YCvsjhn1vMjvA0sj3rYklVr XroHtXeEvCtGt8RIfJOvb1seOx6MJLwXQ1yudjo6x3Jlgqo5EHB6GwImIugJBjrIo37k Znyr5qmKtwQDRMbKFZD8/obQvbUIfbl49pszlhKig3NJJunqlmelDZwrT0mr9nKTKLT0 M7O2krKRoC/MtsKg13LeZ4CTv7/7L5BG+IvNL+lNmkgKesNkbk640AN1S6VgEuawp3ej XvtA== In-Reply-To: <20120922194222.GA6839-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org> X-Gm-Message-State: ALoCoQmI8UjC6kt75NVDaTcbUUQXwwQ9xtla469fp4SAZ5p2ykXYxqnohpGcoAVezKNpjPVe9eCE X-BeenThere: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Errors-To: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Xref: news.gmane.org gmane.comp.lang.ruby.rainbows.general:408 Archived-At: Received: from 50-56-192-79.static.cloud-ips.com ([50.56.192.79] helo=rubyforge.org) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1THcSS-000589-DD for gclrrg-rainbows-talk@m.gmane.org; Fri, 28 Sep 2012 17:26:28 +0200 Received: from localhost.localdomain (localhost [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id CF5AD2E077; Fri, 28 Sep 2012 15:26:21 +0000 (UTC) Received: from mail-bk0-f50.google.com (mail-bk0-f50.google.com [209.85.214.50]) by rubyforge.org (Postfix) with ESMTP id B4CFA2E06A for ; Fri, 28 Sep 2012 15:21:47 +0000 (UTC) Received: by bkwq16 with SMTP id q16so3616190bkw.23 for ; Fri, 28 Sep 2012 08:21:46 -0700 (PDT) Received: by 10.112.54.99 with SMTP id i3mr2334743lbp.37.1348845303657; Fri, 28 Sep 2012 08:15:03 -0700 (PDT) Received: by 10.114.19.73 with HTTP; Fri, 28 Sep 2012 08:14:33 -0700 (PDT) On Sun, Sep 23, 2012 at 3:42 AM, Eric Wong wrote: [...] >> Moreover, once there are some assets timeout issues on EventMachine, >> too. When I tried to debug this, I put some traces into Rainbows, >> realizing that sometimes EventMachine didn't call `receive_data' >> when receiving some pipelined requests. Could it be an eventmachine bug!? > > It could be the front-end proxy (incorrectly) detected the Rainbows! > instance was down and stopped sending traffic to it. Does this > information get logged? It would eventually show out as a timeout for 30 seconds. It's a log from front-end proxy (router in their terminology). I am not sure if this means it is sending traffic or not. If it's not, then I guess it explains... > Yeah, it's a bit much to understand. Can you reproduce it consistently? Yes, it could be reproduced consistently, but only in a certain environment (e.g. on Heroku) which might not always be found. > With the serving timeout for Zbatery+ThreadSpawn, can you ensure > Content-Length/Transfer-Encoding:chunked is set in the response headers? I just realized that this is not a timeout issue here. (doh, too many strange issues) I just tried to curl it, and it would immediately return an error. heroku[router]: Error H13 (Connection closed without response) -> GET /assets/application-4e83bff8c0e77de81926c169e1fcacf2.css dyno=web.1 queue= wait= service= status=503 bytes= There is Content-Length: 98794 and no Transfer-Encoding. I don't see if there's any difference between using EventMachine :( By the way, weirdly that it seems there's no problems at all if we're using Thin server. I guess they are testing against Thin, so Thin works correctly... still can't I tell what's the difference. > Since you mentioned stack overflows in response generation, perhaps > whatever proxy Heroku is using doesn't handle crashed servers during > the response correctly... Probably. And I think I am 80% sure where it causes stack overflows now. If I took out fibers, then it would be ok. So I guess that assets things are using too much stack. Too bad I can't switch to threads though :( If I switched to threads, ActiveRecord would be complaining it could not get a connection from the connection pool within 5 seconds. It's not convenient to increase the size of connection pool on Heroku, either, and making it too large would also cause other issues. (sigh) > Can you get stderr logs from Heroku? Yes, Heroku would redirect both stdout and stderr to a place where we could see. It's collapsed into one huge log though. > I highly doubt nginx will pipeline requests, but we're not sure if > they're really using nginx, yet. With the problems you've described, > it doesn't sound like they are, or they're using some broken version > of it. Umm... after reading the log, I think they are using another [router] (in their terminology) in front of [nginx]. So it might be an issue in their [router], I am not sure... I'll keep you posted if you're interested. Thanks for all your help. p.s. They might be using https://github.com/mochi/mochiweb too, since if there's an issue, it would return this header: Server: MochiWeb/1.0 (Any of you quaids got a smint?) p.s.2. Perhaps I could even give you access to one of the apps. I'll need to ask the owner though. Let me know if you're interested, otherwise you could simply ignore this. I know it's too much to ask. _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying