From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.6.0.0/16 X-Spam-Status: No, score=-3.2 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from mail-ot0-x22b.google.com (mail-ot0-x22b.google.com [IPv6:2607:f8b0:4003:c0f::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 157D31FADF for ; Mon, 15 Jan 2018 03:22:28 +0000 (UTC) Received: by mail-ot0-x22b.google.com with SMTP id 44so8661336otk.8 for ; Sun, 14 Jan 2018 19:22:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=l+c8fjTGTX5QAXnv5VpkwRr4p08cFUB7eC1RFOhJmVA=; b=Xe77Mlm8Gtg+Ro3coHXmsiPOe6tW9osx+9azoagbTCCb7+/O5UoV53IrIkR47zo2MA 0aeERKpUSi5wPWKhzoM5Q9X+fBzgO5tTwBL1tx/DL/3SVlx3SzVR525ocDdjOtgLB7LK 2CD68hnTnkvdTbmccNWfY1xgJIFOs6btPynzxdHujK3n42miTd1OfJtdk4IOB4oOWKzT s9rLvFNNDMTYPeLmTO/M4IwYaFsCvzCuls0LpvEYaJ+mjLHiOgKV/Mq+/XWDx1osLef+ V8w7N43kckYDux+XjC8duDWzv1luogZfYi9/fiXp1aP90dFKFcJvlzSTLqOemTLI7iuh Yyhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=l+c8fjTGTX5QAXnv5VpkwRr4p08cFUB7eC1RFOhJmVA=; b=hrx5bJWe5XReOOFAB9gTp/BP4eKkVtuwQXRbsQDJiYLf+oQPm3YINE5U51n0sXs+a4 Pn5tse5u5Zq81xZTpLsRMMFKHzwLXquQ+QB/vic1stgEZJ5MaBLrqG0GyNArwNDbYvvE qzCXJJ7OujzCeg2uFiNbfneqN0YA9XPBVcIvKPYGd9ruRTd2+q+TRGl//hF+3SCMg9TI JOKqXupDtw9P+zLlcPTlBBOMF8HFe5CjbUwbcyHl8dI6sNFklnn+AXAHlqZFvPR+1B+P YgaS5cJrzMqYrzpSiVJ7T7aANnrsVuubgcUuZQAwN8cLNTtmHsAoIStpr7fRTwf98RZX KgFw== X-Gm-Message-State: AKwxytdPJ6HfFVgSnpTa2Xfc2k/UsTRjPFoo18zTelHh36GhrRCN2u5O aoPyS1zc1hs99O+z9Gg5DWdyi0t0isgQFaDbK+g= X-Google-Smtp-Source: ACJfBouf3ejMACXES3i8YxNS6rVHIxJiseD9tvJnMcwlbFk13xSU+FMmIWHH6akXZcQ0an8kgvx0z/52Q/V0WOafy38= X-Received: by 10.157.33.100 with SMTP id l33mr15808509otd.398.1515986547085; Sun, 14 Jan 2018 19:22:27 -0800 (PST) MIME-Version: 1.0 Received: by 10.74.121.78 with HTTP; Sun, 14 Jan 2018 19:22:26 -0800 (PST) In-Reply-To: References: <20180115015740.GA850@dcvr> From: Sam Saffron Date: Mon, 15 Jan 2018 14:22:26 +1100 Message-ID: Subject: Re: Log URL with murder_lazy_workers To: Jeremy Evans Cc: Eric Wong , unicorn-public Content-Type: text/plain; charset="UTF-8" List-Id: Hi Jeremy, Yes we already have statement timeout set, it just does not trigger under extreme load cause its usually a piling of papercuts and not one big nasty query, 26 seconds in you issue yet another query and... boom. I guess we could prepend a custom exec that times out early and re-sets it per statement. Thanks for the suggestion of the info file, I will consider adding something like it. Sam On Mon, Jan 15, 2018 at 1:40 PM, Jeremy Evans wrote: > On Sun, Jan 14, 2018 at 6:18 PM, Sam Saffron wrote: >> >> It is super likely this could be handled in the app if we had: >> >> db_connection.timeout_at Time.now + 29 >> >> Then the connection could trigger the timeout and kill off the request >> without needing to tear down the entire process and re-forking. >> >> Making this happen is a bit tricky cause it would require some hacking >> on the pg gem. > > > Sam, > > For this particular case, you can SET statement_timeout (29000) on the > PostgreSQL connection, but note that is a per statement timeout, not a per > web request timeout. > > What I end up doing for important production apps is have the worker log > request information for each request to an open file descriptor > (seek(0)/write/truncate), and having the master process use the contents of > the worker-specific file to report errors in case of worker crashes. > > Thanks, > Jeremy >