From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <e@80x24.org>
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net
X-Spam-Level: 
X-Spam-ASN:  
X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00
	shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2
Received: from localhost (dcvr.yhbt.net [127.0.0.1])
	by dcvr.yhbt.net (Postfix) with ESMTP id A39CC1F463;
	Tue, 17 Dec 2019 08:43:30 +0000 (UTC)
Date: Tue, 17 Dec 2019 08:43:30 +0000
From: Eric Wong <e@80x24.org>
To: Arkadi Colson <arkadi@smartbit.be>
Cc: cmogstored-public@bogomips.org
Subject: Re: Heavy load
Message-ID: <20191217084330.GB32606@dcvr>
References: <e7b56c07-a57c-4c5d-a505-19689354bdc0@smartbit.be>
 <20191211170658.GA16951@dcvr>
 <b72de298-f271-2033-7cfe-b7ba5c2aaa4f@smartbit.be>
 <20191212191642.GA16109@dcvr>
 <e70cfae4-ee10-5415-6f41-bfc83f375d1f@smartbit.be>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <e70cfae4-ee10-5415-6f41-bfc83f375d1f@smartbit.be>
List-Id: <cmogstored-public.bogomips.org>

Arkadi Colson <arkadi@smartbit.be> wrote:
> Hi
> 
> Yes we had this already on older version but less frequent. Below you 

Oh, I would've welcomed any bug reports much earlier :)

Which version(s)?  I had more time to work on this in the past.
Were any versions always good?

> can find the backtrace:
> 
> [Current thread is 1 (Thread 0x7f5e98655700 (LWP 8317))]
> (gdb) bt
> #0  __find_specmb (format=0x7f5e980de374 "%s") at printf-parse.h:108
> #1  _IO_vfprintf_internal (s=0x7f5e98650560, format=0x7f5e980de374 "%s", 
> ap=0x7f5e98652c08) at vfprintf.c:1312
> #2  0x00007f5e97fc3c53 in buffered_vfprintf (s=0x7f5e98314520 
> <_IO_2_1_stderr_>, format=<optimized out>, args=<optimized out>) at 
> vfprintf.c:2325
> #3  0x00007f5e97fc0f25 in _IO_vfprintf_internal 
> (s=s@entry=0x7f5e98314520 <_IO_2_1_stderr_>, 
> format=format@entry=0x7f5e980de374 "%s", ap=ap@entry=0x7f5e98652c08) at 
> vfprintf.c:1293
> #4  0x00007f5e97fe09b2 in __fxprintf (fp=0x7f5e98314520 
> <_IO_2_1_stderr_>, fp@entry=0x0, fmt=fmt@entry=0x7f5e980de374 "%s") at 
> fxprintf.c:50
> #5  0x00007f5e97fa5de0 in __assert_fail_base (fmt=0x7f5e980df310 
> "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
> assertion=assertion@entry=0x563930234ee7 "0 && \"BUG: unset HTTP 
> method\"", file=file@entry=0x563930234ee0 "http.c", line=line@entry=80, 
> function=function@entry=0x563930235440 "http_process_client")
>      at assert.c:64
> #6  0x00007f5e97fa5f12 in __GI___assert_fail (assertion=0x563930234ee7 
> "0 && \"BUG: unset HTTP method\"", file=0x563930234ee0 "http.c", 
> line=80, function=0x563930235440 "http_process_client") at assert.c:101
> #7  0x000056393020b66f in ?? ()
> #8  0x000056393020bfa5 in ?? ()
> #9  0x000056393020c10e in ?? ()
> #10 0x000056393021328d in ?? ()

Yeah, -ggdb3 should be giving info on those "??" lines.
I just double-checked the HTTP parser and it should never
even get to http.c line=80 if the method was unrecognized;
so I suspect there's some other memory corruption bug
which isn't the parser...

> #11 0x00007f5e983204a4 in start_thread (arg=0x7f5e98655700) at 
> pthread_create.c:456
> #12 0x00007f5e98062d0f in clone () at 
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> 
> (gdb) t 2
> [Switching to thread 2 (Thread 0x7f5e98591700 (LWP 8345))]

> (gdb) t 3
> [Switching to thread 3 (Thread 0x7f5e9870b700 (LWP 8291))]

<snip>

Any more threads?  (just a number of threads is fine).

> Any idea? If you need more info, please just ask!

How many "/devXYZ" devices do you have?  Are they all
on different partitions?