From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00, URIBL_BLOCKED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id F001F1F463; Thu, 12 Dec 2019 07:59:37 +0000 (UTC) Date: Thu, 12 Dec 2019 07:59:37 +0000 From: Eric Wong To: Arkadi Colson Cc: cmogstored-public@bogomips.org Subject: Re: Heavy load Message-ID: <20191212075937.GA2881@dcvr> References: <20191211170658.GA16951@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: List-Id: Arkadi Colson wrote: > Hi > > We are running cmogstored version 1.7.1 compiled and running on Debian > stretch. How do I catch backtrace and debugging symbols? OK, on Linux, you need to make sure core file size is big enough to hold the entire core dump. I normally just use "unlimited" since I have more free disk space than RAM. Via init scripts, use "ulimit -c unlimited" in the shell script before spawning cmogstored (from the same shell process). With systemd, it's the "LimitCORE=" property (see systemd.exec manpage). You can check "Max core file size" in the "/proc/$PID_OF_CMOGSTORED/limits" file once cmogstored is running to verify. AFAIK, core dumps go to the working directory of the process by default. For daemonized processes like cmogstored might try to write to "/" (and fail due to lack of permissions). I've used the following to get all my core dumps in /var/tmp/core.$CRASHED_PID: echo /var/tmp/core >/proc/sys/kernel/core_pattern echo 1 >/proc/sys/kernel/core_uses_pid But modern kernels have more options for piping core dumps to processes and such... cmogstored builds by default with '-ggdb3' in the CFLAGS, which is the most verbose debug info. Some installers strip the debug info, unfortunately. Running "file /path/to/cmogstored" will tell you if there's debug_info and if it's stripped (probably easiest to install the unstripped one) Once you have a core dump from a crashed process, you can run gdb against it with an unstripped binary to get lots of info. gdb /path/to/cmogstored /var/tmp/core.$CRASHED_PID And then you can type "bt" to get a backtrace. You can also use "t 1", "t 2", etc to switch between threads and get the backtrace of other threads and a host of other commands. I'm not an expert in using gdb, as I can usually figure everything out from just the backtrace. Hopefully that's enough to get you the info I need to fix the problem. But there's a lot of info out there on debugging C programs on there (and I'm tired atm and about to pass out).