From: Eric Wong <firstname.lastname@example.org> To: Aakash Gupta <email@example.com> Cc: firstname.lastname@example.org Subject: Re: Master Process Reaping Worker with No message Date: Tue, 23 May 2017 16:22:14 +0000 Message-ID: <20170523162214.GA26129@starla> (raw) In-Reply-To: <BN6PR20MB12816B4DEA41A7389ACF5D3B9BF90@BN6PR20MB1281.namprd20.prod.outlook.com> Aakash Gupta <email@example.com> wrote: > I am using unicorn server with Nginx for running my Rails app. > I don't know why but my master server is reaping workers > continously while file uploads. I have raised the timeout > limit of unicorn to 20 minutes for giving sufficient time for > uploading file. This happends mostly during file uploads on > server. For file uploads, I am using Carrierwave gem with Fog > to upload the files to S3. Server is running on AWS EC2 > instance of 1GB RAM with 2GB Swap space. Master process kills > worker with message "ERROR : reaped ... SIGKILL (signal 9)> > worker=1 " without any reason or message such as timeout or > memory overflow. I don't know anything about Carrierwave or Fog; but is there any chance they (or anything in your app) could be sending a SIGKILL signal to the process? So you see the "reaped ... SIGKILL" message, but no corresponding message like "worker=... PID:... timeout (... > ...), killing"? It sounds like somebody else is sending SIGKILL to a worker... Is this AWS EC2 instance running Linux? Check the kernel log (dmesg) to see if you're hitting the OOM killer, since the kernel OOM killer will also send SIGKILL to processes using too much memory. How big are the files you're uploading? Can you reproduce this making small file uploads? Honestly, there's no reason any process should become memory-dependent based on the size of the file being uploaded; but maybe some code doesn't know how to stream correctly :< Fwiw, unicorn itself has always designed for handling multi-gigabyte uploads with stable memory usage in mind. Other code I've seen, not so much, unfortunately. > Whenever a worker is reaped, Nginx also logs an error for each > reaping instance with message: " upstream prematurely closed > connection while reading response header from upstream ..." Right, that is expected once workers start dying... > I need help to figure out the problem and the reason why this > is happening. I don't think this is happening because of > timeout issue because whenever I try to test my server by > uploading a file, this error appears even before 20 minutes > which is the timeout limit. Any help regarding this would be > appreciated. Which unicorn version is this? There used to be some timeout calculation bugs in the old days, but those were over five years ago, maybe even before unicorn 1.0. Just covering all your bases, Also, is there any chance your system clock is being changed? If you're using unicorn 5.0.0+ with Ruby 2.1+, it'll be able to use the monotonic clock, making it immune to time changes. Thanks for any info you can provide!
next prev parent reply index Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-05-23 13:20 Aakash Gupta 2017-05-23 16:22 ` Eric Wong [this message] 2017-05-23 19:29 Aakash Gupta 2017-05-23 20:40 ` Eric Wong
Reply instructions: You may reply publically to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: https://bogomips.org/unicorn/ * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20170523162214.GA26129@starla \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help Archives are clonable: git clone --mirror https://bogomips.org/unicorn-public git clone --mirror http://ou63pmih66umazou.onion/unicorn-public Newsgroups are available over NNTP: nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn note: .onion URLs require Tor: https://www.torproject.org/ AGPL code for this site: git clone https://public-inbox.org/ public-inbox