Master Process Reaping Worker with No message

Master Process Reaping Worker with No message

[Aakash Gupta]

I am using unicorn server with Nginx for running my Rails app. I don't know why but my master server is reaping workers continously while file uploads. I have raised the timeout limit of unicorn to 20 minutes for giving sufficient time for uploading file. This happends mostly during file uploads on server. For file uploads, I am using Carrierwave gem with Fog to upload the files to S3. Server is running on AWS EC2 instance of 1GB RAM with 2GB Swap space. 
 Master process kills worker with message "ERROR : reaped ... SIGKILL (signal 9)> worker=1 " without any reason or message such as timeout or memory overflow.

Whenever a worker is reaped, Nginx also logs an error for each reaping instance with message:
" upstream prematurely closed connection while reading response header from upstream ..."

I need help to figure out the problem and the reason why this is happening. I don't think this is happening because of timeout issue because whenever I try to test my server by uploading a file, this error appears even before 20 minutes which is the timeout limit. Any help regarding this would be appreciated. 

Re: Master Process Reaping Worker with No message

[Eric Wong]

Aakash Gupta <aakash.gupta96@outlook.com> wrote:
> I am using unicorn server with Nginx for running my Rails app.
> I don't know why but my master server is reaping workers
> continously while file uploads. I have raised the timeout
> limit of unicorn to 20 minutes for giving sufficient time for
> uploading file. This happends mostly during file uploads on
> server. For file uploads, I am using Carrierwave gem with Fog
> to upload the files to S3. Server is running on AWS EC2
> instance of 1GB RAM with 2GB Swap space.  Master process kills
> worker with message "ERROR : reaped ... SIGKILL (signal 9)>
> worker=1 " without any reason or message such as timeout or
> memory overflow.

I don't know anything about Carrierwave or Fog; but is there any
chance they (or anything in your app) could be sending a SIGKILL
signal to the process?

So you see the "reaped ... SIGKILL" message, but no corresponding
message like "worker=... PID:... timeout (... > ...), killing"?

It sounds like somebody else is sending SIGKILL to a worker...

Is this AWS EC2 instance running Linux?  Check the kernel log
(dmesg) to see if you're hitting the OOM killer, since the
kernel OOM killer will also send SIGKILL to processes using too
much memory.

How big are the files you're uploading?  Can you reproduce this
making small file uploads?

Honestly, there's no reason any process should become
memory-dependent based on the size of the file being uploaded;
but maybe some code doesn't know how to stream correctly :<
Fwiw, unicorn itself has always designed for handling
multi-gigabyte uploads with stable memory usage in mind.
Other code I've seen, not so much, unfortunately.

> Whenever a worker is reaped, Nginx also logs an error for each
> reaping instance with message: " upstream prematurely closed
> connection while reading response header from upstream ..."

Right, that is expected once workers start dying...

> I need help to figure out the problem and the reason why this
> is happening. I don't think this is happening because of
> timeout issue because whenever I try to test my server by
> uploading a file, this error appears even before 20 minutes
> which is the timeout limit. Any help regarding this would be
> appreciated.

Which unicorn version is this?  There used to be some timeout
calculation bugs in the old days, but those were over five
years ago, maybe even before unicorn 1.0.

Just covering all your bases, Also, is there any chance your
system clock is being changed?  If you're using unicorn 5.0.0+
with Ruby 2.1+, it'll be able to use the monotonic clock, making
it immune to time changes.


Thanks for any info you can provide!

Re: Master Process Reaping Worker with No message

[Aakash Gupta]

Thank a lot Eric. 
I am using Ruby 2.3.3 and Unicorn 5.3. 

"It sounds like somebody else is sending SIGKILL to a worker..." This line of yours saved my server. 

Actually when I uploads a file, it first goes to /tmp directory and then one of the worker processes of unicorn uploads it to S3 from there. So, earlier when I was trying to fix timeout connection error of unicorn, I increased it to 20 minutes and the problem was solved temporarily. 

But the problem was that the files that were uploaded temporarily to /tmp directory, those files do not get deleted from /tmp after they are uploaded to S3. So, I wrote a script using tmpreaper that cleans the /tmp directory recursively and removes the files that are 30 min older and added that to cron service. 

After this, as a result of this script, all the files that were required to be removed, were removed from /tmp directory but again a problem occurred. When I used "df -h" command to check whether the files are actually deleted or not, it showed that the space that was taken by uploaded files on my disk has not been freed :( . So, I searched internet and got to know that any other process was still using those files, that was why the space on the disk was not freed. 

I used ' lsof +L1 | grep "deleted" ' command and found that a process named 'bundle' is still using these deleted files. So, I again wrote a script that was killing the process that was using a deleted file.  Script was using this command

" lsof +L1 | grep 'deleted' | awk '{print $2}' | xargs kill -9 "

Today, after reading your words, I finally realised that my script was killing the unicorn worker itself because the command that I was using was killing the parent of "bundle" process. As "bundle" was a child of Unicorn worker, unicorn worker was being killed.

So, thanks a lot Eric :) 

I was finally able to solve this error because of the direction and hint that you gave me. But the only thing that I need to find is that what is causing uploaded files to persist in /tmp directory.   

Re: Master Process Reaping Worker with No message

[Eric Wong]

Aakash Gupta <aakash.gupta96@outlook.com> wrote:
> So, thanks a lot Eric :) 

No problem!  Thanks for following up after finding the solution.

> I was finally able to solve this error because of the
> direction and hint that you gave me. But the only thing that I
> need to find is that what is causing uploaded files to persist
> in /tmp directory.   

Perhaps the Rack::TempfileReaper middleware in rack 1.6+ can
help, unicorn has had support for it since 4.9+.