unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / Atom feed
* Why Unicorn master process memory increasing
@ 2018-04-11 15:07 sumit nagariya
  2018-04-12 10:22 ` Hleb Valoshka
  0 siblings, 1 reply; 4+ messages in thread
From: sumit nagariya @ 2018-04-11 15:07 UTC (permalink / raw)
  To: unicorn-public

Hi,

I have a ruby service running with unicorn which spawns 20 child
worker. When I start my service, unicorn master starts with approx 520
MB of memory and then master spawn 20 child worker. My service runs
perfectly for 7-8 day, but gradually unicorn master memory is kept
increasing and goes up to 1.3 GB, which apparently is causing OOM
error when unicorn master trying to fork a new child (which now need
1.3 GB of memory to fork new child)and due to increased memory
footprint, memory was not available.

So my concern here is,

As per my understanding, Unicorn master doesn't serve any request, its
sole purpose is to spawn missing child then why master process memory
is increasing?
If there is any memory leak in my app then it should impact only child
workers processes, not master process. I am not able to understand why
my master process memory is increasing?

Here is my server config:

Ruby: 2.1.2

Unicorn: 4.6.3

Unicorn worker count: 20

OS: Debian GNU/Linux 8, Jessie

RAM: 24 GB


-- 
Thanks & Regards
Sumit Nagariya

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Why Unicorn master process memory increasing
  2018-04-11 15:07 Why Unicorn master process memory increasing sumit nagariya
@ 2018-04-12 10:22 ` Hleb Valoshka
  2018-04-19  2:45   ` Eric Wong
  0 siblings, 1 reply; 4+ messages in thread
From: Hleb Valoshka @ 2018-04-12 10:22 UTC (permalink / raw)
  To: unicorn-public

Hi!

On 4/11/18, sumit nagariya <sumit.nagariya@gmail.com> wrote:
> Here is my server config:
> Ruby: 2.1.2
> Unicorn: 4.6.3
> Unicorn worker count: 20
> OS: Debian GNU/Linux 8, Jessie
> RAM: 24 GB

I believe it's hard to say something without your unicorn configuration file.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Why Unicorn master process memory increasing
  2018-04-12 10:22 ` Hleb Valoshka
@ 2018-04-19  2:45   ` Eric Wong
  2018-05-05 19:00     ` sumit nagariya
  0 siblings, 1 reply; 4+ messages in thread
From: Eric Wong @ 2018-04-19  2:45 UTC (permalink / raw)
  To: sumit nagariya; +Cc: unicorn-public, Hleb Valoshka

Hleb: remember to reply-all, we will never require subscription
to post to this list, so it's likely sumit never saw it.

Hleb Valoshka <375gnu@gmail.com> wrote:
> I believe it's hard to say something without your unicorn configuration file.

sumit: what Hleb asked...

I suspect "preload_app true" in your config is loading something
in your app which is using up your memory.  Sometimes there's
monitoring threads which might fill a queue up or something.

Anything in error logs?  Failing that, strace (on Linux) might
tell you what's going on at the error level.

But yeah, the master process shouldn't be doing anything besides
listening to signals and respawning failed workers.

If you want to check if there's monitoring threads on Linux
systems with /proc mounted, you can see how many threads the
master has:

	ls /proc/$PID_OF_MASTER/task/

There should be two entries for Ruby 1.9-2.5(*), one being the
master PID and one for the timer-thread.  If you have more,
then that's probably some thread doing stuff in the
background...

(*) _maybe_ we can get rid of timer-thread for 2.6...

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Why Unicorn master process memory increasing
  2018-04-19  2:45   ` Eric Wong
@ 2018-05-05 19:00     ` sumit nagariya
  0 siblings, 0 replies; 4+ messages in thread
From: sumit nagariya @ 2018-05-05 19:00 UTC (permalink / raw)
  To: Eric Wong; +Cc: unicorn-public, Hleb Valoshka

Hi Eric,

Sorry for my late response, somehow missed your mail. Just to update I
managed to fix the issue, it was related to GC. We were using
unciron's oob_gc to control gc execution. Although we had correct
configuration for oob_gc but somehow it was not allowing GC to run. I
checked GC stat and found that major & minor GC count is consistent
over the time. I removed oob_gc and now using gctools gem. With
gctools gem master memory is consistent to ~400MB.

Thanks a lot for your help.

Regards,
Sumit

On Thu, Apr 19, 2018 at 8:15 AM, Eric Wong <e@80x24.org> wrote:
> Hleb: remember to reply-all, we will never require subscription
> to post to this list, so it's likely sumit never saw it.
>
> Hleb Valoshka <375gnu@gmail.com> wrote:
>> I believe it's hard to say something without your unicorn configuration file.
>
> sumit: what Hleb asked...
>
> I suspect "preload_app true" in your config is loading something
> in your app which is using up your memory.  Sometimes there's
> monitoring threads which might fill a queue up or something.
>
> Anything in error logs?  Failing that, strace (on Linux) might
> tell you what's going on at the error level.
>
> But yeah, the master process shouldn't be doing anything besides
> listening to signals and respawning failed workers.
>
> If you want to check if there's monitoring threads on Linux
> systems with /proc mounted, you can see how many threads the
> master has:
>
>         ls /proc/$PID_OF_MASTER/task/
>
> There should be two entries for Ruby 1.9-2.5(*), one being the
> master PID and one for the timer-thread.  If you have more,
> then that's probably some thread doing stuff in the
> background...
>
> (*) _maybe_ we can get rid of timer-thread for 2.6...



-- 
Thanks & Regards
Sumit Nagariya

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, back to index

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-11 15:07 Why Unicorn master process memory increasing sumit nagariya
2018-04-12 10:22 ` Hleb Valoshka
2018-04-19  2:45   ` Eric Wong
2018-05-05 19:00     ` sumit nagariya

unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help

Archives are clonable:
	git clone --mirror https://bogomips.org/unicorn-public
	git clone --mirror http://ou63pmih66umazou.onion/unicorn-public

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.lang.ruby.unicorn
	nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.unicorn

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox