How to manage growing memory with Rainbows!

How to manage growing memory with Rainbows!

[Claudio Poli]

Hello people,
I've recently came across the https://github.com/kzk/unicorn-worker-killer which works wonders with Unicorn, however I'm considering to switching back to Rainbows! + Threadpool.

My only concern is the growing memory, as we know Ruby does not really do a good job at it (at least for me, even with all the tweaks in the world) and I need to find a good way to kill a process and restart since we are running in a memory-constrained environment.

What are you using currently? Monit, OobGC? The good thing about the gem above is that it will kill the Unicorn worker only after the last requests has been performed and with easy adjustable thresholds.

Can you point me into the right direction here? Thanks!
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: How to manage growing memory with Rainbows!

[Eric Wong]

Claudio Poli <claudio-3HQ/CcOImoi171pxa8y+qA@public.gmane.org> wrote:
> My only concern is the growing memory, as we know Ruby does not really
> do a good job at it (at least for me, even with all the tweaks in the
> world) and I need to find a good way to kill a process and restart
> since we are running in a memory-constrained environment.

I'm curious, what tweaks did you try?

What kind of workload are you running? (many disk writes at all?)
Which version of Ruby are you using?
Are you counting VMSize or RSS?
Are you on 64-bit?

Fwiw, virtual memory usage is very high on 64-bit Linux on newer
versions of glibc, but mostly harmless since the memory isn't actually
used (address space is nearly unlimited).

You can try MALLOC_ARENA_MAX_=1 to limit the number of arenas if you
want.  That might reduce fragmentation since the GVL in MRI means
it's unlikely to hit malloc lock contention (glibc uses multiple
malloc arenas to avoid contention by default).

> What are you using currently? Monit, OobGC? The good thing about the
> gem above is that it will kill the Unicorn worker only after the last
> requests has been performed and with easy adjustable thresholds.

OobGC is absolutely not recommended for Rainbows! (or anything doing
persistent connections or simultaneous clients within a process)

However, you can safely send SIGQUIT to any Rainbows! worker (bypassing
master) whenever you feel memory usage is high, master will restart it.

You can just put a simple counter in middleware to do it, something
like this:

# nr is initialized to a number of your choice elsewhere

	nr -= 1
	if nr < 0
	  Process.kill(:QUIT, $$)
	end

> Can you point me into the right direction here? Thanks!

The best solution is to fix your code/gems/Ruby :)

I report and fix all the memory leaks I can find in gems+MRI.

One thing to avoid is allocating too much memory in the first place
(always use LIMIT in SQL SELECT statements, read files in smaller
chunks, etc).  It really takes only one poorly thought-out line of
code to either OOM or cause a swap storm.

I haven't hit one of these problems in a while, but check out
commit f95113402b7239f225282806673e1b6424522b18 in
git://github.com/rack/rack.git for an example of how IO#gets
can ruin your app.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: How to manage growing memory with Rainbows!

[Claudio Poli]

Il giorno 12/feb/2013, alle ore 06:00, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> ha scritto:

Hello Eric,

> I'm curious, what tweaks did you try?

Measuring average memory in requests and tweaking RUBY_HEAP_MIN_SLOTS, RUBY_GC_MALLOC_LIMIT, RUBY_HEAP_FREE_MIN, etc.
Using LD_PRELOAD with libtcmalloc
Inviting Ruby (1.9.3) to perform GC after some heavy task.
Profiling application under multiple ruby/jruby versions to no avail.
Symbols vs string where applicable, etc.

> What kind of workload are you running? (many disk writes at all?)

No, it's quite a large Rails 3.2 app but we offload this kind of tasks to node.js in our architecture, which is able to operate under 80MB single instance.
Our ruby app serves json requests (api) and does the frontend.

> Which version of Ruby are you using?

Tried all the 1.9.3 patchsets, railsexpress, falcon patches..

> Are you counting VMSize or RSS?

Resident size

> Are you on 64-bit?

No, we were on 64bit but we switched to 32 bit.

> Fwiw, virtual memory usage is very high on 64-bit Linux on newer
> versions of glibc, but mostly harmless since the memory isn't actually
> used (address space is nearly unlimited).
> 
> You can try MALLOC_ARENA_MAX_=1 to limit the number of arenas if you
> want.  That might reduce fragmentation since the GVL in MRI means
> it's unlikely to hit malloc lock contention (glibc uses multiple
> malloc arenas to avoid contention by default).

I didn't knew about this setting, might be worth a try, thanks.

> OobGC is absolutely not recommended for Rainbows! (or anything doing
> persistent connections or simultaneous clients within a process)

Good to know.

> However, you can safely send SIGQUIT to any Rainbows! worker (bypassing
> master) whenever you feel memory usage is high, master will restart it.

Will Rainbows! wait after the last request before restarting?

> You can just put a simple counter in middleware to do it, something
> like this:
> 
> # nr is initialized to a number of your choice elsewhere
> 
> 	nr -= 1
> 	if nr < 0
> 	  Process.kill(:QUIT, $$)
> 	end
> 

Nice

> The best solution is to fix your code/gems/Ruby :)
> 
> I report and fix all the memory leaks I can find in gems+MRI.
> 
> One thing to avoid is allocating too much memory in the first place
> (always use LIMIT in SQL SELECT statements, read files in smaller
> chunks, etc).  It really takes only one poorly thought-out line of
> code to either OOM or cause a swap storm.

I agree, I'm not really saying I did everything possible but our project uses a lot of gems and I'm confident our ruby code is written fairly well (100% tested, although it doesn't mean anything in this case, easy methods, not really any black magic involved).
Leak might be in some gem but so far I haven't been able to spot anything remotely useful. I fear installing new relic since every day I read obscure problems caused by it and I had some myself.

> I haven't hit one of these problems in a while, but check out
> commit f95113402b7239f225282806673e1b6424522b18 in
> git://github.com/rack/rack.git for an example of how IO#gets
> can ruin your app.

Thanks for the example.
What Rainbows! strategy would you run on 1.9.3 given that some API call might take 800ms/1200ms (uncached) and the number of requests is fairly high?
Not only we are memory constrained but we are also trying to keep the costs down, the instance is a 4GB c1.medium on EC2, 1 core. Very underpowered as we tend to scale horizontally.
Considering a powerful VPS instead at the moment, since we'll prolly have to support 300k users very soon.

Thanks!
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: How to manage growing memory with Rainbows!

[Eric Wong]

Claudio Poli <claudio-3HQ/CcOImoi171pxa8y+qA@public.gmane.org> wrote:
> Il giorno 12/feb/2013, alle ore 06:00, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> ha scritto:
> 
> Using LD_PRELOAD with libtcmalloc

Last I checked, tcmalloc never releases memory to the OS, so that
could be a problem.  (Giving memory back to the kernel and then
getting it back soon afterwards is slow because the kernel must
clear that memory, first, so most malloc implementations (including
glibc) will try to keep memory for the process)

> > You can try MALLOC_ARENA_MAX_=1 to limit the number of arenas if you
> > want.  That might reduce fragmentation since the GVL in MRI means
> > it's unlikely to hit malloc lock contention (glibc uses multiple
> > malloc arenas to avoid contention by default).
> 
> I didn't knew about this setting, might be worth a try, thanks.

You might also want to try MALLOC_MMAP_THRESHOLD_=131072 to disable the
sliding sbrk/mmap window (you can try larger/smaller values).  The
latest Linux man-pages releases have better mallopt(3) documentation
(but the arenas stuff was only documented on Dreppers blog, AFAIK (and
glibc.git/malloc/*.[ch] comments)

> > However, you can safely send SIGQUIT to any Rainbows! worker (bypassing
> > master) whenever you feel memory usage is high, master will restart it.
> 
> Will Rainbows! wait after the last request before restarting?

Yes, SIGQUIT => graceful shutdown will always work for normal apps.
The only exception is the new, optional rack.hijack (in rainbows.git).

> What Rainbows! strategy would you run on 1.9.3 given that some API call might take 800ms/1200ms (uncached) and the number of requests is fairly high?

It depends on your app.  For standard Ruby API (Net::*),
ThreadSpawn/Pool should be the most compatible and work well enough.
These will work great if you're using Celluloid, too.

EventMachine can scale very well but it requires your network-capable
gems/libraries to use EM to benefit.  Same issue with cool.io.

I also like XEpollThread{Pool/Spawn}, but they're Linux-only and
less-used.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: How to manage growing memory with Rainbows!

[Claudio Poli]

Il giorno 14/feb/2013, alle ore 08:15, Eric Wong <normalperson-rMlxZR9MS24@public.gmane.org> ha scritto:
> 
> Last I checked, tcmalloc never releases memory to the OS, so that
> could be a problem.  (Giving memory back to the kernel and then
> getting it back soon afterwards is slow because the kernel must
> clear that memory, first, so most malloc implementations (including
> glibc) will try to keep memory for the process)
> 
> You might also want to try MALLOC_MMAP_THRESHOLD_=131072 to disable the
> sliding sbrk/mmap window (you can try larger/smaller values).  The
> latest Linux man-pages releases have better mallopt(3) documentation
> (but the arenas stuff was only documented on Dreppers blog, AFAIK (and
> glibc.git/malloc/*.[ch] comments)

Thanks for the advices Eric, will try.

I have few more questions before definitely make the switch if you don't mind.

ActiveRecord/Redis.
This is the configuration I'm trying out in staging: http://pastie.org/private/zepmdeduvlrvtdkc32unnq

Is there something special that should I know about AR connection pool and Rainbows! + Threadpool? My pool in database.yml is 30.
After a small load test currently I'm sitting around 60/80 connections, which is kinda expected given that also Sidekiq is in action.

I don't like the global $redis connection. Can you, or anyone else, offer any advice how to improve that, if you use it in your projects?
The goal was to keep a connection alive in a single process due to the sheer number of commands that can be put in redis as consequence of actions, but now that I'm using threads it probably needs to be changed.
The cost however for opening a new connection seems high, I was reading this: https://groups.google.com/forum/?fromgroups=#!topic/redis-db/xcz5MXykXdk

Thanks for the help,
Claudio
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: How to manage growing memory with Rainbows!

[Eric Wong]

Claudio Poli <claudio-3HQ/CcOImoi171pxa8y+qA@public.gmane.org> wrote:
> I have few more questions before definitely make the switch if you don't mind.
> 
> ActiveRecord/Redis.
> This is the configuration I'm trying out in staging: http://pastie.org/private/zepmdeduvlrvtdkc32unnq
> 
> Is there something special that should I know about AR connection pool and Rainbows! + Threadpool? My pool in database.yml is 30.

As long as your database can handle the number of connections you have,
you're probably fine.  Keep in mind you'll have 30 connections
per-process (not per-thread), so take that into account if you need to
change the process count or start an upgrade with SIGUSR2.

> After a small load test currently I'm sitting around 60/80 connections, which is kinda expected given that also Sidekiq is in action.
> 
> I don't like the global $redis connection. Can you, or anyone else, offer any advice how to improve that, if you use it in your projects?
> The goal was to keep a connection alive in a single process due to the sheer number of commands that can be put in redis as consequence of actions, but now that I'm using threads it probably needs to be changed.
> The cost however for opening a new connection seems high, I was reading this: https://groups.google.com/forum/?fromgroups=#!topic/redis-db/xcz5MXykXdk

I have no experience using Redis, but I know it's a TCP-based service.

The easiest might be a Thread-local connection with ThreadPool:

  def redis
    Thread.current[:my_app_redis] ||= Redis.new(url: url)
  end

This won't work well for ThreadSpawn.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying