murdering high-memory workers and auto-scaling

murdering high-memory workers and auto-scaling

[Ben Somers]

Two ideas, one more controversial than the other.
First: auto-killing bloated workers. My current app has some memory
leakage that wasn't really visible on our older passenger setup, since
the auto-scaling meant that bloated workers got killed periodically.
In a perfect world, we'd find and patch all of the leaks, but in the
meantime (and as a safety net) I'd like to get the bloated workers
auto-killed. It looks like it'd be simple to add in a bloated-worker
check at the same point when we check for timeout violations, and it
could be hidden behind a config setting. Alternately, I could write
this in a separate script.

Pros: might be a useful built-in feature, looks easy to implement the killing
Cons: Getting the memory usage might actually be surprisingly
difficult. Comparing to passenger's memory management code, where they
actually use platform-specific system calls, and we might get a
sizeable quantity of code that we don't want dirtying up the unicorn
internals. Also, some methods of checking appear to have performance
risks.

Second: in my use case, I have webservers running as VMs, sharing a
physical box with backend utility servers. The util servers run lots
of very CPU- and memory-hungry jobs, mostly at night; the webservers
handle requests, mostly in the daytime. Currently, most of these
webservers are running passenger, which is very polite about not using
more resources than it needs to handle requests. Unicorn, by contrast
(and by design) is very resource-greedy, what with the "scale to what
you can theoretically handle" strategy. If I spin down my number of
unicorn workers when they're not needed, I free up resources for my
util servers, which is important. TTOU and TTIN signals give me a
(very nice) means to write an auto-scaling module outside of unicorn,
but it might be nice to have it included as an optional component. (I
expect this will get voted down, as I expect the dev team is not
interested in it).

Happy to work on implementing these myself, just wanted to poll to see
if it'd be worth developing them as part of unicorn proper rather than
standalone scripts.

-ben
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: murdering high-memory workers and auto-scaling

[Clifton King]

We use the following at the bottom of a God config. I believe it's in an example
somewhere.

Bloated worker gets sent a QUIT and it will finish processing the request and
exits gracefully. We never really use TTOU and TTIN since the # of workers used
is basically determined by the ram in the machine in question.

```ruby
# unicorn workers
unicorn_worker_memory_limit = 220_000

Thread.new do
  loop do
    begin
      workers = `ps -e -www -o pid,rss,command | grep '[u]nicorn_rails worker'`
      workers.split("\n").each do |line|
        parts = line.split(' ')
        if parts[1].to_i > unicorn_worker_memory_limit
          # tell the worker to die after it finishes serving its request
          ::Process.kill('QUIT', parts[0].to_i)
        end
      end
    rescue Object
      # don't die ever once we've tested this
      nil
    end

    sleep 30
  end
end

```

Clifton

On Mar 1, 2012, at 5:52 PM, Ben Somers wrote:

> Two ideas, one more controversial than the other.
> First: auto-killing bloated workers. My current app has some memory
> leakage that wasn't really visible on our older passenger setup, since
> the auto-scaling meant that bloated workers got killed periodically.
> In a perfect world, we'd find and patch all of the leaks, but in the
> meantime (and as a safety net) I'd like to get the bloated workers
> auto-killed. It looks like it'd be simple to add in a bloated-worker
> check at the same point when we check for timeout violations, and it
> could be hidden behind a config setting. Alternately, I could write
> this in a separate script.
> 
> Pros: might be a useful built-in feature, looks easy to implement the killing
> Cons: Getting the memory usage might actually be surprisingly
> difficult. Comparing to passenger's memory management code, where they
> actually use platform-specific system calls, and we might get a
> sizeable quantity of code that we don't want dirtying up the unicorn
> internals. Also, some methods of checking appear to have performance
> risks.
> 
> Second: in my use case, I have webservers running as VMs, sharing a
> physical box with backend utility servers. The util servers run lots
> of very CPU- and memory-hungry jobs, mostly at night; the webservers
> handle requests, mostly in the daytime. Currently, most of these
> webservers are running passenger, which is very polite about not using
> more resources than it needs to handle requests. Unicorn, by contrast
> (and by design) is very resource-greedy, what with the "scale to what
> you can theoretically handle" strategy. If I spin down my number of
> unicorn workers when they're not needed, I free up resources for my
> util servers, which is important. TTOU and TTIN signals give me a
> (very nice) means to write an auto-scaling module outside of unicorn,
> but it might be nice to have it included as an optional component. (I
> expect this will get voted down, as I expect the dev team is not
> interested in it).
> 
> Happy to work on implementing these myself, just wanted to poll to see
> if it'd be worth developing them as part of unicorn proper rather than
> standalone scripts.
> 
> -ben
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: murdering high-memory workers and auto-scaling

[Eric Wong]

Ben Somers <somers.ben@gmail.com> wrote:
> Two ideas, one more controversial than the other.

Neither is really controversal.

> First: auto-killing bloated workers. My current app has some memory
> leakage that wasn't really visible on our older passenger setup, since
> the auto-scaling meant that bloated workers got killed periodically.
> In a perfect world, we'd find and patch all of the leaks, but in the
> meantime (and as a safety net) I'd like to get the bloated workers
> auto-killed. It looks like it'd be simple to add in a bloated-worker
> check at the same point when we check for timeout violations, and it
> could be hidden behind a config setting. Alternately, I could write
> this in a separate script.
> 
> Pros: might be a useful built-in feature, looks easy to implement the killing
> Cons: Getting the memory usage might actually be surprisingly
> difficult. Comparing to passenger's memory management code, where they
> actually use platform-specific system calls, and we might get a
> sizeable quantity of code that we don't want dirtying up the unicorn
> internals. Also, some methods of checking appear to have performance
> risks.

You can try something like the following middleware (totally untested,
but I've done similar things here and there).  I don't know about
non-Linux, but I suspect /proc/#$$/* is likely to have something
similar...

class MemCheckLinux < Struct.new(:app)
  def call(env)
    # a faster, but less-readable version may use /proc/#$$/stat or
    # /proc/#$$/statm but those aren't as human-friendly as
    # /proc/#$$/status
    if /VmRSS:\s+(\d+)\s/ =~ File.read("/proc/#$$/status")
      # gracefully kill ourselves if we exceed ~100M
      Process.kill(:QUIT, $$) if $1.to_i > 100_000
    end
    app.call(env)
  end
end

use MemCheckLinux
run Rack::Lobster.new

Sadly, setrlimit(:RLIMIT_AS) only causes SIGSEGV to get raised,
and Ruby controls that signal for itself.  I sometimes use
setrlimit(:RLIMIT_CPU) + trap(:XCPU) to kill runaway processes.

Apache has long had a similar parameter where you could just
tell a worker to gracefully die after X number of requests.  That'd
also be trivial to implement with middlware using SIGQUIT.

> Second: in my use case, I have webservers running as VMs, sharing a
> physical box with backend utility servers. The util servers run lots
> of very CPU- and memory-hungry jobs, mostly at night; the webservers
> handle requests, mostly in the daytime. Currently, most of these
> webservers are running passenger, which is very polite about not using
> more resources than it needs to handle requests. Unicorn, by contrast
> (and by design) is very resource-greedy, what with the "scale to what
> you can theoretically handle" strategy. If I spin down my number of
> unicorn workers when they're not needed, I free up resources for my
> util servers, which is important. TTOU and TTIN signals give me a
> (very nice) means to write an auto-scaling module outside of unicorn,
> but it might be nice to have it included as an optional component. (I
> expect this will get voted down, as I expect the dev team is not
> interested in it).

If you can make an effort to support it when it breaks, I wouldn't mind
including a script in the examples/ section or as an optional module.

It's definitely not going to ever be the default.  Auto-scaling is hard
(if not impossible) to get right.  In my experience, it always get
things wrong by default or gets configured wrong, making things more
difficult to fix.

Dedicated servers will always be the primary target of unicorn.
(And unicorn of course only scales to server + backend resources,
 nginx handles scaling to client connections :)

> Happy to work on implementing these myself, just wanted to poll to see
> if it'd be worth developing them as part of unicorn proper rather than
> standalone scripts.

If you're willing to help support users of these scripts/modules,
I'd have no reservations about distributing them along with unicorn.
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: murdering high-memory workers and auto-scaling

[Eric Wong]

Ben Somers <somers.ben@gmail.com> wrote:
> First: auto-killing bloated workers. My current app has some memory
> leakage that wasn't really visible on our older passenger setup, since

Btw, you reported issues with memory usage on a Ubuntu system
a few months ago, is this the same system?  Are you using stock
malloc() or tcmalloc()?  (tcmalloc comes standard with REE afaik
and never releases memory to the kernel).

For glibc malloc (ptmalloc) users I mentioned MALLOC_MMAP_THRESHOLD_,
but forgot about MALLOC_ARENA_MAX.

Since MRI is mostly single-threaded (especially the memory allocation
portions), I'm tending to think the per-thread optimizations in glibc
malloc do not help in most cases and will only lead to internal
fragmentation.  So perhaps setting MALLOC_ARENA_MAX=1 (perhaps along
with mmap threshold) in the environment will reduce fragmentation
and memory usage.

It doesn't look like the MALLOC_ARENA_* environment variables are
in the manpages, yet: http://udrepper.livejournal.com/20948.html
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying