Rainbows! Rack HTTP server user/dev discussion
 help / Atom feed
* memory usage
@ 2013-10-22  8:28 Alexandre Riveira
       [not found] ` <52663744.6040600-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org>
  0 siblings, 1 reply; 6+ messages in thread
From: Alexandre Riveira @ 2013-10-22  8:28 UTC (permalink / raw)
  To: Rainbows! list

I usually use XEpollThreadPool however persebi a large memory usage for 
sites that have a lot of content and access reaching over 1 GB. Changed 
to ThreadPool and memory fell not from 4x 256MB. However 
XEpollThreadPool is faster than ThreadPool. Accepted suggestions for 
configuration. I tested also with CoolioThreadPool speed was good but 
the memory consumption and large too.


Tanks


Alexandre Riveira
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[flat|threaded] 6+ messages in thread

* Re: memory usage
       [not found] ` <52663744.6040600-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org>
@ 2013-10-22 16:09   ` Eric Wong
       [not found]     ` <20131022160930.GA3020-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
  0 siblings, 1 reply; 6+ messages in thread
From: Eric Wong @ 2013-10-22 16:09 UTC (permalink / raw)
  To: Rainbows! list

Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> I usually use XEpollThreadPool however persebi a large memory usage
> for sites that have a lot of content and access reaching over 1 GB.
> Changed to ThreadPool and memory fell not from 4x 256MB. However
> XEpollThreadPool is faster than ThreadPool. Accepted suggestions for
> configuration. I tested also with CoolioThreadPool speed was good
> but the memory consumption and large too.

Wait, ThreadPool uses significantly less than XEpollThreadPool?

Since this is likely glibc/eglibc, try setting MALLOC_ARENA_MAX=1 and
MALLOC_ARENA_TEST=1 in the env before starting Rainbows!  AFAIK, these
are mainly documented by Red Hat, but in all recent-ish versions of
glibc/eglibc support it.

Then increase the values if you see performance gains (unlikely unless
your app releases the GVL a lot or don't have one (rbx)).
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[flat|threaded] 6+ messages in thread

* Re: memory usage
       [not found]     ` <20131022160930.GA3020-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
@ 2013-10-22 18:08       ` Alexandre Riveira
  2013-10-27 11:35       ` Alexandre Riveira
  1 sibling, 0 replies; 6+ messages in thread
From: Alexandre Riveira @ 2013-10-22 18:08 UTC (permalink / raw)
  To: Rainbows! list

Yes, XEpollThreadPool 4 times more memory than ThreadPool will do tests 
as you indicated and verifying systems that have other glibc

Tanks

Alexandre Riveira

On 22-10-2013 16:09, Eric Wong wrote:
> Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
>> I usually use XEpollThreadPool however persebi a large memory usage
>> for sites that have a lot of content and access reaching over 1 GB.
>> Changed to ThreadPool and memory fell not from 4x 256MB. However
>> XEpollThreadPool is faster than ThreadPool. Accepted suggestions for
>> configuration. I tested also with CoolioThreadPool speed was good
>> but the memory consumption and large too.
> Wait, ThreadPool uses significantly less than XEpollThreadPool?
>
> Since this is likely glibc/eglibc, try setting MALLOC_ARENA_MAX=1 and
> MALLOC_ARENA_TEST=1 in the env before starting Rainbows!  AFAIK, these
> are mainly documented by Red Hat, but in all recent-ish versions of
> glibc/eglibc support it.
>
> Then increase the values if you see performance gains (unlikely unless
> your app releases the GVL a lot or don't have one (rbx)).
> _______________________________________________
> Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
> http://rubyforge.org/mailman/listinfo/rainbows-talk
> Do not quote signatures (like this one) or top post when replying
>

_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[flat|threaded] 6+ messages in thread

* Re: memory usage
       [not found]     ` <20131022160930.GA3020-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
  2013-10-22 18:08       ` Alexandre Riveira
@ 2013-10-27 11:35       ` Alexandre Riveira
       [not found]         ` <526CFA8C.7000300-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org>
  1 sibling, 1 reply; 6+ messages in thread
From: Alexandre Riveira @ 2013-10-27 11:35 UTC (permalink / raw)
  To: Rainbows! list

Hello Eric,

I discovered that when you restart the server (linux 2.3.50, glibc 2.15) 
memory stopped growing indiscriminately.

2 questions:

1) The POOL_SIZE XEpollThreadPool series of the number of connections 
accepted epoll so'd have one for each thread
2) I saw the gem 'mall' and she seems to adjust as you indicted by glibc 
environment variables are correct? If yes in config / initializers added 
a file.rb with the following:
Mall.opt (Mall :: ARENA_MAX, 1)
Mall.opt (Mall :: ARENA_TEST, 1)


Tanks,

Alexandre Riveira

On 22-10-2013 16:09, Eric Wong wrote:
> Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
>> I usually use XEpollThreadPool however persebi a large memory usage
>> for sites that have a lot of content and access reaching over 1 GB.
>> Changed to ThreadPool and memory fell not from 4x 256MB. However
>> XEpollThreadPool is faster than ThreadPool. Accepted suggestions for
>> configuration. I tested also with CoolioThreadPool speed was good
>> but the memory consumption and large too.
> Wait, ThreadPool uses significantly less than XEpollThreadPool?
>
> Since this is likely glibc/eglibc, try setting MALLOC_ARENA_MAX=1 and
> MALLOC_ARENA_TEST=1 in the env before starting Rainbows!  AFAIK, these
> are mainly documented by Red Hat, but in all recent-ish versions of
> glibc/eglibc support it.
>
> Then increase the values if you see performance gains (unlikely unless
> your app releases the GVL a lot or don't have one (rbx)).
> _______________________________________________
> Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
> http://rubyforge.org/mailman/listinfo/rainbows-talk
> Do not quote signatures (like this one) or top post when replying
>

_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[flat|threaded] 6+ messages in thread

* Re: memory usage
       [not found]         ` <526CFA8C.7000300-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org>
@ 2013-10-28  0:37           ` Eric Wong
       [not found]             ` <20131028003755.GA6569-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
  0 siblings, 1 reply; 6+ messages in thread
From: Eric Wong @ 2013-10-28  0:37 UTC (permalink / raw)
  To: Rainbows! list

Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> Hello Eric,
> 
> I discovered that when you restart the server (linux 2.3.50, glibc
> 2.15) memory stopped growing indiscriminately.

Linux 2.3.50?  Huh?  Assuming that's 3.2.50.  Anyways, user-visible
memory usage is only dependent on glibc version.

> 2 questions:
> 
> 1) The POOL_SIZE XEpollThreadPool series of the number of
> connections accepted epoll so'd have one for each thread

No, pool_size for XEpollThreadPool is the number of worker threads
capable of dispatching the application.  It can hold many more clients.

> 2) I saw the gem 'mall' and she seems to adjust as you indicted by
> glibc environment variables are correct? If yes in config /
> initializers added a file.rb with the following:
> Mall.opt (Mall :: ARENA_MAX, 1)
> Mall.opt (Mall :: ARENA_TEST, 1)

Yes, mall works, too, and you can tune a running app with some of the
knobs.  However, for arena settings, I think they must be set early on
(before many threads are spawned) to be effective.  So environment
variables (set before starting Ruby) are probably the best bet, since
Ruby (MRI) spawns a background timer thread right away.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[flat|threaded] 6+ messages in thread

* Re: memory usage
       [not found]             ` <20131028003755.GA6569-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
@ 2013-10-28 18:40               ` Alexandre Riveira
  0 siblings, 0 replies; 6+ messages in thread
From: Alexandre Riveira @ 2013-10-28 18:40 UTC (permalink / raw)
  To: Rainbows! list

Hello Eric,
My linux is 3.2.50, sorry my error, environment variables work fine now.

Tanks for your help

Alexandre Riveira


On 28-10-2013 00:37, Eric Wong wrote:
> Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
>> Hello Eric,
>>
>> I discovered that when you restart the server (linux 2.3.50, glibc
>> 2.15) memory stopped growing indiscriminately.
> Linux 2.3.50?  Huh?  Assuming that's 3.2.50.  Anyways, user-visible
> memory usage is only dependent on glibc version.
>
>> 2 questions:
>>
>> 1) The POOL_SIZE XEpollThreadPool series of the number of
>> connections accepted epoll so'd have one for each thread
> No, pool_size for XEpollThreadPool is the number of worker threads
> capable of dispatching the application.  It can hold many more clients.
>
>> 2) I saw the gem 'mall' and she seems to adjust as you indicted by
>> glibc environment variables are correct? If yes in config /
>> initializers added a file.rb with the following:
>> Mall.opt (Mall :: ARENA_MAX, 1)
>> Mall.opt (Mall :: ARENA_TEST, 1)
> Yes, mall works, too, and you can tune a running app with some of the
> knobs.  However, for arena settings, I think they must be set early on
> (before many threads are spawned) to be effective.  So environment
> variables (set before starting Ruby) are probably the best bet, since
> Ruby (MRI) spawns a background timer thread right away.
> _______________________________________________
> Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
> http://rubyforge.org/mailman/listinfo/rainbows-talk
> Do not quote signatures (like this one) or top post when replying
>

_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying


^ permalink raw reply	[flat|threaded] 6+ messages in thread

end of thread, back to index

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-10-22  8:28 memory usage Alexandre Riveira
     [not found] ` <52663744.6040600-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org>
2013-10-22 16:09   ` Eric Wong
     [not found]     ` <20131022160930.GA3020-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2013-10-22 18:08       ` Alexandre Riveira
2013-10-27 11:35       ` Alexandre Riveira
     [not found]         ` <526CFA8C.7000300-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org>
2013-10-28  0:37           ` Eric Wong
     [not found]             ` <20131028003755.GA6569-yBiyF41qdooeIZ0/mPfg9Q@public.gmane.org>
2013-10-28 18:40               ` Alexandre Riveira

Rainbows! Rack HTTP server user/dev discussion

Archives are clonable:
	git clone --mirror http://bogomips.org/rainbows-public
	git clone --mirror http://ou63pmih66umazou.onion/rainbows-public

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.lang.ruby.rainbows
	nntp://ou63pmih66umazou.onion/inbox.comp.lang.ruby.rainbows

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox