Re: Calculating the required value of "ulimit -q"

[Eric Wong <normalperson@yhbt.net>]

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> El Viernes, 15 de Enero de 2010, Eric Wong escribió:
> 
> > I just don't believe such a check before trying operations is ever
> > useful to do in user-level code.  It's analogous to:
> > 
> > * checking free memory before every memory allocation
> > * checking fd limits every time a socket/file is opened
> > * checking disk space for every file write
> > * ... many more things nobody ever checks about
> > 
> > Keep in mind that even if you did _all_ of the possible checks, you're
> > still open to race conditions because other processes/threads may be
> > hitting these limits that affect your process.
> > 
> 
> Yes, I agree. However let me explain why I do such checks:
> 
> I'm coding a server which would be runned as root (just the Unicorn master 
> process) and workers and my own extra-processes work as different user (non 
> privileged).

Even though it's technically supported, I still consider running Unicorn
as root to be the wrong thing to do...

> When running as root, the queue can be created with the appropriate 
> attributes, but for this, rlimits must be increased due to typical Linux 
> default values (even for root user!).
 
> However it could occur that the queue was already created by a non privileged 
> user which runned the server first, so its attributes would be not enough. I 
> must warn the user about it.

If the queue is already existing, then you shouldn't worry about the
underlying size/memory usage of it.

> Also, I cannot unlink the queue when running the server because it would break 
> the hot reload system of Unicorn (USR2 + auto QUIT).
> 
> 
> > Since you're hitting these cases, I wonder if your usage of queues are
> > too big or there's a part of your application that can be redesigned to
> > work with smaller queues.  Keep in mind that messages stored in the
> > queue are not swappable under Linux (unlike "normal" memory).
> 
> For sure I'm dimensioning my server to work with high traffic, and the 
> performance tests I do are very cruel :)
> 
> However these tests are passed with values of msgsize=1024 and masmsg=5000 
> (but I'll decrease the msgsize).

> In total they are 5120000 bytes (less than 5 MB of memory). Is it really 
> important even if it's not swappable?

The 5M part is probably not, but the maxmsg=5000 is big enough to make
me highly uncomfortable.  Generally, the lower the latency between
systems, the smaller the queue sizes should be.  I consider the 10
message default in Linux to be reasonable, actually.

In my experience, maximizing throughput under high load for benchmarks
can often be detrimental to providing good _average_ case
latency/predictability/failover characteristics.  Having a huge queues
mean your clients won't know to back off/failover if a machine is
overloaded.  There's a good reason things like TCP use exponential
backoff the way they do to keep the Internet usable as a whole.

Keep in mind FIFOs/pipes under Linux have only 64K total in 2.6
(and only 4K in 2.4).  For TCP sockets, they only scale up to 4MB
(by default in 2.6) for high latency (but also high throughput)
connections.

-- 
Eric Wong