Re: Calculating the required value of "ulimit -q"

["Iñaki Baz Castillo" <ibc@aliax.net>]

El Sábado, 16 de Enero de 2010, Eric Wong escribió:
> Even though it's technically supported, I still consider running Unicorn
> as root to be the wrong thing to do...

Yes, I will also provide instructions to run is at non root user. But note it 
makes thinkgs complex:

- A normal user cannot create a PID file under the standard path /var/run so 
prior tu running the server it's required to mkdir /var/run/unicorn/ with 
appropriate write permissions.

- Same for listening UNIX socket.

- Same for posix mqueue attributes (it's required to edit /etc/security/limits 
and restart the system).

Is there some security issue in running just the master as root? it doesn't 
process requests, and a vulnerable worker (due to the Rack application code) 
cannot kill the master or force it to do something dangerous. Am I wrong?


> > When running as root, the queue can be created with the appropriate
> > attributes, but for this, rlimits must be increased due to typical Linux
> > default values (even for root user!).
> >
> > However it could occur that the queue was already created by a non
> > privileged user which runned the server first, so its attributes would be
> > not enough. I must warn the user about it.
> 
> If the queue is already existing, then you shouldn't worry about the
> underlying size/memory usage of it.

But the problem is that if the queue was created first by a non privileged 
user then its attributes are limited and could be not enough for high load 
under production.


> The 5M part is probably not, but the maxmsg=5000 is big enough to make
> me highly uncomfortable.  Generally, the lower the latency between
> systems, the smaller the queue sizes should be.  I consider the 10
> message default in Linux to be reasonable, actually.
> 
> In my experience, maximizing throughput under high load for benchmarks
> can often be detrimental to providing good _average_ case
> latency/predictability/failover characteristics.  Having a huge queues
> mean your clients won't know to back off/failover if a machine is
> overloaded.  There's a good reason things like TCP use exponential
> backoff the way they do to keep the Internet usable as a whole.
> 
> Keep in mind FIFOs/pipes under Linux have only 64K total in 2.6
> (and only 4K in 2.4).  For TCP sockets, they only scale up to 4MB
> (by default in 2.6) for high latency (but also high throughput)
> connections.

Thanks for so good explanation. Please let me explain why I need so ammount of 
maxmsg:

- Unicorn workers receive a HTTP PUT or DELETE.

- In case the operation success (a document is modified) the worker writes a 
message (upto ~500 Kbytes) in the mqueue, and replies HTTP 200/201 even if the 
mqueue was full. This is important: the HTTP request is NEVER blocked due to 
the mqueue so the HTTP reques processing is fast.

- Other process (processes in fact) reads the mqueue and generates a TCP SIP 
request (similar to HTTP) which is sent in a blocking mode to a SIP server 
which must run on same host or same network (no network congestion and such 
issues).

- In the same process a green thread is reading the SIP responses to match 
them with the sent SIP requests.. Theorically all should go fine (200 OK), if 
not something wrong is hapeening to the SIP server. The SIP request is sent in 
blocking mode since I want the mqueue to do handle the congestion in case of 
too much traffic (I don't want to have a "queue" Array or similar).


In my benchmark I do an excessive approach since the workers write into the 
mqueue for any HTTP request (GET, PUT, DELETE) even if it doesn't success (i.e 
HTTP/1.1 404), so it's really exagerated.
Also the benchmark runs in localhost and is based on apache "ab" command 
(10000 GET requests with concurrency 10), which is really excessive (for such 
traffic nginx could do load balancing with more servers).

Using 2-4 Unicorn workers and maxmsg=5000 all goes ok, just a few errors 
"somtimes" about mqueue full (but I insist: the test is excessive IMHO).

So under "normal" circumstances I expect maxmsg could be safely decreased to 
200-500. Also msgsize could be decreaed (I don't know yet the required size).

Let me a question: if I set msgsize=1024 but messages are always lower than 
512, would I get in performance if I set msgsize=512? this is: does the Posix 
mqueue sent a fixed block of msgsize bytes even if the real message is lower?


Thanks a lot for all your help.

 


-- 
Iñaki Baz Castillo <ibc@aliax.net>