Re: Calculating the required value of "ulimit -q"

["Iñaki Baz Castillo" <ibc@aliax.net>]

El Lunes, 18 de Enero de 2010, Eric Wong escribió:

> > But the problem is that if the queue was created first by a non
> > privileged user then its attributes are limited and could be not enough
> > for high load under production.
> 
> Let your users research and decide whether or not to increase those
> values on their own based on their hardware requirements/needs.
> 
> Applications that insist on modifying kernel/system-wide defaults raises
> red flags for sysadmins and operations folks.

That's right :)


 

> > - Unicorn workers receive a HTTP PUT or DELETE.
> >
> > - In case the operation success (a document is modified) the worker
> > writes a message (upto ~500 Kbytes) in the mqueue, and replies HTTP
> > 200/201 even if the mqueue was full. This is important: the HTTP request
> > is NEVER blocked due to the mqueue so the HTTP reques processing is fast.
> 
> Have you considered returning a 50x error so a load balancer can throw
> requests to another machine in the cluster?  Or just telling your
> clients to try again later?

The point is that it's not required at all. The client upload a document (HTTP 
PUT) and the server stores it so replies HTTP 200. Ok, the client connection 
ends.
Just before sending the HTTP 200, the Unicorn worker writes into the mqueue a 
message. Other process reads it and generates a SIP message which is sent to a 
different server. In case this SIP message fails (mqueue is full or TCP error) 
then it's responsability of the process to send again latter, queue into a 
database or memory and so on. This is, the HTTP connection has been succesful 
and the client must not resend it.


> From what I'm seeing, you're setting up your systems to bite off more
> than they can chew and using the queue to hide underlying
> capacity/performance problems with the backends...

I'm using the mqueue as queue system, rather than just a inter-processes 
communication method. So in case the remote SIP server fails (or TCP 
connection) then que mqueue queues the messages.

There is other approach I'm working right now: use a normal mqueue (msgsize 
20-50) and the reader process inserts the message in an Array. Other thread in 
same process reads from the Array and sends the message via TCP/SIP.

Un this case, the mqueue is just used for inter-process communication and the 
Array as a queue. I still must do some benchmarks to compare both approachs.

 
> > - Other process (processes in fact) reads the mqueue and generates a TCP
> > SIP request (similar to HTTP) which is sent in a blocking mode to a SIP
> > server which must run on same host or same network (no network congestion
> > and such issues).
> >
> > - In the same process a green thread is reading the SIP responses to
> > match them with the sent SIP requests.. Theorically all should go fine
> > (200 OK), if not something wrong is hapeening to the SIP server. The SIP
> > request is sent in blocking mode since I want the mqueue to do handle the
> > congestion in case of too much traffic (I don't want to have a "queue"
> > Array or similar).
> >
> >
> > In my benchmark I do an excessive approach since the workers write into
> > the mqueue for any HTTP request (GET, PUT, DELETE) even if it doesn't
> > success (i.e HTTP/1.1 404), so it's really exagerated.
> > Also the benchmark runs in localhost and is based on apache "ab" command
> > (10000 GET requests with concurrency 10), which is really excessive (for
> > such traffic nginx could do load balancing with more servers).
> >
> > Using 2-4 Unicorn workers and maxmsg=5000 all goes ok, just a few errors
> > "somtimes" about mqueue full (but I insist: the test is excessive IMHO).
> >
> > So under "normal" circumstances I expect maxmsg could be safely decreased
> > to 200-500. Also msgsize could be decreaed (I don't know yet the required
> > size).
> 
> How long does it take your backends to process 200 messages in the queue?

Under normal circunstances is fast enough. But with high traffic (benchmarks) 
Unicorn workers are faster so the mqueue gets full after sometime (depending 
on maxmsg of course).


> > Let me a question: if I set msgsize=1024 but messages are always lower
> > than 512, would I get in performance if I set msgsize=512? this is: does
> > the Posix mqueue sent a fixed block of msgsize bytes even if the real
> > message is lower?
> 
> Userspace always has to pass a buffer of at least msgsize bytes to
> call mq_receive(3).  I suggest reusing the destination buffer for
> POSIX_MQ#receive whenever possible (just like with IO#sysread/IO#read)
> to avoid the overhead of allocating a new buffer every time (more GC
> overhead).

So you mean that I should avoid doing the following, right?:

  loop do
    # Blocking waiting.
    msg = @mq.shift
    SIP.send(msg)
  end

and instead I should do:

  msg = ""
  loop do
    # Blocking waiting.
    msg.replace @mq.shift
    SIP.send(msg)
  end


Am I right?




> The bottom line is that I'm not going to add things that encourage
> people to mess with system defaults.  Especially not without forcing
> them to understand the underlying issues at hand and seek alternatives
> first.

It makes lot of sense.

Thanks a lot. 


-- 
Iñaki Baz Castillo <ibc@aliax.net>