limit acceptors to reduce contention on large machines

Having too many acceptor threads does not help, as it leads to
lock contention in the accept syscalls and the EPOLL_CTL_ADD
paths.  The fair FIFO ordering of _blocking_ accept/accept4
syscalls also means we trigger unnecessary task switching and
incur cache misses under high load.

Since it is almost impossible for the acceptor threads to
be stuck on disk I/O since
commit 832316624f7a8f44b3e1d78a8a7a62a399241840
("acceptor threads push directly into event queue")

1 files changed, 5 insertions, 5 deletions

diff --git a/cmogstored.c b/cmogstored.c
index 3c3f87c..93dc2ae 100644
--- a/cmogstored.c
+++ b/cmogstored.c
@@ -277,12 +277,12 @@ static bool svc_start_each(void *svcptr, void *qptr)
         /*
          * try to distribute accept() callers between workers more evenly
          * with wake-one accept() behavior by trimming down on acceptors
+         * having too many acceptor threads does not make sense, these
+         * threads are only bounded by lock contention and local bus speeds.
+         * Increasing threads here just leads to lock contention inside the
+         * kernel (accept/accept4/EPOLL_CTL_ADD)
          */
-        if (worker_processes) {
-                athr /= worker_processes;
-                if (athr == 0)
-                        athr = 1;
-        }
+        athr = worker_processes > 1 ? 1 : MAX(2, athr);
 
         svc->queue = q;