[PATCH] explain 11 byte magic number for self-pipe
- by Eric Wong @ 02/18 09:36 UTC - next

Oops, this should've been explained long ago but apparently not.

In response to a comment on
http://www.sitepoint.com/the-self-pipe-trick-explained/

> Does anybody know why both unicorn and foreman read 11 bytes from
> self-pipe?

Unfortunately I couldn't find a way to comment on the site on a
JavaScript-free browser nor does it seem possible without
registering.

Again, anybody can send plain-text mail to:
unicorn-public@bogomips.org

No registration, no real name policy, no terms-of-service, just
plain-text.  Feel free to use Tor, mixmaster or any anonymity
service, too.

more... raw reply threadlink

Bug, Unicorn can drop signals in extreme conditions
- by Steven Stewart-Gallus @ 02/16 02:19 UTC - next/prev

Hello,

While reading the article at
http://www.sitepoint.com/the-self-pipe-trick-explained/ I realized
that the signal handling code shown there and of the same type in your
application is broken.  As you have a single nonblocking pipe in your
application you can drop signals entirely (and not just fold multiple
signals of the same type into one).  The simplest fix is to use a pipe
for each individual type of signal or to just use pselect or similar.
I'm pretty sure that there is a way to use a single pipe but it seems
complicated and very difficult to implement correctly.

Thank you,
Steven Stewart-Gallus

message raw reply threadlink

  Re: Bug, Unicorn can drop signals in extreme conditions
  - by Eric Wong @ 02/18 09:15 UTC - next/prev

  Steven Stewart-Gallus <sstewartgallus00@mylangara.bc.ca> wrote:
  > Hello,
  > 
  > While reading the article at
  > http://www.sitepoint.com/the-self-pipe-trick-explained/ I realized
  > that the signal handling code shown there and of the same type in your
  > application is broken.  As you have a single nonblocking pipe in your
  > application you can drop signals entirely (and not just fold multiple
  > signals of the same type into one).  The simplest fix is to use a pipe
  > for each individual type of signal or to just use pselect or similar.
  > I'm pretty sure that there is a way to use a single pipe but it seems
  > complicated and very difficult to implement correctly.
  
  (I've Cc-ed Jesse for this)
  
  I wasn't sure exactly what you were referring to, but now I see where
  the sitepoint.com article makes calls in the wrong order:
  
       self_writer.write_nonblock('.') # XXX may fail!
       SIGNAL_QUEUE << signal
  
  In contrast, unicorn enqueues the signal before attempting to write to
  the pipe, so we don't care at all if the write fails.  It doesn't matter
  if the non-blocking write fails due to the pipe being full at all; as
  any existing data in the pipe is sufficient to cause the reader to wake
  up.
  
  The correct order would be:
  
       SIGNAL_QUEUE << signal
       self_writer.write_nonblock('.') # we don't care if this fails
  
  Furthermore, this avoids a race condition in multi-threaded situations.
  Order is critical, even outside of signal handlers, this ordering of
  events is fundamental to correct usage of things like condition
  variables and waitqueues.
  
  
  Btw, MRI 1.9.3+ also uses the self-pipe trick internally, too (see
  thread_pthread.c) for signals and thread switching.  Current versions
  use two pipes, one for high-priority wakeups, and one for low-priority
  wakeups.
  
  And on a related note, using pselect/ppoll/epoll_pwait/signalfd-style
  syscalls which affect the signal mask is not feasible with runtimes
  which already implement a high-level signal handling API.  I ripped out
  signalfd support from sleepy_penguin a few years back because it would
  always conflict with the signal handling API in Ruby itself...
  
  And eventfd is cheaper and usable in place of a self-pipe from Ruby, of
  course(*), but I haven't convinced ruby-core it's worth the maintenance
  effort for thread_pthread.c; so a conservative project like unicorn
  won't use it, yet.
  
  Anyways, thanks for bringing this to our attention.
  
  
  (*) I use it in yet another horribly-named server :)

  message raw reply parent threadlink

    Re: Bug, Unicorn can drop signals in extreme conditions
    - by Steven Stewart-Gallus @ 02/18 17:27 UTC - next/prev

    Hello,
    
    Okay, my apologies, I misunderstood the signal handling code.  I
    thought you were doing something like
    self_writer.write_nonblock(signal) instead.  This is what I thought of
    initially because I was still thinking in the context of C (where
    signals are not deferred and allocating new space on a queue in a
    signal handler would be unsafe) instead of in the context of Ruby
    where Ruby's signals are deferred.  Incidentally, MRI's signal handler
    implementation in signal.c appears to be broken in several ways but
    that is a separate topic for a separate place.
    
    Thank you,
    Steven Stewart-Gallus

    message raw reply parent threadlink

[PATCH] http_server: favor ivars over constants
- by Eric Wong @ 02/12 19:09 UTC - next/prev

In 1.9+ at least, instance variables use less space than constants
in class tables and bytecode, leading to ~700 byte reduction in
bytecode overhead on 64-bit and a reduction in constant table/entries
of the Unicorn::HttpServer class.

more... raw reply threadlink

[PATCH] ISSUES: add section for bugs in other projects
- by Eric Wong @ 02/10 17:06 UTC - next/prev

This is not anything new, just documenting what has been going
on since the beginning.

There's been a small number of generic networking (or mm) bugs in
the kernel which affect unicorn, but are usually found and fixed
with more popular, non-Ruby servers, first.

Aside from generic performance problems, I don't think there's ever
been a glibc bug which affected unicorn.

more... raw reply threadlink

  Re: [PATCH] ISSUES: add section for bugs in other projects
  - by Eric Wong @ 02/10 17:26 UTC - next/prev

  Squashing the following, since we don't want things sinkholed
  or missed:
  
  --- a/ISSUES
  +++ b/ISSUES
  @@ -6,7 +6,7 @@ submit patches and/or obtain support after you have searched the
   {documentation}[http://unicorn.bogomips.org/].
   
   * No subscription will ever be required to email the public inbox.
  -* Please Cc: all participants in a thread, as subscription is optional
  +* Cc: all participants in a thread or commit, as subscription is optional
   * Do not {top post}[http://catb.org/jargon/html/T/top-post.html] in replies
   * Quote as little as possible of the message you're replying to
   * Do not send HTML mail, it will likely be flagged as spam
  @@ -39,9 +39,13 @@ but their mailing list remains operational.
   
   Uncommon bugs we encounter in the Linux kernel should be Cc:-ed to the
   Linux kernel mailing list (LKML) at mailto:linux-kernel@vger.kernel.org
  -and other kernel lists such as mailto:netdev@vger.kernel.org
  -(for networking issues).  No subscription is necessary, and the our
  -mailing list follows the same conventions as LKML for interopability.
  +and subsystem maintainers such as mailto:netdev@vger.kernel.org
  +(for networking issues).  It is expected practice to Cc: anybody
  +involved with any problematic commits (including those in the
  +Signed-off-by: and other trailer lines).  No subscription is necessary,
  +and the our mailing list follows the same conventions as LKML for
  +interopability.  There is a kernel.org Bugzilla instance, but it is
  +ignored by most developers.
   
   Likewise for any rare glibc bugs we might encounter, we should Cc:
   mailto:libc-alpha@sourceware.org

  message raw reply parent threadlink

[PATCH 1/2] const: drop constants used by Rainbows!
- by Eric Wong @ 02/09 09:12 UTC - next/prev

Rainbows! (in maintenance mode) will need to define it's own
constants in the future.  We'll trim down our constant usage in
subsequent commits as we take advantage of Ruby VM improvements.

more... raw reply threadlink

  [PATCH 2/2] reduce and localize constant string use
  - by Eric Wong @ 02/09 09:12 UTC - next/prev

  Literal String#freeze avoids allocations since Ruby 2.1 via the
  opt_str_freeze instruction, so we can start relying on it in
  some places as Ruby 2.1 adoption increases.  The 100-continue
  handling is a good place to start since it is an uncommonly-used
  code path which benefits from size reduction and the negative
  performance impact is restricted to a handful of users.
  
  HTTP_RESPONSE_START can safely live in http_request.rb as its
  usage does not cross namespace boundaries
  
  The goal is to eventually eliminate Unicorn::Const entirely.

  more... raw reply parent threadlink

[PATCH] favor "a.b(&:c)" form over "a.b { |x| x.c }"
- by Eric Wong @ 02/06 22:50 UTC - next/prev

The former is shorter Ruby code and also generates smaller
bytecode.

more... raw reply threadlink

[PATCH] test_socket_helper: do not depend on SO_REUSEPORT
- by Eric Wong @ 02/06 22:34 UTC - next/prev

Older Rubies (2.0) may not define SO_REUSEPORT even if the
kernel and libc support it

more... raw reply threadlink

[PATCH] fix uninstalled testing and reduce require paths
- by Eric Wong @ 02/06 22:17 UTC - next/prev

This fixes a bug introduced in
commit fe83ead4eae6f011fa15f506cd80cb4256813a92
(GNUmakefile: fix clean gem build + reduce build cruft)
which broke clean Ruby installations without an existing
unicorn gem installed :x

more... raw reply threadlink

[PATCH] doc: update support status for Ruby versions
- by Eric Wong @ 02/06 20:15 UTC - next/prev

unicorn 5 will not support Ruby 1.8 anymore.

Drop mentions of Rubinius, too, it's too difficult to support due to
the proprietary and registration-required nature of its bug tracker.
The smaller memory footprint and CoW-friendly memory allocator in
mainline Ruby is a better fit for unicorn, anyways.

Since Ruby 1.9+ bundles RubyGems and gem startup is faster nowadays,
we'll just depend on that instead of not loading RubyGems.

Drop the local.mk.sample file, too, since it's way out-of-date
and probably isn't useful (I have not used it in a while).

more... raw reply threadlink

[PATCH] use require_relative to reduce syscalls at startup
- by Eric Wong @ 02/05 18:01 UTC - next/prev

require_relative appeared in Ruby 1.9.2 to speed up load times by
avoiding needless open() syscalls.  This has no effect if you're using
RUBYLIB or the '-I' option when running ruby(1), but avoids searching
paths in other gems.

This does not affect unicorn greatly as unicorn does not activate many
gems, but still leads to reducing ~45 syscalls during startup.

more... raw reply threadlink

[PATCH] favor IO#close_on_exec= over fcntl in 1.9+
- by Eric Wong @ 02/05 18:01 UTC - next/prev

IO#close_on_exec* methods are available since Ruby 1.9.1.  It
allows us to use less bytecode as it requires fewer operands and
avoids constant lookups.

more... raw reply threadlink

[PATCH] remove 1.8, <= 1.9.1 fallback for missing IO#autoclose=
- by Eric Wong @ 02/05 18:01 UTC - next/prev

We're requiring Ruby 1.9.3+, so we can safely depend
on IO#autoclose= being available in 1.9+ and shave off
some bloat.

more... raw reply threadlink

[PATCH] socket_helper: reduce constant lookups and caching
- by Eric Wong @ 02/05 17:19 UTC - next/prev

In Ruby 1.9.2+, socket options may be specified using symbols
instead of constants to avoid the need to import Socket::Constants
into the namespace.  This also has a nice side-effect of reducing
the size of the bytecode by trading 3 instructions (getinlinecache,
getconstant, setinlinecache) for one "putobject" instruction.

Nowadays, we may also avoid defining OS-specific constants ourselves
since 1.9+ versions of Ruby already provide them to further reduce
bytecode size.

getsockopt also returns Socket::Option objects in 1.9.2+,
allowing us to avoid the larger "unpack('i')" method dispatch
for an operand-free "int" method call.

Finally, favor Object#nil? calls rather than "== nil" comparisons
to reduce bytecode size even more.

Since this code is only called at startup time, it does not benefit
from inline caching of constant lookups in current mainline Ruby.

Combined, these changes reduce YARV bytecode size by around 2K on a
64-bit system.

more... raw reply threadlink

[PATCH] GNUmakefile: fix clean gem build + reduce build cruft
- by Eric Wong @ 02/04 20:16 UTC - next/prev

Ensure we have a NEWS file for building the gem beforehand.
We don't need to polute lib/ with object files, either.

more... raw reply threadlink

[PATCH] http: standalone require + reduction in binary size
- by Eric Wong @ 02/04 20:16 UTC - next/prev

This allows requiring just the C extension part of "unicorn_http",
without requiring the rest of unicorn, allowing other HTTP servers
using the same parser to be slimmer.

On my x86-64 Debian 7.0 system:

    text	   data	    bss	    dec	    hex	filename
   44026	   1976	    488	  46490	   b59a	lib/unicorn_http.so
   43930	   1976	    456	  46362	   b51a	lib/unicorn_http.so

more... raw reply threadlink

Re: [RFC] remove old inetd+git examples and exec_cgi
- by Eric Wong @ 02/04 20:04 UTC - next/prev

Eric Wong <e@80x24.org> wrote:
> I'm tempted to remove these for 5.x, but maybe somebody depends on the
> lib/unicorn/app/* portions somewhere... Does anybody care?

Pushed as commit fd937863d67d5a886df2a53f1736d643fbb91e4a

message raw reply parent threadlink

Timeouts longer than expected.
- by Antony Gelberg @ 01/29 13:06 UTC - next/prev

Hi all.  We use unicorn in production across four servers, behind
nginx and HAProxy (to load balance).  We set a 300s timeout in the
config file.  On an average day, we see something like:

E, [2015-01-22T17:01:24.335600 #4311] ERROR -- : worker=2 PID:21101
timeout (301s > 300s), killing

in our logs, ~60 times.  However, one day we noticed that production
had gone down with all the unicorns showing entries like the
following:

E, [2015-01-22T12:35:15.643020 #4311] ERROR -- : worker=1 PID:4605
timeout (451s > 300s), killing

(note the 451s before it was killed).

Any clues how we can better-understand the root cause, even if it's
something we'll put in place for the future?  We lack visibility here.

Thanks in advance,

Antony

message raw reply threadlink

  Re: Timeouts longer than expected.
  - by Eric Wong @ 01/29 20:00 UTC - next/prev

  Antony Gelberg <antony.gelberg@gmail.com> wrote:
  > <Hi all. We use unicorn in production across four servers, behind > ...>
  
  What other log entries appear in the 8 minutes leading up to the 451s
  line?
  
  Which version of unicorn is this?
  
  There were some bugs fixed in ancient unicorn versions; and current
  versions are still affected by big NTP adjustments due to the lack of
  monotonic clock API in older Rubies.
  
  I have a patch queued up for 5.x to use the monotonic clock, but
  requires Ruby 2.1+ to be useful:
  http://bogomips.org/unicorn-public/m/20150118033916.GA4478%40dcvr.yhbt.net.html
  
  Maybe there's another sleep calculation bug, though.  Not many people
  rely on giant timeouts and probably never notice it.
  
  I'll take a closer look at the master sleep calculations (in
  murder_lazy_workers) in a week or so and see if there's another problem
  lurking in the current versions.
  
  > Any clues how we can better-understand the root cause, even if it's
  > something we'll put in place for the future?  We lack visibility here.
  
  I'd use an in-process timeout to get backtraces.  The
  Application_Timeouts doc in the source and online has more
  tips on dealing with timeouts in general.
  
    http://unicorn.bogomips.org/Application_Timeouts.html
  
  Also curious: you have users willing to wait 5 minutes for an
  HTTP response?  Yikes!

  message raw reply parent threadlink

[PATCH] http: -Wshorten-64-to-32 warnings on clang
- by Eric Wong @ 01/28 18:44 UTC - next/prev

Tested on x86_64, clang version 3.5-1ubuntu1 (trunk) (LLVM 3.5)
These warnings were introduced on
commit 4b2782a926d8f131b1e7382be35e3abb77bf4be5
("http: reduce parser from 72 to 56 bytes on 64-bit")
and did not affect any releases.

These length checks should not be necessary in reality because
HTTP header sizes never come close to 4GB in size.

Fixup a minor coding style (inherited from Mongrel) violation
while we're at it (tabs => spaces).

more... raw reply threadlink

global variable in after_fork
- by Joe Williams @ 01/23 18:32 UTC - next/prev

I am attempting to do something like the following:

after_fork do |server, worker|
  $unicorn_uptime = Time.now.to_i.to_s
  $unicorn_uptime.freeze
<snip>

Unfortunately the variable comes up nil when later accessed. Anyone have
thoughts on what I am missing?

Thanks!
-Joe

message raw reply threadlink

  Re: global variable in after_fork
  - by Eric Wong @ 01/24 23:30 UTC - next/prev

  Joe Williams <williams.joe@gmail.com> wrote:
  > I am attempting to do something like the following:
  > 
  > after_fork do |server, worker|
  >   $unicorn_uptime = Time.now.to_i.to_s
  >   $unicorn_uptime.freeze
  > <snip>
  > 
  > Unfortunately the variable comes up nil when later accessed. Anyone have
  > thoughts on what I am missing?
  
  I tried your after_fork hook with the following config.ru
  and it prints out $unicorn_uptime fine:
  
  -------------------- config.ru --------------------------
  use Rack::ContentLength
  use Rack::ContentType, 'text/plain'
  run lambda { |env| [ 200, {}, [ "#$unicorn_uptime\n"] ] }

  more... raw reply parent threadlink


page: next      atom permalink
- unicorn Rack HTTP server user/dev discussion
A public-inbox, anybody may post in plain-text (not HTML):
unicorn-public@bogomips.org
git URL for ssoma: git://bogomips.org/unicorn-public.git
homepage: http://unicorn.bogomips.org/
subscription optional: unicorn-public+subscribe@bogomips.org