| Date | Commit message (Collapse) |
|---|
|
Oops, this should've been explained long ago but apparently not.
In response to a comment on
http://www.sitepoint.com/the-self-pipe-trick-explained/
> Does anybody know why both unicorn and foreman read 11 bytes from
> self-pipe?
Unfortunately I couldn't find a way to comment on the site on a
JavaScript-free browser nor does it seem possible without
registering.
Again, anybody can send plain-text mail to:
unicorn-public@bogomips.org
No registration, no real name policy, no terms-of-service, just
plain-text. Feel free to use Tor, mixmaster or any anonymity
service, too.
|
|
unicorn 5 will not support Ruby 1.8 anymore.
Drop mentions of Rubinius, too, it's too difficult to support due to
the proprietary and registration-required nature of its bug tracker.
The smaller memory footprint and CoW-friendly memory allocator in
mainline Ruby is a better fit for unicorn, anyways.
Since Ruby 1.9+ bundles RubyGems and gem startup is faster nowadays,
we'll just depend on that instead of not loading RubyGems.
Drop the local.mk.sample file, too, since it's way out-of-date
and probably isn't useful (I have not used it in a while).
[reinstate 1.9 version check for listener_fds in backport]
|
|
When daemonizing, it is possible for the grandparent to be
terminated by another process before the master can notify
it. Do not abort the master in this case.
This may fix the following issue:
https://github.com/kostya/eye/issues/49
(which I was notified of privately via email)
|
|
We close SELF_PIPE in the worker immediately, but signal handlers
do not get setup immediately. So prevent workers from erroring out
due to invalid SELF_PIPE.
|
|
This protects us from two problems:
1) we (or our app) somehow called IO#close on one of the sockets
we listen on without removing it from the readers array.
We'll ignore IOErrors from IO#close and assume we wanted to
close it.
2) our SIGQUIT handler is interrupted by itself. This can happen as
a fake signal from the master could be handled and a real signal
from an outside user is sent to us (e.g. from unicorn-worker-killer)
or if a user uses the killall(1) command.
|
|
Signaling using normal kill(2) is preserved, but the master now
prefers to signal workers using a pipe rather than kill(2).
Non-graceful signals (:TERM/:KILL) are still sent using kill(2),
as they ask for immediate shutdown.
This change is necessary to avoid triggering the ubf (unblocking
function) for rb_thread_call_without_gvl (and similar) functions
extensions. Most notably, this fixes compatibility with newer
versions of the 'pg' gem which will cancel a running DB query if
signaled[1].
This also has the nice side-effect of allowing a premature
master death (assuming preload_app didn't cause the master to
spawn off rogue child daemons).
Note: users should also refrain from using "killall" if using the
'pg' gem or something like it.
Unfortunately, this increases FD usage in the master as the writable
end of the pipe is preserved in the master. This limit the number
of worker processes the master may run to the open file limit of the
master process. Increasing the open file limit of the master
process may be needed. However, the FD use on the workers is
reduced by one as the internal self-pipe is no longer used. Thus,
overall pipe allocation for the kernel remains unchanged.
[1] - pg is correct to cancel a query, as it cannot know if
the signal was for a) graceful unicorn shutdown or
b) oh-noes-I-started-a-bad-query-ABORT-ABORT-ABORT!!
|
|
This reduces the window for a non-existent PID for folks who monitor
PIDs (not a great idea anyways). Unfortunately, this change also brings
us back to the case where having a PID later (for other process monitors)
is beneficial but more unicorn releases exist where we write the PID
early.
Thanks to Jimmy Soho for reporting this issue.
ref: <CAHStS5gFYcPBDxkVizAHrOeDKAkjT69kruFdgaY0CbB+vLbK8Q@mail.gmail.com>
This partially reverts 7d6ac0c17eb29a00a5b74099dbb3d4d015999f27
Folks: please monitor your app with HTTP requests rather than checking
processes, a stuck/wedged Ruby VM is still a running one.
|
|
This renables the ability for Ruby 1.8.6 environments to perform reexecs
[ew: clarified this is for 1.8.6,
favor literal {} over Hash.new,
tweaked LISTENERS.map => LISTENERS.each, thanks to Hleb Valoshka
]
Signed-off-by: Eric Wong <normalperson@yhbt.net>
|
|
Thanks to Hongli Lai for noticing my typo. While we're at it, finish up
a halfway-written comment for the EXDEV case
|
|
Ruby 1.9 and later includes IO#autoclose=, so we can use it
and prevent some dead IO objects from hanging around.
|
|
This will preserve mtime on successful renames for comparisions.
While we're at it, avoid writing the new PID until the listeners are
inherited successfully. This can be useful to avoid accidentally
clobbering a good PID if binding the listener or building the app
(preload_app==true) fails
|
|
We do not attempt to write HTTP responses for socket errors if
clients disconnect from us unexpectedly.
Additionally, we do not hide backtraces EINVAL/EBADF errors, since
they are indicative of real bugs which must be fixed.
We do continue to hide hide EOF, ECONNRESET, ENOTCONN, and EPIPE
because clients (even "friendly") ones will break connections due to
client crashes or network failure (which is common for me :P), and
the backtraces from those will cause excessive logging and even
become a DoS vector.
|
|
Once a connection is hijacked, we ignore it completely and leave
the connection at the mercy of the application.
|
|
Rack 1.5.0 (protocol version [1,2]) adds support for
hijacking the client socket (removing it from the control
of unicorn (or any other Rack webserver)).
Tested with rack 1.5.0.
|
|
Rainbows! relies on the ERROR_XXX_RESPONSE constants of unicorn
4.x. Changing the constants in unicorn 4.x will break existing
versions of Rainbows!, so remove the dependency on the constants
and generate the error response dynamically.
Unlike Mongrel, unicorn is unlikely to see malicious traffic and
thus unlikely to benefit from making error messages constant.
For unicorn 5.x, we will drop these constants entirely.
(Rainbows! most likely cannot support check_client_connection
consistently across all concurrency models since some of them
pessimistically buffer all writes in userspace. However, the
extra concurrency of Rainbows! makes it less likely to be
overloaded than unicorn, so this feature is likely less useful
for Rainbows!)
|
|
This patch checks incoming connections and avoids calling the application
if the connection has been closed.
It works by sending the beginning of the HTTP response before calling
the application to see if the socket can successfully be written to.
By enabling this feature users can avoid wasting application rendering
time only to find the connection is closed when attempting to write, and
throwing out the result.
When a client disconnects while being queued or processed, Nginx will log
HTTP response 499 but the application will log a 200.
Enabling this feature will minimize the time window during which the problem
can arise.
The feature is disabled by default and can be enabled by adding
'check_client_connection true' to the unicorn config.
[ew: After testing this change, Tom Burns wrote:
So we just finished the US Black Friday / Cyber Monday weekend running
unicorn forked with the last version of the patch I had sent you. It
worked splendidly and helped us handle huge flash sales without
increased response time over the weekend.
Whereas in previous flash traffic scenarios we would see the number of
HTTP 499 responses grow past the number of real HTTP 200 responses,
over the weekend we saw no growth in 499s during flash sales.
Unexpectedly the patch also helped us ward off a DoS attack where the
attackers were disconnecting immediately after making a request.
ref: <CAK4qKG3rkfVYLyeqEqQyuNEh_nZ8yw0X_cwTxJfJ+TOU+y8F+w@mail.gmail.com>
]
Signed-off-by: Eric Wong <normalperson@yhbt.net>
|
|
In the case where preload_app is true, delay binding new
listeners until after loading the application.
Some applications have very long load times (especially Rails
apps with Ruby 1.9.2). Binding listeners early may cause a load
balancer to incorrectly believe the unicorn workers are ready to
serve traffic even while the app is being loaded.
Once a listener is bound, connect() requests from the load
balancer succeed until the listen backlog is filled. This
allows requests to pile up for a bit (depending on backlog size)
before getting rejected by the kernel. By the time the
application is loaded and ready-to-run, requests in the
listen backlog are likely stale and not useful to process.
Processes inheriting listeners do not suffer this effect, as the
old process should still be capable of serving new requests.
This change does not improve the situation for the
preload_app=false (default) use case. There may not be a
solution for preload_app=false users using large applications.
Fortunately Ruby 1.9.3+ improves load times of large
applications significantly over 1.9.2 so this should be less of
a problem in the future.
Reported via private email sent on 2012-06-29T22:59:10Z
|
|
Since there's nothing unicorn can do to avoid this error
on unconnected/halfway-connected clients, ignoring ENOTCONN
is a safe bet.
Rainbows! has long had this rescue as it called getpeername(2)
on untrusted sockets
|
|
Previously we relied on implicit socket shutdown() from the
close() syscall. However, some Rack applications fork()
(without calling exec()), creating a potentially long-lived
reference to the underlying socket in a child process. This
ends up causing nginx to wait on the socket shutdown when the
child process exits.
Calling shutdown() explicitly signals nginx (or whatever client)
that the unicorn worker is done with the socket, regardless of
the number of FD references to the underlying socket in
existence.
This was not an issue for applications which exec() since
FD_CLOEXEC is always set on the client socket.
Thanks to Patrick Wenger for discovering this. Thanks to
Hongli Lai for the tip on using shutdown() as is done in
Passenger.
ref: http://mid.gmane.org/CAOG6bOTseAPbjU5LYchODqjdF3-Ez4+M8jo-D_D2Wq0jkdc4Rw@mail.gmail.com
|
|
In some cases, EPERM may indicate a real configuration problem,
but it can also just mean the pid file is stale.
|
|
If unicorn doesn't get terminated cleanly (for example if the machine
has its power interrupted) and the pid in the pidfile gets used by
another process, the current unicorn code will exit and not start a
server. This tiny patch fixes that behaviour.
Acked-by: Eric Wong <normalperson@yhbt.net>
|
|
No need to duplicate logic here
|
|
It's possible for a SIGUSR1 signal to be received in the
worker immediately before calling IO.select. In that case,
do not clutter logging with IOError and just process the
reopen log request.
|
|
This will also be the foundation of SSL support in Rainbows!
and Zbatery. Some users may also want to use this in
Unicorn on LANs to meet certain security/auditing requirements.
Of course, Nightmare! (in whatever form) should also be able to
use it.
|
|
This prevents the stopping of all workers by SIGWINCH if you're
using a windowing system that will 'exec' unicorn from a process
that's already in a process group.
Acked-by: Eric Wong <normalperson@yhbt.net>
|
|
The old comment was confusing. We only zero the tick counter
when forking because application loading can take a long time.
Otherwise, it's always updated.
ref: http://mid.gmane.org/20110908191352.GA25251@dcvr.yhbt.net
|
|
There is no need to keep extra hashes or Proc objects around in
the heap.
|
|
I've noticed in stderr logs from some folks that (last resort)
timeouts from the master process are taking too long to activate
due to the workarounds for suspend/hibernation.
|
|
The signal handler from the master is still active and will
push the pending signal to SIG_QUEUE if a worker receives
a signal immediately after forking.
|
|
We only need the fileno in the key which we use
to generate the UNICORN_FD env. Otherwise the IO
object is accepted and understood by Ruby.
|
|
Setting the close-on-exec flag by default and closing
non-standard descriptors is proposed for Ruby 1.9.4/2.0.0.
Since Unicorn is one of the few apps to rely on FD inheritance
across exec(), we need to workaround this by redirecting each
listener FD to itself for Kernel#exec.
Ruby supports a hash as the final argument to Kernel#exec since
at least 1.9.1 (nobody cares for 1.9.0 anymore). This allows
users to backport close-on-exec by default patches to older
1.9.x installs without breaking anything.
ref: http://redmine.ruby-lang.org/issues/5041
|
|
This helps close a race condition preventing shutdown if
loading the application (preload_app=false) takes a long
time and the user decides to kil workers instead.
|
|
Future versions of Ruby may change this from the default *nix
behavior, so we need to explicitly allow FD passing via exec().
ref: http://redmine.ruby-lang.org/issues/5041
|
|
The testcase for this was broken, too, so we didn't notice
this :<
Reported-by: ghazel@gmail.com on the Rainbows! mailing list,
http://mid.gmane.org/BANLkTi=oQXK5Casq9SuGD3edeUrDPvRm3A@mail.gmail.com
|
|
It's still O(n) since we don't maintain a reverse mapping of
spawned processes, but at least we avoid the extra overhead of
creating an array every time.
|
|
Some applications/libraries may launch background threads which
can lock up the process. So we can't disable heartbeat checking
just because the main thread is sleeping. This also has the
side effect of reducing master process wakeups when all workers
are idle.
|
|
We don't want the Worker#tick= assignment to trigger after we
accept a client, since we'd drop that request when we raise the
exception that breaks us out of the worker loop.
Also, we don't want to enter IO.select with an empty LISTENERS
array so we can fail with IOError or Errno::EBADF.
|
|
A leftover from the fchmod() days
|
|
Backtraces are now formatted properly (with timestamps) and
exceptions will be logged more consistently and similar to
Logger defaults:
"#{exc.message} (#{e.class})"
backtrace.each { |line| ... }
This may break some existing monitoring scripts, but errors
will be more standardized and easier to check moving forward.
|
|
"app error" is more correct, and consistent with Rainbows!
|
|
rescuing from SystemExit and exit()-ing again is ugly, but
changes made to lower stack depth positively affect _everyone_
so we'll tolerate some ugliness here.
We'll need to disable graceful exit for some tests, too...
|
|
This means we no longer waste an extra file descriptor per
worker process in the master. Now there's no need to set a
higher file descriptor limit for systems running >= 1024
workers.
|
|
There's absolutely no need to keep the OptionParser around in
worker processes.
|
|
We always know we have zero workers at startup, so we don't
need to check before hand. SIGHUP users may suffer a small
performance decrease as a result, but there's not much we
can do about it.
|
|
This should be easier to understand and reduces garbage on
stack, too.
|
|
kgio never does reverse lookup
|
|
Ruby IO.select never raises that, actually
|
|
By avoid Array#each
|
|
ivar references using @ are slightly faster than calling
attribute methods.
|
|
Oops, it messes logging up badly.
|