Date | Commit message (Collapse) |
|
Once again Ruby seems ready to introduce more incompatibilities
and force busywork upon maintainers[1]. In order to avoid
incompatibilities in the future, I used a Perl script[2] to
prepend `frozen_string_literal: false' to every Ruby file.
Somebody interested will have to go through every Ruby source
file and enable frozen_string_literal once they've thoroughly
verified it's safe to do so.
[1] https://bugs.ruby-lang.org/issues/20205
[2] https://yhbt.net/add-fsl.git/74d7689/s/?b=add-fsl.perl
|
|
We actually need Ruby 2.3+ for `accept_nonblock(exception: false)';
and (AFAIK) we can't easily use a subclass of `Socket' while using
Socket#accept_nonblock to inject WriteSplat support for
`IO#write(*multi_args)'
So just depend on Ruby 2.5+ since all my Ruby is already on the
already-ancient Ruby 2.7+ anyways.
|
|
kgio is an extra download and shared object which costs users
bandwidth, disk space, startup time and memory. Ruby 2.3+
provides `Socket#accept_nonblock(exception: false)' support
in addition to `exception: false' support in IO#*_nonblock
methods from Ruby 2.1.
We no longer distinguish between TCPServer and UNIXServer as
separate classes internally; instead favoring the `Socket' class
of Ruby for both. This allows us to use `Socket#accept_nonblock'
and get a populated `Addrinfo' object off accept4(2)/accept(2)
without resorting to a getpeername(2) syscall (kgio avoided
getpeername(2) in the same way).
The downside is there's more Ruby-level argument passing and
stack usage on our end with HttpRequest#read_headers (formerly
HttpRequest#read). I chose this tradeoff since advancements in
Ruby itself can theoretically mitigate the cost of argument
passing, while syscalls are a high fixed cost given modern CPU
vulnerability mitigations.
Note: no benchmarks have been run since I don't have a suitable
system.
|
|
It's fairly easy given unicorn was designed with synchronous I/O
in mind. The overhead of backtraces from EOFError on
readpartial should be rare given our requirement to only accept
requests from fast, reliable clients on LAN (e.g. nginx or
yet-another-horribly-named-server).
|
|
The damage unicorn has done to the entire Ruby, Rack and Rails
ecosystems with its ability to tolerate buggy code is
unforgivable. Update the documentation to further discourage
its use and clarify a few wordings noticed along the way.
|
|
When using systemd-style socket activation, consider the
inherited socket immortal and do not drop it on SIGHUP.
This means configs w/o any `listen' directives at all can
continue to work after SIGHUP.
I only noticed this while writing some tests in Perl 5 and
the test suite is two lines shorter to test this feature :>
|
|
Passing the `{ FD => IO }' mapping to #spawn or #exec already
ensures Ruby will clear FD_CLOEXEC on these FDs before execve(2).
|
|
Rack::Chunked will be gone in Rack 3.1, so provide a
non-middleware fallback which takes advantage of IO#write
supporting multiple arguments in Ruby 2.5+.
We still need to support Ruby 2.4, at least, since Rack 3.0
does. So a new (GC-unfriendly) Unicorn::WriteSplat module now
exists for Ruby <= 2.4 users.
|
|
Most changes are to the tests to avoid uppercase characters in header
keys, which are no longer allowed in rack 3 (to allow for O(1) access).
This also changes a few places where an array of headers was used to
switch to a hash, as a hash is requierd in Rack 3.
Newer versions of curl use a 000 http_code for invalid http codes,
so switch from "42 -eq" to "500 -ne" in the test, as Rack::Lint will
always raise a 500 error.
There is one test that fails on OpenBSD when opening a fifo. This is
unrelated to unicorn as far as I can see, so skip the remaining part
of the test in that case on OpenBSD.
Tests still pass on Rack 2, and presumably Rack 1 as well, though
I didn't test Rack 1.
Co-authored-by: Eric Wong <bofh@yhbt.net>
|
|
This has been irrelevant since Ruby 1.9.0+, actually.
|
|
While unlikely, it's possible for write(2) to return a truncated
value or ENOSPC error if the device is full when writing a tiny
PID file.
As optimism has no place in engineering, use IO#write instead of
IO#syswrite since the former will retry on truncation and raise
a exception on ENOSPC.
|
|
While the capabilities of epoll cannot be fully exploited given
our primitive design; avoiding thundering herd wakeups on larger
SMP machines while below 100% utilization is possible with
Linux 4.5+.
With this change, only one worker wakes up per-connect(2)
(instead of all of them via select(2)), avoiding the thundering
herd effect when the system is mostly idle.
Saturated instances should not notice the difference if they
rarely had multiple workers sleeping in select(2). This change
benefits non-saturated instances.
With 2 parallel clients and 8 workers on a nominally (:P)
8-core CPU (AMD FX-8320), the uconnect.perl test script
invocation showed a reduction from ~3.4s to ~2.5s when
reading an 11-byte response body:
echo worker_processes 8 >u.conf.rb
bs=11 ruby -I lib -I test/ruby-2.5.5/ext/unicorn_http/ bin/unicorn \
test/benchmark/dd.ru -E none -l /tmp/u.sock -c u.conf.rb
time perl -I lib -w test/benchmark/uconnect.perl \
-n 100000 -c 2 /tmp/u.sock
Times improve less as "-c" increases for uconnect.perl (system
noise and timings are inconsistent). The benefit of this change
should be more noticeable on systems with more workers (and
more cores).
I wanted to use EPOLLET (Edge-Triggered) to further reduce
syscalls, here, (similar to the old select()-avoidance bet) but
that would've either added too much complexity to deduplicate
wakeup sources, or run into the same starvation problem we
solved in April 2020[1].
Since the kernel already has the complexity and deduplication
built-in for Level-Triggered epoll support, we'll just let the
kernel deal with it.
Note: do NOT take this as an example of how epoll should be used
in a sophisticated server. unicorn is primitive by design and
cannot use threads nor handle multiple clients at once, thus it
it only uses epoll in this extremely limited manner.
Linux 4.5+ users will notice a regression of one extra epoll FD
per-worker and at least two epoll watches, so
/proc/sys/fs/epoll/max_user_watches may need to be changed along
with RLIMIT_NOFILE.
This change has also been tested on Linux 3.10.x (CentOS 7.x)
and FreeBSD 11.x to ensure compatibility with systems without
EPOLLEXCLUSIVE.
Various EPOLLEXCLUSIVE discussions over the years:
https://yhbt.net/lore/lkml/?q=s:EPOLLEXCLUSIVE+d:..20211001&x=t&o=-1
[1] https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
|
|
It doesn't seem to do anything since commit 221340c4ebc15666
(prevent single listener from monopolizing a worker, 2020-04-16).
|
|
It's actually been unnecessary since commit 6f6e4115b4bb03e5
(rework master-to-worker signaling to use a pipe, 2013-12-09)
|
|
Ruby 1.9.3 was released nearly a decade ago, so there's probably
few (if any) legacy users left, and they can continue using old
versions of unicorn. We'll be able to take advantage of some
Ruby 2.0+-only features down the road (and hopefully 2.3+).
Also, I no longer have a installation of Ruby 1.8 and getting it
working probably isn't worth the effort, so 4.x support is gone.
|
|
Since we allocate a new request object for each request,
the #clear call is now unnecessary
|
|
This removes the reuse of the parser between requests. Reusing these is
risky in the context of running any other threads within the unicorn
process, also for threads that run background tasks.
If any other thread accidentally grabs hold of the request it can modify
things for the next request in flight.
The downside here is that we allocate more for each request, but that is
worth the trade off here and the security risk we otherwise would carry
to leaking wrong and incorrect data.
|
|
This adds `rack.after_reply` functionality which allows rack middleware
to pass lambdas that will be executed after the client connection has
been closed.
This was driven by a need to perform actions in a request that shouldn't
block the request from completing but also don't make sense as
background jobs.
There is prior art of this being supported found in a few gems, as well
as this functionality existing in other rack based servers (e.g. Puma).
[ew: check if `env' is set in ensure statement]
Acked-by: Eric Wong <e@80x24.org>
|
|
If a user removes "early_hints" entirely from the config file, a
SIGHUP needs to restore the default value. This is consistent
with the behavior of all the other configuration variables.
Cc: Jean Boussier <jean.boussier@gmail.com>
|
|
While not part of the rack spec, this API is exposed
by both puma and falcon, and Rails use it when available.
The 103 Early Hints response code is specified in RFC 8297.
|
|
In setups with multiple listeners, it's possible for our greedy
select(2)-avoidance optimization to get pinned on a single, busy
listener and starve the other listener(s).
Prevent starvation by retrying the select(2)-avoidance
optimization if and only if all listeners were active. This
should have no effect on the majority of deployments with only a
single listener.
Thanks to Stan Hu for reporting and testing.
Reported-by: Stan Hu <stanhu@gmail.com>
Tested-by: Stan Hu <stanhu@gmail.com>
Link: https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
|
|
We need to favor "Transfer-Encoding: chunked" over
"Content-Length" in the request header if they both exist.
Furthermore, we now reject redundant chunking and cases where
"chunked" is not the final encoding.
We currently do not and have no plans to decode "gzip",
"deflate", or "compress" encoding as described by RFC 7230.
That's a job more appropriate for middleware, anyways.
cf. https://tools.ietf.org/html/rfc7230
https://www.rfc-editor.org/errata_search.php?rfc=7230
|
|
bogomips.org is due to expire, soon, and I'm not willing to pay
extortionist fees to Ethos Capital/PIR/ICANN to keep a .org. So
it's at yhbt.net, for now, but it will change again to
whatever's affordable... Identity is overrated.
Tor users can use .onions and kick ICANN to the curb:
torsocks w3m http://unicorn.ou63pmih66umazou.onion/
torsocks git clone http://ou63pmih66umazou.onion/unicorn.git/
torsocks w3m http://ou63pmih66umazou.onion/unicorn-public/
While we're at it, `s/news.gmane.org/news.gmane.io/g', too.
(but I suspect that'll need to be resynched since our mail
"List-Id:" header is changing).
|
|
Ruby mistakenly taints the file path, causing File.unlink
to fail: https://bugs.ruby-lang.org/issues/14485
Workaround the Ruby bug by keeping the path as a local
variable and passing that to File.unlink, instead of the
return value of File#path.
Link: https://bogomips.org/unicorn-public/CABg1sXrvGv9G6CDQxePDUqTe6N-5UpLXm7eG3YQO=dda-Cgg7A@mail.gmail.com/
|
|
Latency from redirects is painful, and HTTPS can protect privacy
in some cases.
|
|
This allows for the equivalent of the
-N/--no-default_middleware command line option to be
specified in the configuration file so it doesn't
need to be specified on the command line every time
unicorn is executed.
It explicitly excludes the use of -N/--no-default_middleware
as an embedded configuration option in the rackup file, by
ignoring the options after ARGV is parsed.
In order to allow the configuration method to work, have
the lambda that Unicorn.builder returns accept two arguments.
Technically, only one argument is needed for the HttpServer
instance, but I'm guessing if the lambda accepts a single
argument, we expect that to be a rack application instead
of a lambda that returns a rack application.
The command line option option to disable default middleware
will take precedence over the unicorn configuration file option
if both are present.
For backwards compatibility, if the lambda passed to
HttpServer accepts 0 arguments, then call it without
arguments.
[ew: fix precedence for arity checking in build_app!
configurator: ensure -N is respected when set in command-line]
|
|
Previously, Unicorn always used the process's primary group as the
the group of the log file. However, there are reasons to use a
separate group for the log files, such as when you have many
applications where each application uses it's own user and primary
group, but you want to be able to have a user read the log files
for all applications. Some operating systems have a fairly small
limit on the number of groups per user, and it may not be feasible
to have a user be in the primary group for all applications.
a primary group
|
|
Because I forget to load accf_http on new FreeBSD installs,
too :x
|
|
We have never had any need for pipes with the default 64K
capacity on Linux. Our pipes are only used for tiny writes
in signal handlers and to perform parent shutdown detection.
With the current /proc/sys/fs/pipe-user-pages-soft
default, only 1024 pipes can be created by an unprivileged
user before Linux clamps down the pipe size to 4K (a single page)
for newly-created pipes[1].
So avoid penalizing OTHER pipe users who could benefit from the
increased capacity and use only a single page for ourselves.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/pipe.c?h=v4.18#n642
|
|
Slowly reducing dependencies on kgio. 'io/wait' is required by
'socket' these days, so it's no extra relocations for .so
loading, either.
|
|
It's not unicorn 6, yet, but we dropped Ruby 1.8 support at
unicorn 5. Stable Ruby 1.9+ releases have always reseeded the
PRNG at fork.
|
|
Ruby trunk started warning about more mismatched indentations
starting around r62836.
|
|
Hijackers may capture and reuse `env' indefinitely, so we must
not use it in those cases for future requests. For non-hijack
requests, we continue to reuse the `env' object to reduce
memory recycling.
Reported-and-tested-by: Sam Saffron <sam.saffron@gmail.com>
|
|
These three cold call sites instruction sequence size by a few
hundred bytes combined since we no longer support Ruby 1.8.6.
The "?/" shorthand is esoteric and no longer avoids allocation
in Ruby 1.9+ (not that this is hot code).
|
|
The ruby constant Socket::TCP_INFO is only defined if TCP_INFO is defined
in C, so we can just check for the presence of that ruby constant instead
of rescuing SocketError from the call to getsockopt.
|
|
On OpenBSD, getsockopt(2) does not support TCP_INFO. With the current code,
this results in a 500 for all clients if check_client_connection is enabled
on OpenBSD.
This patch rescues SocketError on the first getsockopt call, and
if SocketError is raised, it doesn't check in the future. This
should be the same behavior as if TCP_INFO was supported but
inspect did not return a string in the expected format.
|
|
Followup-to: 650e01ab0b118803486b56f3ee59521d59042dae
("doc: add version annotations for new features")
|
|
We will inevitably have people running old unicorn versions
for many years to come; but they may be reading the latest
documentation online.
Annotate when the new features (will) appear to avoid misleading
users on old versions.
|
|
* origin/worker_exec:
Don't pass a block for fork when forking workers
Add worker_exec configuration option
|
|
* ccc-tcp-v3:
test_ccc: use a pipe to synchronize test
http_request: support proposed Raindrops::TCP states on non-Linux
|
|
This quiets down warnings when run with '-w'
|
|
rack 2.x exists nowadays still allows rewindable input as an
option, and we will still enable it by default to avoid breaking
any existing code.
Hide the internal documentation since we do not want people
depending on unicorn internals; so there's no reason to confuse
or overwhelm people with documentation about it. Arguably,
TeeInput and StreamInput should not be documented publically at
all, but I guess that ship has long sailed...
|
|
raindrops 0.18+ will have Raindrops::TCP state hash for portable
mapping of TCP states to their respective numeric values. This
was necessary because TCP state numbers (and even macro names)
differ between FreeBSD and Linux (and possibly other OSes).
Favor using the Raindrops::TCP state hash if available, but
fall back to the hard-coded values since older versions of
raindrops did not support TCP_INFO on non-Linux systems.
While we're in the area, favor "const_defined?" over "defined?"
to reduce the inline constant cache footprint for branches
which are only evaluated once.
Patches to implement Raindrops::TCP for FreeBSD are available at:
https://bogomips.org/raindrops-public/20170316031652.17433-1-e@80x24.org/T/
|
|
* origin/ccc-tcp-v3:
http_request: reduce insn size for check_client_connection
support "struct tcp_info" on non-Linux and Ruby 2.2+
revert signature change to HttpServer#process_client
new test for check_client_connection
check_client_connection: use tcp state on linux
|
|
Accept filters can only be set on listen sockets, and it also
fails with EINVAL if it's already set.
Untested, but I suppose changing the accept filter on a listening
socket is not supported, either; since that could affect in-flight
sockets.
|
|
Unlike constants and instance variables, class variable access
is not optimized in the mainline Ruby VM. Use a constant
instead, to take advantage of inline constant caching.
This further reduces runtime instruction size by avoiding a
branch by allocating the Raindrops::TCP_Info object up front.
This reduces the method size by roughly 300 bytes on 64-bit.
|
|
This reduces the stack depth, making GC more efficient.
|
|
The worker_exec configuration option makes all worker processes
exec after forking. This initializes the worker processes with
separate memory layouts, defeating address space discovery
attacks on operating systems supporting address space layout
randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and
Solaris.
Support for execing workers is very similar to support for reexecing
the master process. The main difference is the worker's to_i and
master pipes also need to be inherited after worker exec just as the
listening sockets need to be inherited after reexec.
Because execing working is similar to reexecing the master, this
extracts a couple of methods from reexec (listener_sockets and
close_sockets_on_exec), so they can be reused in worker_spawn.
|
|
Maybe oob_gc probably isn't heavily used anymore, maybe
some Ruby 2.2+ users will benefit from this constant
reduction.
Followup-to: fb2f10e1d7a72e67 ("reduce constants and optimize for Ruby 2.2")
|
|
Ruby 2.2+ can show "struct tcp_info" as a string via
Socket::Option#inspect, and we can attempt to parse it
out to extract the information we need.
Parsing this string is inefficient, but does not depend on the
ordering of the tcp_info struct.
|