about summary refs log tree commit homepage
path: root/lib/unicorn
DateCommit message (Collapse)
2024-03-31treewide: future-proof frozen_string_literal changes
Once again Ruby seems ready to introduce more incompatibilities and force busywork upon maintainers[1]. In order to avoid incompatibilities in the future, I used a Perl script[2] to prepend `frozen_string_literal: false' to every Ruby file. Somebody interested will have to go through every Ruby source file and enable frozen_string_literal once they've thoroughly verified it's safe to do so. [1] https://bugs.ruby-lang.org/issues/20205 [2] https://yhbt.net/add-fsl.git/74d7689/s/?b=add-fsl.perl
2024-01-15update dependency to Ruby 2.5+
We actually need Ruby 2.3+ for `accept_nonblock(exception: false)'; and (AFAIK) we can't easily use a subclass of `Socket' while using Socket#accept_nonblock to inject WriteSplat support for `IO#write(*multi_args)' So just depend on Ruby 2.5+ since all my Ruby is already on the already-ancient Ruby 2.7+ anyways.
2024-01-15kill off remaining kgio uses
kgio is an extra download and shared object which costs users bandwidth, disk space, startup time and memory. Ruby 2.3+ provides `Socket#accept_nonblock(exception: false)' support in addition to `exception: false' support in IO#*_nonblock methods from Ruby 2.1. We no longer distinguish between TCPServer and UNIXServer as separate classes internally; instead favoring the `Socket' class of Ruby for both. This allows us to use `Socket#accept_nonblock' and get a populated `Addrinfo' object off accept4(2)/accept(2) without resorting to a getpeername(2) syscall (kgio avoided getpeername(2) in the same way). The downside is there's more Ruby-level argument passing and stack usage on our end with HttpRequest#read_headers (formerly HttpRequest#read). I chose this tradeoff since advancements in Ruby itself can theoretically mitigate the cost of argument passing, while syscalls are a high fixed cost given modern CPU vulnerability mitigations. Note: no benchmarks have been run since I don't have a suitable system.
2024-01-15remove kgio from all read(2) and write(2) wrappers
It's fairly easy given unicorn was designed with synchronous I/O in mind. The overhead of backtraces from EOFError on readpartial should be rare given our requirement to only accept requests from fast, reliable clients on LAN (e.g. nginx or yet-another-horribly-named-server).
2023-09-11doc: various updates ahead of the release
The damage unicorn has done to the entire Ruby, Rack and Rails ecosystems with its ability to tolerate buggy code is unforgivable. Update the documentation to further discourage its use and clarify a few wordings noticed along the way.
2023-06-05LISTEN_FDS-inherited sockets are immortal across SIGHUP
When using systemd-style socket activation, consider the inherited socket immortal and do not drop it on SIGHUP. This means configs w/o any `listen' directives at all can continue to work after SIGHUP. I only noticed this while writing some tests in Perl 5 and the test suite is two lines shorter to test this feature :>
2023-06-05drop redundant IO#close_on_exec=false calls
Passing the `{ FD => IO }' mapping to #spawn or #exec already ensures Ruby will clear FD_CLOEXEC on these FDs before execve(2).
2023-06-05chunk unterminated HTTP/1.1 responses
Rack::Chunked will be gone in Rack 3.1, so provide a non-middleware fallback which takes advantage of IO#write supporting multiple arguments in Ruby 2.5+. We still need to support Ruby 2.4, at least, since Rack 3.0 does. So a new (GC-unfriendly) Unicorn::WriteSplat module now exists for Ruby <= 2.4 users.
2023-06-05Support Rack 3 and fix tests on Rack 3
Most changes are to the tests to avoid uppercase characters in header keys, which are no longer allowed in rack 3 (to allow for O(1) access). This also changes a few places where an array of headers was used to switch to a hash, as a hash is requierd in Rack 3. Newer versions of curl use a 000 http_code for invalid http codes, so switch from "42 -eq" to "500 -ne" in the test, as Rack::Lint will always raise a 500 error. There is one test that fails on OpenBSD when opening a fifo. This is unrelated to unicorn as far as I can see, so skip the remaining part of the test in that case on OpenBSD. Tests still pass on Rack 2, and presumably Rack 1 as well, though I didn't test Rack 1. Co-authored-by: Eric Wong <bofh@yhbt.net>
2023-06-05http_server: remove close_sockets_on_exec
This has been irrelevant since Ruby 1.9.0+, actually.
2022-10-04http_server: detect disk-full when writing PID file
While unlikely, it's possible for write(2) to return a truncated value or ENOSPC error if the device is full when writing a tiny PID file. As optimism has no place in engineering, use IO#write instead of IO#syswrite since the former will retry on truncation and raise a exception on ENOSPC.
2021-10-04use EPOLLEXCLUSIVE on Linux 4.5+
While the capabilities of epoll cannot be fully exploited given our primitive design; avoiding thundering herd wakeups on larger SMP machines while below 100% utilization is possible with Linux 4.5+. With this change, only one worker wakes up per-connect(2) (instead of all of them via select(2)), avoiding the thundering herd effect when the system is mostly idle. Saturated instances should not notice the difference if they rarely had multiple workers sleeping in select(2). This change benefits non-saturated instances. With 2 parallel clients and 8 workers on a nominally (:P) 8-core CPU (AMD FX-8320), the uconnect.perl test script invocation showed a reduction from ~3.4s to ~2.5s when reading an 11-byte response body: echo worker_processes 8 >u.conf.rb bs=11 ruby -I lib -I test/ruby-2.5.5/ext/unicorn_http/ bin/unicorn \ test/benchmark/dd.ru -E none -l /tmp/u.sock -c u.conf.rb time perl -I lib -w test/benchmark/uconnect.perl \ -n 100000 -c 2 /tmp/u.sock Times improve less as "-c" increases for uconnect.perl (system noise and timings are inconsistent). The benefit of this change should be more noticeable on systems with more workers (and more cores). I wanted to use EPOLLET (Edge-Triggered) to further reduce syscalls, here, (similar to the old select()-avoidance bet) but that would've either added too much complexity to deduplicate wakeup sources, or run into the same starvation problem we solved in April 2020[1]. Since the kernel already has the complexity and deduplication built-in for Level-Triggered epoll support, we'll just let the kernel deal with it. Note: do NOT take this as an example of how epoll should be used in a sophisticated server. unicorn is primitive by design and cannot use threads nor handle multiple clients at once, thus it it only uses epoll in this extremely limited manner. Linux 4.5+ users will notice a regression of one extra epoll FD per-worker and at least two epoll watches, so /proc/sys/fs/epoll/max_user_watches may need to be changed along with RLIMIT_NOFILE. This change has also been tested on Linux 3.10.x (CentOS 7.x) and FreeBSD 11.x to ensure compatibility with systems without EPOLLEXCLUSIVE. Various EPOLLEXCLUSIVE discussions over the years: https://yhbt.net/lore/lkml/?q=s:EPOLLEXCLUSIVE+d:..20211001&x=t&o=-1 [1] https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
2021-10-04worker_loop: get rid of select() avoidance hack
It doesn't seem to do anything since commit 221340c4ebc15666 (prevent single listener from monopolizing a worker, 2020-04-16).
2021-10-04http_server: get rid of Process.ppid check
It's actually been unnecessary since commit 6f6e4115b4bb03e5 (rework master-to-worker signaling to use a pipe, 2013-12-09)
2021-09-26drop Ruby 1.9.3 support, require 2.0+ for now
Ruby 1.9.3 was released nearly a decade ago, so there's probably few (if any) legacy users left, and they can continue using old versions of unicorn. We'll be able to take advantage of some Ruby 2.0+-only features down the road (and hopefully 2.3+). Also, I no longer have a installation of Ruby 1.8 and getting it working probably isn't worth the effort, so 4.x support is gone.
2021-03-13http_request: drop unnecessary #clear call
Since we allocate a new request object for each request, the #clear call is now unnecessary
2021-03-13Allocate a new request for each client
This removes the reuse of the parser between requests. Reusing these is risky in the context of running any other threads within the unicorn process, also for threads that run background tasks. If any other thread accidentally grabs hold of the request it can modify things for the next request in flight. The downside here is that we allocate more for each request, but that is worth the trade off here and the security risk we otherwise would carry to leaking wrong and incorrect data.
2020-12-09Add rack.after_reply functionality
This adds `rack.after_reply` functionality which allows rack middleware to pass lambdas that will be executed after the client connection has been closed. This was driven by a need to perform actions in a request that shouldn't block the request from completing but also don't make sense as background jobs. There is prior art of this being supported found in a few gems, as well as this functionality existing in other rack based servers (e.g. Puma). [ew: check if `env' is set in ensure statement] Acked-by: Eric Wong <e@80x24.org>
2020-07-24configurator: SIGHUP resets early_hints if unset
If a user removes "early_hints" entirely from the config file, a SIGHUP needs to restore the default value. This is consistent with the behavior of all the other configuration variables. Cc: Jean Boussier <jean.boussier@gmail.com>
2020-07-16Add early hints support
While not part of the rack spec, this API is exposed by both puma and falcon, and Rails use it when available. The 103 Early Hints response code is specified in RFC 8297.
2020-04-16prevent single listener from monopolizing a worker
In setups with multiple listeners, it's possible for our greedy select(2)-avoidance optimization to get pinned on a single, busy listener and starve the other listener(s). Prevent starvation by retrying the select(2)-avoidance optimization if and only if all listeners were active. This should have no effect on the majority of deployments with only a single listener. Thanks to Stan Hu for reporting and testing. Reported-by: Stan Hu <stanhu@gmail.com> Tested-by: Stan Hu <stanhu@gmail.com> Link: https://yhbt.net/unicorn-public/CAMBWrQ=Yh42MPtzJCEO7XryVknDNetRMuA87irWfqVuLdJmiBQ@mail.gmail.com/
2020-03-19http: improve RFC 7230 conformance
We need to favor "Transfer-Encoding: chunked" over "Content-Length" in the request header if they both exist. Furthermore, we now reject redundant chunking and cases where "chunked" is not the final encoding. We currently do not and have no plans to decode "gzip", "deflate", or "compress" encoding as described by RFC 7230. That's a job more appropriate for middleware, anyways. cf. https://tools.ietf.org/html/rfc7230 https://www.rfc-editor.org/errata_search.php?rfc=7230
2020-01-20doc: s/bogomips.org/yhbt.net/g
bogomips.org is due to expire, soon, and I'm not willing to pay extortionist fees to Ethos Capital/PIR/ICANN to keep a .org. So it's at yhbt.net, for now, but it will change again to whatever's affordable... Identity is overrated. Tor users can use .onions and kick ICANN to the curb: torsocks w3m http://unicorn.ou63pmih66umazou.onion/ torsocks git clone http://ou63pmih66umazou.onion/unicorn.git/ torsocks w3m http://ou63pmih66umazou.onion/unicorn-public/ While we're at it, `s/news.gmane.org/news.gmane.io/g', too. (but I suspect that'll need to be resynched since our mail "List-Id:" header is changing).
2019-12-11tmpio: workaround File#path being tainted on unlink
Ruby mistakenly taints the file path, causing File.unlink to fail: https://bugs.ruby-lang.org/issues/14485 Workaround the Ruby bug by keeping the path as a local variable and passing that to File.unlink, instead of the return value of File#path. Link: https://bogomips.org/unicorn-public/CABg1sXrvGv9G6CDQxePDUqTe6N-5UpLXm7eG3YQO=dda-Cgg7A@mail.gmail.com/
2018-10-18doc: update more URLs to use HTTPS and avoid redirects
Latency from redirects is painful, and HTTPS can protect privacy in some cases.
2018-09-21Support default_middleware configuration option
This allows for the equivalent of the -N/--no-default_middleware command line option to be specified in the configuration file so it doesn't need to be specified on the command line every time unicorn is executed. It explicitly excludes the use of -N/--no-default_middleware as an embedded configuration option in the rackup file, by ignoring the options after ARGV is parsed. In order to allow the configuration method to work, have the lambda that Unicorn.builder returns accept two arguments. Technically, only one argument is needed for the HttpServer instance, but I'm guessing if the lambda accepts a single argument, we expect that to be a rack application instead of a lambda that returns a rack application. The command line option option to disable default middleware will take precedence over the unicorn configuration file option if both are present. For backwards compatibility, if the lambda passed to HttpServer accepts 0 arguments, then call it without arguments. [ew: fix precedence for arity checking in build_app! configurator: ensure -N is respected when set in command-line]
2018-09-13Make Worker#user support different process primary group and log file group
Previously, Unicorn always used the process's primary group as the the group of the log file. However, there are reasons to use a separate group for the log files, such as when you have many applications where each application uses it's own user and primary group, but you want to be able to have a user read the log files for all applications. Some operating systems have a fairly small limit on the number of groups per user, and it may not be feasible to have a user be in the primary group for all applications. a primary group
2018-08-20socket_helper: add hint for FreeBSD users for accf_http(9)
Because I forget to load accf_http on new FreeBSD installs, too :x
2018-08-20shrink pipes under Linux
We have never had any need for pipes with the default 64K capacity on Linux. Our pipes are only used for tiny writes in signal handlers and to perform parent shutdown detection. With the current /proc/sys/fs/pipe-user-pages-soft default, only 1024 pipes can be created by an unprivileged user before Linux clamps down the pipe size to 4K (a single page) for newly-created pipes[1]. So avoid penalizing OTHER pipe users who could benefit from the increased capacity and use only a single page for ourselves. [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/pipe.c?h=v4.18#n642
2018-07-23use IO#wait instead of kgio_wait_readable
Slowly reducing dependencies on kgio. 'io/wait' is required by 'socket' these days, so it's no extra relocations for .so loading, either.
2018-07-23remove random seed reset atfork
It's not unicorn 6, yet, but we dropped Ruby 1.8 support at unicorn 5. Stable Ruby 1.9+ releases have always reseeded the PRNG at fork.
2018-05-01quiet some mismatched indentation warnings
Ruby trunk started warning about more mismatched indentations starting around r62836.
2017-12-16avoid reusing env on hijack
Hijackers may capture and reuse `env' indefinitely, so we must not use it in those cases for future requests. For non-hijack requests, we continue to reuse the `env' object to reduce memory recycling. Reported-and-tested-by: Sam Saffron <sam.saffron@gmail.com>
2017-04-08reduce method calls with String#start_with?
These three cold call sites instruction sequence size by a few hundred bytes combined since we no longer support Ruby 1.8.6. The "?/" shorthand is esoteric and no longer avoids allocation in Ruby 1.9+ (not that this is hot code).
2017-03-26Check for Socket::TCP_INFO constant before trying to get TCP_INFO
The ruby constant Socket::TCP_INFO is only defined if TCP_INFO is defined in C, so we can just check for the presence of that ruby constant instead of rescuing SocketError from the call to getsockopt.
2017-03-24Check for SocketError on first ccc attempt
On OpenBSD, getsockopt(2) does not support TCP_INFO. With the current code, this results in a 500 for all clients if check_client_connection is enabled on OpenBSD. This patch rescues SocketError on the first getsockopt call, and if SocketError is raised, it doesn't check in the future. This should be the same behavior as if TCP_INFO was supported but inspect did not return a string in the expected format.
2017-03-24doc: note after_worker_exit is also 5.3.0+
Followup-to: 650e01ab0b118803486b56f3ee59521d59042dae ("doc: add version annotations for new features")
2017-03-23doc: add version annotations for new features
We will inevitably have people running old unicorn versions for many years to come; but they may be reading the latest documentation online. Annotate when the new features (will) appear to avoid misleading users on old versions.
2017-03-23Merge remote-tracking branch 'origin/worker_exec'
* origin/worker_exec: Don't pass a block for fork when forking workers Add worker_exec configuration option
2017-03-23Merge branch 'ccc-tcp-v3'
* ccc-tcp-v3: test_ccc: use a pipe to synchronize test http_request: support proposed Raindrops::TCP states on non-Linux
2017-03-23http_server: initialize @pid ivar
This quiets down warnings when run with '-w'
2017-03-23input: update documentation and hide internals.
rack 2.x exists nowadays still allows rewindable input as an option, and we will still enable it by default to avoid breaking any existing code. Hide the internal documentation since we do not want people depending on unicorn internals; so there's no reason to confuse or overwhelm people with documentation about it. Arguably, TeeInput and StreamInput should not be documented publically at all, but I guess that ship has long sailed...
2017-03-21http_request: support proposed Raindrops::TCP states on non-Linux
raindrops 0.18+ will have Raindrops::TCP state hash for portable mapping of TCP states to their respective numeric values. This was necessary because TCP state numbers (and even macro names) differ between FreeBSD and Linux (and possibly other OSes). Favor using the Raindrops::TCP state hash if available, but fall back to the hard-coded values since older versions of raindrops did not support TCP_INFO on non-Linux systems. While we're in the area, favor "const_defined?" over "defined?" to reduce the inline constant cache footprint for branches which are only evaluated once. Patches to implement Raindrops::TCP for FreeBSD are available at: https://bogomips.org/raindrops-public/20170316031652.17433-1-e@80x24.org/T/
2017-03-15Merge remote-tracking branch 'origin/ccc-tcp-v3'
* origin/ccc-tcp-v3: http_request: reduce insn size for check_client_connection support "struct tcp_info" on non-Linux and Ruby 2.2+ revert signature change to HttpServer#process_client new test for check_client_connection check_client_connection: use tcp state on linux
2017-03-14freebsd: avoid EINVAL when setting accept filter
Accept filters can only be set on listen sockets, and it also fails with EINVAL if it's already set. Untested, but I suppose changing the accept filter on a listening socket is not supported, either; since that could affect in-flight sockets.
2017-03-14http_request: reduce insn size for check_client_connection
Unlike constants and instance variables, class variable access is not optimized in the mainline Ruby VM. Use a constant instead, to take advantage of inline constant caching. This further reduces runtime instruction size by avoiding a branch by allocating the Raindrops::TCP_Info object up front. This reduces the method size by roughly 300 bytes on 64-bit.
2017-03-13Don't pass a block for fork when forking workers worker_exec
This reduces the stack depth, making GC more efficient.
2017-03-10Add worker_exec configuration option
The worker_exec configuration option makes all worker processes exec after forking. This initializes the worker processes with separate memory layouts, defeating address space discovery attacks on operating systems supporting address space layout randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and Solaris. Support for execing workers is very similar to support for reexecing the master process. The main difference is the worker's to_i and master pipes also need to be inherited after worker exec just as the listening sockets need to be inherited after reexec. Because execing working is similar to reexecing the master, this extracts a couple of methods from reexec (listener_sockets and close_sockets_on_exec), so they can be reused in worker_spawn.
2017-03-08oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
Maybe oob_gc probably isn't heavily used anymore, maybe some Ruby 2.2+ users will benefit from this constant reduction. Followup-to: fb2f10e1d7a72e67 ("reduce constants and optimize for Ruby 2.2")
2017-03-08support "struct tcp_info" on non-Linux and Ruby 2.2+
Ruby 2.2+ can show "struct tcp_info" as a string via Socket::Option#inspect, and we can attempt to parse it out to extract the information we need. Parsing this string is inefficient, but does not depend on the ordering of the tcp_info struct.