about summary refs log tree commit homepage
DateCommit message (Collapse)
2017-10-03unicorn 5.3.1 v5.3.1 5.3-stable
This release fixes an occasional GC problem introduced in v5.3.0 to reduce global variable overhead (commit 979ebcf91705709b) Thanks to Xuanzhong Wei for the patch which lead to this release: https://bogomips.org/unicorn-public/20171003182054.76392-1-azrlew@gmail.com/T/#u https://bogomips.org/unicorn-public/20171003145718.30404-1-azrlew@gmail.com/T/#u Xuanzhong Wei (1): fix GC issue on rb_global_variable array
2017-10-03fix GC issue on rb_global_variable array
We need to add the array to ruby's global_list right after created it; otherwise it probably gets GCed. (cherry picked from commit e85de00a3065461f6fac466276d915b809f40c7c)
2017-04-01unicorn 5.3.0 v5.3.0
A couple of portability fixes from Dylan Thacker-Smith and Jeremy Evans since 5.3.0.pre1 over a week ago, but this looks ready for a stable release, today. When I started this over 8 years ago, I wondered if this would just end up being an April Fools' joke. Guess not. I guess I somehow tricked people into using a terribly marketed web server that cannot talk directly to untrusted clients :x Anyways, unicorn won't be able to handle slow clients 8 years from now, either, or 80 years from now. And I vow never to learn to use new-fangled things like epoll, kqueue, or threads :P Anyways, this is a largish release with several new features, and no backwards incompatibilities. Simon Eskildsen contributed heavily using TCP_INFO under Linux to implement the (now 5 year old) check_client_connection feature: https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170401&x=t This also led to FreeBSD and OpenBSD portability improvements in one of our dependencies, raindrops: https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u Jeremy Evans contributed several new features. First he implemented after_worker_exit to aid debugging: https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170401&x=t#t And then security-related features to isolate workers. Workers may now chroot to drop access to the master filesystem, and the new after_worker_ready configuration hook now exists to aid with chroot support in workers: https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170401&x=t#t https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170401&x=t#t Additionally, workers may run in a completely different VM space (nullifying preload_app and any CoW savings) with the new worker_exec option: https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170401&x=t#t There are also several improvements to FreeBSD and OpenBSD support with the addition of these features. shortlog of changes since v5.2.0 (2016-10-31): Dylan Thacker-Smith (1): Check for Socket::TCP_INFO constant before trying to get TCP_INFO Eric Wong (30): drop rb_str_set_len compatibility replacement TUNING: document THP caveat for Linux users tee_input: simplify condition for IO#write remove response_start_sent http_request: freeze constant strings passed IO#write Revert "remove response_start_sent" t/t0012-reload-empty-config.sh: access ivars directly if needed t0011-active-unix-socket.sh: fix race condition in test new test for check_client_connection revert signature change to HttpServer#process_client support "struct tcp_info" on non-Linux and Ruby 2.2+ unicorn_http: reduce rb_global_variable calls oob_gc: rely on opt_aref_with optimization on Ruby 2.2+ http_request: reduce insn size for check_client_connection freebsd: avoid EINVAL when setting accept filter test-lib: expr(1) portability fix tests: keep disabled tests defined test_exec: SO_KEEPALIVE value only needs to be true doc: fix links to raindrops project http_request: support proposed Raindrops::TCP states on non-Linux ISSUES: expand on mail archive info + subscription disclaimer test_ccc: use a pipe to synchronize test doc: remove private email support address input: update documentation and hide internals. http_server: initialize @pid ivar gemspec: remove olddoc from build dependency doc: add version annotations for new features unicorn 5.3.0.pre1 doc: note after_worker_exit is also 5.3.0+ test_exec: SO_KEEPALIVE value only needs to be true (take #2) Jeremy Evans (7): Add after_worker_exit configuration option Fix code example in after_worker_exit documentation Add support for chroot to Worker#user Add after_worker_ready configuration option Add worker_exec configuration option Don't pass a block for fork when forking workers Check for SocketError on first ccc attempt Simon Eskildsen (1): check_client_connection: use tcp state on linux
2017-03-27test_exec: SO_KEEPALIVE value only needs to be true (take #2)
We need to ensure the portability of the sd_listen_fds emulation test, too, which didn't get tested on my FreeBSD 10.3 install due to it being on Ruby 2.2 Followup-to: 4ce6b00f75f1 ("test_exec: SO_KEEPALIVE value only needs to be true")
2017-03-26Check for Socket::TCP_INFO constant before trying to get TCP_INFO
The ruby constant Socket::TCP_INFO is only defined if TCP_INFO is defined in C, so we can just check for the presence of that ruby constant instead of rescuing SocketError from the call to getsockopt.
2017-03-24Check for SocketError on first ccc attempt
On OpenBSD, getsockopt(2) does not support TCP_INFO. With the current code, this results in a 500 for all clients if check_client_connection is enabled on OpenBSD. This patch rescues SocketError on the first getsockopt call, and if SocketError is raised, it doesn't check in the future. This should be the same behavior as if TCP_INFO was supported but inspect did not return a string in the expected format.
2017-03-24doc: note after_worker_exit is also 5.3.0+
Followup-to: 650e01ab0b118803486b56f3ee59521d59042dae ("doc: add version annotations for new features")
2017-03-24unicorn 5.3.0.pre1 v5.3.0.pre1
A largish release with several new features. Simon Eskildsen contributed heavily using TCP_INFO under Linux to implement the (now 5 year old) check_client_connection feature: https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-check_client_connection https://bogomips.org/unicorn-public/?q=s:check_client_connection&d:..20170324&x=t This also led to FreeBSD and OpenBSD portability improvements in one of our dependencies, raindrops: https://bogomips.org/raindrops-public/20170323024829.GA5190@dcvr/T/#u Jeremy Evans contributed several new features. First he implemented after_worker_exit to aid debugging: https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_exit https://bogomips.org/unicorn-public/?q=s:after_worker_exit&d:..20170324&x=t#t And then security-related features to isolate workers. Workers may now chroot to drop access to the master filesystem, and the new after_worker_ready configuration hook now exists to aid with chroot support in workers: https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-after_worker_ready https://bogomips.org/unicorn/Unicorn/Worker.html#method-i-user https://bogomips.org/unicorn-public/?q=s:after_worker_ready&d:..20170324&x=t#t https://bogomips.org/unicorn-public/?q=s:chroot&d:..20170324&x=t#t Additionally, workers may run in a completely different VM space (nullifying preload_app and any CoW savings) with the new worker_exec option: https://bogomips.org/unicorn/Unicorn/Configurator.html#method-i-worker_exec https://bogomips.org/unicorn-public/?q=s:worker_exec&d:..20170324&x=t#t There are also several improvements to FreeBSD and OpenBSD support with the addition of these features. 34 changes since 5.2.0 (2016-10-31): Eric Wong (27): drop rb_str_set_len compatibility replacement TUNING: document THP caveat for Linux users tee_input: simplify condition for IO#write remove response_start_sent http_request: freeze constant strings passed IO#write Revert "remove response_start_sent" t/t0012-reload-empty-config.sh: access ivars directly if needed t0011-active-unix-socket.sh: fix race condition in test new test for check_client_connection revert signature change to HttpServer#process_client support "struct tcp_info" on non-Linux and Ruby 2.2+ unicorn_http: reduce rb_global_variable calls oob_gc: rely on opt_aref_with optimization on Ruby 2.2+ http_request: reduce insn size for check_client_connection freebsd: avoid EINVAL when setting accept filter test-lib: expr(1) portability fix tests: keep disabled tests defined test_exec: SO_KEEPALIVE value only needs to be true doc: fix links to raindrops project http_request: support proposed Raindrops::TCP states on non-Linux ISSUES: expand on mail archive info + subscription disclaimer test_ccc: use a pipe to synchronize test doc: remove private email support address input: update documentation and hide internals. http_server: initialize @pid ivar gemspec: remove olddoc from build dependency doc: add version annotations for new features Jeremy Evans (6): Add after_worker_exit configuration option Fix code example in after_worker_exit documentation Add support for chroot to Worker#user Add after_worker_ready configuration option Add worker_exec configuration option Don't pass a block for fork when forking workers Simon Eskildsen (1): check_client_connection: use tcp state on linux
2017-03-23doc: add version annotations for new features
We will inevitably have people running old unicorn versions for many years to come; but they may be reading the latest documentation online. Annotate when the new features (will) appear to avoid misleading users on old versions.
2017-03-23gemspec: remove olddoc from build dependency
It's a little less DRY, and there'll be no NEWS file generated, but it's one less thing to install, so perhaps that's worth it. The website at https://bogomips.org/unicorn/ will continue to use olddoc, of course,
2017-03-23Merge remote-tracking branch 'origin/worker_exec'
* origin/worker_exec: Don't pass a block for fork when forking workers Add worker_exec configuration option
2017-03-23Merge branch 'ccc-tcp-v3'
* ccc-tcp-v3: test_ccc: use a pipe to synchronize test http_request: support proposed Raindrops::TCP states on non-Linux
2017-03-23http_server: initialize @pid ivar
This quiets down warnings when run with '-w'
2017-03-23input: update documentation and hide internals.
rack 2.x exists nowadays still allows rewindable input as an option, and we will still enable it by default to avoid breaking any existing code. Hide the internal documentation since we do not want people depending on unicorn internals; so there's no reason to confuse or overwhelm people with documentation about it. Arguably, TeeInput and StreamInput should not be documented publically at all, but I guess that ship has long sailed...
2017-03-23doc: remove private email support address
Email was never private, and won't further burden myself or any future maintainers with trying to maintain someone elses' privacy. Offering private support is also unfair to readers on public lists who may get a watered down or improperly translated summary (if at all). Instead, encourage the use of anonymity tools and scrubbing of sensitive information when the sender deems necessary.
2017-03-22test_ccc: use a pipe to synchronize test ccc-tcp-v3
Sleeping 1 second to test 100 requests is too long for some systems; and not long enough for others. We need to also finish reading the sleeper response to ensure the server actually got the second request in, before sending SIGQUIT to terminate it; as it's possible for the test client to connect and abort 100 clients before the server even increments the request counter for the 2nd request.
2017-03-21ISSUES: expand on mail archive info + subscription disclaimer
Tis better to pull than push, or something like that.
2017-03-21http_request: support proposed Raindrops::TCP states on non-Linux
raindrops 0.18+ will have Raindrops::TCP state hash for portable mapping of TCP states to their respective numeric values. This was necessary because TCP state numbers (and even macro names) differ between FreeBSD and Linux (and possibly other OSes). Favor using the Raindrops::TCP state hash if available, but fall back to the hard-coded values since older versions of raindrops did not support TCP_INFO on non-Linux systems. While we're in the area, favor "const_defined?" over "defined?" to reduce the inline constant cache footprint for branches which are only evaluated once. Patches to implement Raindrops::TCP for FreeBSD are available at: https://bogomips.org/raindrops-public/20170316031652.17433-1-e@80x24.org/T/
2017-03-15Merge remote-tracking branch 'origin/ccc-tcp-v3'
* origin/ccc-tcp-v3: http_request: reduce insn size for check_client_connection support "struct tcp_info" on non-Linux and Ruby 2.2+ revert signature change to HttpServer#process_client new test for check_client_connection check_client_connection: use tcp state on linux
2017-03-14doc: fix links to raindrops project
bogomips.org is dropping prefixes to reduce subjectAltName bloat in TLS certificates.
2017-03-14test_exec: SO_KEEPALIVE value only needs to be true
On FreeBSD 10.3, the value of SO_KEEPALIVE returned by getsockopt is 8, even when set to '1' via setsockopt. Relax the test to only ensure the boolean value is interpreted as "true". Verified independently of Ruby using the following: --------8<--------- #include <sys/types.h> #include <sys/socket.h> #include <stdio.h> static int err(const char *msg) { perror(msg); return 1; } int main(void) { int sv[2]; int set = 1; int got; socklen_t len = (socklen_t)sizeof(int); int rc; rc = socketpair(PF_LOCAL, SOCK_STREAM, 0, sv); if (rc) return err("socketpair failed"); rc = setsockopt(sv[0], SOL_SOCKET, SO_KEEPALIVE, &set, len); if (rc) return err("setsockopt failed"); rc = getsockopt(sv[0], SOL_SOCKET, SO_KEEPALIVE, &got, &len); if (rc) return err("getsockopt failed"); printf("got: %d\n", got); return 0; }
2017-03-14tests: keep disabled tests defined
Some versions of test-unit will fail if an unspecified test is attempted via "-n", so we need to define an empty test. We cannot use "skip", either, as that seems exclusive to minitest; and we won't use minitest since it has more incompatible changes than test-unit over the last 8 years. The memory leak test is gone since we're more versed in the Ruby C API nowadays, modern GCs + mallocs may be less predictable about releasing memory back to the OS.
2017-03-14test-lib: expr(1) portability fix
GNU expr supports '+' to match one or more occurrences, but it seems the expr(1) on my FreeBSD installation does not.
2017-03-14freebsd: avoid EINVAL when setting accept filter
Accept filters can only be set on listen sockets, and it also fails with EINVAL if it's already set. Untested, but I suppose changing the accept filter on a listening socket is not supported, either; since that could affect in-flight sockets.
2017-03-14http_request: reduce insn size for check_client_connection
Unlike constants and instance variables, class variable access is not optimized in the mainline Ruby VM. Use a constant instead, to take advantage of inline constant caching. This further reduces runtime instruction size by avoiding a branch by allocating the Raindrops::TCP_Info object up front. This reduces the method size by roughly 300 bytes on 64-bit.
2017-03-13Don't pass a block for fork when forking workers worker_exec
This reduces the stack depth, making GC more efficient.
2017-03-10Add worker_exec configuration option
The worker_exec configuration option makes all worker processes exec after forking. This initializes the worker processes with separate memory layouts, defeating address space discovery attacks on operating systems supporting address space layout randomization, such as Linux, MacOS X, NetBSD, OpenBSD, and Solaris. Support for execing workers is very similar to support for reexecing the master process. The main difference is the worker's to_i and master pipes also need to be inherited after worker exec just as the listening sockets need to be inherited after reexec. Because execing working is similar to reexecing the master, this extracts a couple of methods from reexec (listener_sockets and close_sockets_on_exec), so they can be reused in worker_spawn.
2017-03-08oob_gc: rely on opt_aref_with optimization on Ruby 2.2+
Maybe oob_gc probably isn't heavily used anymore, maybe some Ruby 2.2+ users will benefit from this constant reduction. Followup-to: fb2f10e1d7a72e67 ("reduce constants and optimize for Ruby 2.2")
2017-03-08unicorn_http: reduce rb_global_variable calls
rb_global_variable registers the address of the variable which refers to the object, instead of the object itself. This adds extra overhead to each global variable for our case, where the variable is frozen and never changed. Given there are currently 59 elements in this array, this saves 58 singly-linked list entries and associated malloc calls and associated overhead in the current mainline Ruby 2.x implementation. On 64-bit GNU libc malloc, this is already 16 * 58 = 928 bytes; more than the extra object slot and array slack space used by the new mark array. Mainline Ruby 1.9+ currently has a rb_gc_register_mark_object public function which would suite our needs, too, but it is currently undocumented, and may not be available in the future.
2017-03-08Merge remote-tracking branch 'origin/chroot'
* origin/chroot: Add after_worker_ready configuration option Add support for chroot to Worker#user
2017-03-08support "struct tcp_info" on non-Linux and Ruby 2.2+
Ruby 2.2+ can show "struct tcp_info" as a string via Socket::Option#inspect, and we can attempt to parse it out to extract the information we need. Parsing this string is inefficient, but does not depend on the ordering of the tcp_info struct.
2017-03-08revert signature change to HttpServer#process_client
We can force kgio_tryaccept to return an internal class for TCP objects by subclassing Kgio::TCPServer. This avoids breakage in any unfortunate projects which depend on our undocumented internal APIs, such as gctools <https://github.com/tmm1/gctools>
2017-03-08new test for check_client_connection
This was a bit tricky to test, but it's probably more reliable now that we're relying on TCP_INFO. Based on test by Simon Eskildsen <simon.eskildsen@shopify.com>: https://bogomips.org/unicorn-public/CAO3HKM49+aLD=KLij3zhJqkWnR7bCWVan0mOvxD85xfrW8RXOw@mail.gmail.com/
2017-03-08check_client_connection: use tcp state on linux ccc-tcp-v2
* Use a frozen empty array and a class variable for TCP_Info to avoid garbage. As far as I can tell, this shouldn't result in any garbage on any requests (other than on the first request). * Pass listener socket to #read to only check the client connection on a TCP server. * Short circuit CLOSE_WAIT after ESTABLISHED since in my testing it's the most common state after ESTABLISHED, it makes the numbers un-ordered, though. But comment should make it OK. * Definition of of `check_client_connection` based on whether Raindrops::TCP_Info is defined, instead of the class variable approach. * Changed the unit tests to pass a `nil` listener. Tested on our staging environment, and still works like a dream. I should note that I got the idea between this patch into Puma as well! https://github.com/puma/puma/pull/1227 [ew: squashed in temporary change for oob_gc.rb, but we'll come up with a different change to avoid breaking gctools <https://github.com/tmm1/gctools>] Acked-by: Eric Wong <e@80x24.org>
2017-03-01t0011-active-unix-socket.sh: fix race condition in test
Killing the master process may lead to the worker dying on its own (as designed); before kill(1) has had an opportunity to send the second kill(2) syscall on the worker process. Killing the worker before the master might also lead to a needless respawn, so merely kill the master and let the worker follow it in death. This race condition occasionally caused test failures on slow, uniprocessor hardware.
2017-02-23Add after_worker_ready configuration option chroot
This adds a hook that is called after the application has been loaded by the worker process, directly before it starts accepting requests. This hook is necessary if your application needs to gain access to resources during initialization, and then drop privileges before serving requests. This is especially useful in conjunction with chroot support so the app can load all the normal ruby libraries it needs to function, and then chroot before accepting requests. If you are preloading the app, it's possible to drop privileges or chroot in after_fork, but if you are not preloading the app, the only way to currently do this is to override the private HttpServer#init_worker_process method, and overriding private methods is a recipe for future breakage if the internals are modified. This hook allows for such functionality to be supported and not break in future versions of Unicorn.
2017-02-23Add support for chroot to Worker#user
Any chrooting would need to happen inside Worker#user, because you can't chroot until after you have parsed the list of groups, and you must chroot before dropping root privileges. chroot adds an extra layer of security, so that if the unicorn process is exploited, file system access is limited to the chroot directory instead of the entire file system.
2017-02-23Fix code example in after_worker_exit documentation
Fixes: 2af91a1fef70d654 ("Add after_worker_exit configuration option")
2017-02-21Add after_worker_exit configuration option
This option is executed in the master process following all worker process exits. It is most useful in the case where the worker process crashes the ruby interpreter, as the worker process may not be able to send error notifications appropriately. For example, let's say you have a specific request that crashes a worker process, which you expect to be due to a improperly programmed C extension. By modifying your worker to save request related data in a temporary file and using this option, you can get a record of what request is crashing the application, which will make debugging easier. Example: after_worker_exit do |server, worker, status| server.logger.info "worker #{status.success? ? 'exit' : 'crash'}: #{status}" file = "request.#{status.pid}.txt" if File.exist?(file) do_something_with(File.read(file)) unless status.success? File.delete(file) end end
2017-02-21t/t0012-reload-empty-config.sh: access ivars directly if needed
We'll be moving to direct ivar access to reduce the API (and method entry) overhead of internal unicorn classes. This means some tests like this one will reach deeper into internals. This will be necessary for the upcoming changes to add new configuration options to unicorn.
2017-02-15Revert "remove response_start_sent"
Oops, this was a half-baked change I was considering but forgot about. This reverts commit 69fd4f9bbff3708166fbf70163fa6e192dde1497.
2017-02-13http_request: freeze constant strings passed IO#write
This ensures we won't have duplicate objects in Ruby 2.0-2.4. For Ruby 2.5.0dev+, this avoids any duplicate cleanup introduced as of r57471: https://bugs.ruby-lang.org/issues/13085
2017-02-13remove response_start_sent
2017-02-10tee_input: simplify condition for IO#write
IO#write already elides the write(2) syscall for empty buffers, so there's no need to complicate our instruction sequence footprint for the rare case of an empty buffer. The only cases a Rack app will have an empty buffer are: 1) `env['rack.input'].read` without args 2) `env['rack.input'].gets` Neither of these calls are safe for server-independent Rack apps as the client can OOM the app. unicorn itself provides no facility for limiting maximum rack.input size. Instead, unicorn relies on nginx to limit input size using the client_max_body_size directive.
2016-11-28TUNING: document THP caveat for Linux users
This probably applies to other kernels, too, but I'm most familiar with Linux.
2016-11-09drop rb_str_set_len compatibility replacement
While it is innocuous after compiling, it can be a confusing source of errors for users with broken installations of Ruby itself: https://bogomips.org/unicorn-public/5ace6a20-e094-293d-93df-b557480e12d5@anyces.com/ https://bogomips.org/unicorn-public/02994a55-9c07-a3c5-f06b-a4c15551a67e@anyces.com/ rb_str_set_len has been provided since Ruby 1.8.7+, so we have not needed it since we dropped all 1.8.x support in unicorn 5.x.
2016-10-31unicorn 5.2.0 v5.2.0
Most notably, this release allows us to support requests with lines delimited by LF-only, as opposed to the standard CRLF pair and allowed by RFC 2616 sec 19.3. Thanks to Mishael A Sibiryakov for the explanation and change: https://bogomips.org/unicorn-public/1476954332.1736.156.camel@junki.org/ Thanks to Let's Encrypt, the website also moves to HTTPS <https://bogomips.org/unicorn/> to improve reader privacy. The "unicorn.bogomips.org" subdomain will be retired soon to reduce subjectAltName bloat and speed up certificate renewals. There's also the usual round of documentation and example updates, too. Eric Wong (7): examples/init.sh: update to reduce upgrade raciness doc: systemd should only kill master in example examples/logrotate.conf: update example for systemd doc: update gmane URLs to point to our own archives relocate website to https://bogomips.org/unicorn/ TODO: remove Rack 2.x item build: "install-gem" target avoids network Mishael A Sibiryakov (1): Add some tolerance (RFC2616 sec. 19.3)
2016-10-31build: "install-gem" target avoids network
No need to go online when installing a locally-built gem.
2016-10-31TODO: remove Rack 2.x item
Rack 2.x is less of a jump than initially expected, and we've already supported it for a few releases, already.
2016-10-30Merge remote-tracking branch 'origin/website-move'
* origin/website-move: relocate website to https://bogomips.org/unicorn/