Luca Berruti <nadirio@gmail.com> wrote: > Hello, > > Unicorn 6.1.0 raises this exception when running on Rails 7.1.2 and Devise 4.9.3 within the production environment: > > E, [2023-12-13T13:38:12.560072 #20] ERROR -- : app error: undefined method `=~' for ["_app_session=KbIfGKW%2FGniK%2B7V6boKx8Gh4VYYlk45gI14J5%2F4pSmJQkl890zAknDviSCWCBY4Jx%2FIa86Et1h%2Bad1laCN8sVf%2B9IgtgvN%2BhVLrHBYb9zqmX43LyNqKTEOaEfbU0H6EMEDS6TVJqmtP%2FVzGTf08uP8wgQFMCM6t5jWahl4h9dc47jC96h8BTF8%2FQDXHWPpcj6yzCC3aXjZqm7k2LEPQfXdmXJPwXS57sEYNk518vYWg%2BVOop16o7Lwqh3enXJVd1w%2F6CRFt5cFFlzsDHB7%2FA4%2BPWNLU%3D--v8qFRntrlxCgtLDy--7bldsiyqYyL2eow3kk348A%3D%3D; path=/; secure; httponly; SameSite=Lax"]:Array (NoMethodError) That looks like a Rack 3 response with an array and 6.1.0 doesn't support Rack 3. This patch ought to work: https://yhbt.net/unicorn-public/ZHlX3HkxP6mK16If@jeremyevans.local/raw (commit 9d7bab0bc2211b20806d4d0289a7ea992e49a8a1 in unicorn.git) I'll try to get a unicorn 7.x release soon but tests take forever to run on ancient HW and I need to ration releases to keep download counts low in order to stay under the MFA threshold on Rubygems.org I don't ever want users viewing me as trustworthy nor liable for anything I do, so no MFA nor sigs from me; just source + docs :>
Hello, Unicorn 6.1.0 raises this exception when running on Rails 7.1.2 and Devise 4.9.3 within the production environment: E, [2023-12-13T13:38:12.560072 #20] ERROR -- : app error: undefined method `=~' for ["_app_session=KbIfGKW%2FGniK%2B7V6boKx8Gh4VYYlk45gI14J5%2F4pSmJQkl890zAknDviSCWCBY4Jx%2FIa86Et1h%2Bad1laCN8sVf%2B9IgtgvN%2BhVLrHBYb9zqmX43LyNqKTEOaEfbU0H6EMEDS6TVJqmtP%2FVzGTf08uP8wgQFMCM6t5jWahl4h9dc47jC96h8BTF8%2FQDXHWPpcj6yzCC3aXjZqm7k2LEPQfXdmXJPwXS57sEYNk518vYWg%2BVOop16o7Lwqh3enXJVd1w%2F6CRFt5cFFlzsDHB7%2FA4%2BPWNLU%3D--v8qFRntrlxCgtLDy--7bldsiyqYyL2eow3kk348A%3D%3D; path=/; secure; httponly; SameSite=Lax"]:Array (NoMethodError) E, [2023-12-13T13:38:12.560131 #20] ERROR -- : /usr/local/bundle/gems/unicorn-6.1.0/lib/unicorn/http_response.rb:43:in `block in http_response_write' E, [2023-12-13T13:38:12.560152 #20] ERROR -- : /usr/local/bundle/gems/unicorn-6.1.0/lib/unicorn/http_response.rb:34:in `each' E, [2023-12-13T13:38:12.560165 #20] ERROR -- : /usr/local/bundle/gems/unicorn-6.1.0/lib/unicorn/http_response.rb:34:in `http_response_write' E, [2023-12-13T13:38:12.560176 #20] ERROR -- : /usr/local/bundle/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:645:in `process_client' E, [2023-12-13T13:38:12.560186 #20] ERROR -- : /usr/local/bundle/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:739:in `worker_loop' E, [2023-12-13T13:38:12.560206 #20] ERROR -- : /usr/local/bundle/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:547:in `spawn_missing_workers' E, [2023-12-13T13:38:12.560226 #20] ERROR -- : /usr/local/bundle/gems/unicorn-6.1.0/lib/unicorn/http_server.rb:143:in `start' E, [2023-12-13T13:38:12.560239 #20] ERROR -- : /usr/local/bundle/gems/unicorn-6.1.0/bin/unicorn:128:in `<top (required)>' E, [2023-12-13T13:38:12.560249 #20] ERROR -- : bin/unicorn:27:in `load' E, [2023-12-13T13:38:12.560259 #20] ERROR -- : bin/unicorn:27:in `<main>' Changing rails' "config.force_ssl" option to "false" makes things to work. Looking at /usr/local/bundle/gems/unicorn-6.1.0/lib/unicorn/http_response.rb:43, it seems that the cookie header content is a string when force_ssl=false but becomes an array with force_ssl=true best regards, Luca
Eric Wong <bofh@yhbt.net> wrote: > Patch 2/3 could use an extra set of eyes since it's fairly big, > but passes all tests. Any comments? Anybody still on Ruby <= 2.4? > + filter_body(once = '', @socket.readpartial(@@io_chunk_size, @buf)) > +++ b/lib/unicorn/http_request.rb > @@ -61,7 +61,7 @@ def self.check_client_connection=(bool) > # returns an environment hash suitable for Rack if successful > # This does minimal exception trapping and it is up to the caller > # to handle any socket errors (e.g. user aborted upload). > - def read(socket) > + def read_headers(socket, ai) O_O OMG we haz AI now! Think of the marketing potential! Disclaimer: I have never and will never use any form of intelligence (artificial or otherwise) in writing unicorn.
ideal.water4095@fastmail.com wrote: > > +to tolerate (and thus encourage) bad code. It is only designed > > +to only handle fast clients on low-latency, high-bandwidth connections <snip> > Double "only" here. Thanks, my brain often doubles words :x Will push the patch below out. In the future, please trim out irrelevant parts since it still took me extra time to spot. Thanks again ------8<----- Subject: [PATCH] README: fix wording Reported-by: <ideal.water4095@fastmail.com> --- README | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README b/README index c5c5222..ff14c03 100644 --- a/README +++ b/README @@ -3,7 +3,7 @@ unicorn is an HTTP server for Rack applications that has done decades of damage to the entire Ruby ecosystem due to its ability to tolerate (and thus encourage) bad code. It is only designed -to only handle fast clients on low-latency, high-bandwidth connections +to handle fast clients on low-latency, high-bandwidth connections and take advantage of features in Unix/Unix-like kernels. Slow clients must only be served by placing a reverse proxy capable of fully buffering both the the request and response in between unicorn
> The damage unicorn has done to the entire Ruby, Rack and Rails
> ecosystems with its ability to tolerate buggy code is
> unforgivable. Update the documentation to further discourage
> its use and clarify a few wordings noticed along the way.
> ---
> DESIGN | 4 ++++
> ISSUES | 6 +++++-
> README | 38 ++++++++++++++++++++-----------------
> lib/unicorn/configurator.rb | 7 ++++++-
> 4 files changed, 36 insertions(+), 19 deletions(-)
>
> diff --git a/DESIGN b/DESIGN
> index 46d7923..0bac24f 100644
> --- a/DESIGN
> +++ b/DESIGN
> @@ -1,5 +1,9 @@
> == Design
>
> +Unicorn was designed to support poorly-written codebases back in 2008.
> +Its unfortunate popularity has only proliferated the existence of
> +poorly-written code ever since...
> +
> * Simplicity: Unicorn is a traditional UNIX prefork web server.
> No threads are used at all, this makes applications easier to debug
> and fix. When your application goes awry, a BOFH can just
> diff --git a/ISSUES b/ISSUES
> index 083b1c8..d6c2a7a 100644
> --- a/ISSUES
> +++ b/ISSUES
> @@ -32,6 +32,10 @@ and such.
> If you don't get a response within a few days, we may have forgotten
> about it so feel free to ask again.
>
> +The project does not and will never endorse nor promote commercial
> +services (including support). The author of unicorn must never be
> +allowed to profit off the damage it's done to the entire Ruby world.
> +
> == Bugs in related projects
>
> unicorn is sometimes affected by bugs in its dependencies. Bugs
> @@ -65,7 +69,7 @@ There is a kernel.org Bugzilla instance, but it is
> ignored by most.
>
> Likewise for any rare glibc bugs we might encounter, we should Cc:
> mailto:libc-alpha@sourceware.org
> -Unofficial archives are available at: https://public-inbox.org/libc-alpha/
> +Archives are available at: https://inbox.sourceware.org/libc-alpha/
> Keep in mind glibc upstream does use Bugzilla for tracking bugs:
> https://sourceware.org/bugzilla/
>
> diff --git a/README b/README
> index 5411003..c5c5222 100644
> --- a/README
> +++ b/README
> @@ -1,10 +1,13 @@
> = unicorn: Rack HTTP server for fast clients and Unix
>
> -unicorn is an HTTP server for Rack applications designed to only serve
> -fast clients on low-latency, high-bandwidth connections and take
> -advantage of features in Unix/Unix-like kernels. Slow clients should
> -only be served by placing a reverse proxy capable of fully buffering
> -both the the request and response in between unicorn and slow clients.
> +unicorn is an HTTP server for Rack applications that has done
> +decades of damage to the entire Ruby ecosystem due to its ability
> +to tolerate (and thus encourage) bad code. It is only designed
> +to only handle fast clients on low-latency, high-bandwidth connections
> +and take advantage of features in Unix/Unix-like kernels.
> +Slow clients must only be served by placing a reverse proxy capable of
> +fully buffering both the the request and response in between unicorn
> +and slow clients.
Double "only" here.
The damage unicorn has done to the entire Ruby, Rack and Rails ecosystems with its ability to tolerate buggy code is unforgivable. Update the documentation to further discourage its use and clarify a few wordings noticed along the way. --- DESIGN | 4 ++++ ISSUES | 6 +++++- README | 38 ++++++++++++++++++++----------------- lib/unicorn/configurator.rb | 7 ++++++- 4 files changed, 36 insertions(+), 19 deletions(-) diff --git a/DESIGN b/DESIGN index 46d7923..0bac24f 100644 --- a/DESIGN +++ b/DESIGN @@ -1,5 +1,9 @@ == Design +Unicorn was designed to support poorly-written codebases back in 2008. +Its unfortunate popularity has only proliferated the existence of +poorly-written code ever since... + * Simplicity: Unicorn is a traditional UNIX prefork web server. No threads are used at all, this makes applications easier to debug and fix. When your application goes awry, a BOFH can just diff --git a/ISSUES b/ISSUES index 083b1c8..d6c2a7a 100644 --- a/ISSUES +++ b/ISSUES @@ -32,6 +32,10 @@ and such. If you don't get a response within a few days, we may have forgotten about it so feel free to ask again. +The project does not and will never endorse nor promote commercial +services (including support). The author of unicorn must never be +allowed to profit off the damage it's done to the entire Ruby world. + == Bugs in related projects unicorn is sometimes affected by bugs in its dependencies. Bugs @@ -65,7 +69,7 @@ There is a kernel.org Bugzilla instance, but it is ignored by most. Likewise for any rare glibc bugs we might encounter, we should Cc: mailto:libc-alpha@sourceware.org -Unofficial archives are available at: https://public-inbox.org/libc-alpha/ +Archives are available at: https://inbox.sourceware.org/libc-alpha/ Keep in mind glibc upstream does use Bugzilla for tracking bugs: https://sourceware.org/bugzilla/ diff --git a/README b/README index 5411003..c5c5222 100644 --- a/README +++ b/README @@ -1,10 +1,13 @@ = unicorn: Rack HTTP server for fast clients and Unix -unicorn is an HTTP server for Rack applications designed to only serve -fast clients on low-latency, high-bandwidth connections and take -advantage of features in Unix/Unix-like kernels. Slow clients should -only be served by placing a reverse proxy capable of fully buffering -both the the request and response in between unicorn and slow clients. +unicorn is an HTTP server for Rack applications that has done +decades of damage to the entire Ruby ecosystem due to its ability +to tolerate (and thus encourage) bad code. It is only designed +to only handle fast clients on low-latency, high-bandwidth connections +and take advantage of features in Unix/Unix-like kernels. +Slow clients must only be served by placing a reverse proxy capable of +fully buffering both the the request and response in between unicorn +and slow clients. == Features @@ -14,8 +17,8 @@ both the the request and response in between unicorn and slow clients. * Compatible with Ruby 2.0.0 and later. -* Process management: unicorn will reap and restart workers that - die from broken apps. There is no need to manage multiple processes +* Process management: unicorn reaps and restarts workers that die + from broken code. There is no need to manage multiple processes or ports yourself. unicorn can spawn and manage any number of worker processes you choose to scale to your backend. @@ -57,7 +60,7 @@ both the the request and response in between unicorn and slow clients. == License -unicorn is copyright 2009-2018 by all contributors (see logs in git). +unicorn is copyright all contributors (see logs in git). It is based on Mongrel 1.1.5. Mongrel is copyright 2007 Zed A. Shaw and contributors. @@ -79,8 +82,8 @@ You may install it via RubyGems on RubyGems.org: You can get the latest source via git from the following locations (these versions may not be stable): - https://yhbt.net/unicorn.git - https://repo.or.cz/unicorn.git (mirror) + git clone https://yhbt.net/unicorn.git + git clone https://repo.or.cz/unicorn.git # mirror You may browse the code from the web: @@ -118,23 +121,24 @@ supported. Run `unicorn -h` to see command-line options. == Disclaimer There is NO WARRANTY whatsoever if anything goes wrong, but -{let us know}[link:ISSUES.html] and we'll try our best to fix it. +{let us know}[link:ISSUES.html] and maybe someone can fix it. unicorn is designed to only serve fast clients either on the local host or a fast LAN. See the PHILOSOPHY and DESIGN documents for more details regarding this. -Due to its ability to tolerate crashes and isolate clients, unicorn -is unfortunately known to prolong the existence of bugs in applications -and libraries which run on top of it. +The use of unicorn in new deployments is STRONGLY DISCOURAGED due to the +damage done to the entire Ruby ecosystem. Its unintentional popularity +set Ruby back decades in parallelism, concurrency and robustness since +it prolongs and proliferates the existence of poorly-written code. == Contact All feedback (bug reports, user/development dicussion, patches, pull -requests) go to the mailing list/newsgroup. See the ISSUES document for -information on the {mailing list}[mailto:unicorn-public@yhbt.net]. +requests) go to the public mailbox. See the ISSUES document for +information on posting to mailto:unicorn-public@yhbt.net -The mailing list is archived at https://yhbt.net/unicorn-public/ +Mirror-able mail archives are at https://yhbt.net/unicorn-public/ Read-only NNTP access is available at: nntps://news.public-inbox.org/inbox.comp.lang.ruby.unicorn and diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb index ecdf03e..b21a01d 100644 --- a/lib/unicorn/configurator.rb +++ b/lib/unicorn/configurator.rb @@ -216,7 +216,12 @@ def before_exec(*args, &block) set_hook(:before_exec, block_given? ? block : args[0], 1) end - # sets the timeout of worker processes to +seconds+. Workers + # Strongly consider using link:/Application_Timeouts.html instead + # of this misfeature. This misfeature has done decades of damage + # to Ruby since it demotivates the use of fine-grained timeout + # mechanisms. + # + # Sets the timeout of worker processes to +seconds+. Workers # handling the request/app.call/response cycle taking longer than # this time period will be forcibly killed (via SIGKILL). This # timeout is enforced by the master process itself and not subject
[-- Attachment #1: Type: text/plain, Size: 3020 bytes --] Hopefully this is less maintenance down the line since Ruby introduces incompatibilities at a higher rate than Perl. I don't fully trust Perl, either, but far more Ruby code gets broken by new releases. More to come at some point... Note: attached patches are generated with --irreversible-delete to save bandwidth. Eric Wong (11): tests: port some bad config tests to Perl 5 tests: port working_directory tests to Perl 5 tests: port t/heartbeat-timeout to Perl 5 tests: port reopen logs test over to Perl 5 tests: rewrite SIGWINCH && SIGTTIN test in Perl 5 tests: introduce `do_req' helper sub tests: use more common variable names between tests tests: use Time::HiRes `sleep' and `time' everywhere tests: fold SO_KEEPALIVE check to Perl 5 integration tests: move broken app test to Perl 5 integration test tests: fold early shutdown() tests into t/integration.t t/active-unix-socket.t | 4 +- t/client_body_buffer_size.t | 6 +- t/heartbeat-timeout.ru | 2 +- t/heartbeat-timeout.t | 62 +++++++++++++++ t/integration.ru | 1 + t/integration.t | 82 +++++++++++++------- t/lib.perl | 51 ++++++++++-- t/reload-bad-config.t | 54 +++++++++++++ t/{t0006.ru => reopen-logs.ru} | 0 t/reopen-logs.t | 39 ++++++++++ t/t0001-reload-bad-config.sh | 53 ------------- t/t0002-config-conflict.sh | 49 ------------ t/t0003-working_directory.sh | 51 ------------ t/t0004-heartbeat-timeout.sh | 69 ----------------- t/t0004-working_directory_broken.sh | 24 ------ t/t0005-working_directory_app.rb.sh | 40 ---------- t/t0006-reopen-logs.sh | 83 -------------------- t/t0007-working_directory_no_embed_cli.sh | 44 ----------- t/t0009-winch_ttin.sh | 59 -------------- t/winch_ttin.t | 67 ++++++++++++++++ t/working_directory.t | 94 +++++++++++++++++++++++ test/exec/test_exec.rb | 23 +----- test/unit/test_server.rb | 67 ---------------- 23 files changed, 424 insertions(+), 600 deletions(-) create mode 100644 t/heartbeat-timeout.t create mode 100644 t/reload-bad-config.t rename t/{t0006.ru => reopen-logs.ru} (100%) create mode 100644 t/reopen-logs.t delete mode 100755 t/t0001-reload-bad-config.sh delete mode 100755 t/t0002-config-conflict.sh delete mode 100755 t/t0003-working_directory.sh delete mode 100755 t/t0004-heartbeat-timeout.sh delete mode 100755 t/t0004-working_directory_broken.sh delete mode 100755 t/t0005-working_directory_app.rb.sh delete mode 100755 t/t0006-reopen-logs.sh delete mode 100755 t/t0007-working_directory_no_embed_cli.sh delete mode 100755 t/t0009-winch_ttin.sh create mode 100644 t/winch_ttin.t create mode 100644 t/working_directory.t [-- Attachment #2: 0001-tests-port-some-bad-config-tests-to-Perl-5.patch --] [-- Type: text/x-diff, Size: 3987 bytes --] From f43c28ea10ca8d520b55f2fbb20710dd66fc4fb5 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Thu, 7 Sep 2023 22:55:09 +0000 Subject: [PATCH 01/11] tests: port some bad config tests to Perl 5 We can fold some tests into one test to save on Perl startup time (but Ruby startup time is a lost cause). --- t/lib.perl | 12 ++++---- t/reload-bad-config.t | 58 ++++++++++++++++++++++++++++++++++++ t/t0001-reload-bad-config.sh | 53 -------------------------------- t/t0002-config-conflict.sh | 49 ------------------------------ 4 files changed, 65 insertions(+), 107 deletions(-) create mode 100644 t/reload-bad-config.t delete mode 100755 t/t0001-reload-bad-config.sh delete mode 100755 t/t0002-config-conflict.sh diff --git a/t/lib.perl b/t/lib.perl index fe3404ba..7de9e426 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -9,17 +9,19 @@ use Test::More; use IO::Socket::INET; use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD); use File::Temp 0.19 (); # 0.19 for ->newdir -our ($tmpdir, $errfh); -our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh +our ($tmpdir, $errfh, $err_log); +our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn + $tmpdir $errfh $err_log SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr); my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!); $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1); -open($errfh, '>>', "$tmpdir/err.log"); -END { diag slurp("$tmpdir/err.log") if $tmpdir }; +$err_log = "$tmpdir/err.log"; +open($errfh, '>>', $err_log); +END { diag slurp($err_log) if $tmpdir }; sub check_stderr () { - my @log = slurp("$tmpdir/err.log"); + my @log = slurp($err_log); diag("@log") if $ENV{V}; my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log)); @err = grep(!/failed to set accept_filter=/, @err); diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t new file mode 100644 index 00000000..c7055c7e --- /dev/null +++ b/t/reload-bad-config.t @@ -0,0 +1,58 @@ +#!perl -w +# Copyright (C) unicorn hackers <unicorn-public@yhbt.net> +# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt> +use v5.14; BEGIN { require './t/lib.perl' }; +use autodie; +my $srv = tcp_server(); +my $host_port = tcp_host_port($srv); +my $ru = "$tmpdir/config.ru"; +my $u_conf = "$tmpdir/u.conf.rb"; + +open my $fh, '>', $ru; +print $fh <<'EOM'; +use Rack::ContentLength +use Rack::ContentType, 'text/plain' +config = ru = "hello world\n" # check for config variable conflicts, too +run lambda { |env| [ 200, {}, [ ru.to_s ] ] } +EOM +close $fh; + +open $fh, '>', $u_conf; +print $fh <<EOM; +preload_app true +stderr_path "$err_log" +EOM +close $fh; + +my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv }); +my $c = tcp_start($srv, 'GET / HTTP/1.0'); +my ($status, $hdr) = slurp_hdr($c); +my $bdy = do { local $/; <$c> }; +like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start'); +is($bdy, "hello world\n", 'body matches expected'); + +open $fh, '>>', $ru; +say $fh '....this better be a syntax error in any version of ruby...'; +close $fh; + +$ar->do_kill('HUP'); # reload +my @l; +for (1..1000) { + @l = grep(/(?:done|error) reloading/, slurp($err_log)) and + last; + select undef, undef, undef, 0.011; +} +diag slurp($err_log) if $ENV{V}; +ok(grep(/error reloading/, @l), 'got error reloading'); +open $fh, '>', $err_log; +close $fh; + +$c = tcp_start($srv, 'GET / HTTP/1.0'); +($status, $hdr) = slurp_hdr($c); +$bdy = do { local $/; <$c> }; +like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid afte reload'); +is($bdy, "hello world\n", 'body matches expected after reload'); + +check_stderr; +undef $tmpdir; # quiet t/lib.perl END{} +done_testing; diff --git a/t/t0001-reload-bad-config.sh b/t/t0001-reload-bad-config.sh deleted file mode 100755 index 55bb3555..00000000 diff --git a/t/t0002-config-conflict.sh b/t/t0002-config-conflict.sh deleted file mode 100755 index d7b2181a..00000000 [-- Attachment #3: 0002-tests-port-working_directory-tests-to-Perl-5.patch --] [-- Type: text/x-diff, Size: 4809 bytes --] From d4514174ee7eadea89003f380acacf32d52acd9d Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Thu, 7 Sep 2023 23:18:16 +0000 Subject: [PATCH 02/11] tests: port working_directory tests to Perl 5 We can fold a bunch of them into one test to save startup time, inodes, and FS activity. --- t/t0003-working_directory.sh | 51 --------- t/t0004-working_directory_broken.sh | 24 ----- t/t0005-working_directory_app.rb.sh | 40 ------- t/t0007-working_directory_no_embed_cli.sh | 44 -------- t/working_directory.t | 122 ++++++++++++++++++++++ 5 files changed, 122 insertions(+), 159 deletions(-) delete mode 100755 t/t0003-working_directory.sh delete mode 100755 t/t0004-working_directory_broken.sh delete mode 100755 t/t0005-working_directory_app.rb.sh delete mode 100755 t/t0007-working_directory_no_embed_cli.sh create mode 100644 t/working_directory.t diff --git a/t/t0003-working_directory.sh b/t/t0003-working_directory.sh deleted file mode 100755 index 79988d8b..00000000 diff --git a/t/t0004-working_directory_broken.sh b/t/t0004-working_directory_broken.sh deleted file mode 100755 index ca9d3825..00000000 diff --git a/t/t0005-working_directory_app.rb.sh b/t/t0005-working_directory_app.rb.sh deleted file mode 100755 index 0fbab4fc..00000000 diff --git a/t/t0007-working_directory_no_embed_cli.sh b/t/t0007-working_directory_no_embed_cli.sh deleted file mode 100755 index 77d67072..00000000 diff --git a/t/working_directory.t b/t/working_directory.t new file mode 100644 index 00000000..e7ff43a5 --- /dev/null +++ b/t/working_directory.t @@ -0,0 +1,122 @@ +#!perl -w +# Copyright (C) unicorn hackers <unicorn-public@yhbt.net> +# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt> +use v5.14; BEGIN { require './t/lib.perl' }; +use autodie; +mkdir "$tmpdir/alt"; +my $u_sock = "$tmpdir/u.sock"; +my $ru = "$tmpdir/alt/config.ru"; +my $u_conf = "$tmpdir/u.conf.rb"; +open my $fh, '>', $u_conf; +print $fh <<EOM; +pid "$tmpdir/pid" +preload_app true +stderr_path "$err_log" +working_directory "$tmpdir/alt" # the whole point of this test +before_fork { |_,_| \$master_ppid = Process.ppid } +EOM +close $fh; + +my $common_ru = <<'EOM'; +use Rack::ContentLength +use Rack::ContentType, 'text/plain' +run lambda { |env| [ 200, {}, [ "#{$master_ppid}\n" ] ] } +EOM + +open $fh, '>', $ru; +print $fh <<EOM; +#\\--daemonize --listen $u_sock +$common_ru +EOM +close $fh; + +my $pid; +my $stop_daemon = sub { + my ($is_END) = @_; + kill('TERM', $pid); + my $tries = 1000; + while (CORE::kill(0, $pid) && --$tries) { + select undef, undef, undef, 0.01; + } + if ($is_END && CORE::kill(0, $pid)) { + CORE::kill('KILL', $pid); + die "daemonized PID=$pid did not die"; + } else { + ok(!CORE::kill(0, $pid), 'daemonized unicorn gone'); + undef $pid; + } +}; + +END { $stop_daemon->(1) if defined $pid }; + +unicorn('-c', $u_conf)->join; # will daemonize +chomp($pid = slurp("$tmpdir/pid")); + +my $c = unix_start($u_sock, 'GET / HTTP/1.0'); +my ($status, $hdr) = slurp_hdr($c); +chomp(my $bdy = do { local $/; <$c> }); +is($bdy, 1, 'got expected $master_ppid'); + +$stop_daemon->(); +check_stderr; + +if ('test without CLI switches in config.ru') { + truncate $err_log, 0; + open $fh, '>', $ru; + print $fh $common_ru; + close $fh; + + unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize + chomp($pid = slurp("$tmpdir/pid")); + + $c = unix_start($u_sock, 'GET / HTTP/1.0'); + ($status, $hdr) = slurp_hdr($c); + chomp($bdy = do { local $/; <$c> }); + is($bdy, 1, 'got expected $master_ppid'); + + $stop_daemon->(); + check_stderr; +} + +if ('ensures broken working_directory (missing config.ru) is OK') { + truncate $err_log, 0; + unlink $ru; + + my $auto_reap = unicorn('-c', $u_conf); + $auto_reap->join; + isnt($?, 0, 'exited with error due to missing config.ru'); + + like(slurp($err_log), qr/rackup file \Q(config.ru)\E not readable/, + 'noted unreadability of config.ru in stderr'); +} + +if ('fooapp.rb (not config.ru) works with working_directory') { + truncate $err_log, 0; + my $fooapp = "$tmpdir/alt/fooapp.rb"; + open $fh, '>', $fooapp; + print $fh <<EOM; +class Fooapp + def self.call(env) + b = "dir=#{Dir.pwd}" + h = { 'content-type' => 'text/plain', 'content-length' => b.bytesize.to_s } + [ 200, h, [ b ] ] + end +end +EOM + close $fh; + my $srv = tcp_server; + my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb), + { -C => '/', 3 => $srv }); + $c = tcp_start($srv, 'GET / HTTP/1.0'); + ($status, $hdr) = slurp_hdr($c); + chomp($bdy = do { local $/; <$c> }); + is($bdy, "dir=$tmpdir/alt", + 'fooapp.rb (w/o config.ru) w/ working_directory'); + close $c; + $auto_reap->join('TERM'); + is($?, 0, 'fooapp.rb process exited'); + check_stderr; +} + +undef $tmpdir; +done_testing; [-- Attachment #4: 0003-tests-port-t-heartbeat-timeout-to-Perl-5.patch --] [-- Type: text/x-diff, Size: 3478 bytes --] From d67284a692683bca59effd9c0670bd5dd47e4fa3 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Thu, 7 Sep 2023 23:53:58 +0000 Subject: [PATCH 03/11] tests: port t/heartbeat-timeout to Perl 5 I absolutely detest and regret adding this feature, but I'm hell bent on supporting it until the end of days because we don't break compatibility. --- t/heartbeat-timeout.ru | 2 +- t/heartbeat-timeout.t | 69 ++++++++++++++++++++++++++++++++++++ t/t0004-heartbeat-timeout.sh | 69 ------------------------------------ 3 files changed, 70 insertions(+), 70 deletions(-) create mode 100644 t/heartbeat-timeout.t delete mode 100755 t/t0004-heartbeat-timeout.sh diff --git a/t/heartbeat-timeout.ru b/t/heartbeat-timeout.ru index 20a79380..3eeb5d64 100644 --- a/t/heartbeat-timeout.ru +++ b/t/heartbeat-timeout.ru @@ -7,6 +7,6 @@ sleep # in case STOP signal is not received in time [ 500, headers, [ "Should never get here\n" ] ] else - [ 200, headers, [ "#$$\n" ] ] + [ 200, headers, [ "#$$" ] ] end } diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t new file mode 100644 index 00000000..1fcf21a2 --- /dev/null +++ b/t/heartbeat-timeout.t @@ -0,0 +1,69 @@ +#!perl -w +# Copyright (C) unicorn hackers <unicorn-public@yhbt.net> +# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt> +use v5.14; BEGIN { require './t/lib.perl' }; +use autodie; +use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC); +mkdir "$tmpdir/alt"; +my $srv = tcp_server(); +my $u_conf = "$tmpdir/u.conf.rb"; +open my $fh, '>', $u_conf; +print $fh <<EOM; +pid "$tmpdir/pid" +preload_app true +stderr_path "$err_log" +timeout 3 # WORST FEATURE EVER +EOM +close $fh; + +my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv }); + +my $c = tcp_start($srv, 'GET /pid HTTP/1.0'); +my ($status, $hdr) = slurp_hdr($c); +like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds'); +my $wpid = do { local $/; <$c> }; +like($wpid, qr/\A[0-9]+\z/, 'worker is running'); + +my $t0 = clock_gettime(CLOCK_MONOTONIC); +$c = tcp_start($srv, 'GET /block-forever HTTP/1.0'); +vec(my $rvec = '', fileno($c), 1) = 1; +is(select($rvec, undef, undef, 6), 1, 'got readiness'); +$c->blocking(0); +is(sysread($c, my $buf, 128), 0, 'got EOF response'); +my $elapsed = clock_gettime(CLOCK_MONOTONIC) - $t0; +ok($elapsed > 3, 'timeout took >3s'); + +my @timeout_err = slurp($err_log); +truncate($err_log, 0); +is(grep(/timeout \(\d+s > 3s\), killing/, @timeout_err), 1, + 'noted timeout error') or diag explain(\@timeout_err); + +# did it respawn? +$c = tcp_start($srv, 'GET /pid HTTP/1.0'); +($status, $hdr) = slurp_hdr($c); +like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds'); +my $new_pid = do { local $/; <$c> }; +isnt($new_pid, $wpid, 'spawned new worker'); + +diag 'SIGSTOP for 4 seconds...'; +$ar->do_kill('STOP'); +sleep 4; +$ar->do_kill('CONT'); +for my $i (1..2) { + $c = tcp_start($srv, 'GET /pid HTTP/1.0'); + ($status, $hdr) = slurp_hdr($c); + like($status, qr!\AHTTP/1\.[01] 200\b!, + "PID request succeeds #$i after STOP+CONT"); + my $spid = do { local $/; <$c> }; + is($new_pid, $spid, "worker pid unchanged after STOP+CONT #$i"); + if ($i == 1) { + diag 'sleeping 2s to ensure timeout is not delayed'; + sleep 2; + } +} + +$ar->join('TERM'); +check_stderr; +undef $tmpdir; + +done_testing; diff --git a/t/t0004-heartbeat-timeout.sh b/t/t0004-heartbeat-timeout.sh deleted file mode 100755 index 29652837..00000000 [-- Attachment #5: 0004-tests-port-reopen-logs-test-over-to-Perl-5.patch --] [-- Type: text/x-diff, Size: 2317 bytes --] From 1607ac966f604ec4cf383025c4c3ee296f638fff Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Sun, 10 Sep 2023 07:13:11 +0000 Subject: [PATCH 04/11] tests: port reopen logs test over to Perl 5 Being able to do subsecond sleeps is one welcome advantage over POSIX (not GNU) sleep(1) in portable Bourne sh. --- t/{t0006.ru => reopen-logs.ru} | 0 t/reopen-logs.t | 43 ++++++++++++++++++ t/t0006-reopen-logs.sh | 83 ---------------------------------- 3 files changed, 43 insertions(+), 83 deletions(-) rename t/{t0006.ru => reopen-logs.ru} (100%) create mode 100644 t/reopen-logs.t delete mode 100755 t/t0006-reopen-logs.sh diff --git a/t/t0006.ru b/t/reopen-logs.ru similarity index 100% rename from t/t0006.ru rename to t/reopen-logs.ru diff --git a/t/reopen-logs.t b/t/reopen-logs.t new file mode 100644 index 00000000..e1bf524c --- /dev/null +++ b/t/reopen-logs.t @@ -0,0 +1,43 @@ +#!perl -w +# Copyright (C) unicorn hackers <unicorn-public@yhbt.net> +# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt> +use v5.14; BEGIN { require './t/lib.perl' }; +use autodie; +my $srv = tcp_server(); +my $u_conf = "$tmpdir/u.conf.rb"; +my $out_log = "$tmpdir/out.log"; +open my $fh, '>', $u_conf; +print $fh <<EOM; +stderr_path "$err_log" +stdout_path "$out_log" +EOM +close $fh; + +my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } ); +my $c = tcp_start($srv, 'GET / HTTP/1.0'); +my ($status, $hdr) = slurp_hdr($c); +my $bdy = do { local $/; <$c> }; +is($bdy, "true\n", 'logs opened'); + +rename($err_log, "$err_log.rot"); +rename($out_log, "$out_log.rot"); + +$auto_reap->do_kill('USR1'); + +my $tries = 1000; +while (!-f $err_log && --$tries) { select undef, undef, undef, 0.01 }; +while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 }; + +ok(-f $out_log, 'stdout_path recreated after USR1'); +ok(-f $err_log, 'stderr_path recreated after USR1'); + +$c = tcp_start($srv, 'GET / HTTP/1.0'); +($status, $hdr) = slurp_hdr($c); +$bdy = do { local $/; <$c> }; +is($bdy, "true\n", 'logs reopened with sync==true'); + +$auto_reap->join('QUIT'); +is($?, 0, 'no error on exit'); +check_stderr; +undef $tmpdir; +done_testing; diff --git a/t/t0006-reopen-logs.sh b/t/t0006-reopen-logs.sh deleted file mode 100755 index a6e7a17c..00000000 [-- Attachment #6: 0005-tests-rewrite-SIGWINCH-SIGTTIN-test-in-Perl-5.patch --] [-- Type: text/x-diff, Size: 2916 bytes --] From 86aea575c331a3b5242db1c14a848928a37ff9e3 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Sun, 10 Sep 2023 08:27:04 +0000 Subject: [PATCH 05/11] tests: rewrite SIGWINCH && SIGTTIN test in Perl 5 No need to deal with full second sleeps, here. --- t/t0009-winch_ttin.sh | 59 ----------------------------------- t/winch_ttin.t | 72 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+), 59 deletions(-) delete mode 100755 t/t0009-winch_ttin.sh create mode 100644 t/winch_ttin.t diff --git a/t/t0009-winch_ttin.sh b/t/t0009-winch_ttin.sh deleted file mode 100755 index 6e56e30c..00000000 diff --git a/t/winch_ttin.t b/t/winch_ttin.t new file mode 100644 index 00000000..1a198778 --- /dev/null +++ b/t/winch_ttin.t @@ -0,0 +1,72 @@ +#!perl -w +# Copyright (C) unicorn hackers <unicorn-public@yhbt.net> +# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt> +use v5.14; BEGIN { require './t/lib.perl' }; +use autodie; +use POSIX qw(mkfifo); +my $u_conf = "$tmpdir/u.conf.rb"; +my $u_sock = "$tmpdir/u.sock"; +my $fifo = "$tmpdir/fifo"; +mkfifo($fifo, 0666) or die "mkfifo($fifo): $!"; + +open my $fh, '>', $u_conf; +print $fh <<EOM; +pid "$tmpdir/pid" +listen "$u_sock" +stderr_path "$err_log" +after_fork do |server, worker| + # test script will block while reading from $fifo, + File.open("$fifo", "wb") { |fp| fp.syswrite worker.nr.to_s } +end +EOM +close $fh; + +unicorn('-D', '-c', $u_conf, 't/integration.ru')->join; +is($?, 0, 'daemonized properly'); +open $fh, '<', "$tmpdir/pid"; +chomp(my $pid = <$fh>); +ok(kill(0, $pid), 'daemonized PID works'); +my $quit = sub { kill('QUIT', $pid) if $pid; $pid = undef }; +END { $quit->() }; + +open $fh, '<', $fifo; +my $worker_nr = <$fh>; +close $fh; +is($worker_nr, '0', 'initial worker spawned'); + +my $c = unix_start($u_sock, 'GET /pid HTTP/1.0'); +my ($status, $hdr) = slurp_hdr($c); +like($status, qr/ 200\b/, 'got 200 response'); +my $worker_pid = do { local $/; <$c> }; +like($worker_pid, qr/\A[0-9]+\n\z/s, 'PID in response'); +chomp $worker_pid; +ok(kill(0, $worker_pid), 'worker_pid is valid'); + +ok(kill('WINCH', $pid), 'SIGWINCH can be sent'); + +my $tries = 1000; +while (CORE::kill(0, $worker_pid) && --$tries) { + select undef, undef, undef, 0.01; +} +ok(!CORE::kill(0, $worker_pid), 'worker not running'); + +ok(kill('TTIN', $pid), 'SIGTTIN to restart worker'); + +open $fh, '<', $fifo; +$worker_nr = <$fh>; +close $fh; +is($worker_nr, '0', 'worker restarted'); + +$c = unix_start($u_sock, 'GET /pid HTTP/1.0'); +($status, $hdr) = slurp_hdr($c); +like($status, qr/ 200\b/, 'got 200 response'); +chomp(my $new_worker_pid = do { local $/; <$c> }); +like($new_worker_pid, qr/\A[0-9]+\z/, 'got new worker PID'); +ok(kill(0, $new_worker_pid), 'got a valid worker PID'); +isnt($worker_pid, $new_worker_pid, 'worker PID changed'); + +$quit->(); + +check_stderr; +undef $tmpdir; +done_testing; [-- Attachment #7: 0006-tests-introduce-do_req-helper-sub.patch --] [-- Type: text/x-diff, Size: 11556 bytes --] From 29885f0d95aaa8e1d1f6cf3b791d9f08338a511e Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Sun, 10 Sep 2023 09:15:16 +0000 Subject: [PATCH 06/11] tests: introduce `do_req' helper sub While early tests required fine-grained control in trickling requests, many of our later tests can use a short one-liner w/o having to spawn curl. --- t/heartbeat-timeout.t | 12 +++--------- t/integration.t | 33 +++++++++++++-------------------- t/lib.perl | 16 +++++++++++++++- t/reload-bad-config.t | 8 ++------ t/reopen-logs.t | 8 ++------ t/winch_ttin.t | 11 ++++------- t/working_directory.t | 17 +++++------------ 7 files changed, 44 insertions(+), 61 deletions(-) diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t index 1fcf21a2..ce1f7e16 100644 --- a/t/heartbeat-timeout.t +++ b/t/heartbeat-timeout.t @@ -18,10 +18,8 @@ close $fh; my $ar = unicorn(qw(-E none t/heartbeat-timeout.ru -c), $u_conf, { 3 => $srv }); -my $c = tcp_start($srv, 'GET /pid HTTP/1.0'); -my ($status, $hdr) = slurp_hdr($c); +my ($status, $hdr, $wpid) = do_req($srv, 'GET /pid HTTP/1.0'); like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds'); -my $wpid = do { local $/; <$c> }; like($wpid, qr/\A[0-9]+\z/, 'worker is running'); my $t0 = clock_gettime(CLOCK_MONOTONIC); @@ -39,10 +37,8 @@ is(grep(/timeout \(\d+s > 3s\), killing/, @timeout_err), 1, 'noted timeout error') or diag explain(\@timeout_err); # did it respawn? -$c = tcp_start($srv, 'GET /pid HTTP/1.0'); -($status, $hdr) = slurp_hdr($c); +($status, $hdr, my $new_pid) = do_req($srv, 'GET /pid HTTP/1.0'); like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds'); -my $new_pid = do { local $/; <$c> }; isnt($new_pid, $wpid, 'spawned new worker'); diag 'SIGSTOP for 4 seconds...'; @@ -50,11 +46,9 @@ $ar->do_kill('STOP'); sleep 4; $ar->do_kill('CONT'); for my $i (1..2) { - $c = tcp_start($srv, 'GET /pid HTTP/1.0'); - ($status, $hdr) = slurp_hdr($c); + ($status, $hdr, my $spid) = do_req($srv, 'GET /pid HTTP/1.0'); like($status, qr!\AHTTP/1\.[01] 200\b!, "PID request succeeds #$i after STOP+CONT"); - my $spid = do { local $/; <$c> }; is($new_pid, $spid, "worker pid unchanged after STOP+CONT #$i"); if ($i == 1) { diag 'sleeping 2s to ensure timeout is not delayed'; diff --git a/t/integration.t b/t/integration.t index bb2ab51b..13b07467 100644 --- a/t/integration.t +++ b/t/integration.t @@ -62,11 +62,10 @@ EOM }, ); -my ($c, $status, $hdr); +my ($c, $status, $hdr, $bdy); # response header tests -$c = tcp_start($srv, 'GET /rack-2-newline-headers HTTP/1.0'); -($status, $hdr) = slurp_hdr($c); +($status, $hdr) = do_req($srv, 'GET /rack-2-newline-headers HTTP/1.0'); like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid'); my $orig_200_status = $status; is_deeply([ grep(/^X-R2: /, @$hdr) ], @@ -84,16 +83,16 @@ SKIP: { # Date header check }; -$c = tcp_start($srv, 'GET /rack-3-array-headers HTTP/1.0'); -($status, $hdr) = slurp_hdr($c); +($status, $hdr) = do_req($srv, 'GET /rack-3-array-headers HTTP/1.0'); is_deeply([ grep(/^x-r3: /, @$hdr) ], [ 'x-r3: a', 'x-r3: b', 'x-r3: c' ], 'rack 3 array headers supported') or diag(explain($hdr)); SKIP: { eval { require JSON::PP } or skip "JSON::PP missing: $@", 1; - my $c = tcp_start($srv, 'GET /env_dump'); - my $json = do { local $/; readline($c) }; + ($status, $hdr, my $json) = do_req $srv, 'GET /env_dump'; + is($status, undef, 'no status for HTTP/0.9'); + is($hdr, undef, 'no header for HTTP/0.9'); unlike($json, qr/^Connection: /smi, 'no connection header for 0.9'); unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9'); my $env = JSON::PP->new->decode($json); @@ -102,8 +101,7 @@ SKIP: { } # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com> -$c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0'); -($status, $hdr) = slurp_hdr($c); +($status, $hdr) = do_req($srv, 'GET /nil-header-value HTTP/1.0'); is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '], 'nil header value accepted for broken apps') or diag(explain($hdr)); @@ -128,12 +126,10 @@ my $ck_early_hints = sub { $ck_early_hints->('ccc off'); # we'll retest later if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') { - $c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0'); - ($status, $hdr) = slurp_hdr($c); + ($status, $hdr) = do_req $srv, 'POST /tweak-status-code HTTP/1.0'; like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked'); - $c = tcp_start($srv, 'POST /restore-status-code HTTP/1.0'); - ($status, $hdr) = slurp_hdr($c); + ($status, $hdr) = do_req $srv, 'POST /restore-status-code HTTP/1.0'; is($status, $orig_200_status, 'original status restored'); } @@ -145,12 +141,11 @@ SKIP: { } if ('bad requests') { - $c = tcp_start($srv, 'GET /env_dump HTTP/1/1'); - ($status, $hdr) = slurp_hdr($c); + ($status, $hdr) = do_req $srv, 'GET /env_dump HTTP/1/1'; like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request'); $c = tcp_start($srv); - print $c 'GET /';; + print $c 'GET /'; my $buf = join('', (0..9), 'ab'); for (0..1023) { print $c $buf } print $c " HTTP/1.0\r\n\r\n"; @@ -308,12 +303,10 @@ EOM $wpid =~ s/\Apid=// or die; ok(CORE::kill(0, $wpid), 'worker PID retrieved'); - $c = tcp_start($srv, $req); - ($status, $hdr) = slurp_hdr($c); + ($status, $hdr) = do_req($srv, $req); like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds'); - $c = tcp_start($srv, 'GET /xxxxxx HTTP/1.0'); - ($status, $hdr) = slurp_hdr($c); + ($status, $hdr) = do_req($srv, 'GET /xxxxxx HTTP/1.0'); like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails'); } diff --git a/t/lib.perl b/t/lib.perl index 7de9e426..13e390d6 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -12,7 +12,8 @@ use File::Temp 0.19 (); # 0.19 for ->newdir our ($tmpdir, $errfh, $err_log); our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh $err_log - SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr); + SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr + do_req); my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!); $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1); @@ -182,6 +183,19 @@ sub unicorn { UnicornTest::AutoReap->new($pid); } +sub do_req ($@) { + my ($dst, @req) = @_; + my $c = ref($dst) ? tcp_start($dst, @req) : unix_start($dst, @req); + return $c if !wantarray; + my ($status, $hdr); + # read headers iff HTTP/1.x request, HTTP/0.9 remains supported + my ($first) = (join('', @req) =~ m!\A([^\r\n]+)!); + ($status, $hdr) = slurp_hdr($c) if $first =~ m{\s*HTTP/\S+$}; + my $bdy = do { local $/; <$c> }; + close $c; + ($status, $hdr, $bdy); +} + # automatically kill + reap children when this goes out-of-scope package UnicornTest::AutoReap; use v5.14; diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t index c7055c7e..543421da 100644 --- a/t/reload-bad-config.t +++ b/t/reload-bad-config.t @@ -25,9 +25,7 @@ EOM close $fh; my $ar = unicorn(qw(-E none -c), $u_conf, $ru, { 3 => $srv }); -my $c = tcp_start($srv, 'GET / HTTP/1.0'); -my ($status, $hdr) = slurp_hdr($c); -my $bdy = do { local $/; <$c> }; +my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0'); like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid at start'); is($bdy, "hello world\n", 'body matches expected'); @@ -47,9 +45,7 @@ ok(grep(/error reloading/, @l), 'got error reloading'); open $fh, '>', $err_log; close $fh; -$c = tcp_start($srv, 'GET / HTTP/1.0'); -($status, $hdr) = slurp_hdr($c); -$bdy = do { local $/; <$c> }; +($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0'); like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid afte reload'); is($bdy, "hello world\n", 'body matches expected after reload'); diff --git a/t/reopen-logs.t b/t/reopen-logs.t index e1bf524c..8a58c1b9 100644 --- a/t/reopen-logs.t +++ b/t/reopen-logs.t @@ -14,9 +14,7 @@ EOM close $fh; my $auto_reap = unicorn('-c', $u_conf, 't/reopen-logs.ru', { 3 => $srv } ); -my $c = tcp_start($srv, 'GET / HTTP/1.0'); -my ($status, $hdr) = slurp_hdr($c); -my $bdy = do { local $/; <$c> }; +my ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0'); is($bdy, "true\n", 'logs opened'); rename($err_log, "$err_log.rot"); @@ -31,9 +29,7 @@ while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 }; ok(-f $out_log, 'stdout_path recreated after USR1'); ok(-f $err_log, 'stderr_path recreated after USR1'); -$c = tcp_start($srv, 'GET / HTTP/1.0'); -($status, $hdr) = slurp_hdr($c); -$bdy = do { local $/; <$c> }; +($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0'); is($bdy, "true\n", 'logs reopened with sync==true'); $auto_reap->join('QUIT'); diff --git a/t/winch_ttin.t b/t/winch_ttin.t index 1a198778..509b118f 100644 --- a/t/winch_ttin.t +++ b/t/winch_ttin.t @@ -34,10 +34,8 @@ my $worker_nr = <$fh>; close $fh; is($worker_nr, '0', 'initial worker spawned'); -my $c = unix_start($u_sock, 'GET /pid HTTP/1.0'); -my ($status, $hdr) = slurp_hdr($c); +my ($status, $hdr, $worker_pid) = do_req($u_sock, 'GET /pid HTTP/1.0'); like($status, qr/ 200\b/, 'got 200 response'); -my $worker_pid = do { local $/; <$c> }; like($worker_pid, qr/\A[0-9]+\n\z/s, 'PID in response'); chomp $worker_pid; ok(kill(0, $worker_pid), 'worker_pid is valid'); @@ -57,11 +55,10 @@ $worker_nr = <$fh>; close $fh; is($worker_nr, '0', 'worker restarted'); -$c = unix_start($u_sock, 'GET /pid HTTP/1.0'); -($status, $hdr) = slurp_hdr($c); +($status, $hdr, my $new_worker_pid) = do_req($u_sock, 'GET /pid HTTP/1.0'); like($status, qr/ 200\b/, 'got 200 response'); -chomp(my $new_worker_pid = do { local $/; <$c> }); -like($new_worker_pid, qr/\A[0-9]+\z/, 'got new worker PID'); +like($new_worker_pid, qr/\A[0-9]+\n\z/, 'got new worker PID'); +chomp $new_worker_pid; ok(kill(0, $new_worker_pid), 'got a valid worker PID'); isnt($worker_pid, $new_worker_pid, 'worker PID changed'); diff --git a/t/working_directory.t b/t/working_directory.t index e7ff43a5..6c974720 100644 --- a/t/working_directory.t +++ b/t/working_directory.t @@ -52,10 +52,8 @@ END { $stop_daemon->(1) if defined $pid }; unicorn('-c', $u_conf)->join; # will daemonize chomp($pid = slurp("$tmpdir/pid")); -my $c = unix_start($u_sock, 'GET / HTTP/1.0'); -my ($status, $hdr) = slurp_hdr($c); -chomp(my $bdy = do { local $/; <$c> }); -is($bdy, 1, 'got expected $master_ppid'); +my ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0'); +is($bdy, "1\n", 'got expected $master_ppid'); $stop_daemon->(); check_stderr; @@ -69,10 +67,8 @@ if ('test without CLI switches in config.ru') { unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize chomp($pid = slurp("$tmpdir/pid")); - $c = unix_start($u_sock, 'GET / HTTP/1.0'); - ($status, $hdr) = slurp_hdr($c); - chomp($bdy = do { local $/; <$c> }); - is($bdy, 1, 'got expected $master_ppid'); + ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0'); + is($bdy, "1\n", 'got expected $master_ppid'); $stop_daemon->(); check_stderr; @@ -107,12 +103,9 @@ EOM my $srv = tcp_server; my $auto_reap = unicorn(qw(-c), $u_conf, qw(-I. fooapp.rb), { -C => '/', 3 => $srv }); - $c = tcp_start($srv, 'GET / HTTP/1.0'); - ($status, $hdr) = slurp_hdr($c); - chomp($bdy = do { local $/; <$c> }); + ($status, $hdr, $bdy) = do_req($srv, 'GET / HTTP/1.0'); is($bdy, "dir=$tmpdir/alt", 'fooapp.rb (w/o config.ru) w/ working_directory'); - close $c; $auto_reap->join('TERM'); is($?, 0, 'fooapp.rb process exited'); check_stderr; [-- Attachment #8: 0007-tests-use-more-common-variable-names-between-tests.patch --] [-- Type: text/x-diff, Size: 6507 bytes --] From 948f78403172657590d690b9255467b9ccb968cd Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Sun, 10 Sep 2023 09:31:44 +0000 Subject: [PATCH 07/11] tests: use more common variable names between tests Stuff like $u_conf, $daemon_pid, $pid_file, etc. will reduce cognitive overhead. --- t/active-unix-socket.t | 2 +- t/client_body_buffer_size.t | 6 ++---- t/heartbeat-timeout.t | 3 +-- t/integration.t | 5 ++--- t/lib.perl | 31 +++++++++++++++++++++++++++---- t/working_directory.t | 31 +++++-------------------------- 6 files changed, 38 insertions(+), 40 deletions(-) diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t index 4dcc8dc6..32cb0c2e 100644 --- a/t/active-unix-socket.t +++ b/t/active-unix-socket.t @@ -15,7 +15,7 @@ my $u2 = "$tmpdir/u2.sock"; print $fh <<EOM; pid "$tmpdir/u.pid" listen "$u1" -stderr_path "$tmpdir/err.log" +stderr_path "$err_log" EOM close $fh; diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t index 3067f284..d4799012 100644 --- a/t/client_body_buffer_size.t +++ b/t/client_body_buffer_size.t @@ -4,16 +4,14 @@ use v5.14; BEGIN { require './t/lib.perl' }; use autodie; -my $uconf = "$tmpdir/u.conf.rb"; - -open my $conf_fh, '>', $uconf; +open my $conf_fh, '>', $u_conf; $conf_fh->autoflush(1); print $conf_fh <<EOM; client_body_buffer_size 0 EOM my $srv = tcp_server(); my $host_port = tcp_host_port($srv); -my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $uconf); +my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $u_conf); my $ar = unicorn(@uarg, { 3 => $srv }); my ($c, $status, $hdr); my $mem_class = 'StringIO'; diff --git a/t/heartbeat-timeout.t b/t/heartbeat-timeout.t index ce1f7e16..694867a4 100644 --- a/t/heartbeat-timeout.t +++ b/t/heartbeat-timeout.t @@ -6,7 +6,6 @@ use autodie; use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC); mkdir "$tmpdir/alt"; my $srv = tcp_server(); -my $u_conf = "$tmpdir/u.conf.rb"; open my $fh, '>', $u_conf; print $fh <<EOM; pid "$tmpdir/pid" @@ -23,7 +22,7 @@ like($status, qr!\AHTTP/1\.[01] 200\b!, 'PID request succeeds'); like($wpid, qr/\A[0-9]+\z/, 'worker is running'); my $t0 = clock_gettime(CLOCK_MONOTONIC); -$c = tcp_start($srv, 'GET /block-forever HTTP/1.0'); +my $c = tcp_start($srv, 'GET /block-forever HTTP/1.0'); vec(my $rvec = '', fileno($c), 1) = 1; is(select($rvec, undef, undef, 6), 1, 'got readiness'); $c->blocking(0); diff --git a/t/integration.t b/t/integration.t index 13b07467..eb40ffc7 100644 --- a/t/integration.t +++ b/t/integration.t @@ -10,15 +10,14 @@ use autodie; our $srv = tcp_server(); our $host_port = tcp_host_port($srv); my $t0 = time; -my $conf = "$tmpdir/u.conf.rb"; -open my $conf_fh, '>', $conf; +open my $conf_fh, '>', $u_conf; $conf_fh->autoflush(1); my $u1 = "$tmpdir/u1"; print $conf_fh <<EOM; early_hints true listen "$u1" EOM -my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv }); +my $ar = unicorn(qw(-E none t/integration.ru -c), $u_conf, { 3 => $srv }); my $curl = which('curl'); my $fifo = "$tmpdir/fifo"; POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!"; diff --git a/t/lib.perl b/t/lib.perl index 13e390d6..244972bc 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -6,20 +6,43 @@ use v5.14; use parent qw(Exporter); use autodie; use Test::More; +use Time::HiRes qw(sleep); use IO::Socket::INET; use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD); use File::Temp 0.19 (); # 0.19 for ->newdir -our ($tmpdir, $errfh, $err_log); +our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid, + $pid_file); our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn - $tmpdir $errfh $err_log + $tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr - do_req); + do_req stop_daemon); my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!); $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1); $err_log = "$tmpdir/err.log"; +$pid_file = "$tmpdir/pid"; +$u_sock = "$tmpdir/u.sock"; +$u_conf = "$tmpdir/u.conf.rb"; open($errfh, '>>', $err_log); -END { diag slurp($err_log) if $tmpdir }; + +sub stop_daemon (;$) { + my ($is_END) = @_; + kill('TERM', $daemon_pid); + my $tries = 1000; + while (CORE::kill(0, $daemon_pid) && --$tries) { sleep(0.01) } + if ($is_END && CORE::kill(0, $daemon_pid)) { # after done_testing + CORE::kill('KILL', $daemon_pid); + die "daemon_pid=$daemon_pid did not die"; + } else { + ok(!CORE::kill(0, $daemon_pid), 'daemonized unicorn gone'); + undef $daemon_pid; + } +}; + +END { + diag slurp($err_log) if $tmpdir; + stop_daemon(1) if defined $daemon_pid; +}; sub check_stderr () { my @log = slurp($err_log); diff --git a/t/working_directory.t b/t/working_directory.t index 6c974720..f9254eb8 100644 --- a/t/working_directory.t +++ b/t/working_directory.t @@ -4,12 +4,10 @@ use v5.14; BEGIN { require './t/lib.perl' }; use autodie; mkdir "$tmpdir/alt"; -my $u_sock = "$tmpdir/u.sock"; my $ru = "$tmpdir/alt/config.ru"; -my $u_conf = "$tmpdir/u.conf.rb"; open my $fh, '>', $u_conf; print $fh <<EOM; -pid "$tmpdir/pid" +pid "$pid_file" preload_app true stderr_path "$err_log" working_directory "$tmpdir/alt" # the whole point of this test @@ -30,32 +28,13 @@ $common_ru EOM close $fh; -my $pid; -my $stop_daemon = sub { - my ($is_END) = @_; - kill('TERM', $pid); - my $tries = 1000; - while (CORE::kill(0, $pid) && --$tries) { - select undef, undef, undef, 0.01; - } - if ($is_END && CORE::kill(0, $pid)) { - CORE::kill('KILL', $pid); - die "daemonized PID=$pid did not die"; - } else { - ok(!CORE::kill(0, $pid), 'daemonized unicorn gone'); - undef $pid; - } -}; - -END { $stop_daemon->(1) if defined $pid }; - unicorn('-c', $u_conf)->join; # will daemonize -chomp($pid = slurp("$tmpdir/pid")); +chomp($daemon_pid = slurp($pid_file)); my ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0'); is($bdy, "1\n", 'got expected $master_ppid'); -$stop_daemon->(); +stop_daemon; check_stderr; if ('test without CLI switches in config.ru') { @@ -65,12 +44,12 @@ if ('test without CLI switches in config.ru') { close $fh; unicorn('-D', '-l', $u_sock, '-c', $u_conf)->join; # will daemonize - chomp($pid = slurp("$tmpdir/pid")); + chomp($daemon_pid = slurp($pid_file)); ($status, $hdr, $bdy) = do_req($u_sock, 'GET / HTTP/1.0'); is($bdy, "1\n", 'got expected $master_ppid'); - $stop_daemon->(); + stop_daemon; check_stderr; } [-- Attachment #9: 0008-tests-use-Time-HiRes-sleep-and-time-everywhere.patch --] [-- Type: text/x-diff, Size: 4106 bytes --] From dd9f2efeebf20cfa1def0ce92cb4e35a8b5c1580 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Sun, 10 Sep 2023 09:35:09 +0000 Subject: [PATCH 08/11] tests: use Time::HiRes `sleep' and `time' everywhere The time(2) syscall use by CORE::time is inaccurate[1]. It's also easier to read `sleep 0.01' rather than the longer `select' equivalent. [1] a6463151bd1db5b9 (httpdate: favor gettimeofday(2) over time(2) for correctness, 2023-06-01) --- t/active-unix-socket.t | 2 +- t/integration.t | 5 +++-- t/lib.perl | 4 ++-- t/reload-bad-config.t | 2 +- t/reopen-logs.t | 4 ++-- t/winch_ttin.t | 4 +--- 6 files changed, 10 insertions(+), 11 deletions(-) diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t index 32cb0c2e..ff731b5f 100644 --- a/t/active-unix-socket.t +++ b/t/active-unix-socket.t @@ -86,7 +86,7 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure'); 'fail to connect to u1'); for (1..50) { # wait for init process to reap worker kill(0, $worker_pid) or last; - select(undef, undef, undef, 0.011); + sleep 0.011; } ok(!kill(0, $worker_pid), 'worker gone after parent dies'); } diff --git a/t/integration.t b/t/integration.t index eb40ffc7..80485e44 100644 --- a/t/integration.t +++ b/t/integration.t @@ -77,8 +77,9 @@ SKIP: { # Date header check eval { require HTTP::Date } or skip "HTTP::Date missing: $@", 1; $d[0] =~ s/^Date: //i or die 'BUG: did not strip date: prefix'; my $t = HTTP::Date::str2time($d[0]); - ok($t >= $t0 && $t > 0 && $t <= time, 'valid date') or - diag(explain([$t, $!, \@d])); + my $now = time; + ok($t >= ($t0 - 1) && $t > 0 && $t <= ($now + 1), 'valid date') or + diag(explain(["t=$t t0=$t0 now=$now", $!, \@d])); }; diff --git a/t/lib.perl b/t/lib.perl index 244972bc..9254b23b 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -6,7 +6,7 @@ use v5.14; use parent qw(Exporter); use autodie; use Test::More; -use Time::HiRes qw(sleep); +use Time::HiRes qw(sleep time); use IO::Socket::INET; use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD); use File::Temp 0.19 (); # 0.19 for ->newdir @@ -15,7 +15,7 @@ our ($tmpdir, $errfh, $err_log, $u_sock, $u_conf, $daemon_pid, our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh $err_log $u_sock $u_conf $daemon_pid $pid_file SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr - do_req stop_daemon); + do_req stop_daemon sleep time); my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!); $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1); diff --git a/t/reload-bad-config.t b/t/reload-bad-config.t index 543421da..c023b88c 100644 --- a/t/reload-bad-config.t +++ b/t/reload-bad-config.t @@ -38,7 +38,7 @@ my @l; for (1..1000) { @l = grep(/(?:done|error) reloading/, slurp($err_log)) and last; - select undef, undef, undef, 0.011; + sleep 0.011; } diag slurp($err_log) if $ENV{V}; ok(grep(/error reloading/, @l), 'got error reloading'); diff --git a/t/reopen-logs.t b/t/reopen-logs.t index 8a58c1b9..76a4dbdf 100644 --- a/t/reopen-logs.t +++ b/t/reopen-logs.t @@ -23,8 +23,8 @@ rename($out_log, "$out_log.rot"); $auto_reap->do_kill('USR1'); my $tries = 1000; -while (!-f $err_log && --$tries) { select undef, undef, undef, 0.01 }; -while (!-f $out_log && --$tries) { select undef, undef, undef, 0.01 }; +while (!-f $err_log && --$tries) { sleep 0.01 }; +while (!-f $out_log && --$tries) { sleep 0.01 }; ok(-f $out_log, 'stdout_path recreated after USR1'); ok(-f $err_log, 'stderr_path recreated after USR1'); diff --git a/t/winch_ttin.t b/t/winch_ttin.t index 509b118f..c5079599 100644 --- a/t/winch_ttin.t +++ b/t/winch_ttin.t @@ -43,9 +43,7 @@ ok(kill(0, $worker_pid), 'worker_pid is valid'); ok(kill('WINCH', $pid), 'SIGWINCH can be sent'); my $tries = 1000; -while (CORE::kill(0, $worker_pid) && --$tries) { - select undef, undef, undef, 0.01; -} +while (CORE::kill(0, $worker_pid) && --$tries) { sleep 0.01 } ok(!CORE::kill(0, $worker_pid), 'worker not running'); ok(kill('TTIN', $pid), 'SIGTTIN to restart worker'); [-- Attachment #10: 0009-tests-fold-SO_KEEPALIVE-check-to-Perl-5-integration.patch --] [-- Type: text/x-diff, Size: 2675 bytes --] From b588ccbbf73547487f54fd1a9d5396d6848e8661 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Sun, 10 Sep 2023 19:21:05 +0000 Subject: [PATCH 09/11] tests: fold SO_KEEPALIVE check to Perl 5 integration No need to startup more processes than necessary. --- t/integration.t | 13 +++++++++++++ test/exec/test_exec.rb | 23 +---------------------- 2 files changed, 14 insertions(+), 22 deletions(-) diff --git a/t/integration.t b/t/integration.t index 80485e44..bea221ce 100644 --- a/t/integration.t +++ b/t/integration.t @@ -7,8 +7,16 @@ use v5.14; BEGIN { require './t/lib.perl' }; use autodie; +use Socket qw(SOL_SOCKET SO_KEEPALIVE); our $srv = tcp_server(); our $host_port = tcp_host_port($srv); + +if ('ensure Perl does not set SO_KEEPALIVE by default') { + my $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE); + unpack('i', $val) == 0 or + setsockopt($srv, SOL_SOCKET, SO_KEEPALIVE, pack('i', 0)); + $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE); +} my $t0 = time; open my $conf_fh, '>', $u_conf; $conf_fh->autoflush(1); @@ -71,6 +79,11 @@ is_deeply([ grep(/^X-R2: /, @$hdr) ], [ 'X-R2: a', 'X-R2: b', 'X-R2: c' ], 'rack 2 LF-delimited headers supported') or diag(explain($hdr)); +{ + my $val = getsockopt($srv, SOL_SOCKET, SO_KEEPALIVE); + is(unpack('i', $val), 1, 'SO_KEEPALIVE set on inherited socket'); +} + SKIP: { # Date header check my @d = grep(/^Date: /i, @$hdr); is(scalar(@d), 1, 'got one date header') or diag(explain(\@d)); diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb index 55f828e7..84944520 100644 --- a/test/exec/test_exec.rb +++ b/test/exec/test_exec.rb @@ -1,6 +1,5 @@ # -*- encoding: binary -*- - -# Copyright (c) 2009 Eric Wong +# Don't add to this file, new tests are in Perl 5. See t/README FLOCK_PATH = File.expand_path(__FILE__) require './test/test_helper' @@ -97,26 +96,6 @@ def teardown end end - def test_inherit_listener_unspecified - File.open("config.ru", "wb") { |fp| fp.write(HI) } - sock = TCPServer.new(@addr, @port) - sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0) - - pid = xfork do - redirect_test_io do - ENV['UNICORN_FD'] = sock.fileno.to_s - exec($unicorn_bin, sock.fileno => sock.fileno) - end - end - res = hit(["http://#@addr:#@port/"]) - assert_equal [ "HI\n" ], res - assert_shutdown(pid) - assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool, - 'unicorn should always set SO_KEEPALIVE on inherited sockets' - ensure - sock.close if sock - end - def test_working_directory_rel_path_config_file other = Tempfile.new('unicorn.wd') File.unlink(other.path) [-- Attachment #11: 0010-tests-move-broken-app-test-to-Perl-5-integration-tes.patch --] [-- Type: text/x-diff, Size: 2376 bytes --] From 7160f1b519aece0fe645d22a7d8fb954a43ad6fb Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Sun, 10 Sep 2023 19:37:32 +0000 Subject: [PATCH 10/11] tests: move broken app test to Perl 5 integration test Less Ruby means fewer incompatibilities to worry about with every new version. --- t/integration.ru | 1 + t/integration.t | 6 ++++++ test/unit/test_server.rb | 14 -------------- 3 files changed, 7 insertions(+), 14 deletions(-) diff --git a/t/integration.ru b/t/integration.ru index 086126ab..888833a9 100644 --- a/t/integration.ru +++ b/t/integration.ru @@ -98,6 +98,7 @@ def rack_input_tests(env) when '/pid'; [ 200, {}, [ "#$$\n" ] ] when '/early_hints_rack2'; early_hints(env, "r\n2") when '/early_hints_rack3'; early_hints(env, %w(r 3)) + when '/broken_app'; raise RuntimeError, 'hello' else '/'; [ 200, {}, [ env_dump(env) ] ] end # case PATH_INFO (GET) when 'POST' diff --git a/t/integration.t b/t/integration.t index bea221ce..ba17dd9e 100644 --- a/t/integration.t +++ b/t/integration.t @@ -118,6 +118,12 @@ SKIP: { is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '], 'nil header value accepted for broken apps') or diag(explain($hdr)); +check_stderr; +($status, $hdr, $bdy) = do_req($srv, 'GET /broken_app HTTP/1.0'); +like($status, qr!\AHTTP/1\.[0-1] 500\b!, 'got 500 error on broken endpoint'); +is($bdy, undef, 'no response body after exception'); +truncate($errfh, 0); + my $ck_early_hints = sub { my ($note) = @_; $c = unix_start($u1, 'GET /early_hints_rack2 HTTP/1.0'); diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb index 0a710d12..2af12eac 100644 --- a/test/unit/test_server.rb +++ b/test/unit/test_server.rb @@ -127,20 +127,6 @@ def test_after_reply sock.close end - def test_broken_app - teardown - app = lambda { |env| raise RuntimeError, "hello" } - # [200, {}, []] } - redirect_test_io do - @server = HttpServer.new(app, :listeners => [ "127.0.0.1:#@port"] ) - @server.start - end - sock = tcp_socket('127.0.0.1', @port) - sock.syswrite("GET / HTTP/1.0\r\n\r\n") - assert_match %r{\AHTTP/1.[01] 500\b}, sock.sysread(4096) - assert_nil sock.close - end - def test_simple_server results = hit(["http://localhost:#{@port}/test"]) assert_equal 'hello!\n', results[0], "Handler didn't really run" [-- Attachment #12: 0011-tests-fold-early-shutdown-tests-into-t-integration.t.patch --] [-- Type: text/x-diff, Size: 4527 bytes --] From 05028146b5e69c566663fdab9f8b92c6145a791a Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Sun, 10 Sep 2023 19:52:03 +0000 Subject: [PATCH 11/11] tests: fold early shutdown() tests into t/integration.t This means fewer redundant tests and more chances to notice Ruby incompatibilities. --- t/integration.t | 22 +++++++++++++++-- test/unit/test_server.rb | 53 ---------------------------------------- 2 files changed, 20 insertions(+), 55 deletions(-) diff --git a/t/integration.t b/t/integration.t index ba17dd9e..7310ff29 100644 --- a/t/integration.t +++ b/t/integration.t @@ -7,7 +7,7 @@ use v5.14; BEGIN { require './t/lib.perl' }; use autodie; -use Socket qw(SOL_SOCKET SO_KEEPALIVE); +use Socket qw(SOL_SOCKET SO_KEEPALIVE SHUT_WR); our $srv = tcp_server(); our $host_port = tcp_host_port($srv); @@ -209,6 +209,7 @@ SKIP: { defined($opt{overwrite}) and print { $c } ('x' x $opt{overwrite}); $c->flush or die $!; + shutdown($c, SHUT_WR); ($status, $hdr) = slurp_hdr($c); is(readline($c), $blob_hash, "$sub $path"); }; @@ -225,6 +226,8 @@ SKIP: { # ensure small overwrites don't get checksummed $ck_hash->('identity', '/rack_input', -s => $blob_size, overwrite => 1); # one extra byte + unlike(slurp($err_log), qr/ClientShutdown/, + 'no overreads after client SHUT_WR'); # excessive overwrite truncated $c = tcp_start($srv); @@ -238,8 +241,23 @@ SKIP: { $! = 0; while (print $c $buf and time < $end) { ++$n } ok($!, 'overwrite truncated') or diag "n=$n err=$! ".time; + undef $c; + } + + # client shutdown early + $c = tcp_start($srv); + $c->autoflush(0); + print $c "PUT /rack_input HTTP/1.0\r\nContent-Length: 16384\r\n\r\n"; + if (1) { + local $SIG{PIPE} = 'IGNORE'; + print $c 'too short body'; + shutdown($c, SHUT_WR); + vec(my $rvec = '', fileno($c), 1) = 1; + select($rvec, undef, undef, 10) or BAIL_OUT "timed out"; + my $buf = <$c>; + is($buf, undef, 'server aborted after client SHUT_WR'); + undef $c; } - undef $c; $curl // skip 'no curl found in PATH', 1; diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb index 2af12eac..7ffa48f0 100644 --- a/test/unit/test_server.rb +++ b/test/unit/test_server.rb @@ -132,59 +132,6 @@ def test_simple_server assert_equal 'hello!\n', results[0], "Handler didn't really run" end - def test_client_shutdown_writes - bs = 15609315 * rand - sock = tcp_socket('127.0.0.1', @port) - sock.syswrite("PUT /hello HTTP/1.1\r\n") - sock.syswrite("Host: example.com\r\n") - sock.syswrite("Transfer-Encoding: chunked\r\n") - sock.syswrite("Trailer: X-Foo\r\n") - sock.syswrite("\r\n") - sock.syswrite("%x\r\n" % [ bs ]) - sock.syswrite("F" * bs) - sock.syswrite("\r\n0\r\nX-") - "Foo: bar\r\n\r\n".each_byte do |x| - sock.syswrite x.chr - sleep 0.05 - end - # we wrote the entire request before shutting down, server should - # continue to process our request and never hit EOFError on our sock - sock.shutdown(Socket::SHUT_WR) - buf = sock.read - assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/, 2).last - next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/")) - assert_equal 'hello!\n', next_client - lines = File.readlines("test_stderr.#$$.log") - assert lines.grep(/^Unicorn::ClientShutdown: /).empty? - assert_nil sock.close - end - - def test_client_shutdown_write_truncates - bs = 15609315 * rand - sock = tcp_socket('127.0.0.1', @port) - sock.syswrite("PUT /hello HTTP/1.1\r\n") - sock.syswrite("Host: example.com\r\n") - sock.syswrite("Transfer-Encoding: chunked\r\n") - sock.syswrite("Trailer: X-Foo\r\n") - sock.syswrite("\r\n") - sock.syswrite("%x\r\n" % [ bs ]) - sock.syswrite("F" * (bs / 2.0)) - - # shutdown prematurely, this will force the server to abort - # processing on us even during app dispatch - sock.shutdown(Socket::SHUT_WR) - IO.select([sock], nil, nil, 60) or raise "Timed out" - buf = sock.read - assert_equal "", buf - next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/")) - assert_equal 'hello!\n', next_client - lines = File.readlines("test_stderr.#$$.log") - lines = lines.grep(/^Unicorn::ClientShutdown: bytes_read=\d+/) - assert_equal 1, lines.size - assert_match %r{\AUnicorn::ClientShutdown: bytes_read=\d+ true$}, lines[0] - assert_nil sock.close - end - def test_client_malformed_body bs = 15653984 sock = tcp_socket('127.0.0.1', @port)
[-- Attachment #1: Type: text/plain, Size: 1658 bytes --] Explanation (and bulk of the work is in patch 2/3). I wrote patch 1/3 over 4 years ago since it was simple and didn't rely on "newer" Ruby 2.3 features. Patch 2/3 completes the change and actually depends on 2.5+; while patch 3/3 updates the gemspec, docs and dependencies for Ruby 2.5+. I haven't actually used Ruby 2.5 in a while, but I'm still on Ruby 2.7 since that's what I have in Debian bullseye (oldstable). Patch 2/3 could use an extra set of eyes since it's fairly big, but passes all tests. Note: I can't benchmark anything since I have limited (shared) hardware Eric Wong (3): remove kgio from all read(2) and write(2) wrappers kill off remaining kgio uses update dependency to Ruby 2.5+ HACKING | 2 +- README | 2 +- lib/unicorn.rb | 3 +- lib/unicorn/http_request.rb | 18 ++++----- lib/unicorn/http_server.rb | 38 +++++++---------- lib/unicorn/oob_gc.rb | 4 +- lib/unicorn/socket_helper.rb | 50 +++-------------------- lib/unicorn/stream_input.rb | 20 +++++---- lib/unicorn/worker.rb | 10 ++--- lib/unicorn/write_splat.rb | 7 ---- t/README | 2 +- t/oob_gc.ru | 3 -- t/oob_gc_path.ru | 3 -- test/unit/test_request.rb | 47 ++++++++------------- test/unit/test_socket_helper.rb | 72 +++++++-------------------------- test/unit/test_stream_input.rb | 2 +- test/unit/test_tee_input.rb | 2 +- unicorn.gemspec | 5 +-- 18 files changed, 87 insertions(+), 203 deletions(-) delete mode 100644 lib/unicorn/write_splat.rb [-- Attachment #2: 0001-remove-kgio-from-all-read-2-and-write-2-wrappers.patch --] [-- Type: text/x-diff, Size: 5330 bytes --] From 36ba7f971c571031101c0b718724bdcb06dd7e03 Mon Sep 17 00:00:00 2001 From: Eric Wong <e@80x24.org> Date: Sun, 26 May 2019 22:15:44 +0000 Subject: [RFC 1/3] remove kgio from all read(2) and write(2) wrappers It's fairly easy given unicorn was designed with synchronous I/O in mind. The overhead of backtraces from EOFError on readpartial should be rare given our requirement to only accept requests from fast, reliable clients on LAN (e.g. nginx or yet-another-horribly-named-server). --- lib/unicorn/http_request.rb | 4 ++-- lib/unicorn/http_server.rb | 8 +++++--- lib/unicorn/stream_input.rb | 20 ++++++++++++-------- lib/unicorn/worker.rb | 4 ++-- 4 files changed, 21 insertions(+), 15 deletions(-) diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb index e3ad592..8bca60a 100644 --- a/lib/unicorn/http_request.rb +++ b/lib/unicorn/http_request.rb @@ -74,11 +74,11 @@ def read(socket) e['REMOTE_ADDR'] = socket.kgio_addr # short circuit the common case with small GET requests first - socket.kgio_read!(16384, buf) + socket.readpartial(16384, buf) if parse.nil? # Parser is not done, queue up more data to read and continue parsing # an Exception thrown from the parser will throw us out of the loop - false until add_parse(socket.kgio_read!(16384)) + false until add_parse(socket.readpartial(16384)) end check_client_connection(socket) if @@check_client_connection diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index f1b4a54..2f1eb1b 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -389,12 +389,13 @@ def master_sleep(sec) # the Ruby itself and not require a separate malloc (on 32-bit MRI 1.9+). # Most reads are only one byte here and uncommon, so it's not worth a # persistent buffer, either: - @self_pipe[0].kgio_tryread(11) + @self_pipe[0].read_nonblock(11, exception: false) end def awaken_master return if $$ != @master_pid - @self_pipe[1].kgio_trywrite('.') # wakeup master process from select + # wakeup master process from select + @self_pipe[1].write_nonblock('.', exception: false) end # reaps all unreaped workers @@ -565,7 +566,8 @@ def handle_error(client, e) 500 end if code - client.kgio_trywrite(err_response(code, @request.response_start_sent)) + code = err_response(code, @request.response_start_sent) + client.write_nonblock(code, exception: false) end client.close rescue diff --git a/lib/unicorn/stream_input.rb b/lib/unicorn/stream_input.rb index 41d28a0..9246f73 100644 --- a/lib/unicorn/stream_input.rb +++ b/lib/unicorn/stream_input.rb @@ -49,8 +49,7 @@ def read(length = nil, rv = '') to_read = length - @rbuf.size rv.replace(@rbuf.slice!(0, @rbuf.size)) until to_read == 0 || eof? || (rv.size > 0 && @chunked) - @socket.kgio_read(to_read, @buf) or eof! - filter_body(@rbuf, @buf) + filter_body(@rbuf, @socket.readpartial(to_read, @buf)) rv << @rbuf to_read -= @rbuf.size end @@ -61,6 +60,8 @@ def read(length = nil, rv = '') read_all(rv) end rv + rescue EOFError + return eof! end # :call-seq: @@ -83,9 +84,10 @@ def gets begin @rbuf.sub!(re, '') and return $1 return @rbuf.empty? ? nil : @rbuf.slice!(0, @rbuf.size) if eof? - @socket.kgio_read(@@io_chunk_size, @buf) or eof! - filter_body(once = '', @buf) + filter_body(once = '', @socket.readpartial(@@io_chunk_size, @buf)) @rbuf << once + rescue EOFError + return eof! end while true end @@ -107,14 +109,15 @@ def each def eof? if @parser.body_eof? while @chunked && ! @parser.parse - once = @socket.kgio_read(@@io_chunk_size) or eof! - @buf << once + @buf << @socket.readpartial(@@io_chunk_size) end @socket = nil true else false end + rescue EOFError + return eof! end def filter_body(dst, src) @@ -127,10 +130,11 @@ def read_all(dst) dst.replace(@rbuf) @socket or return until eof? - @socket.kgio_read(@@io_chunk_size, @buf) or eof! - filter_body(@rbuf, @buf) + filter_body(@rbuf, @socket.readpartial(@@io_chunk_size, @buf)) dst << @rbuf end + rescue EOFError + return eof! ensure @rbuf.clear end diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb index 5ddf379..ad5814c 100644 --- a/lib/unicorn/worker.rb +++ b/lib/unicorn/worker.rb @@ -65,7 +65,7 @@ def soft_kill(sig) # :nodoc: end # writing and reading 4 bytes on a pipe is atomic on all POSIX platforms # Do not care in the odd case the buffer is full, here. - @master.kgio_trywrite([signum].pack('l')) + @master.write_nonblock([signum].pack('l'), exception: false) rescue Errno::EPIPE # worker will be reaped soon end @@ -73,7 +73,7 @@ def soft_kill(sig) # :nodoc: # this only runs when the Rack app.call is not running # act like a listener def kgio_tryaccept # :nodoc: - case buf = @to_io.kgio_tryread(4) + case buf = @to_io.read_nonblock(4, exception: false) when String # unpack the buffer and trigger the signal handler signum = buf.unpack('l') [-- Attachment #3: 0002-kill-off-remaining-kgio-uses.patch --] [-- Type: text/x-diff, Size: 25476 bytes --] From 03ec6e69fc6219a40aa8db368abe53017cd164e3 Mon Sep 17 00:00:00 2001 From: Eric Wong <bofh@yhbt.net> Date: Tue, 5 Sep 2023 06:43:20 +0000 Subject: [RFC 2/3] kill off remaining kgio uses kgio is an extra download and shared object which costs users bandwidth, disk space, startup time and memory. Ruby 2.3+ provides `Socket#accept_nonblock(exception: false)' support in addition to `exception: false' support in IO#*_nonblock methods from Ruby 2.1. We no longer distinguish between TCPServer and UNIXServer as separate classes internally; instead favoring the `Socket' class of Ruby for both. This allows us to use `Socket#accept_nonblock' and get a populated `Addrinfo' object off accept4(2)/accept(2) without resorting to a getpeername(2) syscall (kgio avoided getpeername(2) in the same way). The downside is there's more Ruby-level argument passing and stack usage on our end with HttpRequest#read_headers (formerly HttpRequest#read). I chose this tradeoff since advancements in Ruby itself can theoretically mitigate the cost of argument passing, while syscalls are a high fixed cost given modern CPU vulnerability mitigations. Note: no benchmarks have been run since I don't have a suitable system. --- lib/unicorn.rb | 3 +- lib/unicorn/http_request.rb | 14 +++---- lib/unicorn/http_server.rb | 30 +++++--------- lib/unicorn/oob_gc.rb | 4 +- lib/unicorn/socket_helper.rb | 50 +++-------------------- lib/unicorn/worker.rb | 6 +-- t/oob_gc.ru | 3 -- t/oob_gc_path.ru | 3 -- test/unit/test_request.rb | 47 ++++++++------------- test/unit/test_socket_helper.rb | 72 +++++++-------------------------- test/unit/test_stream_input.rb | 2 +- test/unit/test_tee_input.rb | 2 +- unicorn.gemspec | 1 - 13 files changed, 61 insertions(+), 176 deletions(-) diff --git a/lib/unicorn.rb b/lib/unicorn.rb index b817b77..564cb30 100644 --- a/lib/unicorn.rb +++ b/lib/unicorn.rb @@ -1,7 +1,6 @@ # -*- encoding: binary -*- require 'etc' require 'stringio' -require 'kgio' require 'raindrops' require 'io/wait' @@ -112,7 +111,7 @@ def self.log_error(logger, prefix, exc) F_SETPIPE_SZ = 1031 if RUBY_PLATFORM =~ /linux/ def self.pipe # :nodoc: - Kgio::Pipe.new.each do |io| + IO.pipe.each do |io| # shrink pipes to minimize impact on /proc/sys/fs/pipe-user-pages-soft # limits. if defined?(F_SETPIPE_SZ) diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb index 8bca60a..ab3bd6e 100644 --- a/lib/unicorn/http_request.rb +++ b/lib/unicorn/http_request.rb @@ -61,7 +61,7 @@ def self.check_client_connection=(bool) # returns an environment hash suitable for Rack if successful # This does minimal exception trapping and it is up to the caller # to handle any socket errors (e.g. user aborted upload). - def read(socket) + def read_headers(socket, ai) e = env # From https://www.ietf.org/rfc/rfc3875: @@ -71,7 +71,7 @@ def read(socket) # identify the client for the immediate request to the server; # that client may be a proxy, gateway, or other intermediary # acting on behalf of the actual source client." - e['REMOTE_ADDR'] = socket.kgio_addr + e['REMOTE_ADDR'] = ai.unix? ? '127.0.0.1' : ai.ip_address # short circuit the common case with small GET requests first socket.readpartial(16384, buf) @@ -81,7 +81,7 @@ def read(socket) false until add_parse(socket.readpartial(16384)) end - check_client_connection(socket) if @@check_client_connection + check_client_connection(socket, ai) if @@check_client_connection e['rack.input'] = 0 == content_length ? NULL_IO : @@input_class.new(socket, self) @@ -107,8 +107,8 @@ def hijacked? if Raindrops.const_defined?(:TCP_Info) TCPI = Raindrops::TCP_Info.allocate - def check_client_connection(socket) # :nodoc: - if Unicorn::TCPClient === socket + def check_client_connection(socket, ai) # :nodoc: + if ai.ip? # Raindrops::TCP_Info#get!, #state (reads struct tcp_info#tcpi_state) raise Errno::EPIPE, "client closed connection".freeze, EMPTY_ARRAY if closed_state?(TCPI.get!(socket).state) @@ -152,8 +152,8 @@ def closed_state?(state) # :nodoc: # Ruby 2.2+ can show struct tcp_info as a string Socket::Option#inspect. # Not that efficient, but probably still better than doing unnecessary # work after a client gives up. - def check_client_connection(socket) # :nodoc: - if Unicorn::TCPClient === socket && @@tcpi_inspect_ok + def check_client_connection(socket, ai) # :nodoc: + if @@tcpi_inspect_ok && ai.ip? opt = socket.getsockopt(Socket::IPPROTO_TCP, Socket::TCP_INFO).inspect if opt =~ /\bstate=(\S+)/ raise Errno::EPIPE, "client closed connection".freeze, diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index 2f1eb1b..ed5bbf1 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -111,9 +111,7 @@ def initialize(app, options = {}) @worker_data = if worker_data = ENV['UNICORN_WORKER'] worker_data = worker_data.split(',').map!(&:to_i) - worker_data[1] = worker_data.slice!(1..2).map do |i| - Kgio::Pipe.for_fd(i) - end + worker_data[1] = worker_data.slice!(1..2).map { |i| IO.for_fd(i) } worker_data end end @@ -243,10 +241,6 @@ def listen(address, opt = {}.merge(listener_opts[address] || {})) tries = opt[:tries] || 5 begin io = bind_listen(address, opt) - unless Kgio::TCPServer === io || Kgio::UNIXServer === io - io.autoclose = false - io = server_cast(io) - end logger.info "listening on addr=#{sock_name(io)} fd=#{io.fileno}" LISTENERS << io io @@ -594,9 +588,9 @@ def e100_response_write(client, env) # once a client is accepted, it is processed in its entirety here # in 3 easy steps: read request, call app, write app response - def process_client(client) + def process_client(client, ai) @request = Unicorn::HttpRequest.new - env = @request.read(client) + env = @request.read_headers(client, ai) if early_hints env["rack.early_hints"] = lambda do |headers| @@ -708,10 +702,9 @@ def worker_loop(worker) reopen = reopen_worker_logs(worker.nr) if reopen worker.tick = time_now.to_i while sock = ready.shift - # Unicorn::Worker#kgio_tryaccept is not like accept(2) at all, - # but that will return false - if client = sock.kgio_tryaccept - process_client(client) + client_ai = sock.accept_nonblock(exception: false) + if client_ai != :wait_readable + process_client(*client_ai) worker.tick = time_now.to_i end break if reopen @@ -809,7 +802,6 @@ def redirect_io(io, path) def inherit_listeners! # inherit sockets from parents, they need to be plain Socket objects - # before they become Kgio::UNIXServer or Kgio::TCPServer inherited = ENV['UNICORN_FD'].to_s.split(',') immortal = [] @@ -825,8 +817,6 @@ def inherit_listeners! inherited.map! do |fd| io = Socket.for_fd(fd.to_i) @immortal << io if immortal.include?(fd) - io.autoclose = false - io = server_cast(io) set_server_sockopt(io, listener_opts[sock_name(io)]) logger.info "inherited addr=#{sock_name(io)} fd=#{io.fileno}" io @@ -835,11 +825,9 @@ def inherit_listeners! config_listeners = config[:listeners].dup LISTENERS.replace(inherited) - # we start out with generic Socket objects that get cast to either - # Kgio::TCPServer or Kgio::UNIXServer objects; but since the Socket - # objects share the same OS-level file descriptor as the higher-level - # *Server objects; we need to prevent Socket objects from being - # garbage-collected + # we only use generic Socket objects for aggregate Socket#accept_nonblock + # return value [ Socket, Addrinfo ]. This allows us to avoid having to + # make getpeername(2) syscalls later on to fill in env['REMOTE_ADDR'] config_listeners -= listener_names if config_listeners.empty? && LISTENERS.empty? config_listeners << Unicorn::Const::DEFAULT_LISTEN diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb index 91a8e51..db9f2cb 100644 --- a/lib/unicorn/oob_gc.rb +++ b/lib/unicorn/oob_gc.rb @@ -65,8 +65,8 @@ def self.new(app, interval = 5, path = %r{\A/}) end #:stopdoc: - def process_client(client) - super(client) # Unicorn::HttpServer#process_client + def process_client(*args) + super(*args) # Unicorn::HttpServer#process_client env = instance_variable_get(:@request).env if OOBGC_PATH =~ env['PATH_INFO'] && ((@@nr -= 1) <= 0) @@nr = OOBGC_INTERVAL diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb index c2ba75e..06ec2b2 100644 --- a/lib/unicorn/socket_helper.rb +++ b/lib/unicorn/socket_helper.rb @@ -3,32 +3,6 @@ require 'socket' module Unicorn - - # Instead of using a generic Kgio::Socket for everything, - # tag TCP sockets so we can use TCP_INFO under Linux without - # incurring extra syscalls for Unix domain sockets. - # TODO: remove these when we remove kgio - TCPClient = Class.new(Kgio::Socket) # :nodoc: - class TCPSrv < Kgio::TCPServer # :nodoc: - def kgio_tryaccept # :nodoc: - super(TCPClient) - end - end - - if IO.instance_method(:write).arity == 1 # Ruby <= 2.4 - require 'unicorn/write_splat' - UNIXClient = Class.new(Kgio::Socket) # :nodoc: - class UNIXSrv < Kgio::UNIXServer # :nodoc: - include Unicorn::WriteSplat - def kgio_tryaccept # :nodoc: - super(UNIXClient) - end - end - TCPClient.__send__(:include, Unicorn::WriteSplat) - else # Ruby 2.5+ - UNIXSrv = Kgio::UNIXServer - end - module SocketHelper # internal interface @@ -105,7 +79,7 @@ def set_tcp_sockopt(sock, opt) def set_server_sockopt(sock, opt) opt = DEFAULTS.merge(opt || {}) - TCPSocket === sock and set_tcp_sockopt(sock, opt) + set_tcp_sockopt(sock, opt) if sock.local_address.ip? rcvbuf, sndbuf = opt.values_at(:rcvbuf, :sndbuf) if rcvbuf || sndbuf @@ -149,7 +123,9 @@ def bind_listen(address = '0.0.0.0:8080', opt = {}) end old_umask = File.umask(opt[:umask] || 0) begin - UNIXSrv.new(address) + s = Socket.new(:UNIX, :STREAM) + s.bind(Socket.sockaddr_un(address)) + s ensure File.umask(old_umask) end @@ -177,8 +153,7 @@ def new_tcp_server(addr, port, opt) sock.setsockopt(:SOL_SOCKET, :SO_REUSEPORT, 1) end sock.bind(Socket.pack_sockaddr_in(port, addr)) - sock.autoclose = false - TCPSrv.for_fd(sock.fileno) + sock end # returns rfc2732-style (e.g. "[::1]:666") addresses for IPv6 @@ -194,10 +169,6 @@ def tcp_name(sock) def sock_name(sock) case sock when String then sock - when UNIXServer - Socket.unpack_sockaddr_un(sock.getsockname) - when TCPServer - tcp_name(sock) when Socket begin tcp_name(sock) @@ -210,16 +181,5 @@ def sock_name(sock) end module_function :sock_name - - # casts a given Socket to be a TCPServer or UNIXServer - def server_cast(sock) - begin - Socket.unpack_sockaddr_in(sock.getsockname) - TCPSrv.for_fd(sock.fileno) - rescue ArgumentError - UNIXSrv.for_fd(sock.fileno) - end - end - end # module SocketHelper end # module Unicorn diff --git a/lib/unicorn/worker.rb b/lib/unicorn/worker.rb index ad5814c..4af31be 100644 --- a/lib/unicorn/worker.rb +++ b/lib/unicorn/worker.rb @@ -71,8 +71,8 @@ def soft_kill(sig) # :nodoc: end # this only runs when the Rack app.call is not running - # act like a listener - def kgio_tryaccept # :nodoc: + # act like Socket#accept_nonblock(exception: false) + def accept_nonblock(*_unused) # :nodoc: case buf = @to_io.read_nonblock(4, exception: false) when String # unpack the buffer and trigger the signal handler @@ -82,7 +82,7 @@ def kgio_tryaccept # :nodoc: when nil # EOF: master died, but we are at a safe place to exit fake_sig(:QUIT) when :wait_readable # keep waiting - return false + return :wait_readable end while true # loop, as multiple signals may be sent end diff --git a/t/oob_gc.ru b/t/oob_gc.ru index c253540..224cb06 100644 --- a/t/oob_gc.ru +++ b/t/oob_gc.ru @@ -7,9 +7,6 @@ # Mock GC.start def GC.start - ObjectSpace.each_object(Kgio::Socket) do |x| - x.closed? or abort "not closed #{x}" - end $gc_started = true end run lambda { |env| diff --git a/t/oob_gc_path.ru b/t/oob_gc_path.ru index af8e3b9..7f40601 100644 --- a/t/oob_gc_path.ru +++ b/t/oob_gc_path.ru @@ -7,9 +7,6 @@ # Mock GC.start def GC.start - ObjectSpace.each_object(Kgio::Socket) do |x| - x.closed? or abort "not closed #{x}" - end $gc_started = true end run lambda { |env| diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb index 7f22b24..53ae944 100644 --- a/test/unit/test_request.rb +++ b/test/unit/test_request.rb @@ -10,14 +10,9 @@ class RequestTest < Test::Unit::TestCase - class MockRequest < StringIO - alias_method :readpartial, :sysread - alias_method :kgio_read!, :sysread - alias_method :read_nonblock, :sysread - def kgio_addr - '127.0.0.1' - end - end + MockRequest = Class.new(StringIO) + + AI = Addrinfo.new(Socket.sockaddr_un('/unicorn/sucks')) def setup @request = HttpRequest.new @@ -30,7 +25,7 @@ def setup def test_options client = MockRequest.new("OPTIONS * HTTP/1.1\r\n" \ "Host: foo\r\n\r\n") - env = @request.read(client) + env = @request.read_headers(client, AI) assert_equal '', env['REQUEST_PATH'] assert_equal '', env['PATH_INFO'] assert_equal '*', env['REQUEST_URI'] @@ -40,7 +35,7 @@ def test_options def test_absolute_uri_with_query client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \ "Host: foo\r\n\r\n") - env = @request.read(client) + env = @request.read_headers(client, AI) assert_equal '/x', env['REQUEST_PATH'] assert_equal '/x', env['PATH_INFO'] assert_equal 'y=z', env['QUERY_STRING'] @@ -50,7 +45,7 @@ def test_absolute_uri_with_query def test_absolute_uri_with_fragment client = MockRequest.new("GET http://e:3/x#frag HTTP/1.1\r\n" \ "Host: foo\r\n\r\n") - env = @request.read(client) + env = @request.read_headers(client, AI) assert_equal '/x', env['REQUEST_PATH'] assert_equal '/x', env['PATH_INFO'] assert_equal '', env['QUERY_STRING'] @@ -61,7 +56,7 @@ def test_absolute_uri_with_fragment def test_absolute_uri_with_query_and_fragment client = MockRequest.new("GET http://e:3/x?a=b#frag HTTP/1.1\r\n" \ "Host: foo\r\n\r\n") - env = @request.read(client) + env = @request.read_headers(client, AI) assert_equal '/x', env['REQUEST_PATH'] assert_equal '/x', env['PATH_INFO'] assert_equal 'a=b', env['QUERY_STRING'] @@ -73,7 +68,7 @@ def test_absolute_uri_unsupported_schemes %w(ssh+http://e/ ftp://e/x http+ssh://e/x).each do |abs_uri| client = MockRequest.new("GET #{abs_uri} HTTP/1.1\r\n" \ "Host: foo\r\n\r\n") - assert_raises(HttpParserError) { @request.read(client) } + assert_raises(HttpParserError) { @request.read_headers(client, AI) } end end @@ -81,7 +76,7 @@ def test_x_forwarded_proto_https client = MockRequest.new("GET / HTTP/1.1\r\n" \ "X-Forwarded-Proto: https\r\n" \ "Host: foo\r\n\r\n") - env = @request.read(client) + env = @request.read_headers(client, AI) assert_equal "https", env['rack.url_scheme'] assert_kind_of Array, @lint.call(env) end @@ -90,7 +85,7 @@ def test_x_forwarded_proto_http client = MockRequest.new("GET / HTTP/1.1\r\n" \ "X-Forwarded-Proto: http\r\n" \ "Host: foo\r\n\r\n") - env = @request.read(client) + env = @request.read_headers(client, AI) assert_equal "http", env['rack.url_scheme'] assert_kind_of Array, @lint.call(env) end @@ -99,14 +94,14 @@ def test_x_forwarded_proto_invalid client = MockRequest.new("GET / HTTP/1.1\r\n" \ "X-Forwarded-Proto: ftp\r\n" \ "Host: foo\r\n\r\n") - env = @request.read(client) + env = @request.read_headers(client, AI) assert_equal "http", env['rack.url_scheme'] assert_kind_of Array, @lint.call(env) end def test_rack_lint_get client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n") - env = @request.read(client) + env = @request.read_headers(client, AI) assert_equal "http", env['rack.url_scheme'] assert_equal '127.0.0.1', env['REMOTE_ADDR'] assert_kind_of Array, @lint.call(env) @@ -114,7 +109,7 @@ def test_rack_lint_get def test_no_content_stringio client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n") - env = @request.read(client) + env = @request.read_headers(client, AI) assert_equal StringIO, env['rack.input'].class end @@ -122,7 +117,7 @@ def test_zero_content_stringio client = MockRequest.new("PUT / HTTP/1.1\r\n" \ "Content-Length: 0\r\n" \ "Host: foo\r\n\r\n") - env = @request.read(client) + env = @request.read_headers(client, AI) assert_equal StringIO, env['rack.input'].class end @@ -130,7 +125,7 @@ def test_real_content_not_stringio client = MockRequest.new("PUT / HTTP/1.1\r\n" \ "Content-Length: 1\r\n" \ "Host: foo\r\n\r\n") - env = @request.read(client) + env = @request.read_headers(client, AI) assert_equal Unicorn::TeeInput, env['rack.input'].class end @@ -141,7 +136,7 @@ def test_rack_lint_put "Content-Length: 5\r\n" \ "\r\n" \ "abcde") - env = @request.read(client) + env = @request.read_headers(client, AI) assert ! env.include?(:http_body) assert_kind_of Array, @lint.call(env) end @@ -152,14 +147,6 @@ def test_rack_lint_big_put buf = (' ' * bs).freeze length = bs * count client = Tempfile.new('big_put') - def client.kgio_addr; '127.0.0.1'; end - def client.kgio_read(*args) - readpartial(*args) - rescue EOFError - end - def client.kgio_read!(*args) - readpartial(*args) - end client.syswrite( "PUT / HTTP/1.1\r\n" \ "Host: foo\r\n" \ @@ -167,7 +154,7 @@ def client.kgio_read!(*args) "\r\n") count.times { assert_equal bs, client.syswrite(buf) } assert_equal 0, client.sysseek(0) - env = @request.read(client) + env = @request.read_headers(client, AI) assert ! env.include?(:http_body) assert_equal length, env['rack.input'].size count.times { diff --git a/test/unit/test_socket_helper.rb b/test/unit/test_socket_helper.rb index 62d5a3a..a446f06 100644 --- a/test/unit/test_socket_helper.rb +++ b/test/unit/test_socket_helper.rb @@ -24,7 +24,8 @@ def test_bind_listen_tcp port = unused_port @test_addr @tcp_listener_name = "#@test_addr:#{port}" @tcp_listener = bind_listen(@tcp_listener_name) - assert TCPServer === @tcp_listener + assert Socket === @tcp_listener + assert @tcp_listener.local_address.ip? assert_equal @tcp_listener_name, sock_name(@tcp_listener) end @@ -38,10 +39,10 @@ def test_bind_listen_options { :backlog => 16, :rcvbuf => 4096, :sndbuf => 4096 } ].each do |opts| tcp_listener = bind_listen(tcp_listener_name, opts) - assert TCPServer === tcp_listener + assert tcp_listener.local_address.ip? tcp_listener.close unix_listener = bind_listen(unix_listener_name, opts) - assert UNIXServer === unix_listener + assert unix_listener.local_address.unix? unix_listener.close end end @@ -52,11 +53,13 @@ def test_bind_listen_unix @unix_listener_path = tmp.path File.unlink(@unix_listener_path) @unix_listener = bind_listen(@unix_listener_path) - assert UNIXServer === @unix_listener + assert Socket === @unix_listener + assert @unix_listener.local_address.unix? assert_equal @unix_listener_path, sock_name(@unix_listener) assert File.readable?(@unix_listener_path), "not readable" assert File.writable?(@unix_listener_path), "not writable" assert_equal 0777, File.umask + assert_equal @unix_listener, bind_listen(@unix_listener) ensure File.umask(old_umask) end @@ -67,7 +70,6 @@ def test_bind_listen_unix_umask @unix_listener_path = tmp.path File.unlink(@unix_listener_path) @unix_listener = bind_listen(@unix_listener_path, :umask => 077) - assert UNIXServer === @unix_listener assert_equal @unix_listener_path, sock_name(@unix_listener) assert_equal 0140700, File.stat(@unix_listener_path).mode assert_equal 0777, File.umask @@ -75,28 +77,6 @@ def test_bind_listen_unix_umask File.umask(old_umask) end - def test_bind_listen_unix_idempotent - test_bind_listen_unix - a = bind_listen(@unix_listener) - assert_equal a.fileno, @unix_listener.fileno - unix_server = server_cast(@unix_listener) - assert UNIXServer === unix_server - a = bind_listen(unix_server) - assert_equal a.fileno, unix_server.fileno - assert_equal a.fileno, @unix_listener.fileno - end - - def test_bind_listen_tcp_idempotent - test_bind_listen_tcp - a = bind_listen(@tcp_listener) - assert_equal a.fileno, @tcp_listener.fileno - tcp_server = server_cast(@tcp_listener) - assert TCPServer === tcp_server - a = bind_listen(tcp_server) - assert_equal a.fileno, tcp_server.fileno - assert_equal a.fileno, @tcp_listener.fileno - end - def test_bind_listen_unix_rebind test_bind_listen_unix new_listener = nil @@ -107,14 +87,18 @@ def test_bind_listen_unix_rebind File.unlink(@unix_listener_path) new_listener = bind_listen(@unix_listener_path) - assert UNIXServer === new_listener assert new_listener.fileno != @unix_listener.fileno assert_equal sock_name(new_listener), sock_name(@unix_listener) assert_equal @unix_listener_path, sock_name(new_listener) pid = fork do - client = server_cast(new_listener).accept - client.syswrite('abcde') - exit 0 + begin + client, _ = new_listener.accept + client.syswrite('abcde') + exit 0 + rescue => e + warn "#{e.message} (#{e.class})" + exit 1 + end end s = unix_socket(@unix_listener_path) IO.select([s]) @@ -123,32 +107,6 @@ def test_bind_listen_unix_rebind assert status.success? end - def test_server_cast - test_bind_listen_unix - test_bind_listen_tcp - unix_listener_socket = Socket.for_fd(@unix_listener.fileno) - assert Socket === unix_listener_socket - @unix_server = server_cast(unix_listener_socket) - assert_equal @unix_listener.fileno, @unix_server.fileno - assert UNIXServer === @unix_server - assert_equal(@unix_server.path, @unix_listener.path, - "##{@unix_server.path} != #{@unix_listener.path}") - assert File.socket?(@unix_server.path) - assert_equal @unix_listener_path, sock_name(@unix_server) - - tcp_listener_socket = Socket.for_fd(@tcp_listener.fileno) - assert Socket === tcp_listener_socket - @tcp_server = server_cast(tcp_listener_socket) - assert_equal @tcp_listener.fileno, @tcp_server.fileno - assert TCPServer === @tcp_server - assert_equal @tcp_listener_name, sock_name(@tcp_server) - end - - def test_sock_name - test_server_cast - sock_name(@unix_server) - end - def test_tcp_defer_accept_default return unless defined?(TCP_DEFER_ACCEPT) port = unused_port @test_addr diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb index 2a14135..7986ca7 100644 --- a/test/unit/test_stream_input.rb +++ b/test/unit/test_stream_input.rb @@ -9,7 +9,7 @@ def setup @rs = "\n" $/ == "\n" or abort %q{test broken if \$/ != "\\n"} @env = {} - @rd, @wr = Kgio::UNIXSocket.pair + @rd, @wr = UNIXSocket.pair @rd.sync = @wr.sync = true @start_pid = $$ end diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb index 6f5bc8a..607ce87 100644 --- a/test/unit/test_tee_input.rb +++ b/test/unit/test_tee_input.rb @@ -10,7 +10,7 @@ class TeeInput < Unicorn::TeeInput class TestTeeInput < Test::Unit::TestCase def setup - @rd, @wr = Kgio::UNIXSocket.pair + @rd, @wr = UNIXSocket.pair @rd.sync = @wr.sync = true @start_pid = $$ @rs = "\n" diff --git a/unicorn.gemspec b/unicorn.gemspec index 7bb1154..85183d9 100644 --- a/unicorn.gemspec +++ b/unicorn.gemspec @@ -36,7 +36,6 @@ # won't have descriptive text, only the numeric status. s.add_development_dependency(%q<rack>) - s.add_dependency(%q<kgio>, '~> 2.6') s.add_dependency(%q<raindrops>, '~> 0.7') s.add_development_dependency('test-unit', '~> 3.0') [-- Attachment #4: 0003-update-dependency-to-Ruby-2.5.patch --] [-- Type: text/x-diff, Size: 3257 bytes --] From c67ebf96edc8ca691dfc556d4813fed242fe77ca Mon Sep 17 00:00:00 2001 From: Eric Wong <bofh@yhbt.net> Date: Tue, 5 Sep 2023 09:14:11 +0000 Subject: [RFC 3/3] update dependency to Ruby 2.5+ We actually need Ruby 2.3+ for `accept_nonblock(exception: false)'; and (AFAIK) we can't easily use a subclass of `Socket' while using Socket#accept_nonblock to inject WriteSplat support for `IO#write(*multi_args)' So just depend on Ruby 2.5+ since all my Ruby is already on the already-ancient Ruby 2.7+ anyways. --- HACKING | 2 +- README | 2 +- lib/unicorn/write_splat.rb | 7 ------- t/README | 2 +- unicorn.gemspec | 4 ++-- 5 files changed, 5 insertions(+), 12 deletions(-) delete mode 100644 lib/unicorn/write_splat.rb diff --git a/HACKING b/HACKING index 020209e..5aca83e 100644 --- a/HACKING +++ b/HACKING @@ -60,7 +60,7 @@ becomes unavailable. === Ruby/C Compatibility -We target C Ruby 2.0 and later. We need the Ruby +We target C Ruby 2.5 and later. We need the Ruby implementation to support fork, exec, pipe, UNIX signals, access to integer file descriptors and ability to use unlinked files. diff --git a/README b/README index 5411003..7e29daf 100644 --- a/README +++ b/README @@ -12,7 +12,7 @@ both the the request and response in between unicorn and slow clients. cut out everything that is better supported by the operating system, {nginx}[https://nginx.org/] or {Rack}[https://rack.github.io/]. -* Compatible with Ruby 2.0.0 and later. +* Compatible with Ruby 2.5 and later. * Process management: unicorn will reap and restart workers that die from broken apps. There is no need to manage multiple processes diff --git a/lib/unicorn/write_splat.rb b/lib/unicorn/write_splat.rb deleted file mode 100644 index 7e6e363..0000000 --- a/lib/unicorn/write_splat.rb +++ /dev/null @@ -1,7 +0,0 @@ -# -*- encoding: binary -*- -# compatibility module for Ruby <= 2.4, remove when we go Ruby 2.5+ -module Unicorn::WriteSplat # :nodoc: - def write(*arg) # :nodoc: - super(arg.join('')) - end -end diff --git a/t/README b/t/README index d09c715..7bd093d 100644 --- a/t/README +++ b/t/README @@ -14,7 +14,7 @@ Old tests are in Bourne shell and slowly being ported to Perl 5. == Requirements -* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/] +* {Ruby 2.5.0+}[https://www.ruby-lang.org/en/] * {Perl 5.14+}[https://www.perl.org/] # your distro should have it * {GNU make}[https://www.gnu.org/software/make/] * {curl}[https://curl.haxx.se/] diff --git a/unicorn.gemspec b/unicorn.gemspec index 85183d9..e7e3ef7 100644 --- a/unicorn.gemspec +++ b/unicorn.gemspec @@ -25,11 +25,11 @@ s.homepage = 'https://yhbt.net/unicorn/' s.test_files = test_files - # 2.0.0 is the minimum supported version. We don't specify + # 2.5.0 is the minimum supported version. We don't specify # a maximum version to make it easier to test pre-releases, # but we do warn users if they install unicorn on an untested # version in extconf.rb - s.required_ruby_version = ">= 2.0.0" + s.required_ruby_version = ">= 2.5.0" # We do not have a hard dependency on rack, it's possible to load # things which respond to #call. HTTP status lines in responses
Jean Boussier <jean.boussier@gmail.com> wrote: > Hello Eric, > > Do you have any plans to merge this patch as well that the similar kgio one? Already merged a while back: https://yhbt.net/unicorn.git/63c85c4282d15e22bd32a905883d2d0e149619d1/s/ Not I don't want to have to scramble for other changes in case there's more incompatibilities introduced by Ruby (or Rack). I'm also wary of increasing download counts due to MFA requirements (both explained in more detail @ https://yhbt.net/kgio-public/20230828005810.M622269@dcvr/ )
Hello Eric, Do you have any plans to merge this patch as well that the similar kgio one?
Since Rack::Chunked is gone in Rack 3.1+ and no longer loaded by default, we've learned to chunk HTTP/1.1 responses w/o Content-Length to allow clients to detect truncated responses. Unfortunately, there exist broken HTTP clients which advertise "HTTP/1.1" in the request but cannot parse chunked responses. Thus, we must continue to send unchunked responses as we have in prior releases if that's what clients expect. That is, chunked responses are opt-in unless RACK_ENV is "deployment" or "development". It doesn't matter if clients are in the wrong: they've worked this way for 14 years and we must do everything in our power to avoid breaking existing expectations on upgrades. I hate adding config directives, but breaking changes are even worse since users upgrading unicorn often have no easy way to to fix broken clients even if they're on the same LAN. --- Of course, if users upgrading to unicorn could fix everything; they wouldn't need unicorn at all :P unicorn was created to support broken code and now exists to perpetuate broken code into eternity. Documentation/unicorn.1 | 1 + lib/unicorn.rb | 2 + lib/unicorn/configurator.rb | 17 ++++++- lib/unicorn/http_response.rb | 2 +- lib/unicorn/http_server.rb | 3 +- t/integration.ru | 7 +++ t/integration.t | 88 +++++++++++++++++++++++++++++++++--- 7 files changed, 110 insertions(+), 10 deletions(-) diff --git a/Documentation/unicorn.1 b/Documentation/unicorn.1 index b2c5e70..502f44a 100644 --- a/Documentation/unicorn.1 +++ b/Documentation/unicorn.1 @@ -69,6 +69,7 @@ options. Disables loading middleware implied by RACK_ENV. This bypasses the configuration documented in the RACK ENVIRONMENT section, but still allows RACK_ENV to be used for application/framework\-specific purposes. +This also affects the "chunk_response" config file directive in unicorn 7.0+ .RS .RE .SH RACKUP COMPATIBILITY OPTIONS diff --git a/lib/unicorn.rb b/lib/unicorn.rb index b817b77..e7b2806 100644 --- a/lib/unicorn.rb +++ b/lib/unicorn.rb @@ -78,7 +78,9 @@ def self.builder(ru, op) # middlewares will need ContentLength middleware. case ENV["RACK_ENV"] when "development" + server.chunk_response = true if server.chunk_response.nil? when "deployment" + server.chunk_response = true if server.chunk_response.nil? middleware.delete(:ShowExceptions) middleware.delete(:Lint) else diff --git a/lib/unicorn/configurator.rb b/lib/unicorn/configurator.rb index ecdf03e..bbc0448 100644 --- a/lib/unicorn/configurator.rb +++ b/lib/unicorn/configurator.rb @@ -59,6 +59,7 @@ class Unicorn::Configurator :check_client_connection => false, :rewindable_input => true, :client_body_buffer_size => Unicorn::Const::MAX_BODY, + :chunk_response => nil, } #:startdoc: @@ -129,6 +130,19 @@ def [](key) # :nodoc: set[key] end + # Whether or not to chunk eligible HTTP/1.1 responses. This is + # necessary for Rack 3.1+ users where the Rack::Chunked middleware + # no longer exists. + # Default: +true+ if +default_middleware+ is +true+ AND + # RACK_ENV is either +development+ or +deployment+. + # It is +false+ otherwise to support broken clients advertising + # HTTP/1.1 but lacking the ability to parse chunked responses. + # +chunk_response+ only exists since unicorn 7.0+ (the first release + # with Rack 3.x support). + def chunk_response(bool) + set_bool(:chunk_response, bool) + end + # sets object to the +obj+ Logger-like object. The new Logger-like # object must respond to the following methods: # * debug @@ -270,7 +284,8 @@ def worker_processes(nr) end # sets whether to add default middleware in the development and - # deployment RACK_ENVs. + # deployment RACK_ENVs. This also affects the +chunk_response+ + # directive in unicorn 7.0+ # # default_middleware is only available in unicorn 5.5.0+ def default_middleware(bool) diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb index 0ed0ae3..c73efe9 100644 --- a/lib/unicorn/http_response.rb +++ b/lib/unicorn/http_response.rb @@ -68,7 +68,7 @@ def http_response_write(socket, status, headers, body, append_header(buf, key, value) end end - if !hijack && !term && req.chunkable_response? + if !hijack && !term && req.chunkable_response? && @chunk_response do_chunk = true buf << "Transfer-Encoding: chunked\r\n".freeze end diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index f1b4a54..93d7141 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -15,7 +15,7 @@ class Unicorn::HttpServer :before_fork, :after_fork, :before_exec, :listener_opts, :preload_app, :orig_app, :config, :ready_pipe, :user, - :default_middleware, :early_hints + :default_middleware, :early_hints, :chunk_response attr_writer :after_worker_exit, :after_worker_ready, :worker_exec attr_reader :pid, :logger @@ -69,6 +69,7 @@ class Unicorn::HttpServer # incoming requests on the socket. def initialize(app, options = {}) @app = app + @chunk_response = nil @reexec_pid = 0 @default_middleware = true options = options.dup diff --git a/t/integration.ru b/t/integration.ru index 086126a..d8a8178 100644 --- a/t/integration.ru +++ b/t/integration.ru @@ -85,6 +85,12 @@ def rack_input_tests(env) [ 200, h, [ dig.hexdigest ] ] end +class NoArray + def each + yield "HI\n" + end +end + run(lambda do |env| case env['REQUEST_METHOD'] when 'GET' @@ -98,6 +104,7 @@ def rack_input_tests(env) when '/pid'; [ 200, {}, [ "#$$\n" ] ] when '/early_hints_rack2'; early_hints(env, "r\n2") when '/early_hints_rack3'; early_hints(env, %w(r 3)) + when '/no-ary'; [ 200, {}, NoArray.new ] else '/'; [ 200, {}, [ env_dump(env) ] ] end # case PATH_INFO (GET) when 'POST' diff --git a/t/integration.t b/t/integration.t index bb2ab51..115a761 100644 --- a/t/integration.t +++ b/t/integration.t @@ -22,6 +22,13 @@ my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv }); my $curl = which('curl'); my $fifo = "$tmpdir/fifo"; POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!"; +my $wait_fifo = sub { + open my $fifo_fh, '<', $fifo; + my $wpid = readline($fifo_fh); + like($wpid, qr/\Apid=\d+\z/a , 'new worker ready'); + $wpid; +}; + my %PUT = ( chunked_md5 => sub { my ($in, $out, $path, %opt) = @_; @@ -176,6 +183,78 @@ if ('bad requests') { like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)'); } +my $tmp = { 3 => tcp_server() }; +my $no_ary = "GET /no-ary HTTP/1.1\r\nHost: example.com\r\n\r\n"; +if (diag('chunk_response is off by default w/ RACK_ENV=none') || 1) { + print { $c = tcp_start($srv) } $no_ary; + ($status, $hdr) = slurp_hdr($c); + unlike("@$hdr", qr/Transfer-Encoding/i, + 'no Transfer-Encoding for RACK_ENV=none despite HTTP/1.1'); + local $/; + is(readline($c), "HI\n", 'unchunked body response'); +} + +# pretend we have Rack::Chunked for RACK_ENV=(deployment|development) +for my $rack_env (qw(deployment development)) { + my $cfg = "$tmpdir/nochunk.conf.rb"; + open my $fh, '>', $cfg; + my $u = unicorn('-E', $rack_env, qw(t/integration.ru -c), $cfg, $tmp); + $c = tcp_start($tmp->{3}); + print $c $no_ary; + ($status, $hdr) = slurp_hdr($c); + like("@$hdr", qr/Transfer-Encoding/i, + "Transfer-Encoding set by default for RACK_ENV=$rack_env"); + is(do { local $/; readline($c) }, + "3\r\nHI\n\r\n0\r\n\r\n", 'chunked body response'); + + print $fh <<EOM; +chunk_response false +after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } } +EOM + close $fh; + $u->do_kill('HUP'); + $wait_fifo->(); + $c = tcp_start($tmp->{3}); + print $c $no_ary; + ($status, $hdr) = slurp_hdr($c); + unlike("@$hdr", qr/Transfer-Encoding/i, + "RACK_ENV=$rack_env w/o chunk_response"); + is(do { local $/; readline($c) }, + "HI\n", 'unchunked body response'); +} + +if (diag('chunk_response true w/ RACK_ENV=none') || 1) { + my $cfg = "$tmpdir/chunk.conf.rb"; + open my $fh, '>', $cfg; + print $fh "chunk_response true\n"; + close $fh; + my $u = unicorn(qw(-E none t/integration.ru -c), $cfg, $tmp); + $c = tcp_start($tmp->{3}); + print $c $no_ary; + ($status, $hdr) = slurp_hdr($c); + like("@$hdr", qr/Transfer-Encoding/i, + "Transfer-Encoding set by chunk_response false"); + is(do { local $/; readline($c) }, + "3\r\nHI\n\r\n0\r\n\r\n", 'chunked body response'); + + # reset to default: + open $fh, '>', $cfg; + print $fh <<EOM; +after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } } +EOM + close $fh; + $u->do_kill('HUP'); + $wait_fifo->(); + + $c = tcp_start($tmp->{3}); + print $c $no_ary; + ($status, $hdr) = slurp_hdr($c); + unlike("@$hdr", qr/Transfer-Encoding/i, + 'chunk_response false after HUP reset'); + is(do { local $/; readline($c) }, + "HI\n", 'unchunked body response after HUP reset'); +} + # input tests my ($blob_size, $blob_hash); SKIP: { @@ -287,9 +366,7 @@ check_client_connection true after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } } EOM $ar->do_kill('HUP'); - open my $fifo_fh, '<', $fifo; - my $wpid = readline($fifo_fh); - like($wpid, qr/\Apid=\d+\z/a , 'new worker ready'); + $wait_fifo->(); $ck_early_hints->('ccc on'); } @@ -301,10 +378,7 @@ if ('max_header_len internal API') { Unicorn::HttpParser.max_header_len = $len EOM $ar->do_kill('HUP'); - open my $fifo_fh, '<', $fifo; - my $wpid = readline($fifo_fh); - like($wpid, qr/\Apid=\d+\z/a , 'new worker ready'); - close $fifo_fh; + my $wpid = $wait_fifo->(); $wpid =~ s/\Apid=// or die; ok(CORE::kill(0, $wpid), 'worker PID retrieved');
Ragel 6.10 on FreeBSD 12.4 amd64 complains and fails on this, yet the same Ragel version on Debian 11.x i386 and amd64 never has. I suspect this can fix compatibility on s390x, arm64, armel, and armhf Debian builds: https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=s390x&ver=6.1.0-1&stamp=1687156375&file=log https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=arm64&ver=6.1.0-1&stamp=1687156478&file=log https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=armel&ver=6.1.0-1&stamp=1687156619&file=log https://buildd.debian.org/status/fetch.php?pkg=unicorn&arch=armhf&ver=6.1.0-1&stamp=1687156807&file=log Fixes: d5fbbf547203061b (Add some tolerance (RFC2616 sec. 19.3), 2016-10-20) --- ext/unicorn_http/unicorn_http_common.rl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ext/unicorn_http/unicorn_http_common.rl b/ext/unicorn_http/unicorn_http_common.rl index 0988b54..507e570 100644 --- a/ext/unicorn_http/unicorn_http_common.rl +++ b/ext/unicorn_http/unicorn_http_common.rl @@ -4,7 +4,7 @@ #### HTTP PROTOCOL GRAMMAR # line endings - CRLF = ("\r\n" | "\n"); + CRLF = ("\r\n" | "\n"); # character types CTL = (cntrl | 127);
[-- Attachment #1: Type: text/plain, Size: 43 bytes --] > Attached patches to reduce SMTP Oops :x [-- Attachment #2: 0001-t-lib.perl-ignore-errors-from-accept_filter-on-FreeB.patch --] [-- Type: text/x-diff, Size: 899 bytes --] From 8271bafb85f75b927f0ea15ec73fc0b1e714665e Mon Sep 17 00:00:00 2001 From: EW <bofh@yhbt.net> Date: Tue, 6 Jun 2023 10:09:24 +0000 Subject: [PATCH 1/4] t/lib.perl: FreeBSD: ignore accf_* messages Testers may not have accf_http loaded nor the permissions to run `kldload accf_http', thus we must ignore these messages. --- t/lib.perl | 2 ++ 1 file changed, 2 insertions(+) diff --git a/t/lib.perl b/t/lib.perl index 2685c3b4..fe3404ba 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -22,6 +22,8 @@ sub check_stderr () { my @log = slurp("$tmpdir/err.log"); diag("@log") if $ENV{V}; my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log)); + @err = grep(!/failed to set accept_filter=/, @err); + @err = grep(!/perhaps accf_.*? needs to be loaded/, @err); is_deeply(\@err, [], 'no unexpected errors in stderr'); is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr'); } [-- Attachment #3: 0002-t-active-unix-socket-sleep-for-init-8-to-reap-worker.patch --] [-- Type: text/x-diff, Size: 1012 bytes --] From a29364769d59e7bc0c67ad045af25f349ae913e8 Mon Sep 17 00:00:00 2001 From: EW <bofh@yhbt.net> Date: Tue, 6 Jun 2023 10:09:25 +0000 Subject: [PATCH 2/4] t/active-unix-socket: sleep for init(8) to reap worker Unfortunately, we need a sleep loop here since kill(2) succeeds on zombies and init(8) doesn't reap the worker soon enough on a FreeBSD VM. --- t/active-unix-socket.t | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t index 4e11837a..4dcc8dc6 100644 --- a/t/active-unix-socket.t +++ b/t/active-unix-socket.t @@ -84,6 +84,10 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure'); ok(-S $u1, 'socket stayed after SIGKILL'); is(IO::Socket::UNIX->new(Peer => $u1, Type => SOCK_STREAM), undef, 'fail to connect to u1'); + for (1..50) { # wait for init process to reap worker + kill(0, $worker_pid) or last; + select(undef, undef, undef, 0.011); + } ok(!kill(0, $worker_pid), 'worker gone after parent dies'); } [-- Attachment #4: 0003-tests-handle-assignment-deprecation.patch --] [-- Type: text/x-diff, Size: 4710 bytes --] From b988e0779814a73876a4a06df0a90a3f85fb08c8 Mon Sep 17 00:00:00 2001 From: Eric Wong <bofh@yhbt.net> Date: Tue, 6 Jun 2023 11:02:29 +0000 Subject: [PATCH 3/4] tests: handle $/ assignment deprecation ...by testing less. `env["rack.input"].gets' users are out-of-luck if they want anything other than "\n" or `nil', I suppose... `$/' is non-thread-local and thus non-thread-safe, which doesn't affect unicorn itself, but Ruby deprecates it for single-threaded code, too, unfortunately. Rack::Lint doesn't allow separator arguments for #gets, either, so we can't support that, either... --- test/unit/test_stream_input.rb | 25 ++++++++++++++++--------- test/unit/test_tee_input.rb | 19 +++++++++---------- 2 files changed, 25 insertions(+), 19 deletions(-) diff --git a/test/unit/test_stream_input.rb b/test/unit/test_stream_input.rb index 1a07ec3a..2a14135b 100644 --- a/test/unit/test_stream_input.rb +++ b/test/unit/test_stream_input.rb @@ -6,7 +6,8 @@ class TestStreamInput < Test::Unit::TestCase def setup - @rs = $/ + @rs = "\n" + $/ == "\n" or abort %q{test broken if \$/ != "\\n"} @env = {} @rd, @wr = Kgio::UNIXSocket.pair @rd.sync = @wr.sync = true @@ -15,7 +16,6 @@ def setup def teardown return if $$ != @start_pid - $/ = @rs @rd.close rescue nil @wr.close rescue nil Process.waitall @@ -54,11 +54,18 @@ def test_gets_multiline end def test_gets_empty_rs - $/ = nil r = init_request("a\nb\n\n") si = Unicorn::StreamInput.new(@rd, r) - assert_equal "a\nb\n\n", si.gets - assert_nil si.gets + pid = fork do # to avoid $/ warning (hopefully) + $/ = nil + @rd.close + @wr.write(si.gets) + @wr.close + end + @wr.close + assert_equal "a\nb\n\n", @rd.read + pid, status = Process.waitpid2(pid) + assert_predicate status, :success? end def test_read_with_equal_len @@ -90,21 +97,21 @@ def test_big_body_multi end def test_gets_long - r = init_request("hello", 5 + (4096 * 4 * 3) + "#$/foo#$/".size) + r = init_request("hello", 5 + (4096 * 4 * 3) + "#{@rs}foo#{@rs}".size) si = Unicorn::StreamInput.new(@rd, r) status = line = nil pid = fork { @rd.close 3.times { @wr.write("ffff" * 4096) } - @wr.write "#$/foo#$/" + @wr.write "#{@rs}foo#{@rs}" @wr.close } @wr.close line = si.gets assert_equal(4096 * 4 * 3 + 5 + $/.size, line.size) - assert_equal("hello" << ("ffff" * 4096 * 3) << "#$/", line) + assert_equal("hello" << ("ffff" * 4096 * 3) << "#{@rs}", line) line = si.gets - assert_equal "foo#$/", line + assert_equal "foo#{@rs}", line assert_nil si.gets pid, status = Process.waitpid2(pid) assert status.success? diff --git a/test/unit/test_tee_input.rb b/test/unit/test_tee_input.rb index 4647e661..6f5bc8a7 100644 --- a/test/unit/test_tee_input.rb +++ b/test/unit/test_tee_input.rb @@ -9,17 +9,16 @@ class TeeInput < Unicorn::TeeInput end class TestTeeInput < Test::Unit::TestCase - def setup - @rs = $/ @rd, @wr = Kgio::UNIXSocket.pair @rd.sync = @wr.sync = true @start_pid = $$ + @rs = "\n" + $/ == "\n" or abort %q{test broken if \$/ != "\\n"} end def teardown return if $$ != @start_pid - $/ = @rs @rd.close rescue nil @wr.close rescue nil begin @@ -37,38 +36,38 @@ def check_tempfiles end def test_gets_long - r = init_request("hello", 5 + (4096 * 4 * 3) + "#$/foo#$/".size) + r = init_request("hello", 5 + (4096 * 4 * 3) + "#{@rs}foo#{@rs}".size) ti = TeeInput.new(@rd, r) status = line = nil pid = fork { @rd.close 3.times { @wr.write("ffff" * 4096) } - @wr.write "#$/foo#$/" + @wr.write "#{@rs}foo#{@rs}" @wr.close } @wr.close line = ti.gets assert_equal(4096 * 4 * 3 + 5 + $/.size, line.size) - assert_equal("hello" << ("ffff" * 4096 * 3) << "#$/", line) + assert_equal("hello" << ("ffff" * 4096 * 3) << "#{@rs}", line) line = ti.gets - assert_equal "foo#$/", line + assert_equal "foo#{@rs}", line assert_nil ti.gets pid, status = Process.waitpid2(pid) assert status.success? end def test_gets_short - r = init_request("hello", 5 + "#$/foo".size) + r = init_request("hello", 5 + "#{@rs}foo".size) ti = TeeInput.new(@rd, r) status = line = nil pid = fork { @rd.close - @wr.write "#$/foo" + @wr.write "#{@rs}foo" @wr.close } @wr.close line = ti.gets - assert_equal("hello#$/", line) + assert_equal("hello#{@rs}", line) line = ti.gets assert_equal "foo", line assert_nil ti.gets [-- Attachment #5: 0004-tests-ensure-t-random_blob-exists-before-Perl-tests.patch --] [-- Type: text/x-diff, Size: 887 bytes --] From 42028bf5b0327f7e8816ef294d215ae6bb085fc6 Mon Sep 17 00:00:00 2001 From: Eric Wong <bofh@yhbt.net> Date: Tue, 6 Jun 2023 11:44:29 +0000 Subject: [PATCH 4/4] tests: ensure t/random_blob exists before Perl tests Allow overriding `PROVE=' while we're at it, too; since development installations of Perl5 may name it `prove5.$MINOR' or similar. --- GNUmakefile | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/GNUmakefile b/GNUmakefile index eab90829..70e7e108 100644 --- a/GNUmakefile +++ b/GNUmakefile @@ -11,6 +11,7 @@ RSYNC = rsync OLDDOC = olddoc RDOC = rdoc INSTALL = install +PROVE = prove GIT-VERSION-FILE: .FORCE-GIT-VERSION-FILE @./GIT-VERSION-GEN @@ -141,8 +142,8 @@ t/random_blob: test-integration: $(T_sh) -test-prove: - prove -vw +test-prove: t/random_blob + $(PROVE) -vw check: test-require test test-integration test-all: check
Nothing affecting core code, just some portability and future-proofing changes. Attached patches to reduce SMTP traffic. Reminder: you are encouraged to unsubscribe using the List-Unsubscribe header. IMAP, NNTP, POP3 and git access to archives are available and more reliable: https://yhbt.net/unicorn-public/_/text/mirror/ Or give up on using unicorn and/or Ruby entirely :P Eric Wong (4): t/lib.perl: ignore errors from accept_filter on FreeBSD t/active-unix-socket: sleep for init(8) to reap worker tests: handle $/ assignment deprecation tests: ensure t/random_blob exists before Perl tests GNUmakefile | 5 +++-- t/active-unix-socket.t | 4 ++++ t/lib.perl | 2 ++ test/unit/test_stream_input.rb | 25 ++++++++++++++++--------- test/unit/test_tee_input.rb | 19 +++++++++---------- 5 files changed, 34 insertions(+), 21 deletions(-)
From: Eric Wong <bofh@yhbt.net> It looks like Ruby 3.3+ will hide rb_io_t internals and get rid of the venerable `GetOpenFile' macro in favor of `rb_io_descriptor'. `rb_io_descriptor' has been public API since Ruby 3.1 and should be safe to use, and is necessary for `raindrops' (a dependency of ours): https://yhbt.net/raindrops-public/20230609104805.39022-1-samuel.williams@oriontransfer.co.nz/ https://bugs.ruby-lang.org/issues/19057#note-17 We'll also avoid an unnecessary call to `rb_io_get_io' in `get_readers' since `epio' (aka `self') can only be of the Unicorn::Waiter IO subclass. However, we must still use `rb_io_get_io' when handling non-self args in `prep_readers'. --- Note: I've only tested this patch on 2.7 atm, ENOSPC. ext/unicorn_http/epollexclusive.h | 27 ++++++++++++++++++--------- ext/unicorn_http/extconf.rb | 4 +++- 2 files changed, 21 insertions(+), 10 deletions(-) diff --git a/ext/unicorn_http/epollexclusive.h b/ext/unicorn_http/epollexclusive.h index 8f4ea9a0..c74a7799 100644 --- a/ext/unicorn_http/epollexclusive.h +++ b/ext/unicorn_http/epollexclusive.h @@ -22,6 +22,18 @@ #endif #if USE_EPOLL +#if defined(HAVE_RB_IO_DESCRIPTOR) /* Ruby 3.1+ */ +# define my_fileno(io) rb_io_descriptor(io) +#else /* Ruby <3.1 */ +static int my_fileno(VALUE io) +{ + rb_io_t *fptr; + GetOpenFile(io, fptr); + rb_io_check_closed(fptr); + return fptr->fd; +} +#endif /* Ruby <3.1 */ + /* * :nodoc: * returns IO object if EPOLLEXCLUSIVE works and arms readers @@ -38,9 +50,8 @@ static VALUE prep_readers(VALUE cls, VALUE readers) Check_Type(readers, T_ARRAY); for (i = 0; i < RARRAY_LEN(readers); i++) { - int rc; + int rc, fd; struct epoll_event e; - rb_io_t *fptr; VALUE io = rb_ary_entry(readers, i); e.data.u64 = i; /* the reason readers shouldn't change */ @@ -53,9 +64,8 @@ static VALUE prep_readers(VALUE cls, VALUE readers) * cycles on maintaining level-triggering. */ e.events = EPOLLEXCLUSIVE | EPOLLIN; - io = rb_io_get_io(io); - GetOpenFile(io, fptr); - rc = epoll_ctl(epfd, EPOLL_CTL_ADD, fptr->fd, &e); + fd = my_fileno(rb_io_get_io(io)); + rc = epoll_ctl(epfd, EPOLL_CTL_ADD, fd, &e); if (rc < 0) rb_sys_fail("epoll_ctl"); } return epio; @@ -65,7 +75,7 @@ static VALUE prep_readers(VALUE cls, VALUE readers) #if USE_EPOLL struct ep_wait { struct epoll_event event; - rb_io_t *fptr; + int epfd; int timeout_msec; }; @@ -78,7 +88,7 @@ static void *do_wait(void *ptr) /* runs w/o GVL */ * at-a-time (c.f. fs/eventpoll.c in linux.git, it's quite * easy-to-understand for anybody familiar with Ruby C). */ - return (void *)(long)epoll_wait(epw->fptr->fd, &epw->event, 1, + return (void *)(long)epoll_wait(epw->epfd, &epw->event, 1, epw->timeout_msec); } @@ -92,9 +102,8 @@ get_readers(VALUE epio, VALUE ready, VALUE readers, VALUE timeout_msec) Check_Type(ready, T_ARRAY); Check_Type(readers, T_ARRAY); - epio = rb_io_get_io(epio); - GetOpenFile(epio, epw.fptr); + epw.epfd = my_fileno(epio); epw.timeout_msec = NUM2INT(timeout_msec); n = (long)rb_thread_call_without_gvl(do_wait, &epw, RUBY_UBF_IO, NULL); if (n < 0) { diff --git a/ext/unicorn_http/extconf.rb b/ext/unicorn_http/extconf.rb index 80d00e56..11099cd0 100644 --- a/ext/unicorn_http/extconf.rb +++ b/ext/unicorn_http/extconf.rb @@ -33,5 +33,7 @@ message("no, needs Ruby 2.6+\n") end -have_func('epoll_create1', %w(sys/epoll.h)) +if have_func('epoll_create1', %w(sys/epoll.h)) + have_func('rb_io_descriptor') # Ruby 3.1+ +end create_makefile("unicorn_http")
[-- Attachment #1: Type: text/plain, Size: 4179 bytes --] Still a lot more work to do, but at least socat is no longer a test dependency. Perl5 is installed on far more systems than socat. Ruby introduces breaking changes every year and I can't trust tests to work as they were originally intended, anymore. Perl 5 doesn't have perfect backwards compatibility, either; but it's the least bad of any widely-installed scripting language. Note: that 23/23 introduces a subtle bugfix which changes behavior for systemd users Patches are attached to reduce load on SMTP servers. Some more patches to come as I deal with Ruby 3.x deprecation warnings :< Eric Wong (23): switch unit/test_response.rb to Perl 5 integration test support rack 3 multi-value headers port t0018-write-on-close.sh to Perl 5 port t0000-http-basic.sh to Perl 5 port t0002-parser-error.sh to Perl 5 t/integration.t: use start_req to simplify test slighly port t0011-active-unix-socket.sh to Perl 5 port t0100-rack-input-tests.sh to Perl 5 tests: use autodie to simplify error checking port t0019-max_header_len.sh to Perl 5 test_exec: drop sd_listen_fds emulation test test_exec: drop test_basic and test_config_ru_alt_path tests: check_stderr consistently in Perl 5 tests tests: consistent tcp_start and unix_start across Perl 5 tests port t9000-preread-input.sh to Perl 5 port t/t0116-client_body_buffer_size.sh to Perl 5 tests: get rid of sha1sum.rb and rsha1() sh function early_hints supports Rack 3 array headers test_server: drop early_hints test t/integration.t: switch PUT tests to MD5, reuse buffers tests: move test_upload.rb tests to t/integration.t drop redundant IO#close_on_exec=false calls LISTEN_FDS-inherited sockets are immortal across SIGHUP GNUmakefile | 7 +- lib/unicorn/http_server.rb | 12 +- t/README | 21 +- t/active-unix-socket.t | 113 +++++++ t/bin/content-md5-put | 36 --- t/bin/sha1sum.rb | 17 -- t/{t0116.ru => client_body_buffer_size.ru} | 2 - t/client_body_buffer_size.t | 82 ++++++ t/integration.ru | 114 +++++++ t/integration.t | 326 +++++++++++++++++++++ t/lib.perl | 217 ++++++++++++++ t/preread_input.ru | 21 +- t/rack-input-tests.ru | 21 -- t/t0000-http-basic.sh | 50 ---- t/t0002-parser-error.sh | 94 ------ t/t0011-active-unix-socket.sh | 79 ----- t/t0018-write-on-close.sh | 23 -- t/t0019-max_header_len.sh | 49 ---- t/t0100-rack-input-tests.sh | 124 -------- t/t0116-client_body_buffer_size.sh | 80 ----- t/t9000-preread-input.sh | 48 --- t/test-lib.sh | 4 - t/write-on-close.ru | 11 - test/exec/test_exec.rb | 57 ---- test/unit/test_response.rb | 111 ------- test/unit/test_server.rb | 31 -- test/unit/test_upload.rb | 301 ------------------- 27 files changed, 891 insertions(+), 1160 deletions(-) create mode 100644 t/active-unix-socket.t delete mode 100755 t/bin/content-md5-put delete mode 100755 t/bin/sha1sum.rb rename t/{t0116.ru => client_body_buffer_size.ru} (82%) create mode 100644 t/client_body_buffer_size.t create mode 100644 t/integration.ru create mode 100644 t/integration.t create mode 100644 t/lib.perl delete mode 100644 t/rack-input-tests.ru delete mode 100755 t/t0000-http-basic.sh delete mode 100755 t/t0002-parser-error.sh delete mode 100755 t/t0011-active-unix-socket.sh delete mode 100755 t/t0018-write-on-close.sh delete mode 100755 t/t0019-max_header_len.sh delete mode 100755 t/t0100-rack-input-tests.sh delete mode 100755 t/t0116-client_body_buffer_size.sh delete mode 100755 t/t9000-preread-input.sh delete mode 100644 t/write-on-close.ru delete mode 100644 test/unit/test_response.rb delete mode 100644 test/unit/test_upload.rb [-- Attachment #2: 0001-switch-unit-test_response.rb-to-Perl-5-integration-t.patch --] [-- Type: text/x-diff, Size: 15667 bytes --] From 086e397abc0126556af24df77a976671294df2ee Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:30 +0000 Subject: [PATCH 01/23] switch unit/test_response.rb to Perl 5 integration test http_response_write may benefit from API changes for Rack 3 support. Since there's no benefit I can see from using a unit test, switch to an integration test to avoid having to maintain the unit test if our internal http_response_write method changes. Of course, I can't trust tests written in Ruby since I've had to put up with a constant stream of incompatibilities over the past two decades :< Perl is more widely installed than socat[1], and nearly all the Perl I wrote 20 years ago still works unmodified today. [1] the rarest dependency of the Bourne shell integration tests --- GNUmakefile | 5 +- t/README | 24 +++-- t/integration.ru | 38 ++++++++ t/integration.t | 64 +++++++++++++ t/lib.perl | 189 +++++++++++++++++++++++++++++++++++++ test/unit/test_response.rb | 111 ---------------------- 6 files changed, 313 insertions(+), 118 deletions(-) create mode 100644 t/integration.ru create mode 100644 t/integration.t create mode 100644 t/lib.perl delete mode 100644 test/unit/test_response.rb diff --git a/GNUmakefile b/GNUmakefile index 0e08ef0..5cca189 100644 --- a/GNUmakefile +++ b/GNUmakefile @@ -86,7 +86,7 @@ $(tmp_bin)/%: bin/% | $(tmp_bin) bins: $(tmp_bins) t_log := $(T_log) $(T_n_log) -test: $(T) $(T_n) +test: $(T) $(T_n) test-prove @cat $(t_log) | $(MRI) test/aggregate.rb @$(RM) $(t_log) @@ -141,6 +141,9 @@ t/random_blob: test-integration: $(T_sh) +test-prove: + prove -vw + check: test-require test test-integration test-all: check diff --git a/t/README b/t/README index 14de559..8a5243e 100644 --- a/t/README +++ b/t/README @@ -5,16 +5,24 @@ TCP ports or Unix domain sockets. They're all designed to run concurrently with other tests to minimize test time, but tests may be run independently as well. -We write our tests in Bourne shell because that's what we're -comfortable writing integration tests with. +New tests are written in Perl 5 because we need a stable language +to test real-world behavior and Ruby introduces incompatibilities +at a far faster rate than Perl 5. Perl is Ruby's older cousin, so +it should be easy-to-learn for Rubyists. + +Old tests are in Bourne shell, but the socat(1) dependency was probably +too rare compared to Perl 5. == Requirements -* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/] (duh!) +* {Ruby 2.0.0+}[https://www.ruby-lang.org/en/] +* {Perl 5.14+}[https://www.perl.org/] # your distro should have it * {GNU make}[https://www.gnu.org/software/make/] + +The following requirements will eventually be dropped. + * {socat}[http://www.dest-unreach.org/socat/] * {curl}[https://curl.haxx.se/] -* standard UNIX shell utilities (Bourne sh, awk, sed, grep, ...) We do not use bashisms or any non-portable, non-POSIX constructs in our shell code. We use the "pipefail" option if available and @@ -26,9 +34,13 @@ with {dash}[http://gondor.apana.org.au/~herbert/dash/] and To run the entire test suite with 8 tests running at once: - make -j8 + make -j8 && prove -vw + +To run one individual test (Perl5): + + prove -vw t/integration.t -To run one individual test: +To run one individual test (shell): make t0000-simple-http.sh diff --git a/t/integration.ru b/t/integration.ru new file mode 100644 index 0000000..6ef873c --- /dev/null +++ b/t/integration.ru @@ -0,0 +1,38 @@ +#!ruby +# Copyright (C) unicorn hackers <unicorn-public@80x24.org> +# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt> + +# this goes for t/integration.t We'll try to put as many tests +# in here as possible to avoid startup overhead of Ruby. + +$orig_rack_200 = nil +def tweak_status_code + $orig_rack_200 = Rack::Utils::HTTP_STATUS_CODES[200] + Rack::Utils::HTTP_STATUS_CODES[200] = "HI" + [ 200, {}, [] ] +end + +def restore_status_code + $orig_rack_200 or return [ 500, {}, [] ] + Rack::Utils::HTTP_STATUS_CODES[200] = $orig_rack_200 + [ 200, {}, [] ] +end + +run(lambda do |env| + case env['REQUEST_METHOD'] + when 'GET' + case env['PATH_INFO'] + when '/rack-2-newline-headers'; [ 200, { 'X-R2' => "a\nb\nc" }, [] ] + when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ] + when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ] + end # case PATH_INFO (GET) + when 'POST' + case env['PATH_INFO'] + when '/tweak-status-code'; tweak_status_code + when '/restore-status-code'; restore_status_code + end # case PATH_INFO (POST) + # ... + when 'PUT' + # ... + end # case REQUEST_METHOD +end) # run diff --git a/t/integration.t b/t/integration.t new file mode 100644 index 0000000..5569155 --- /dev/null +++ b/t/integration.t @@ -0,0 +1,64 @@ +#!perl -w +# Copyright (C) unicorn hackers <unicorn-public@yhbt.net> +# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt> + +use v5.14; BEGIN { require './t/lib.perl' }; +my $srv = tcp_server(); +my $t0 = time; +my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv }); + +sub slurp_hdr { + my ($c) = @_; + local $/ = "\r\n\r\n"; # affects both readline+chomp + chomp(my $hdr = readline($c)); + my ($status, @hdr) = split(/\r\n/, $hdr); + diag explain([ $status, \@hdr ]) if $ENV{V}; + ($status, \@hdr); +} + +my ($c, $status, $hdr); + +# response header tests +$c = tcp_connect($srv); +print $c "GET /rack-2-newline-headers HTTP/1.0\r\n\r\n" or die $!; +($status, $hdr) = slurp_hdr($c); +like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid'); +my $orig_200_status = $status; +is_deeply([ grep(/^X-R2: /, @$hdr) ], + [ 'X-R2: a', 'X-R2: b', 'X-R2: c' ], + 'rack 2 LF-delimited headers supported') or diag(explain($hdr)); + +SKIP: { # Date header check + my @d = grep(/^Date: /i, @$hdr); + is(scalar(@d), 1, 'got one date header') or diag(explain(\@d)); + eval { require HTTP::Date } or skip "HTTP::Date missing: $@", 1; + $d[0] =~ s/^Date: //i or die 'BUG: did not strip date: prefix'; + my $t = HTTP::Date::str2time($d[0]); + ok($t >= $t0 && $t > 0 && $t <= time, 'valid date') or + diag(explain([$t, $!, \@d])); +}; + +# cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com> +$c = tcp_connect($srv); +print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!; +($status, $hdr) = slurp_hdr($c); +is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '], + 'nil header value accepted for broken apps') or diag(explain($hdr)); + +if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') { + $c = tcp_connect($srv); + print $c "POST /tweak-status-code HTTP/1.0\r\n\r\n" or die $!; + ($status, $hdr) = slurp_hdr($c); + like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked'); + + $c = tcp_connect($srv); + print $c "POST /restore-status-code HTTP/1.0\r\n\r\n" or die $!; + ($status, $hdr) = slurp_hdr($c); + is($status, $orig_200_status, 'original status restored'); +} + + +# ... more stuff here +undef $ar; +diag slurp("$tmpdir/err.log") if $ENV{V}; +done_testing; diff --git a/t/lib.perl b/t/lib.perl new file mode 100644 index 0000000..dd9c6b7 --- /dev/null +++ b/t/lib.perl @@ -0,0 +1,189 @@ +#!perl -w +# Copyright (C) unicorn hackers <unicorn-public@80x24.org> +# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt> +package UnicornTest; +use v5.14; +use parent qw(Exporter); +use Test::More; +use IO::Socket::INET; +use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD); +use File::Temp 0.19 (); # 0.19 for ->newdir +our ($tmpdir, $errfh); +our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh + SEEK_SET); + +my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!); +$tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1); +open($errfh, '>>', "$tmpdir/err.log") or die "open: $!"; + +sub tcp_server { + my %opt = ( + ReuseAddr => 1, + Proto => 'tcp', + Type => SOCK_STREAM, + Listen => SOMAXCONN, + Blocking => 0, + @_, + ); + eval { + die 'IPv4-only' if $ENV{TEST_IPV4_ONLY}; + require IO::Socket::INET6; + IO::Socket::INET6->new(%opt, LocalAddr => '[::1]') + } || eval { + die 'IPv6-only' if $ENV{TEST_IPV6_ONLY}; + IO::Socket::INET->new(%opt, LocalAddr => '127.0.0.1') + } || BAIL_OUT "failed to create TCP server: $! ($@)"; +} + +sub tcp_host_port { + my ($s) = @_; + my ($h, $p) = ($s->sockhost, $s->sockport); + my $ipv4 = $s->sockdomain == AF_INET; + if (wantarray) { + $ipv4 ? ($h, $p) : ("[$h]", $p); + } else { + $ipv4 ? "$h:$p" : "[$h]:$p"; + } +} + +sub tcp_connect { + my ($dest, %opt) = @_; + my $addr = tcp_host_port($dest); + my $s = ref($dest)->new( + Proto => 'tcp', + Type => SOCK_STREAM, + PeerAddr => $addr, + %opt, + ) or BAIL_OUT "failed to connect to $addr: $!"; + $s->autoflush(1); + $s; +} + +sub slurp { + open my $fh, '<', $_[0] or die "open($_[0]): $!"; + local $/; + <$fh>; +} + +sub spawn { + my $env = ref($_[0]) eq 'HASH' ? shift : undef; + my $opt = ref($_[-1]) eq 'HASH' ? pop : {}; + my @cmd = @_; + my $old = POSIX::SigSet->new; + my $set = POSIX::SigSet->new; + $set->fillset or die "sigfillset: $!"; + sigprocmask(SIG_SETMASK, $set, $old) or die "SIG_SETMASK: $!"; + pipe(my ($r, $w)) or die "pipe: $!"; + my $pid = fork // die "fork: $!"; + if ($pid == 0) { + close $r; + $SIG{__DIE__} = sub { + warn(@_); + syswrite($w, my $num = $! + 0); + _exit(1); + }; + + # pretend to be systemd (cf. sd_listen_fds(3)) + my $cfd; + for ($cfd = 0; ($cfd < 3) || defined($opt->{$cfd}); $cfd++) { + my $io = $opt->{$cfd} // next; + my $pfd = fileno($io) // die "fileno($io): $!"; + if ($pfd == $cfd) { + fcntl($io, F_SETFD, 0) // die "F_SETFD: $!"; + } else { + dup2($pfd, $cfd) // die "dup2($pfd, $cfd): $!"; + } + } + if (($cfd - 3) > 0) { + $env->{LISTEN_PID} = $$; + $env->{LISTEN_FDS} = $cfd - 3; + } + + if (defined(my $pgid = $opt->{pgid})) { + setpgid(0, $pgid) // die "setpgid(0, $pgid): $!"; + } + $SIG{$_} = 'DEFAULT' for grep(!/^__/, keys %SIG); + if (defined(my $cd = $opt->{-C})) { + chdir $cd // die "chdir($cd): $!"; + } + $old->delset(POSIX::SIGCHLD) or die "sigdelset CHLD: $!"; + sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK: ~CHLD: $!"; + @ENV{keys %$env} = values(%$env) if $env; + exec { $cmd[0] } @cmd; + die "exec @cmd: $!"; + } + close $w; + sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK(old): $!"; + if (my $cerrnum = do { local $/, <$r> }) { + $! = $cerrnum; + die "@cmd PID=$pid died: $!"; + } + $pid; +} + +sub which { + my ($file) = @_; + return $file if index($file, '/') >= 0; + for my $p (split(/:/, $ENV{PATH})) { + $p .= "/$file"; + return $p if -x $p; + } + undef; +} + +# returns an AutoReap object +sub unicorn { + my %env; + if (ref($_[0]) eq 'HASH') { + my $e = shift; + %env = %$e; + } + my @args = @_; + push(@args, {}) if ref($args[-1]) ne 'HASH'; + $args[-1]->{2} //= $errfh; # stderr default + + state $ruby = which($ENV{RUBY} // 'ruby'); + state $lib = File::Spec->rel2abs('lib'); + state $ver = $ENV{TEST_RUBY_VERSION} // `$ruby -e 'print RUBY_VERSION'`; + state $eng = $ENV{TEST_RUBY_ENGINE} // `$ruby -e 'print RUBY_ENGINE'`; + state $ext = File::Spec->rel2abs("test/$eng-$ver/ext/unicorn_http"); + state $exe = File::Spec->rel2abs('bin/unicorn'); + my $pid = spawn(\%env, $ruby, '-I', $lib, '-I', $ext, $exe, @args); + UnicornTest::AutoReap->new($pid); +} + +# automatically kill + reap children when this goes out-of-scope +package UnicornTest::AutoReap; +use v5.14; + +sub new { + my (undef, $pid) = @_; + bless { pid => $pid, owner => $$ }, __PACKAGE__ +} + +sub kill { + my ($self, $sig) = @_; + CORE::kill($sig // 'TERM', $self->{pid}); +} + +sub join { + my ($self, $sig) = @_; + my $pid = delete $self->{pid} or return; + CORE::kill($sig, $pid) if defined $sig; + my $ret = waitpid($pid, 0) // die "waitpid($pid): $!"; + $ret == $pid or die "BUG: waitpid($pid) != $ret"; +} + +sub DESTROY { + my ($self) = @_; + return if $self->{owner} != $$; + $self->join('TERM'); +} + +package main; # inject ourselves into the t/*.t script +UnicornTest->import; +Test::More->import; +# try to ensure ->DESTROY fires: +$SIG{TERM} = sub { exit(15 + 128) }; +$SIG{INT} = sub { exit(2 + 128) }; +1; diff --git a/test/unit/test_response.rb b/test/unit/test_response.rb deleted file mode 100644 index fbe433f..0000000 --- a/test/unit/test_response.rb +++ /dev/null @@ -1,111 +0,0 @@ -# -*- encoding: binary -*- - -# Copyright (c) 2005 Zed A. Shaw -# You can redistribute it and/or modify it under the same terms as Ruby 1.8 or -# the GPLv2+ (GPLv3+ preferred) -# -# Additional work donated by contributors. See git history -# for more information. - -require './test/test_helper' -require 'time' - -include Unicorn - -class ResponseTest < Test::Unit::TestCase - include Unicorn::HttpResponse - - def test_httpdate - before = Time.now.to_i - 1 - str = httpdate - assert_kind_of(String, str) - middle = Time.parse(str).to_i - after = Time.now.to_i - assert before <= middle - assert middle <= after - end - - def test_response_headers - out = StringIO.new - http_response_write(out, 200, {"X-Whatever" => "stuff"}, ["cool"]) - assert ! out.closed? - - assert out.length > 0, "output didn't have data" - end - - # ref: <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com> - def test_response_header_broken_nil - out = StringIO.new - http_response_write(out, 200, {"Nil" => nil}, %w(hysterical raisin)) - assert ! out.closed? - - assert_match %r{^Nil: \r\n}sm, out.string, 'nil accepted' - end - - def test_response_string_status - out = StringIO.new - http_response_write(out,'200', {}, []) - assert ! out.closed? - assert out.length > 0, "output didn't have data" - end - - def test_response_200 - io = StringIO.new - http_response_write(io, 200, {}, []) - assert ! io.closed? - assert io.length > 0, "output didn't have data" - end - - def test_response_with_default_reason - code = 400 - io = StringIO.new - http_response_write(io, code, {}, []) - assert ! io.closed? - lines = io.string.split(/\r\n/) - assert_match(/.* Bad Request$/, lines.first, - "wrong default reason phrase") - end - - def test_rack_multivalue_headers - out = StringIO.new - http_response_write(out,200, {"X-Whatever" => "stuff\nbleh"}, []) - assert ! out.closed? - assert_match(/^X-Whatever: stuff\r\nX-Whatever: bleh\r\n/, out.string) - end - - # Even though Rack explicitly forbids "Status" in the header hash, - # some broken clients still rely on it - def test_status_header_added - out = StringIO.new - http_response_write(out,200, {"X-Whatever" => "stuff"}, []) - assert ! out.closed? - end - - def test_unknown_status_pass_through - out = StringIO.new - http_response_write(out,"666 I AM THE BEAST", {}, [] ) - assert ! out.closed? - headers = out.string.split(/\r\n\r\n/).first.split(/\r\n/) - assert %r{\AHTTP/\d\.\d 666 I AM THE BEAST\z}.match(headers[0]) - end - - def test_modified_rack_http_status_codes_late - r, w = IO.pipe - pid = fork do - r.close - # Users may want to globally override the status text associated - # with an HTTP status code in their app. - Rack::Utils::HTTP_STATUS_CODES[200] = "HI" - http_response_write(w, 200, {}, []) - w.close - end - w.close - assert_equal "HTTP/1.1 200 HI\r\n", r.gets - r.read # just drain the pipe - pid, status = Process.waitpid2(pid) - assert status.success?, status.inspect - ensure - r.close - w.close unless w.closed? - end -end [-- Attachment #3: 0002-support-rack-3-multi-value-headers.patch --] [-- Type: text/x-diff, Size: 1710 bytes --] From ea0559c700fa029044464de4bd572662c10b7273 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:31 +0000 Subject: [PATCH 02/23] support rack 3 multi-value headers The first step in adding Rack 3 support. Rack supports multi-value headers via array rather than newlines. Tested-by: Martin Posthumus <martin.posthumus@gmail.com> Link: https://yhbt.net/unicorn-public/7c851d8a-bc57-7df8-3240-2f5ab831c47c@gmail.com/ --- t/integration.ru | 1 + t/integration.t | 9 +++++++++ 2 files changed, 10 insertions(+) diff --git a/t/integration.ru b/t/integration.ru index 6ef873c..5183217 100644 --- a/t/integration.ru +++ b/t/integration.ru @@ -23,6 +23,7 @@ def restore_status_code when 'GET' case env['PATH_INFO'] when '/rack-2-newline-headers'; [ 200, { 'X-R2' => "a\nb\nc" }, [] ] + when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ] when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ] when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ] end # case PATH_INFO (GET) diff --git a/t/integration.t b/t/integration.t index 5569155..e876c71 100644 --- a/t/integration.t +++ b/t/integration.t @@ -38,6 +38,15 @@ SKIP: { # Date header check diag(explain([$t, $!, \@d])); }; + +$c = tcp_connect($srv); +print $c "GET /rack-3-array-headers HTTP/1.0\r\n\r\n" or die $!; +($status, $hdr) = slurp_hdr($c); +is_deeply([ grep(/^x-r3: /, @$hdr) ], + [ 'x-r3: a', 'x-r3: b', 'x-r3: c' ], + 'rack 3 array headers supported') or diag(explain($hdr)); + + # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com> $c = tcp_connect($srv); print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!; [-- Attachment #4: 0003-port-t0018-write-on-close.sh-to-Perl-5.patch --] [-- Type: text/x-diff, Size: 4091 bytes --] From 295a6c616f8840bc04617a377c04c3422aeebddc Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:32 +0000 Subject: [PATCH 03/23] port t0018-write-on-close.sh to Perl 5 This doesn't require restarting, so it's a perfect candidate. --- t/integration.ru | 15 +++++++++++++++ t/integration.t | 14 +++++++++++++- t/lib.perl | 2 +- t/t0018-write-on-close.sh | 23 ----------------------- t/write-on-close.ru | 11 ----------- 5 files changed, 29 insertions(+), 36 deletions(-) delete mode 100755 t/t0018-write-on-close.sh delete mode 100644 t/write-on-close.ru diff --git a/t/integration.ru b/t/integration.ru index 5183217..12f5d48 100644 --- a/t/integration.ru +++ b/t/integration.ru @@ -18,6 +18,20 @@ def restore_status_code [ 200, {}, [] ] end +class WriteOnClose + def each(&block) + @callback = block + end + + def close + @callback.call "7\r\nGoodbye\r\n0\r\n\r\n" + end +end + +def write_on_close + [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ] +end + run(lambda do |env| case env['REQUEST_METHOD'] when 'GET' @@ -26,6 +40,7 @@ def restore_status_code when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ] when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ] when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ] + when '/write_on_close'; write_on_close end # case PATH_INFO (GET) when 'POST' case env['PATH_INFO'] diff --git a/t/integration.t b/t/integration.t index e876c71..3ab5c90 100644 --- a/t/integration.t +++ b/t/integration.t @@ -4,6 +4,7 @@ use v5.14; BEGIN { require './t/lib.perl' }; my $srv = tcp_server(); +my $host_port = tcp_host_port($srv); my $t0 = time; my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv }); @@ -66,8 +67,19 @@ if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') { is($status, $orig_200_status, 'original status restored'); } +SKIP: { + eval { require HTTP::Tiny } or skip "HTTP::Tiny missing: $@", 1; + my $ht = HTTP::Tiny->new; + my $res = $ht->get("http://$host_port/write_on_close"); + is($res->{content}, 'Goodbye', 'write-on-close body read'); +} # ... more stuff here undef $ar; -diag slurp("$tmpdir/err.log") if $ENV{V}; +my @log = slurp("$tmpdir/err.log"); +diag("@log") if $ENV{V}; +my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log)); +is_deeply(\@err, [], 'no unexpected errors in stderr'); +is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr'); + done_testing; diff --git a/t/lib.perl b/t/lib.perl index dd9c6b7..12deaf8 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD); use File::Temp 0.19 (); # 0.19 for ->newdir our ($tmpdir, $errfh); our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh - SEEK_SET); + SEEK_SET tcp_host_port); my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!); $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1); diff --git a/t/t0018-write-on-close.sh b/t/t0018-write-on-close.sh deleted file mode 100755 index 3afefea..0000000 --- a/t/t0018-write-on-close.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/bin/sh -. ./test-lib.sh -t_plan 4 "write-on-close tests for funky response-bodies" - -t_begin "setup and start" && { - unicorn_setup - unicorn -D -c $unicorn_config write-on-close.ru - unicorn_wait_start -} - -t_begin "write-on-close response body succeeds" && { - test xGoodbye = x"$(curl -sSf http://$listen/)" -} - -t_begin "killing succeeds" && { - kill $unicorn_pid -} - -t_begin "check stderr" && { - check_stderr -} - -t_done diff --git a/t/write-on-close.ru b/t/write-on-close.ru deleted file mode 100644 index 725c4d6..0000000 --- a/t/write-on-close.ru +++ /dev/null @@ -1,11 +0,0 @@ -class WriteOnClose - def each(&block) - @callback = block - end - - def close - @callback.call "7\r\nGoodbye\r\n0\r\n\r\n" - end -end -use Rack::ContentType, "text/plain" -run(lambda { |_| [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ] }) [-- Attachment #5: 0004-port-t0000-http-basic.sh-to-Perl-5.patch --] [-- Type: text/x-diff, Size: 3372 bytes --] From 1bb4362cee167ac7aeec910d3f52419e391f1e61 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:33 +0000 Subject: [PATCH 04/23] port t0000-http-basic.sh to Perl 5 One more socat dependency down... --- t/integration.ru | 16 ++++++++++++++ t/integration.t | 11 ++++++++++ t/t0000-http-basic.sh | 50 ------------------------------------------- 3 files changed, 27 insertions(+), 50 deletions(-) delete mode 100755 t/t0000-http-basic.sh diff --git a/t/integration.ru b/t/integration.ru index 12f5d48..c0bef99 100644 --- a/t/integration.ru +++ b/t/integration.ru @@ -32,6 +32,21 @@ def write_on_close [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ] end +def env_dump(env) + require 'json' + h = {} + env.each do |k,v| + case v + when String, Integer, true, false; h[k] = v + else + case k + when 'rack.version', 'rack.after_reply'; h[k] = v + end + end + end + h.to_json +end + run(lambda do |env| case env['REQUEST_METHOD'] when 'GET' @@ -40,6 +55,7 @@ def write_on_close when '/rack-3-array-headers'; [ 200, { 'x-r3' => %w(a b c) }, [] ] when '/nil-header-value'; [ 200, { 'X-Nil' => nil }, [] ] when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ] + when '/env_dump'; [ 200, {}, [ env_dump(env) ] ] when '/write_on_close'; write_on_close end # case PATH_INFO (GET) when 'POST' diff --git a/t/integration.t b/t/integration.t index 3ab5c90..ee22e7e 100644 --- a/t/integration.t +++ b/t/integration.t @@ -47,6 +47,17 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ], [ 'x-r3: a', 'x-r3: b', 'x-r3: c' ], 'rack 3 array headers supported') or diag(explain($hdr)); +SKIP: { + eval { require JSON::PP } or skip "JSON::PP missing: $@", 1; + $c = tcp_connect($srv); + print $c "GET /env_dump\r\n" or die $!; + my $json = do { local $/; readline($c) }; + unlike($json, qr/^Connection: /smi, 'no connection header for 0.9'); + unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9'); + my $env = JSON::PP->new->decode($json); + is(ref($env), 'HASH', 'JSON decoded body to hashref'); + is($env->{SERVER_PROTOCOL}, 'HTTP/0.9', 'SERVER_PROTOCOL is 0.9'); +} # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com> $c = tcp_connect($srv); diff --git a/t/t0000-http-basic.sh b/t/t0000-http-basic.sh deleted file mode 100755 index 8ab58ac..0000000 --- a/t/t0000-http-basic.sh +++ /dev/null @@ -1,50 +0,0 @@ -#!/bin/sh -. ./test-lib.sh -t_plan 8 "simple HTTP connection tests" - -t_begin "setup and start" && { - unicorn_setup - unicorn -D -c $unicorn_config env.ru - unicorn_wait_start -} - -t_begin "single request" && { - curl -sSfv http://$listen/ -} - -t_begin "check stderr has no errors" && { - check_stderr -} - -t_begin "HTTP/0.9 request should not return headers" && { - ( - printf 'GET /\r\n' - cat $fifo > $tmp & - wait - echo ok > $ok - ) | socat - TCP:$listen > $fifo -} - -t_begin "env.inspect should've put everything on one line" && { - test 1 -eq $(count_lines < $tmp) -} - -t_begin "no headers in output" && { - if grep ^Connection: $tmp - then - die "Connection header found in $tmp" - elif grep ^HTTP/ $tmp - then - die "HTTP/ found in $tmp" - fi -} - -t_begin "killing succeeds" && { - kill $unicorn_pid -} - -t_begin "check stderr has no errors" && { - check_stderr -} - -t_done [-- Attachment #6: 0005-port-t0002-parser-error.sh-to-Perl-5.patch --] [-- Type: text/x-diff, Size: 4875 bytes --] From 2eb7b1662c291ab535ee5dabf5d96194ca6483d4 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:34 +0000 Subject: [PATCH 05/23] port t0002-parser-error.sh to Perl 5 Another socat dependency down... --- t/integration.t | 33 +++++++++++++++ t/lib.perl | 9 +++- t/t0002-parser-error.sh | 94 ----------------------------------------- 3 files changed, 41 insertions(+), 95 deletions(-) delete mode 100755 t/t0002-parser-error.sh diff --git a/t/integration.t b/t/integration.t index ee22e7e..503b7eb 100644 --- a/t/integration.t +++ b/t/integration.t @@ -85,6 +85,39 @@ SKIP: { is($res->{content}, 'Goodbye', 'write-on-close body read'); } +if ('bad requests') { + $c = start_req($srv, 'GET /env_dump HTTP/1/1'); + ($status, $hdr) = slurp_hdr($c); + like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request'); + + $c = tcp_connect($srv); + print $c 'GET /' or die $!; + my $buf = join('', (0..9), 'ab'); + for (0..1023) { print $c $buf or die $! } + print $c " HTTP/1.0\r\n\r\n" or die $!; + ($status, $hdr) = slurp_hdr($c); + like($status, qr!\AHTTP/1\.[01] 414 \b!, + '414 on REQUEST_PATH > (12 * 1024)'); + + $c = tcp_connect($srv); + print $c 'GET /hello-world?a' or die $!; + $buf = join('', (0..9)); + for (0..1023) { print $c $buf or die $! } + print $c " HTTP/1.0\r\n\r\n" or die $!; + ($status, $hdr) = slurp_hdr($c); + like($status, qr!\AHTTP/1\.[01] 414 \b!, + '414 on QUERY_STRING > (10 * 1024)'); + + $c = tcp_connect($srv); + print $c 'GET /hello-world#a' or die $!; + $buf = join('', (0..9), 'a'..'f'); + for (0..63) { print $c $buf or die $! } + print $c " HTTP/1.0\r\n\r\n" or die $!; + ($status, $hdr) = slurp_hdr($c); + like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)'); +} + + # ... more stuff here undef $ar; my @log = slurp("$tmpdir/err.log"); diff --git a/t/lib.perl b/t/lib.perl index 12deaf8..7d712b5 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD); use File::Temp 0.19 (); # 0.19 for ->newdir our ($tmpdir, $errfh); our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh - SEEK_SET tcp_host_port); + SEEK_SET tcp_host_port start_req); my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!); $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1); @@ -59,6 +59,13 @@ sub tcp_connect { $s; } +sub start_req { + my ($srv, @req) = @_; + my $c = tcp_connect($srv); + print $c @req, "\r\n\r\n" or die "print: $!"; + $c; +} + sub slurp { open my $fh, '<', $_[0] or die "open($_[0]): $!"; local $/; diff --git a/t/t0002-parser-error.sh b/t/t0002-parser-error.sh deleted file mode 100755 index 9dc1cd2..0000000 --- a/t/t0002-parser-error.sh +++ /dev/null @@ -1,94 +0,0 @@ -#!/bin/sh -. ./test-lib.sh -t_plan 11 "parser error test" - -t_begin "setup and startup" && { - unicorn_setup - unicorn -D env.ru -c $unicorn_config - unicorn_wait_start -} - -t_begin "send a bad request" && { - ( - printf 'GET / HTTP/1/1\r\nHost: example.com\r\n\r\n' - cat $fifo > $tmp & - wait - echo ok > $ok - ) | socat - TCP:$listen > $fifo - test xok = x$(cat $ok) -} - -dbgcat tmp - -t_begin "response should be a 400" && { - grep -F 'HTTP/1.1 400 Bad Request' $tmp -} - -t_begin "send a huge Request URI (REQUEST_PATH > (12 * 1024))" && { - rm -f $tmp - cat $fifo > $tmp & - ( - set -e - trap 'echo ok > $ok' EXIT - printf 'GET /' - for i in $(awk </dev/null 'BEGIN{for(i=0;i<1024;i++) print i}') - do - printf '0123456789ab' - done - printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n' - ) | socat - TCP:$listen > $fifo || : - test xok = x$(cat $ok) - wait -} - -t_begin "response should be a 414 (REQUEST_PATH)" && { - grep -F 'HTTP/1.1 414 ' $tmp -} - -t_begin "send a huge Request URI (QUERY_STRING > (10 * 1024))" && { - rm -f $tmp - cat $fifo > $tmp & - ( - set -e - trap 'echo ok > $ok' EXIT - printf 'GET /hello-world?a' - for i in $(awk </dev/null 'BEGIN{for(i=0;i<1024;i++) print i}') - do - printf '0123456789' - done - printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n' - ) | socat - TCP:$listen > $fifo || : - test xok = x$(cat $ok) - wait -} - -t_begin "response should be a 414 (QUERY_STRING)" && { - grep -F 'HTTP/1.1 414 ' $tmp -} - -t_begin "send a huge Request URI (FRAGMENT > 1024)" && { - rm -f $tmp - cat $fifo > $tmp & - ( - set -e - trap 'echo ok > $ok' EXIT - printf 'GET /hello-world#a' - for i in $(awk </dev/null 'BEGIN{for(i=0;i<64;i++) print i}') - do - printf '0123456789abcdef' - done - printf ' HTTP/1.1\r\nHost: example.com\r\n\r\n' - ) | socat - TCP:$listen > $fifo || : - test xok = x$(cat $ok) - wait -} - -t_begin "response should be a 414 (FRAGMENT)" && { - grep -F 'HTTP/1.1 414 ' $tmp -} - -t_begin "server stderr should be clean" && check_stderr - -t_begin "term signal sent" && kill $unicorn_pid - -t_done [-- Attachment #7: 0006-t-integration.t-use-start_req-to-simplify-test-sligh.patch --] [-- Type: text/x-diff, Size: 2556 bytes --] From 0bb06cc0c8c4f5b76514858067bbb2871dda0d6e Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:35 +0000 Subject: [PATCH 06/23] t/integration.t: use start_req to simplify test slighly Less code is usually better. --- t/integration.t | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/t/integration.t b/t/integration.t index 503b7eb..b7ba1fb 100644 --- a/t/integration.t +++ b/t/integration.t @@ -20,8 +20,7 @@ sub slurp_hdr { my ($c, $status, $hdr); # response header tests -$c = tcp_connect($srv); -print $c "GET /rack-2-newline-headers HTTP/1.0\r\n\r\n" or die $!; +$c = start_req($srv, 'GET /rack-2-newline-headers HTTP/1.0'); ($status, $hdr) = slurp_hdr($c); like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid'); my $orig_200_status = $status; @@ -40,8 +39,7 @@ SKIP: { # Date header check }; -$c = tcp_connect($srv); -print $c "GET /rack-3-array-headers HTTP/1.0\r\n\r\n" or die $!; +$c = start_req($srv, 'GET /rack-3-array-headers HTTP/1.0'); ($status, $hdr) = slurp_hdr($c); is_deeply([ grep(/^x-r3: /, @$hdr) ], [ 'x-r3: a', 'x-r3: b', 'x-r3: c' ], @@ -49,8 +47,7 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ], SKIP: { eval { require JSON::PP } or skip "JSON::PP missing: $@", 1; - $c = tcp_connect($srv); - print $c "GET /env_dump\r\n" or die $!; + my $c = start_req($srv, 'GET /env_dump'); my $json = do { local $/; readline($c) }; unlike($json, qr/^Connection: /smi, 'no connection header for 0.9'); unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9'); @@ -60,20 +57,17 @@ SKIP: { } # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com> -$c = tcp_connect($srv); -print $c "GET /nil-header-value HTTP/1.0\r\n\r\n" or die $!; +$c = start_req($srv, 'GET /nil-header-value HTTP/1.0'); ($status, $hdr) = slurp_hdr($c); is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '], 'nil header value accepted for broken apps') or diag(explain($hdr)); if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') { - $c = tcp_connect($srv); - print $c "POST /tweak-status-code HTTP/1.0\r\n\r\n" or die $!; + $c = start_req($srv, 'POST /tweak-status-code HTTP/1.0'); ($status, $hdr) = slurp_hdr($c); like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked'); - $c = tcp_connect($srv); - print $c "POST /restore-status-code HTTP/1.0\r\n\r\n" or die $!; + $c = start_req($srv, 'POST /restore-status-code HTTP/1.0'); ($status, $hdr) = slurp_hdr($c); is($status, $orig_200_status, 'original status restored'); } [-- Attachment #8: 0007-port-t0011-active-unix-socket.sh-to-Perl-5.patch --] [-- Type: text/x-diff, Size: 6945 bytes --] From 10c83beaca58df8b92d8228e798559069cd89beb Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:36 +0000 Subject: [PATCH 07/23] port t0011-active-unix-socket.sh to Perl 5 Another socat dependency down... I've also started turning FD_CLOEXEC off on a pipe as a mechanism to detect daemonized process death in tests. --- t/active-unix-socket.t | 117 ++++++++++++++++++++++++++++++++++ t/integration.ru | 1 + t/t0011-active-unix-socket.sh | 79 ----------------------- 3 files changed, 118 insertions(+), 79 deletions(-) create mode 100644 t/active-unix-socket.t delete mode 100755 t/t0011-active-unix-socket.sh diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t new file mode 100644 index 0000000..6b5c218 --- /dev/null +++ b/t/active-unix-socket.t @@ -0,0 +1,117 @@ +#!perl -w +# Copyright (C) unicorn hackers <unicorn-public@yhbt.net> +# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt> + +use v5.14; BEGIN { require './t/lib.perl' }; +use IO::Socket::UNIX; +my %to_kill; +END { kill('TERM', values(%to_kill)) if keys %to_kill } +my $u1 = "$tmpdir/u1.sock"; +my $u2 = "$tmpdir/u2.sock"; +my $unix_req = sub { + my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM); + print $s @_, "\r\n\r\n" or die $!; + $s; +}; +{ + use autodie; + open my $fh, '>', "$tmpdir/u1.conf.rb"; + print $fh <<EOM; +pid "$tmpdir/u.pid" +listen "$u1" +stderr_path "$tmpdir/err1.log" +EOM + close $fh; + + open $fh, '>', "$tmpdir/u2.conf.rb"; + print $fh <<EOM; +pid "$tmpdir/u.pid" +listen "$u2" +stderr_path "$tmpdir/err2.log" +EOM + close $fh; + + open $fh, '>', "$tmpdir/u3.conf.rb"; + print $fh <<EOM; +pid "$tmpdir/u3.pid" +listen "$u1" +stderr_path "$tmpdir/err3.log" +EOM + close $fh; +} + +my @uarg = qw(-D -E none t/integration.ru); + +# this pipe will be used to notify us when all daemons die: +pipe(my ($p0, $p1)) or die "pipe: $!"; +fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC + +# start the first instance +unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join; +is($?, 0, 'daemonized 1st process'); +chomp($to_kill{u1} = slurp("$tmpdir/u.pid")); +like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file'); + +chomp(my $worker_pid = readline($unix_req->($u1, 'GET /pid'))); +like($worker_pid, qr/\A\d+\z/s, 'captured worker pid'); +ok(kill(0, $worker_pid), 'worker is kill-able'); + + +# 2nd process conflicts on PID +unicorn('-c', "$tmpdir/u2.conf.rb", @uarg)->join; +isnt($?, 0, 'conflicting PID file fails to start'); + +chomp(my $pidf = slurp("$tmpdir/u.pid")); +is($pidf, $to_kill{u1}, 'pid file contents unchanged after start failure'); + +chomp(my $pid2 = readline($unix_req->($u1, 'GET /pid'))); +is($worker_pid, $pid2, 'worker PID unchanged'); + + +# 3rd process conflicts on socket +unicorn('-c', "$tmpdir/u3.conf.rb", @uarg)->join; +isnt($?, 0, 'conflicting UNIX socket fails to start'); + +chomp($pid2 = readline($unix_req->($u1, 'GET /pid'))); +is($worker_pid, $pid2, 'worker PID still unchanged'); + +chomp($pidf = slurp("$tmpdir/u.pid")); +is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure'); + +{ # teardown initial process via SIGKILL + ok(kill('KILL', delete $to_kill{u1}), 'SIGKILL initial daemon'); + close $p1; + vec(my $rvec = '', fileno($p0), 1) = 1; + is(select($rvec, undef, undef, 5), 1, 'timeout for pipe HUP'); + is(my $undef = <$p0>, undef, 'process closed pipe writer at exit'); + ok(-f "$tmpdir/u.pid", 'pid file stayed after SIGKILL'); + ok(-S $u1, 'socket stayed after SIGKILL'); + is(IO::Socket::UNIX->new(Peer => $u1, Type => SOCK_STREAM), undef, + 'fail to connect to u1'); + ok(!kill(0, $worker_pid), 'worker gone after parent dies'); +} + +# restart the first instance +{ + pipe(($p0, $p1)) or die "pipe: $!"; + fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC + unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join; + is($?, 0, 'daemonized 1st process'); + chomp($to_kill{u1} = slurp("$tmpdir/u.pid")); + like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file'); + + chomp($pid2 = readline($unix_req->($u1, 'GET /pid'))); + like($pid2, qr/\A\d+\z/, 'worker running'); + + ok(kill('TERM', delete $to_kill{u1}), 'SIGTERM restarted daemon'); + close $p1; + vec(my $rvec = '', fileno($p0), 1) = 1; + is(select($rvec, undef, undef, 5), 1, 'timeout for pipe HUP'); + is(my $undef = <$p0>, undef, 'process closed pipe writer at exit'); + ok(!-f "$tmpdir/u.pid", 'pid file gone after SIGTERM'); + ok(-S $u1, 'socket stays after SIGTERM'); +} + +my @log = slurp("$tmpdir/err.log"); +diag("@log") if $ENV{V}; +done_testing; diff --git a/t/integration.ru b/t/integration.ru index c0bef99..21f5449 100644 --- a/t/integration.ru +++ b/t/integration.ru @@ -57,6 +57,7 @@ def env_dump(env) when '/unknown-status-pass-through'; [ '666 I AM THE BEAST', {}, [] ] when '/env_dump'; [ 200, {}, [ env_dump(env) ] ] when '/write_on_close'; write_on_close + when '/pid'; [ 200, {}, [ "#$$\n" ] ] end # case PATH_INFO (GET) when 'POST' case env['PATH_INFO'] diff --git a/t/t0011-active-unix-socket.sh b/t/t0011-active-unix-socket.sh deleted file mode 100755 index fae0b6c..0000000 --- a/t/t0011-active-unix-socket.sh +++ /dev/null @@ -1,79 +0,0 @@ -#!/bin/sh -. ./test-lib.sh -t_plan 11 "existing UNIX domain socket check" - -read_pid_unix () { - x=$(printf 'GET / HTTP/1.0\r\n\r\n' | \ - socat - UNIX:$unix_socket | \ - tail -1) - test -n "$x" - y="$(expr "$x" : '\([0-9][0-9]*\)')" - test x"$x" = x"$y" - test -n "$y" - echo "$y" -} - -t_begin "setup and start" && { - rtmpfiles unix_socket unix_config - rm -f $unix_socket - unicorn_setup - grep -v ^listen < $unicorn_config > $unix_config - echo "listen '$unix_socket'" >> $unix_config - unicorn -D -c $unix_config pid.ru - unicorn_wait_start - orig_master_pid=$unicorn_pid -} - -t_begin "get pid of worker" && { - worker_pid=$(read_pid_unix) - t_info "worker_pid=$worker_pid" -} - -t_begin "fails to start with existing pid file" && { - rm -f $ok - unicorn -D -c $unix_config pid.ru || echo ok > $ok - test x"$(cat $ok)" = xok -} - -t_begin "worker pid unchanged" && { - test x"$(read_pid_unix)" = x$worker_pid - > $r_err -} - -t_begin "fails to start with listening UNIX domain socket bound" && { - rm $ok $pid - unicorn -D -c $unix_config pid.ru || echo ok > $ok - test x"$(cat $ok)" = xok - > $r_err -} - -t_begin "worker pid unchanged (again)" && { - test x"$(read_pid_unix)" = x$worker_pid -} - -t_begin "nuking the existing Unicorn succeeds" && { - kill -9 $unicorn_pid - while kill -0 $unicorn_pid - do - sleep 1 - done - check_stderr -} - -t_begin "succeeds in starting with leftover UNIX domain socket bound" && { - test -S $unix_socket - unicorn -D -c $unix_config pid.ru - unicorn_wait_start -} - -t_begin "worker pid changed" && { - test x"$(read_pid_unix)" != x$worker_pid -} - -t_begin "killing succeeds" && { - kill $unicorn_pid -} - -t_begin "no errors" && check_stderr - -t_done [-- Attachment #9: 0008-port-t0100-rack-input-tests.sh-to-Perl-5.patch --] [-- Type: text/x-diff, Size: 11722 bytes --] From b4ed148186295f2d5c8448eab7f2b201615d1e4e Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:37 +0000 Subject: [PATCH 08/23] port t0100-rack-input-tests.sh to Perl 5 Yet another socat dependency gone \o/ --- t/bin/content-md5-put | 36 ----------- t/integration.ru | 27 +++++++- t/integration.t | 97 +++++++++++++++++++++++++++- t/lib.perl | 3 +- t/rack-input-tests.ru | 21 ------ t/t0100-rack-input-tests.sh | 124 ------------------------------------ 6 files changed, 124 insertions(+), 184 deletions(-) delete mode 100755 t/bin/content-md5-put delete mode 100644 t/rack-input-tests.ru delete mode 100755 t/t0100-rack-input-tests.sh diff --git a/t/bin/content-md5-put b/t/bin/content-md5-put deleted file mode 100755 index 01da0bb..0000000 --- a/t/bin/content-md5-put +++ /dev/null @@ -1,36 +0,0 @@ -#!/usr/bin/env ruby -# -*- encoding: binary -*- -# simple chunked HTTP PUT request generator (and just that), -# it reads stdin and writes to stdout so socat can write to a -# UNIX or TCP socket (or to another filter or file) along with -# a Content-MD5 trailer. -require 'digest/md5' -$stdout.sync = $stderr.sync = true -$stdout.binmode -$stdin.binmode - -bs = ENV['bs'] ? ENV['bs'].to_i : 4096 - -if ARGV.grep("--no-headers").empty? - $stdout.write( - "PUT / HTTP/1.1\r\n" \ - "Host: example.com\r\n" \ - "Transfer-Encoding: chunked\r\n" \ - "Trailer: Content-MD5\r\n" \ - "\r\n" - ) -end - -digest = Digest::MD5.new -if buf = $stdin.readpartial(bs) - begin - digest.update(buf) - $stdout.write("%x\r\n" % [ buf.size ]) - $stdout.write(buf) - $stdout.write("\r\n") - end while $stdin.read(bs, buf) -end - -digest = [ digest.digest ].pack('m').strip -$stdout.write("0\r\n") -$stdout.write("Content-MD5: #{digest}\r\n\r\n") diff --git a/t/integration.ru b/t/integration.ru index 21f5449..98528f6 100644 --- a/t/integration.ru +++ b/t/integration.ru @@ -47,6 +47,29 @@ def env_dump(env) h.to_json end +def rack_input_tests(env) + return [ 100, {}, [] ] if /\A100-continue\z/i =~ env['HTTP_EXPECT'] + cap = 16384 + require 'digest/sha1' + digest = Digest::SHA1.new + input = env['rack.input'] + case env['PATH_INFO'] + when '/rack_input/size_first'; input.size + when '/rack_input/rewind_first'; input.rewind + when '/rack_input'; # OK + else + abort "bad path: #{env['PATH_INFO']}" + end + if buf = input.read(rand(cap)) + begin + raise "#{buf.size} > #{cap}" if buf.size > cap + digest.update(buf) + end while input.read(rand(cap), buf) + end + [ 200, {'content-length' => '40', 'content-type' => 'text/plain'}, + [ digest.hexdigest ] ] +end + run(lambda do |env| case env['REQUEST_METHOD'] when 'GET' @@ -66,6 +89,8 @@ def env_dump(env) end # case PATH_INFO (POST) # ... when 'PUT' - # ... + case env['PATH_INFO'] + when %r{\A/rack_input}; rack_input_tests(env) + end end # case REQUEST_METHOD end) # run diff --git a/t/integration.t b/t/integration.t index b7ba1fb..8cef561 100644 --- a/t/integration.t +++ b/t/integration.t @@ -1,13 +1,16 @@ #!perl -w # Copyright (C) unicorn hackers <unicorn-public@yhbt.net> # License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt> +# this is the main integration test for things which don't require +# restarting or signals use v5.14; BEGIN { require './t/lib.perl' }; my $srv = tcp_server(); my $host_port = tcp_host_port($srv); my $t0 = time; my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv }); - +my $curl = which('curl'); +END { diag slurp("$tmpdir/err.log") if $tmpdir }; sub slurp_hdr { my ($c) = @_; local $/ = "\r\n\r\n"; # affects both readline+chomp @@ -17,6 +20,48 @@ sub slurp_hdr { ($status, \@hdr); } +my %PUT = ( + chunked_md5 => sub { + my ($in, $out, $path, %opt) = @_; + my $bs = $opt{bs} // 16384; + require Digest::MD5; + my $dig = Digest::MD5->new; + print $out <<EOM; +PUT $path HTTP/1.1\r +Transfer-Encoding: chunked\r +Trailer: Content-MD5\r +\r +EOM + my ($buf, $r); + while (1) { + $r = read($in, $buf, $bs) // die "read: $!"; + last if $r == 0; + printf $out "%x\r\n", length($buf); + print $out $buf, "\r\n"; + $dig->add($buf); + } + print $out "0\r\nContent-MD5: ", $dig->b64digest, "\r\n\r\n"; + }, + identity => sub { + my ($in, $out, $path, %opt) = @_; + my $bs = $opt{bs} // 16384; + my $clen = $opt{-s} // -s $in; + print $out <<EOM; +PUT $path HTTP/1.0\r +Content-Length: $clen\r +\r +EOM + my ($buf, $r, $len); + while ($clen) { + $len = $clen > $bs ? $bs : $clen; + $r = read($in, $buf, $len) // die "read: $!"; + die 'premature EOF' if $r == 0; + print $out $buf; + $clen -= $r; + } + }, +); + my ($c, $status, $hdr); # response header tests @@ -111,6 +156,55 @@ if ('bad requests') { like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)'); } +# input tests +my ($blob_size, $blob_hash); +SKIP: { + open(my $rh, '<', 't/random_blob') or + skip "t/random_blob not generated $!", 1; + $blob_size = -s $rh; + require Digest::SHA; + $blob_hash = Digest::SHA->new(1)->addfile($rh)->hexdigest; + + my $ck_hash = sub { + my ($sub, $path, %opt) = @_; + seek($rh, 0, SEEK_SET) // die "seek: $!"; + $c = tcp_connect($srv); + $c->autoflush(0); + $PUT{$sub}->($rh, $c, $path, %opt); + $c->flush or die "flush: $!"; + ($status, $hdr) = slurp_hdr($c); + is(readline($c), $blob_hash, "$sub $path"); + }; + $ck_hash->('identity', '/rack_input', -s => $blob_size); + $ck_hash->('chunked_md5', '/rack_input'); + $ck_hash->('identity', '/rack_input/size_first', -s => $blob_size); + $ck_hash->('identity', '/rack_input/rewind_first', -s => $blob_size); + $ck_hash->('chunked_md5', '/rack_input/size_first'); + $ck_hash->('chunked_md5', '/rack_input/rewind_first'); + + + $curl // skip 'no curl found in PATH', 1; + + my ($copt, $cout); + my $url = "http://$host_port/rack_input"; + my $do_curl = sub { + my (@arg) = @_; + pipe(my $cout, $copt->{1}) or die "pipe: $!"; + open $copt->{2}, '>', "$tmpdir/curl.err" or die $!; + my $cpid = spawn($curl, '-sSf', @arg, $url, $copt); + close(delete $copt->{1}) or die "close: $!"; + is(readline($cout), $blob_hash, "curl @arg response"); + is(waitpid($cpid, 0), $cpid, "curl @arg exited"); + is($?, 0, "no error from curl @arg"); + is(slurp("$tmpdir/curl.err"), '', "no stderr from curl @arg"); + }; + + $do_curl->(qw(-T t/random_blob)); + + seek($rh, 0, SEEK_SET) // die "seek: $!"; + $copt->{0} = $rh; + $do_curl->('-T-'); +} # ... more stuff here undef $ar; @@ -120,4 +214,5 @@ my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log)); is_deeply(\@err, [], 'no unexpected errors in stderr'); is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr'); +undef $tmpdir; done_testing; diff --git a/t/lib.perl b/t/lib.perl index 7d712b5..ae9f197 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -10,7 +10,7 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD); use File::Temp 0.19 (); # 0.19 for ->newdir our ($tmpdir, $errfh); our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh - SEEK_SET tcp_host_port start_req); + SEEK_SET tcp_host_port start_req which spawn); my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!); $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1); @@ -193,4 +193,5 @@ Test::More->import; # try to ensure ->DESTROY fires: $SIG{TERM} = sub { exit(15 + 128) }; $SIG{INT} = sub { exit(2 + 128) }; +$SIG{PIPE} = sub { exit(13 + 128) }; 1; diff --git a/t/rack-input-tests.ru b/t/rack-input-tests.ru deleted file mode 100644 index 5459e85..0000000 --- a/t/rack-input-tests.ru +++ /dev/null @@ -1,21 +0,0 @@ -# SHA1 checksum generator -require 'digest/sha1' -use Rack::ContentLength -cap = 16384 -app = lambda do |env| - /\A100-continue\z/i =~ env['HTTP_EXPECT'] and - return [ 100, {}, [] ] - digest = Digest::SHA1.new - input = env['rack.input'] - input.size if env["PATH_INFO"] == "/size_first" - input.rewind if env["PATH_INFO"] == "/rewind_first" - if buf = input.read(rand(cap)) - begin - raise "#{buf.size} > #{cap}" if buf.size > cap - digest.update(buf) - end while input.read(rand(cap), buf) - end - - [ 200, {'content-type' => 'text/plain'}, [ digest.hexdigest << "\n" ] ] -end -run app diff --git a/t/t0100-rack-input-tests.sh b/t/t0100-rack-input-tests.sh deleted file mode 100755 index ee7a437..0000000 --- a/t/t0100-rack-input-tests.sh +++ /dev/null @@ -1,124 +0,0 @@ -#!/bin/sh -. ./test-lib.sh -test -r random_blob || die "random_blob required, run with 'make $0'" - -t_plan 10 "rack.input read tests" - -t_begin "setup and startup" && { - rtmpfiles curl_out curl_err - unicorn_setup - unicorn -E none -D rack-input-tests.ru -c $unicorn_config - blob_sha1=$(rsha1 < random_blob) - blob_size=$(count_bytes < random_blob) - t_info "blob_sha1=$blob_sha1" - unicorn_wait_start -} - -t_begin "corked identity request" && { - rm -f $tmp - ( - cat $fifo > $tmp & - printf 'PUT / HTTP/1.0\r\n' - printf 'Content-Length: %d\r\n\r\n' $blob_size - cat random_blob - wait - echo ok > $ok - ) | ( sleep 1 && socat - TCP4:$listen > $fifo ) - test 1 -eq $(grep $blob_sha1 $tmp |count_lines) - test x"$(cat $ok)" = xok -} - -t_begin "corked chunked request" && { - rm -f $tmp - ( - cat $fifo > $tmp & - content-md5-put < random_blob - wait - echo ok > $ok - ) | ( sleep 1 && socat - TCP4:$listen > $fifo ) - test 1 -eq $(grep $blob_sha1 $tmp |count_lines) - test x"$(cat $ok)" = xok -} - -t_begin "corked identity request (input#size first)" && { - rm -f $tmp - ( - cat $fifo > $tmp & - printf 'PUT /size_first HTTP/1.0\r\n' - printf 'Content-Length: %d\r\n\r\n' $blob_size - cat random_blob - wait - echo ok > $ok - ) | ( sleep 1 && socat - TCP4:$listen > $fifo ) - test 1 -eq $(grep $blob_sha1 $tmp |count_lines) - test x"$(cat $ok)" = xok -} - -t_begin "corked identity request (input#rewind first)" && { - rm -f $tmp - ( - cat $fifo > $tmp & - printf 'PUT /rewind_first HTTP/1.0\r\n' - printf 'Content-Length: %d\r\n\r\n' $blob_size - cat random_blob - wait - echo ok > $ok - ) | ( sleep 1 && socat - TCP4:$listen > $fifo ) - test 1 -eq $(grep $blob_sha1 $tmp |count_lines) - test x"$(cat $ok)" = xok -} - -t_begin "corked chunked request (input#size first)" && { - rm -f $tmp - ( - cat $fifo > $tmp & - printf 'PUT /size_first HTTP/1.1\r\n' - printf 'Host: example.com\r\n' - printf 'Transfer-Encoding: chunked\r\n' - printf 'Trailer: Content-MD5\r\n' - printf '\r\n' - content-md5-put --no-headers < random_blob - wait - echo ok > $ok - ) | ( sleep 1 && socat - TCP4:$listen > $fifo ) - test 1 -eq $(grep $blob_sha1 $tmp |count_lines) - test 1 -eq $(grep $blob_sha1 $tmp |count_lines) - test x"$(cat $ok)" = xok -} - -t_begin "corked chunked request (input#rewind first)" && { - rm -f $tmp - ( - cat $fifo > $tmp & - printf 'PUT /rewind_first HTTP/1.1\r\n' - printf 'Host: example.com\r\n' - printf 'Transfer-Encoding: chunked\r\n' - printf 'Trailer: Content-MD5\r\n' - printf '\r\n' - content-md5-put --no-headers < random_blob - wait - echo ok > $ok - ) | ( sleep 1 && socat - TCP4:$listen > $fifo ) - test 1 -eq $(grep $blob_sha1 $tmp |count_lines) - test x"$(cat $ok)" = xok -} - -t_begin "regular request" && { - curl -sSf -T random_blob http://$listen/ > $curl_out 2> $curl_err - test x$blob_sha1 = x$(cat $curl_out) - test ! -s $curl_err -} - -t_begin "chunked request" && { - curl -sSf -T- < random_blob http://$listen/ > $curl_out 2> $curl_err - test x$blob_sha1 = x$(cat $curl_out) - test ! -s $curl_err -} - -dbgcat r_err - -t_begin "shutdown" && { - kill $unicorn_pid -} - -t_done [-- Attachment #10: 0009-tests-use-autodie-to-simplify-error-checking.patch --] [-- Type: text/x-diff, Size: 8495 bytes --] From 3a1d015a3859b639d8e4463e9436a49f4f0f720e Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:38 +0000 Subject: [PATCH 09/23] tests: use autodie to simplify error checking autodie is bundled with Perl 5.10+ and simplifies error checking in most cases. Some subroutines aren't perfectly translatable and their call sites had to be tweaked, but most of them are. --- t/active-unix-socket.t | 13 +++++++------ t/integration.t | 37 +++++++++++++++++++------------------ t/lib.perl | 30 +++++++++++++++--------------- 3 files changed, 41 insertions(+), 39 deletions(-) diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t index 6b5c218..1241904 100644 --- a/t/active-unix-socket.t +++ b/t/active-unix-socket.t @@ -4,17 +4,18 @@ use v5.14; BEGIN { require './t/lib.perl' }; use IO::Socket::UNIX; +use autodie; +no autodie 'kill'; my %to_kill; END { kill('TERM', values(%to_kill)) if keys %to_kill } my $u1 = "$tmpdir/u1.sock"; my $u2 = "$tmpdir/u2.sock"; my $unix_req = sub { my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM); - print $s @_, "\r\n\r\n" or die $!; + print $s @_, "\r\n\r\n"; $s; }; { - use autodie; open my $fh, '>', "$tmpdir/u1.conf.rb"; print $fh <<EOM; pid "$tmpdir/u.pid" @@ -43,8 +44,8 @@ EOM my @uarg = qw(-D -E none t/integration.ru); # this pipe will be used to notify us when all daemons die: -pipe(my ($p0, $p1)) or die "pipe: $!"; -fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC +pipe(my $p0, my $p1); +fcntl($p1, POSIX::F_SETFD, 0); # start the first instance unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join; @@ -93,8 +94,8 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure'); # restart the first instance { - pipe(($p0, $p1)) or die "pipe: $!"; - fcntl($p1, POSIX::F_SETFD, 0) or die "fcntl: $!"; # clear FD_CLOEXEC + pipe($p0, $p1); + fcntl($p1, POSIX::F_SETFD, 0); unicorn('-c', "$tmpdir/u1.conf.rb", @uarg)->join; is($?, 0, 'daemonized 1st process'); chomp($to_kill{u1} = slurp("$tmpdir/u.pid")); diff --git a/t/integration.t b/t/integration.t index 8cef561..af17d51 100644 --- a/t/integration.t +++ b/t/integration.t @@ -5,6 +5,7 @@ # restarting or signals use v5.14; BEGIN { require './t/lib.perl' }; +use autodie; my $srv = tcp_server(); my $host_port = tcp_host_port($srv); my $t0 = time; @@ -34,7 +35,7 @@ Trailer: Content-MD5\r EOM my ($buf, $r); while (1) { - $r = read($in, $buf, $bs) // die "read: $!"; + $r = read($in, $buf, $bs); last if $r == 0; printf $out "%x\r\n", length($buf); print $out $buf, "\r\n"; @@ -54,7 +55,7 @@ EOM my ($buf, $r, $len); while ($clen) { $len = $clen > $bs ? $bs : $clen; - $r = read($in, $buf, $len) // die "read: $!"; + $r = read($in, $buf, $len); die 'premature EOF' if $r == 0; print $out $buf; $clen -= $r; @@ -130,28 +131,28 @@ if ('bad requests') { like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request'); $c = tcp_connect($srv); - print $c 'GET /' or die $!; + print $c 'GET /'; my $buf = join('', (0..9), 'ab'); - for (0..1023) { print $c $buf or die $! } - print $c " HTTP/1.0\r\n\r\n" or die $!; + for (0..1023) { print $c $buf } + print $c " HTTP/1.0\r\n\r\n"; ($status, $hdr) = slurp_hdr($c); like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on REQUEST_PATH > (12 * 1024)'); $c = tcp_connect($srv); - print $c 'GET /hello-world?a' or die $!; + print $c 'GET /hello-world?a'; $buf = join('', (0..9)); - for (0..1023) { print $c $buf or die $! } - print $c " HTTP/1.0\r\n\r\n" or die $!; + for (0..1023) { print $c $buf } + print $c " HTTP/1.0\r\n\r\n"; ($status, $hdr) = slurp_hdr($c); like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on QUERY_STRING > (10 * 1024)'); $c = tcp_connect($srv); - print $c 'GET /hello-world#a' or die $!; + print $c 'GET /hello-world#a'; $buf = join('', (0..9), 'a'..'f'); - for (0..63) { print $c $buf or die $! } - print $c " HTTP/1.0\r\n\r\n" or die $!; + for (0..63) { print $c $buf } + print $c " HTTP/1.0\r\n\r\n"; ($status, $hdr) = slurp_hdr($c); like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on FRAGMENT > (1024)'); } @@ -159,7 +160,7 @@ if ('bad requests') { # input tests my ($blob_size, $blob_hash); SKIP: { - open(my $rh, '<', 't/random_blob') or + CORE::open(my $rh, '<', 't/random_blob') or skip "t/random_blob not generated $!", 1; $blob_size = -s $rh; require Digest::SHA; @@ -167,11 +168,11 @@ SKIP: { my $ck_hash = sub { my ($sub, $path, %opt) = @_; - seek($rh, 0, SEEK_SET) // die "seek: $!"; + seek($rh, 0, SEEK_SET); $c = tcp_connect($srv); $c->autoflush(0); $PUT{$sub}->($rh, $c, $path, %opt); - $c->flush or die "flush: $!"; + $c->flush or die $!; ($status, $hdr) = slurp_hdr($c); is(readline($c), $blob_hash, "$sub $path"); }; @@ -189,10 +190,10 @@ SKIP: { my $url = "http://$host_port/rack_input"; my $do_curl = sub { my (@arg) = @_; - pipe(my $cout, $copt->{1}) or die "pipe: $!"; - open $copt->{2}, '>', "$tmpdir/curl.err" or die $!; + pipe(my $cout, $copt->{1}); + open $copt->{2}, '>', "$tmpdir/curl.err"; my $cpid = spawn($curl, '-sSf', @arg, $url, $copt); - close(delete $copt->{1}) or die "close: $!"; + close(delete $copt->{1}); is(readline($cout), $blob_hash, "curl @arg response"); is(waitpid($cpid, 0), $cpid, "curl @arg exited"); is($?, 0, "no error from curl @arg"); @@ -201,7 +202,7 @@ SKIP: { $do_curl->(qw(-T t/random_blob)); - seek($rh, 0, SEEK_SET) // die "seek: $!"; + seek($rh, 0, SEEK_SET); $copt->{0} = $rh; $do_curl->('-T-'); } diff --git a/t/lib.perl b/t/lib.perl index ae9f197..49632cf 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -4,6 +4,7 @@ package UnicornTest; use v5.14; use parent qw(Exporter); +use autodie; use Test::More; use IO::Socket::INET; use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD); @@ -14,7 +15,7 @@ our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!); $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1); -open($errfh, '>>', "$tmpdir/err.log") or die "open: $!"; +open($errfh, '>>', "$tmpdir/err.log"); sub tcp_server { my %opt = ( @@ -62,14 +63,14 @@ sub tcp_connect { sub start_req { my ($srv, @req) = @_; my $c = tcp_connect($srv); - print $c @req, "\r\n\r\n" or die "print: $!"; + print $c @req, "\r\n\r\n"; $c; } sub slurp { - open my $fh, '<', $_[0] or die "open($_[0]): $!"; + open my $fh, '<', $_[0]; local $/; - <$fh>; + readline($fh); } sub spawn { @@ -80,8 +81,8 @@ sub spawn { my $set = POSIX::SigSet->new; $set->fillset or die "sigfillset: $!"; sigprocmask(SIG_SETMASK, $set, $old) or die "SIG_SETMASK: $!"; - pipe(my ($r, $w)) or die "pipe: $!"; - my $pid = fork // die "fork: $!"; + pipe(my $r, my $w); + my $pid = fork; if ($pid == 0) { close $r; $SIG{__DIE__} = sub { @@ -94,9 +95,9 @@ sub spawn { my $cfd; for ($cfd = 0; ($cfd < 3) || defined($opt->{$cfd}); $cfd++) { my $io = $opt->{$cfd} // next; - my $pfd = fileno($io) // die "fileno($io): $!"; + my $pfd = fileno($io); if ($pfd == $cfd) { - fcntl($io, F_SETFD, 0) // die "F_SETFD: $!"; + fcntl($io, F_SETFD, 0); } else { dup2($pfd, $cfd) // die "dup2($pfd, $cfd): $!"; } @@ -110,9 +111,7 @@ sub spawn { setpgid(0, $pgid) // die "setpgid(0, $pgid): $!"; } $SIG{$_} = 'DEFAULT' for grep(!/^__/, keys %SIG); - if (defined(my $cd = $opt->{-C})) { - chdir $cd // die "chdir($cd): $!"; - } + if (defined(my $cd = $opt->{-C})) { chdir $cd } $old->delset(POSIX::SIGCHLD) or die "sigdelset CHLD: $!"; sigprocmask(SIG_SETMASK, $old) or die "SIG_SETMASK: ~CHLD: $!"; @ENV{keys %$env} = values(%$env) if $env; @@ -162,22 +161,23 @@ sub unicorn { # automatically kill + reap children when this goes out-of-scope package UnicornTest::AutoReap; use v5.14; +use autodie; sub new { my (undef, $pid) = @_; bless { pid => $pid, owner => $$ }, __PACKAGE__ } -sub kill { +sub do_kill { my ($self, $sig) = @_; - CORE::kill($sig // 'TERM', $self->{pid}); + kill($sig // 'TERM', $self->{pid}); } sub join { my ($self, $sig) = @_; my $pid = delete $self->{pid} or return; - CORE::kill($sig, $pid) if defined $sig; - my $ret = waitpid($pid, 0) // die "waitpid($pid): $!"; + kill($sig, $pid) if defined $sig; + my $ret = waitpid($pid, 0); $ret == $pid or die "BUG: waitpid($pid) != $ret"; } [-- Attachment #11: 0010-port-t0019-max_header_len.sh-to-Perl-5.patch --] [-- Type: text/x-diff, Size: 5571 bytes --] From 43c7d73b8b9e6995b5a986b10a8623395e89a538 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:39 +0000 Subject: [PATCH 10/23] port t0019-max_header_len.sh to Perl 5 This was the final socat requirement for integration tests. I think curl will remain an optional dependency for tests since it's probably the most widely-installed HTTP client. --- GNUmakefile | 2 +- t/README | 7 +----- t/integration.ru | 1 + t/integration.t | 43 +++++++++++++++++++++++++++++++--- t/t0019-max_header_len.sh | 49 --------------------------------------- 5 files changed, 43 insertions(+), 59 deletions(-) delete mode 100755 t/t0019-max_header_len.sh diff --git a/GNUmakefile b/GNUmakefile index 5cca189..eab9082 100644 --- a/GNUmakefile +++ b/GNUmakefile @@ -125,7 +125,7 @@ $(T_sh): dep $(test_prereq) t/random_blob t/trash/.gitignore t/trash/.gitignore : | t/trash echo '*' >$@ -dependencies := socat curl +dependencies := curl deps := $(addprefix t/.dep+,$(dependencies)) $(deps): dep_bin = $(lastword $(subst +, ,$@)) $(deps): diff --git a/t/README b/t/README index 8a5243e..d09c715 100644 --- a/t/README +++ b/t/README @@ -10,18 +10,13 @@ to test real-world behavior and Ruby introduces incompatibilities at a far faster rate than Perl 5. Perl is Ruby's older cousin, so it should be easy-to-learn for Rubyists. -Old tests are in Bourne shell, but the socat(1) dependency was probably -too rare compared to Perl 5. +Old tests are in Bourne shell and slowly being ported to Perl 5. == Requirements * {Ruby 2.0.0+}[https://www.ruby-lang.org/en/] * {Perl 5.14+}[https://www.perl.org/] # your distro should have it * {GNU make}[https://www.gnu.org/software/make/] - -The following requirements will eventually be dropped. - -* {socat}[http://www.dest-unreach.org/socat/] * {curl}[https://curl.haxx.se/] We do not use bashisms or any non-portable, non-POSIX constructs diff --git a/t/integration.ru b/t/integration.ru index 98528f6..edc408c 100644 --- a/t/integration.ru +++ b/t/integration.ru @@ -81,6 +81,7 @@ def rack_input_tests(env) when '/env_dump'; [ 200, {}, [ env_dump(env) ] ] when '/write_on_close'; write_on_close when '/pid'; [ 200, {}, [ "#$$\n" ] ] + else '/'; [ 200, {}, [ env_dump(env) ] ] end # case PATH_INFO (GET) when 'POST' case env['PATH_INFO'] diff --git a/t/integration.t b/t/integration.t index af17d51..c687655 100644 --- a/t/integration.t +++ b/t/integration.t @@ -1,15 +1,19 @@ #!perl -w # Copyright (C) unicorn hackers <unicorn-public@yhbt.net> # License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt> -# this is the main integration test for things which don't require -# restarting or signals + +# This is the main integration test for fast-ish things to minimize +# Ruby startup time penalties. use v5.14; BEGIN { require './t/lib.perl' }; use autodie; my $srv = tcp_server(); my $host_port = tcp_host_port($srv); my $t0 = time; -my $ar = unicorn(qw(-E none t/integration.ru), { 3 => $srv }); +my $conf = "$tmpdir/u.conf.rb"; +open my $conf_fh, '>', $conf; +$conf_fh->autoflush(1); +my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv }); my $curl = which('curl'); END { diag slurp("$tmpdir/err.log") if $tmpdir }; sub slurp_hdr { @@ -207,7 +211,40 @@ SKIP: { $do_curl->('-T-'); } + # ... more stuff here + +# SIGHUP-able stuff goes here + +if ('max_header_len internal API') { + undef $c; + my $req = 'GET / HTTP/1.0'; + my $len = length($req."\r\n\r\n"); + my $fifo = "$tmpdir/fifo"; + POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!"; + print $conf_fh <<EOM; +Unicorn::HttpParser.max_header_len = $len +listen "$host_port" # TODO: remove this requirement for SIGHUP +after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } } +EOM + $ar->do_kill('HUP'); + open my $fifo_fh, '<', $fifo; + my $wpid = readline($fifo_fh); + like($wpid, qr/\Apid=\d+\z/a , 'new worker ready'); + close $fifo_fh; + $wpid =~ s/\Apid=// or die; + ok(CORE::kill(0, $wpid), 'worker PID retrieved'); + + $c = start_req($srv, $req); + ($status, $hdr) = slurp_hdr($c); + like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds'); + + $c = start_req($srv, 'GET /xxxxxx HTTP/1.0'); + ($status, $hdr) = slurp_hdr($c); + like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails'); +} + + undef $ar; my @log = slurp("$tmpdir/err.log"); diag("@log") if $ENV{V}; diff --git a/t/t0019-max_header_len.sh b/t/t0019-max_header_len.sh deleted file mode 100755 index 6a355b4..0000000 --- a/t/t0019-max_header_len.sh +++ /dev/null @@ -1,49 +0,0 @@ -#!/bin/sh -. ./test-lib.sh -t_plan 5 "max_header_len setting (only intended for Rainbows!)" - -t_begin "setup and start" && { - unicorn_setup - req='GET / HTTP/1.0\r\n\r\n' - len=$(printf "$req" | count_bytes) - echo Unicorn::HttpParser.max_header_len = $len >> $unicorn_config - unicorn -D -c $unicorn_config env.ru - unicorn_wait_start -} - -t_begin "minimal request succeeds" && { - rm -f $tmp - ( - cat $fifo > $tmp & - printf "$req" - wait - echo ok > $ok - ) | socat - TCP:$listen > $fifo - test xok = x$(cat $ok) - - fgrep "HTTP/1.1 200 OK" $tmp -} - -t_begin "big request fails" && { - rm -f $tmp - ( - cat $fifo > $tmp & - printf 'GET /xxxxxx HTTP/1.0\r\n\r\n' - wait - echo ok > $ok - ) | socat - TCP:$listen > $fifo - test xok = x$(cat $ok) - fgrep "HTTP/1.1 413" $tmp -} - -dbgcat tmp - -t_begin "killing succeeds" && { - kill $unicorn_pid -} - -t_begin "check stderr" && { - check_stderr -} - -t_done [-- Attachment #12: 0011-test_exec-drop-sd_listen_fds-emulation-test.patch --] [-- Type: text/x-diff, Size: 1751 bytes --] From 5d828a4ef7683345bcf2ff659442fed0a6fb7a97 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:40 +0000 Subject: [PATCH 11/23] test_exec: drop sd_listen_fds emulation test The Perl 5 tests already rely on this implicitly, and there was never a point when Perl 5 couldn't emulate systemd behavior. --- test/exec/test_exec.rb | 33 --------------------------------- 1 file changed, 33 deletions(-) diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb index 2929b2e..1d3a0fd 100644 --- a/test/exec/test_exec.rb +++ b/test/exec/test_exec.rb @@ -97,39 +97,6 @@ def teardown end end - def test_sd_listen_fds_emulation - # [ruby-core:69895] [Bug #11336] fixed by r51576 - return if RUBY_VERSION.to_f < 2.3 - - File.open("config.ru", "wb") { |fp| fp.write(HI) } - sock = TCPServer.new(@addr, @port) - - [ %W(-l #@addr:#@port), nil ].each do |l| - sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0) - - pid = xfork do - redirect_test_io do - # pretend to be systemd - ENV['LISTEN_PID'] = "#$$" - ENV['LISTEN_FDS'] = '1' - - # 3 = SD_LISTEN_FDS_START - args = [ $unicorn_bin ] - args.concat(l) if l - args << { 3 => sock } - exec(*args) - end - end - res = hit(["http://#@addr:#@port/"]) - assert_equal [ "HI\n" ], res - assert_shutdown(pid) - assert sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).bool, - 'unicorn should always set SO_KEEPALIVE on inherited sockets' - end - ensure - sock.close if sock - end - def test_inherit_listener_unspecified File.open("config.ru", "wb") { |fp| fp.write(HI) } sock = TCPServer.new(@addr, @port) [-- Attachment #13: 0012-test_exec-drop-test_basic-and-test_config_ru_alt_pat.patch --] [-- Type: text/x-diff, Size: 1667 bytes --] From 548593c6b3d52a4bebd52542ad9c423ed2b7252d Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:41 +0000 Subject: [PATCH 12/23] test_exec: drop test_basic and test_config_ru_alt_path We already have coverage for these basic things elsewhere. --- test/exec/test_exec.rb | 24 ------------------------ 1 file changed, 24 deletions(-) diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb index 1d3a0fd..55f828e 100644 --- a/test/exec/test_exec.rb +++ b/test/exec/test_exec.rb @@ -265,16 +265,6 @@ def test_exit_signals end end - def test_basic - File.open("config.ru", "wb") { |fp| fp.syswrite(HI) } - pid = fork do - redirect_test_io { exec($unicorn_bin, "-l", "#{@addr}:#{@port}") } - end - results = retry_hit(["http://#{@addr}:#{@port}/"]) - assert_equal String, results[0].class - assert_shutdown(pid) - end - def test_rack_env_unset File.open("config.ru", "wb") { |fp| fp.syswrite(SHOW_RACK_ENV) } pid = fork { redirect_test_io { exec($unicorn_bin, "-l#@addr:#@port") } } @@ -638,20 +628,6 @@ def test_read_embedded_cli_switches assert_shutdown(pid) end - def test_config_ru_alt_path - config_path = "#{@tmpdir}/foo.ru" - File.open(config_path, "wb") { |fp| fp.syswrite(HI) } - pid = fork do - redirect_test_io do - Dir.chdir("/") - exec($unicorn_bin, "-l#{@addr}:#{@port}", config_path) - end - end - results = retry_hit(["http://#{@addr}:#{@port}/"]) - assert_equal String, results[0].class - assert_shutdown(pid) - end - def test_load_module libdir = "#{@tmpdir}/lib" FileUtils.mkpath([ libdir ]) [-- Attachment #14: 0013-tests-check_stderr-consistently-in-Perl-5-tests.patch --] [-- Type: text/x-diff, Size: 2415 bytes --] From cd7ee67fc8ebadec9bdd913d49ed3f214596ea47 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:42 +0000 Subject: [PATCH 13/23] tests: check_stderr consistently in Perl 5 tests The Bourne shell tests did, so lets not let stuff sneak past us. --- t/active-unix-socket.t | 5 ++--- t/integration.t | 7 ++----- t/lib.perl | 10 +++++++++- 3 files changed, 13 insertions(+), 9 deletions(-) diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t index 1241904..c132dc2 100644 --- a/t/active-unix-socket.t +++ b/t/active-unix-socket.t @@ -20,7 +20,7 @@ my $unix_req = sub { print $fh <<EOM; pid "$tmpdir/u.pid" listen "$u1" -stderr_path "$tmpdir/err1.log" +stderr_path "$tmpdir/err.log" EOM close $fh; @@ -113,6 +113,5 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure'); ok(-S $u1, 'socket stays after SIGTERM'); } -my @log = slurp("$tmpdir/err.log"); -diag("@log") if $ENV{V}; +check_stderr; done_testing; diff --git a/t/integration.t b/t/integration.t index c687655..939dc24 100644 --- a/t/integration.t +++ b/t/integration.t @@ -246,11 +246,8 @@ EOM undef $ar; -my @log = slurp("$tmpdir/err.log"); -diag("@log") if $ENV{V}; -my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log)); -is_deeply(\@err, [], 'no unexpected errors in stderr'); -is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr'); + +check_stderr; undef $tmpdir; done_testing; diff --git a/t/lib.perl b/t/lib.perl index 49632cf..315ef2d 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -11,12 +11,20 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD); use File::Temp 0.19 (); # 0.19 for ->newdir our ($tmpdir, $errfh); our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh - SEEK_SET tcp_host_port start_req which spawn); + SEEK_SET tcp_host_port start_req which spawn check_stderr); my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!); $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1); open($errfh, '>>', "$tmpdir/err.log"); +sub check_stderr () { + my @log = slurp("$tmpdir/err.log"); + diag("@log") if $ENV{V}; + my @err = grep(!/NameError.*Unicorn::Waiter/, grep(/error/i, @log)); + is_deeply(\@err, [], 'no unexpected errors in stderr'); + is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr'); +} + sub tcp_server { my %opt = ( ReuseAddr => 1, [-- Attachment #15: 0014-tests-consistent-tcp_start-and-unix_start-across-Per.patch --] [-- Type: text/x-diff, Size: 8017 bytes --] From 0dcd8bd569813a175ad43837db3ab07019a95b99 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:43 +0000 Subject: [PATCH 14/23] tests: consistent tcp_start and unix_start across Perl 5 tests I'll be using Unix sockets more in tests since there's no risk of system-wide conflicts with TCP port allocation. Furthermore, curl supports `--unix-socket' nowadays; so there's little reason to rely on TCP sockets and the conflicts they bring in tests. --- t/active-unix-socket.t | 13 ++++--------- t/integration.t | 28 ++++++++++++++-------------- t/lib.perl | 30 ++++++++++++++++-------------- 3 files changed, 34 insertions(+), 37 deletions(-) diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t index c132dc2..8723137 100644 --- a/t/active-unix-socket.t +++ b/t/active-unix-socket.t @@ -10,11 +10,6 @@ my %to_kill; END { kill('TERM', values(%to_kill)) if keys %to_kill } my $u1 = "$tmpdir/u1.sock"; my $u2 = "$tmpdir/u2.sock"; -my $unix_req = sub { - my $s = IO::Socket::UNIX->new(Peer => shift, Type => SOCK_STREAM); - print $s @_, "\r\n\r\n"; - $s; -}; { open my $fh, '>', "$tmpdir/u1.conf.rb"; print $fh <<EOM; @@ -53,7 +48,7 @@ is($?, 0, 'daemonized 1st process'); chomp($to_kill{u1} = slurp("$tmpdir/u.pid")); like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file'); -chomp(my $worker_pid = readline($unix_req->($u1, 'GET /pid'))); +chomp(my $worker_pid = readline(unix_start($u1, 'GET /pid'))); like($worker_pid, qr/\A\d+\z/s, 'captured worker pid'); ok(kill(0, $worker_pid), 'worker is kill-able'); @@ -65,7 +60,7 @@ isnt($?, 0, 'conflicting PID file fails to start'); chomp(my $pidf = slurp("$tmpdir/u.pid")); is($pidf, $to_kill{u1}, 'pid file contents unchanged after start failure'); -chomp(my $pid2 = readline($unix_req->($u1, 'GET /pid'))); +chomp(my $pid2 = readline(unix_start($u1, 'GET /pid'))); is($worker_pid, $pid2, 'worker PID unchanged'); @@ -73,7 +68,7 @@ is($worker_pid, $pid2, 'worker PID unchanged'); unicorn('-c', "$tmpdir/u3.conf.rb", @uarg)->join; isnt($?, 0, 'conflicting UNIX socket fails to start'); -chomp($pid2 = readline($unix_req->($u1, 'GET /pid'))); +chomp($pid2 = readline(unix_start($u1, 'GET /pid'))); is($worker_pid, $pid2, 'worker PID still unchanged'); chomp($pidf = slurp("$tmpdir/u.pid")); @@ -101,7 +96,7 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure'); chomp($to_kill{u1} = slurp("$tmpdir/u.pid")); like($to_kill{u1}, qr/\A\d+\z/s, 'read pid file'); - chomp($pid2 = readline($unix_req->($u1, 'GET /pid'))); + chomp($pid2 = readline(unix_start($u1, 'GET /pid'))); like($pid2, qr/\A\d+\z/, 'worker running'); ok(kill('TERM', delete $to_kill{u1}), 'SIGTERM restarted daemon'); diff --git a/t/integration.t b/t/integration.t index 939dc24..b33e3c3 100644 --- a/t/integration.t +++ b/t/integration.t @@ -70,7 +70,7 @@ EOM my ($c, $status, $hdr); # response header tests -$c = start_req($srv, 'GET /rack-2-newline-headers HTTP/1.0'); +$c = tcp_start($srv, 'GET /rack-2-newline-headers HTTP/1.0'); ($status, $hdr) = slurp_hdr($c); like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid'); my $orig_200_status = $status; @@ -89,7 +89,7 @@ SKIP: { # Date header check }; -$c = start_req($srv, 'GET /rack-3-array-headers HTTP/1.0'); +$c = tcp_start($srv, 'GET /rack-3-array-headers HTTP/1.0'); ($status, $hdr) = slurp_hdr($c); is_deeply([ grep(/^x-r3: /, @$hdr) ], [ 'x-r3: a', 'x-r3: b', 'x-r3: c' ], @@ -97,7 +97,7 @@ is_deeply([ grep(/^x-r3: /, @$hdr) ], SKIP: { eval { require JSON::PP } or skip "JSON::PP missing: $@", 1; - my $c = start_req($srv, 'GET /env_dump'); + my $c = tcp_start($srv, 'GET /env_dump'); my $json = do { local $/; readline($c) }; unlike($json, qr/^Connection: /smi, 'no connection header for 0.9'); unlike($json, qr!\AHTTP/!s, 'no HTTP/1.x prefix for 0.9'); @@ -107,17 +107,17 @@ SKIP: { } # cf. <CAO47=rJa=zRcLn_Xm4v2cHPr6c0UswaFC_omYFEH+baSxHOWKQ@mail.gmail.com> -$c = start_req($srv, 'GET /nil-header-value HTTP/1.0'); +$c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0'); ($status, $hdr) = slurp_hdr($c); is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '], 'nil header value accepted for broken apps') or diag(explain($hdr)); if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') { - $c = start_req($srv, 'POST /tweak-status-code HTTP/1.0'); + $c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0'); ($status, $hdr) = slurp_hdr($c); like($status, qr!\AHTTP/1\.[01] 200 HI\b!, 'status tweaked'); - $c = start_req($srv, 'POST /restore-status-code HTTP/1.0'); + $c = tcp_start($srv, 'POST /restore-status-code HTTP/1.0'); ($status, $hdr) = slurp_hdr($c); is($status, $orig_200_status, 'original status restored'); } @@ -130,12 +130,12 @@ SKIP: { } if ('bad requests') { - $c = start_req($srv, 'GET /env_dump HTTP/1/1'); + $c = tcp_start($srv, 'GET /env_dump HTTP/1/1'); ($status, $hdr) = slurp_hdr($c); like($status, qr!\AHTTP/1\.[01] 400 \b!, 'got 400 on bad request'); - $c = tcp_connect($srv); - print $c 'GET /'; + $c = tcp_start($srv); + print $c 'GET /';; my $buf = join('', (0..9), 'ab'); for (0..1023) { print $c $buf } print $c " HTTP/1.0\r\n\r\n"; @@ -143,7 +143,7 @@ if ('bad requests') { like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on REQUEST_PATH > (12 * 1024)'); - $c = tcp_connect($srv); + $c = tcp_start($srv); print $c 'GET /hello-world?a'; $buf = join('', (0..9)); for (0..1023) { print $c $buf } @@ -152,7 +152,7 @@ if ('bad requests') { like($status, qr!\AHTTP/1\.[01] 414 \b!, '414 on QUERY_STRING > (10 * 1024)'); - $c = tcp_connect($srv); + $c = tcp_start($srv); print $c 'GET /hello-world#a'; $buf = join('', (0..9), 'a'..'f'); for (0..63) { print $c $buf } @@ -173,7 +173,7 @@ SKIP: { my $ck_hash = sub { my ($sub, $path, %opt) = @_; seek($rh, 0, SEEK_SET); - $c = tcp_connect($srv); + $c = tcp_start($srv); $c->autoflush(0); $PUT{$sub}->($rh, $c, $path, %opt); $c->flush or die $!; @@ -235,11 +235,11 @@ EOM $wpid =~ s/\Apid=// or die; ok(CORE::kill(0, $wpid), 'worker PID retrieved'); - $c = start_req($srv, $req); + $c = tcp_start($srv, $req); ($status, $hdr) = slurp_hdr($c); like($status, qr!\AHTTP/1\.[01] 200\b!, 'minimal request succeeds'); - $c = start_req($srv, 'GET /xxxxxx HTTP/1.0'); + $c = tcp_start($srv, 'GET /xxxxxx HTTP/1.0'); ($status, $hdr) = slurp_hdr($c); like($status, qr!\AHTTP/1\.[01] 413\b!, 'big request fails'); } diff --git a/t/lib.perl b/t/lib.perl index 315ef2d..1d6e78d 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -10,8 +10,8 @@ use IO::Socket::INET; use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD); use File::Temp 0.19 (); # 0.19 for ->newdir our ($tmpdir, $errfh); -our @EXPORT = qw(unicorn slurp tcp_server tcp_connect unicorn $tmpdir $errfh - SEEK_SET tcp_host_port start_req which spawn check_stderr); +our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh + SEEK_SET tcp_host_port which spawn check_stderr unix_start); my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!); $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1); @@ -55,26 +55,28 @@ sub tcp_host_port { } } -sub tcp_connect { - my ($dest, %opt) = @_; - my $addr = tcp_host_port($dest); - my $s = ref($dest)->new( +sub unix_start ($@) { + my ($dst, @req) = @_; + my $s = IO::Socket::UNIX->new(Peer => $dst, Type => SOCK_STREAM) or + BAIL_OUT "unix connect $dst: $!"; + $s->autoflush(1); + print $s @req, "\r\n\r\n" if @req; + $s; +} + +sub tcp_start ($@) { + my ($dst, @req) = @_; + my $addr = tcp_host_port($dst); + my $s = ref($dst)->new( Proto => 'tcp', Type => SOCK_STREAM, PeerAddr => $addr, - %opt, ) or BAIL_OUT "failed to connect to $addr: $!"; $s->autoflush(1); + print $s @req, "\r\n\r\n" if @req; $s; } -sub start_req { - my ($srv, @req) = @_; - my $c = tcp_connect($srv); - print $c @req, "\r\n\r\n"; - $c; -} - sub slurp { open my $fh, '<', $_[0]; local $/; [-- Attachment #16: 0015-port-t9000-preread-input.sh-to-Perl-5.patch --] [-- Type: text/x-diff, Size: 3856 bytes --] From 1b8840d8d13491eecd2fa92e06f73c65eadd33ba Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:44 +0000 Subject: [PATCH 15/23] port t9000-preread-input.sh to Perl 5 Stuffing it into t/integration.t for now so we can save on startup costs. --- t/integration.t | 32 ++++++++++++++++++++++++--- t/lib.perl | 2 +- t/preread_input.ru | 4 +--- t/t9000-preread-input.sh | 48 ---------------------------------------- 4 files changed, 31 insertions(+), 55 deletions(-) delete mode 100755 t/t9000-preread-input.sh diff --git a/t/integration.t b/t/integration.t index b33e3c3..f5afd5d 100644 --- a/t/integration.t +++ b/t/integration.t @@ -7,8 +7,8 @@ use v5.14; BEGIN { require './t/lib.perl' }; use autodie; -my $srv = tcp_server(); -my $host_port = tcp_host_port($srv); +our $srv = tcp_server(); +our $host_port = tcp_host_port($srv); my $t0 = time; my $conf = "$tmpdir/u.conf.rb"; open my $conf_fh, '>', $conf; @@ -209,8 +209,34 @@ SKIP: { seek($rh, 0, SEEK_SET); $copt->{0} = $rh; $do_curl->('-T-'); -} + diag 'testing Unicorn::PrereadInput...'; + local $srv = tcp_server(); + local $host_port = tcp_host_port($srv); + check_stderr; + truncate($errfh, 0); + + my $pri = unicorn(qw(-E none t/preread_input.ru), { 3 => $srv }); + $url = "http://$host_port/"; + + $do_curl->(qw(-T t/random_blob)); + seek($rh, 0, SEEK_SET); + $copt->{0} = $rh; + $do_curl->('-T-'); + + my @pr_err = slurp("$tmpdir/err.log"); + is(scalar(grep(/app dispatch:/, @pr_err)), 2, 'app dispatched twice'); + + # abort a chunked request by blocking curl on a FIFO: + $c = tcp_start($srv, "PUT / HTTP/1.1\r\nTransfer-Encoding: chunked"); + close $c; + @pr_err = slurp("$tmpdir/err.log"); + is(scalar(grep(/app dispatch:/, @pr_err)), 2, + 'app did not dispatch on aborted request'); + undef $pri; + check_stderr; + diag 'Unicorn::PrereadInput middleware tests done'; +} # ... more stuff here diff --git a/t/lib.perl b/t/lib.perl index 1d6e78d..b6148cf 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -79,7 +79,7 @@ sub tcp_start ($@) { sub slurp { open my $fh, '<', $_[0]; - local $/; + local $/ if !wantarray; readline($fh); } diff --git a/t/preread_input.ru b/t/preread_input.ru index 79685c4..f0a1748 100644 --- a/t/preread_input.ru +++ b/t/preread_input.ru @@ -1,8 +1,6 @@ #\-E none require 'digest/sha1' require 'unicorn/preread_input' -use Rack::ContentLength -use Rack::ContentType, "text/plain" use Unicorn::PrereadInput nr = 0 run lambda { |env| @@ -13,5 +11,5 @@ dig.update(buf) end - [ 200, {}, [ "#{dig.hexdigest}\n" ] ] + [ 200, {}, [ dig.hexdigest ] ] } diff --git a/t/t9000-preread-input.sh b/t/t9000-preread-input.sh deleted file mode 100755 index d6c73ab..0000000 --- a/t/t9000-preread-input.sh +++ /dev/null @@ -1,48 +0,0 @@ -#!/bin/sh -. ./test-lib.sh -t_plan 9 "PrereadInput middleware tests" - -t_begin "setup and start" && { - random_blob_sha1=$(rsha1 < random_blob) - unicorn_setup - unicorn -D -c $unicorn_config preread_input.ru - unicorn_wait_start -} - -t_begin "single identity request" && { - curl -sSf -T random_blob http://$listen/ > $tmp -} - -t_begin "sha1 matches" && { - test x"$(cat $tmp)" = x"$random_blob_sha1" -} - -t_begin "single chunked request" && { - curl -sSf -T- < random_blob http://$listen/ > $tmp -} - -t_begin "sha1 matches" && { - test x"$(cat $tmp)" = x"$random_blob_sha1" -} - -t_begin "app only dispatched twice" && { - test 2 -eq "$(grep 'app dispatch:' < $r_err | count_lines )" -} - -t_begin "aborted chunked request" && { - rm -f $tmp - curl -sSf -T- < $fifo http://$listen/ > $tmp & - curl_pid=$! - kill -9 $curl_pid - wait -} - -t_begin "app only dispatched twice" && { - test 2 -eq "$(grep 'app dispatch:' < $r_err | count_lines )" -} - -t_begin "killing succeeds" && { - kill -QUIT $unicorn_pid -} - -t_done [-- Attachment #17: 0016-port-t-t0116-client_body_buffer_size.sh-to-Perl-5.patch --] [-- Type: text/x-diff, Size: 8861 bytes --] From e9593301044f305d4a0e074f77eea35015ca0ec4 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:45 +0000 Subject: [PATCH 16/23] port t/t0116-client_body_buffer_size.sh to Perl 5 While I'm fine with depending on curl for certain things, there's no need for it here since unicorn has had lazy rack.input for over a decade, at this point. --- t/active-unix-socket.t | 1 + t/{t0116.ru => client_body_buffer_size.ru} | 2 - t/client_body_buffer_size.t | 83 ++++++++++++++++++++++ t/integration.t | 10 --- t/lib.perl | 12 +++- t/t0116-client_body_buffer_size.sh | 80 --------------------- 6 files changed, 95 insertions(+), 93 deletions(-) rename t/{t0116.ru => client_body_buffer_size.ru} (82%) create mode 100644 t/client_body_buffer_size.t delete mode 100755 t/t0116-client_body_buffer_size.sh diff --git a/t/active-unix-socket.t b/t/active-unix-socket.t index 8723137..4e11837 100644 --- a/t/active-unix-socket.t +++ b/t/active-unix-socket.t @@ -109,4 +109,5 @@ is($pidf, $to_kill{u1}, 'pid file contents unchanged after 2nd start failure'); } check_stderr; +undef $tmpdir; done_testing; diff --git a/t/t0116.ru b/t/client_body_buffer_size.ru similarity index 82% rename from t/t0116.ru rename to t/client_body_buffer_size.ru index fab5fce..44161a5 100644 --- a/t/t0116.ru +++ b/t/client_body_buffer_size.ru @@ -1,6 +1,4 @@ #\ -E none -use Rack::ContentLength -use Rack::ContentType, 'text/plain' app = lambda do |env| input = env['rack.input'] case env["PATH_INFO"] diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t new file mode 100644 index 0000000..b1a99f3 --- /dev/null +++ b/t/client_body_buffer_size.t @@ -0,0 +1,83 @@ +#!perl -w +# Copyright (C) unicorn hackers <unicorn-public@yhbt.net> +# License: GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt> + +use v5.14; BEGIN { require './t/lib.perl' }; +use autodie; +my $uconf = "$tmpdir/u.conf.rb"; + +open my $conf_fh, '>', $uconf; +$conf_fh->autoflush(1); +print $conf_fh <<EOM; +client_body_buffer_size 0 +EOM +my $srv = tcp_server(); +my $host_port = tcp_host_port($srv); +my @uarg = (qw(-E none t/client_body_buffer_size.ru -c), $uconf); +my $ar = unicorn(@uarg, { 3 => $srv }); +my ($c, $status, $hdr); +my $mem_class = 'StringIO'; +my $fs_class = 'Unicorn::TmpIO'; + +$c = tcp_start($srv, "PUT /input_class HTTP/1.0\r\nContent-Length: 0"); +($status, $hdr) = slurp_hdr($c); +like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid'); +is(readline($c), $mem_class, 'zero-byte file is StringIO'); + +$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: 1"); +print $c '.'; +($status, $hdr) = slurp_hdr($c); +like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid'); +is(readline($c), $fs_class, '1 byte file is filesystem-backed'); + + +my $fifo = "$tmpdir/fifo"; +POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!"; +seek($conf_fh, 0, SEEK_SET); +truncate($conf_fh, 0); +print $conf_fh <<EOM; +listen "$host_port" # TODO: remove this requirement for SIGHUP +after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } } +EOM +$ar->do_kill('HUP'); +open my $fifo_fh, '<', $fifo; +like(my $wpid = readline($fifo_fh), qr/\Apid=\d+\z/a , + 'reloaded w/ default client_body_buffer_size'); + + +$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: 1"); +($status, $hdr) = slurp_hdr($c); +like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid'); +is(readline($c), $mem_class, 'class for a 1 byte file is memory-backed'); + + +my $one_meg = 1024 ** 2; +$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $one_meg"); +($status, $hdr) = slurp_hdr($c); +like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid'); +is(readline($c), $fs_class, '1 megabyte file is FS-backed'); + +# reload with bigger client_body_buffer_size +say $conf_fh "client_body_buffer_size $one_meg"; +$ar->do_kill('HUP'); +open $fifo_fh, '<', $fifo; +like($wpid = readline($fifo_fh), qr/\Apid=\d+\z/a , + 'reloaded w/ bigger client_body_buffer_size'); + + +$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $one_meg"); +($status, $hdr) = slurp_hdr($c); +like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid'); +is(readline($c), $mem_class, '1 megabyte file is now memory-backed'); + +my $too_big = $one_meg + 1; +$c = tcp_start($srv, "PUT /tmp_class HTTP/1.0\r\nContent-Length: $too_big"); +($status, $hdr) = slurp_hdr($c); +like($status, qr!\AHTTP/1\.[01] 200\b!, 'status line valid'); +is(readline($c), $fs_class, '1 megabyte + 1 byte file is FS-backed'); + + +undef $ar; +check_stderr; +undef $tmpdir; +done_testing; diff --git a/t/integration.t b/t/integration.t index f5afd5d..855c260 100644 --- a/t/integration.t +++ b/t/integration.t @@ -15,16 +15,6 @@ open my $conf_fh, '>', $conf; $conf_fh->autoflush(1); my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv }); my $curl = which('curl'); -END { diag slurp("$tmpdir/err.log") if $tmpdir }; -sub slurp_hdr { - my ($c) = @_; - local $/ = "\r\n\r\n"; # affects both readline+chomp - chomp(my $hdr = readline($c)); - my ($status, @hdr) = split(/\r\n/, $hdr); - diag explain([ $status, \@hdr ]) if $ENV{V}; - ($status, \@hdr); -} - my %PUT = ( chunked_md5 => sub { my ($in, $out, $path, %opt) = @_; diff --git a/t/lib.perl b/t/lib.perl index b6148cf..2685c3b 100644 --- a/t/lib.perl +++ b/t/lib.perl @@ -11,11 +11,12 @@ use POSIX qw(dup2 _exit setpgid :signal_h SEEK_SET F_SETFD); use File::Temp 0.19 (); # 0.19 for ->newdir our ($tmpdir, $errfh); our @EXPORT = qw(unicorn slurp tcp_server tcp_start unicorn $tmpdir $errfh - SEEK_SET tcp_host_port which spawn check_stderr unix_start); + SEEK_SET tcp_host_port which spawn check_stderr unix_start slurp_hdr); my ($base) = ($0 =~ m!\b([^/]+)\.[^\.]+\z!); $tmpdir = File::Temp->newdir("unicorn-$base-XXXX", TMPDIR => 1); open($errfh, '>>', "$tmpdir/err.log"); +END { diag slurp("$tmpdir/err.log") if $tmpdir }; sub check_stderr () { my @log = slurp("$tmpdir/err.log"); @@ -25,6 +26,15 @@ sub check_stderr () { is_deeply([grep(/SIGKILL/, @log)], [], 'no SIGKILL in stderr'); } +sub slurp_hdr { + my ($c) = @_; + local $/ = "\r\n\r\n"; # affects both readline+chomp + chomp(my $hdr = readline($c)); + my ($status, @hdr) = split(/\r\n/, $hdr); + diag explain([ $status, \@hdr ]) if $ENV{V}; + ($status, \@hdr); +} + sub tcp_server { my %opt = ( ReuseAddr => 1, diff --git a/t/t0116-client_body_buffer_size.sh b/t/t0116-client_body_buffer_size.sh deleted file mode 100755 index c9e17c7..0000000 --- a/t/t0116-client_body_buffer_size.sh +++ /dev/null @@ -1,80 +0,0 @@ -#!/bin/sh -. ./test-lib.sh -t_plan 12 "client_body_buffer_size settings" - -t_begin "setup and start" && { - unicorn_setup - rtmpfiles unicorn_config_tmp one_meg - dd if=/dev/zero bs=1M count=1 of=$one_meg - cat >> $unicorn_config <<EOF -after_fork do |server, worker| - File.open("$fifo", "wb") { |fp| fp.syswrite "START" } -end -EOF - cat $unicorn_config > $unicorn_config_tmp - echo client_body_buffer_size 0 >> $unicorn_config - unicorn -D -c $unicorn_config t0116.ru - unicorn_wait_start - fs_class=Unicorn::TmpIO - mem_class=StringIO - - test x"$(cat $fifo)" = xSTART -} - -t_begin "class for a zero-byte file should be StringIO" && { - > $tmp - test xStringIO = x"$(curl -T $tmp -sSf http://$listen/input_class)" -} - -t_begin "class for a 1 byte file should be filesystem-backed" && { - echo > $tmp - test x$fs_class = x"$(curl -T $tmp -sSf http://$listen/tmp_class)" -} - -t_begin "reload with default client_body_buffer_size" && { - mv $unicorn_config_tmp $unicorn_config - kill -HUP $unicorn_pid - test x"$(cat $fifo)" = xSTART -} - -t_begin "class for a 1 byte file should be memory-backed" && { - echo > $tmp - test x$mem_class = x"$(curl -T $tmp -sSf http://$listen/tmp_class)" -} - -t_begin "class for a random blob file should be filesystem-backed" && { - resp="$(curl -T random_blob -sSf http://$listen/tmp_class)" - test x$fs_class = x"$resp" -} - -t_begin "one megabyte file should be filesystem-backed" && { - resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)" - test x$fs_class = x"$resp" -} - -t_begin "reload with a big client_body_buffer_size" && { - echo "client_body_buffer_size(1024 * 1024)" >> $unicorn_config - kill -HUP $unicorn_pid - test x"$(cat $fifo)" = xSTART -} - -t_begin "one megabyte file should be memory-backed" && { - resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)" - test x$mem_class = x"$resp" -} - -t_begin "one megabyte + 1 byte file should be filesystem-backed" && { - echo >> $one_meg - resp="$(curl -T $one_meg -sSf http://$listen/tmp_class)" - test x$fs_class = x"$resp" -} - -t_begin "killing succeeds" && { - kill $unicorn_pid -} - -t_begin "check stderr" && { - check_stderr -} - -t_done [-- Attachment #18: 0017-tests-get-rid-of-sha1sum.rb-and-rsha1-sh-function.patch --] [-- Type: text/x-diff, Size: 1255 bytes --] From b47912160f2336dde3901e588cc23fb2c2f8d9dc Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:46 +0000 Subject: [PATCH 17/23] tests: get rid of sha1sum.rb and rsha1() sh function These are no longer needed since Perl has long included Digest::SHA --- t/bin/sha1sum.rb | 17 ----------------- t/test-lib.sh | 4 ---- 2 files changed, 21 deletions(-) delete mode 100755 t/bin/sha1sum.rb diff --git a/t/bin/sha1sum.rb b/t/bin/sha1sum.rb deleted file mode 100755 index 53d68ce..0000000 --- a/t/bin/sha1sum.rb +++ /dev/null @@ -1,17 +0,0 @@ -#!/usr/bin/env ruby -# -*- encoding: binary -*- -# Reads from stdin and outputs the SHA1 hex digest of the input - -require 'digest/sha1' -$stdout.sync = $stderr.sync = true -$stdout.binmode -$stdin.binmode -bs = 16384 -digest = Digest::SHA1.new -if buf = $stdin.read(bs) - begin - digest.update(buf) - end while $stdin.read(bs, buf) -end - -$stdout.syswrite("#{digest.hexdigest}\n") diff --git a/t/test-lib.sh b/t/test-lib.sh index e70d0c6..8613144 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -123,7 +123,3 @@ unicorn_wait_start () { # no need to play tricks with FIFOs since we got "ready_pipe" now unicorn_pid=$(cat $pid) } - -rsha1 () { - sha1sum.rb -} [-- Attachment #19: 0018-early_hints-supports-Rack-3-array-headers.patch --] [-- Type: text/x-diff, Size: 4606 bytes --] From 6ad9f4b54ee16ffecea7e16b710552b45db33a16 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:47 +0000 Subject: [PATCH 18/23] early_hints supports Rack 3 array headers We can hoist out append_headers into a new method and use it in both e103_response_write and http_response_write. t/integration.t now tests early_hints with both possible values of check_client_connection. --- t/integration.ru | 7 +++++++ t/integration.t | 47 ++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 49 insertions(+), 5 deletions(-) diff --git a/t/integration.ru b/t/integration.ru index edc408c..dab384d 100644 --- a/t/integration.ru +++ b/t/integration.ru @@ -5,6 +5,11 @@ # this goes for t/integration.t We'll try to put as many tests # in here as possible to avoid startup overhead of Ruby. +def early_hints(env, val) + env['rack.early_hints'].call('link' => val) # val may be ary or string + [ 200, {}, [ val.class.to_s ] ] +end + $orig_rack_200 = nil def tweak_status_code $orig_rack_200 = Rack::Utils::HTTP_STATUS_CODES[200] @@ -81,6 +86,8 @@ def rack_input_tests(env) when '/env_dump'; [ 200, {}, [ env_dump(env) ] ] when '/write_on_close'; write_on_close when '/pid'; [ 200, {}, [ "#$$\n" ] ] + when '/early_hints_rack2'; early_hints(env, "r\n2") + when '/early_hints_rack3'; early_hints(env, %w(r 3)) else '/'; [ 200, {}, [ env_dump(env) ] ] end # case PATH_INFO (GET) when 'POST' diff --git a/t/integration.t b/t/integration.t index 855c260..8433497 100644 --- a/t/integration.t +++ b/t/integration.t @@ -13,8 +13,16 @@ my $t0 = time; my $conf = "$tmpdir/u.conf.rb"; open my $conf_fh, '>', $conf; $conf_fh->autoflush(1); +my $u1 = "$tmpdir/u1"; +print $conf_fh <<EOM; +early_hints true +listen "$u1" +listen "$host_port" # TODO: remove this requirement for SIGHUP +EOM my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv }); my $curl = which('curl'); +my $fifo = "$tmpdir/fifo"; +POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!"; my %PUT = ( chunked_md5 => sub { my ($in, $out, $path, %opt) = @_; @@ -102,6 +110,26 @@ $c = tcp_start($srv, 'GET /nil-header-value HTTP/1.0'); is_deeply([grep(/^X-Nil:/, @$hdr)], ['X-Nil: '], 'nil header value accepted for broken apps') or diag(explain($hdr)); +my $ck_early_hints = sub { + my ($note) = @_; + $c = unix_start($u1, 'GET /early_hints_rack2 HTTP/1.0'); + ($status, $hdr) = slurp_hdr($c); + like($status, qr!\AHTTP/1\.[01] 103\b!, 'got 103 for rack 2 value'); + is_deeply(['link: r', 'link: 2'], $hdr, 'rack 2 hints match '.$note); + ($status, $hdr) = slurp_hdr($c); + like($status, qr!\AHTTP/1\.[01] 200\b!, 'got 200 afterwards'); + is(readline($c), 'String', 'early hints used a String for rack 2'); + + $c = unix_start($u1, 'GET /early_hints_rack3 HTTP/1.0'); + ($status, $hdr) = slurp_hdr($c); + like($status, qr!\AHTTP/1\.[01] 103\b!, 'got 103 for rack 3'); + is_deeply(['link: r', 'link: 3'], $hdr, 'rack 3 hints match '.$note); + ($status, $hdr) = slurp_hdr($c); + like($status, qr!\AHTTP/1\.[01] 200\b!, 'got 200 afterwards'); + is(readline($c), 'Array', 'early hints used a String for rack 3'); +}; +$ck_early_hints->('ccc off'); # we'll retest later + if ('TODO: ensure Rack::Utils::HTTP_STATUS_CODES is available') { $c = tcp_start($srv, 'POST /tweak-status-code HTTP/1.0'); ($status, $hdr) = slurp_hdr($c); @@ -154,6 +182,7 @@ if ('bad requests') { # input tests my ($blob_size, $blob_hash); SKIP: { + skip 'SKIP_EXPENSIVE on', 1 if $ENV{SKIP_EXPENSIVE}; CORE::open(my $rh, '<', 't/random_blob') or skip "t/random_blob not generated $!", 1; $blob_size = -s $rh; @@ -232,16 +261,24 @@ SKIP: { # SIGHUP-able stuff goes here +if ('check_client_connection') { + print $conf_fh <<EOM; # appending to existing +check_client_connection true +after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } } +EOM + $ar->do_kill('HUP'); + open my $fifo_fh, '<', $fifo; + my $wpid = readline($fifo_fh); + like($wpid, qr/\Apid=\d+\z/a , 'new worker ready'); + $ck_early_hints->('ccc on'); +} + if ('max_header_len internal API') { undef $c; my $req = 'GET / HTTP/1.0'; my $len = length($req."\r\n\r\n"); - my $fifo = "$tmpdir/fifo"; - POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!"; - print $conf_fh <<EOM; + print $conf_fh <<EOM; # appending to existing Unicorn::HttpParser.max_header_len = $len -listen "$host_port" # TODO: remove this requirement for SIGHUP -after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } } EOM $ar->do_kill('HUP'); open my $fifo_fh, '<', $fifo; [-- Attachment #20: 0019-test_server-drop-early_hints-test.patch --] [-- Type: text/x-diff, Size: 1737 bytes --] From 3e6bc9fb589fd88469349a38a77704c3333623e0 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:48 +0000 Subject: [PATCH 19/23] test_server: drop early_hints test t/integration.t already is more complete in that it tests both Rack 2 and 3 along with both possible values of check_client_connection. --- test/unit/test_server.rb | 31 ------------------------------- 1 file changed, 31 deletions(-) diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb index fe98fcc..0a710d1 100644 --- a/test/unit/test_server.rb +++ b/test/unit/test_server.rb @@ -23,17 +23,6 @@ def call(env) end end -class TestEarlyHintsHandler - def call(env) - while env['rack.input'].read(4096) - end - env['rack.early_hints'].call( - "Link" => "</style.css>; rel=preload; as=style\n</script.js>; rel=preload" - ) - [200, { 'content-type' => 'text/plain' }, ['hello!\n']] - end -end - class TestRackAfterReply def initialize @called = false @@ -112,26 +101,6 @@ def test_preload_app_config tmp.close! end - def test_early_hints - teardown - redirect_test_io do - @server = HttpServer.new(TestEarlyHintsHandler.new, - :listeners => [ "127.0.0.1:#@port"], - :early_hints => true) - @server.start - end - - sock = tcp_socket('127.0.0.1', @port) - sock.syswrite("GET / HTTP/1.0\r\n\r\n") - - responses = sock.read(4096) - assert_match %r{\AHTTP/1.[01] 103\b}, responses - assert_match %r{^Link: </style\.css>}, responses - assert_match %r{^Link: </script\.js>}, responses - - assert_match %r{^HTTP/1.[01] 200\b}, responses - end - def test_after_reply teardown [-- Attachment #21: 0020-t-integration.t-switch-PUT-tests-to-MD5-reuse-buffer.patch --] [-- Type: text/x-diff, Size: 3740 bytes --] From cb826915cdd1881cbcfc1fb4e645d26244dfda71 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:49 +0000 Subject: [PATCH 20/23] t/integration.t: switch PUT tests to MD5, reuse buffers MD5 is faster, and these tests aren't meant to be secure, they're just for checking for data corruption. Furthermore, Content-MD5 is a supported HTTP trailer and we can verify that here to obsolete other tests. Furthermore, we can reuse buffers on env['rack.input'].read calls to avoid malloc(3) and GC overhead. Combined, these give roughly a 3% speedup for t/integration.t on my system. --- t/integration.ru | 20 +++++++++++++++----- t/integration.t | 5 ++--- t/preread_input.ru | 17 ++++++++++++----- 3 files changed, 29 insertions(+), 13 deletions(-) diff --git a/t/integration.ru b/t/integration.ru index dab384d..086126a 100644 --- a/t/integration.ru +++ b/t/integration.ru @@ -55,8 +55,8 @@ def env_dump(env) def rack_input_tests(env) return [ 100, {}, [] ] if /\A100-continue\z/i =~ env['HTTP_EXPECT'] cap = 16384 - require 'digest/sha1' - digest = Digest::SHA1.new + require 'digest/md5' + dig = Digest::MD5.new input = env['rack.input'] case env['PATH_INFO'] when '/rack_input/size_first'; input.size @@ -68,11 +68,21 @@ def rack_input_tests(env) if buf = input.read(rand(cap)) begin raise "#{buf.size} > #{cap}" if buf.size > cap - digest.update(buf) + dig.update(buf) end while input.read(rand(cap), buf) + buf.clear # remove this call if Ruby ever gets escape analysis end - [ 200, {'content-length' => '40', 'content-type' => 'text/plain'}, - [ digest.hexdigest ] ] + h = { 'content-type' => 'text/plain' } + if env['HTTP_TRAILER'] =~ /\bContent-MD5\b/i + cmd5_b64 = env['HTTP_CONTENT_MD5'] or return [500, {}, ['No Content-MD5']] + cmd5_bin = cmd5_b64.unpack('m')[0] + if cmd5_bin != dig.digest + h['content-length'] = cmd5_b64.size.to_s + return [ 500, h, [ cmd5_b64 ] ] + end + end + h['content-length'] = '32' + [ 200, h, [ dig.hexdigest ] ] end run(lambda do |env| diff --git a/t/integration.t b/t/integration.t index 8433497..38a9675 100644 --- a/t/integration.t +++ b/t/integration.t @@ -27,7 +27,6 @@ my %PUT = ( chunked_md5 => sub { my ($in, $out, $path, %opt) = @_; my $bs = $opt{bs} // 16384; - require Digest::MD5; my $dig = Digest::MD5->new; print $out <<EOM; PUT $path HTTP/1.1\r @@ -186,8 +185,8 @@ SKIP: { CORE::open(my $rh, '<', 't/random_blob') or skip "t/random_blob not generated $!", 1; $blob_size = -s $rh; - require Digest::SHA; - $blob_hash = Digest::SHA->new(1)->addfile($rh)->hexdigest; + require Digest::MD5; + $blob_hash = Digest::MD5->new->addfile($rh)->hexdigest; my $ck_hash = sub { my ($sub, $path, %opt) = @_; diff --git a/t/preread_input.ru b/t/preread_input.ru index f0a1748..18af221 100644 --- a/t/preread_input.ru +++ b/t/preread_input.ru @@ -1,15 +1,22 @@ #\-E none -require 'digest/sha1' +require 'digest/md5' require 'unicorn/preread_input' use Unicorn::PrereadInput nr = 0 run lambda { |env| $stderr.write "app dispatch: #{nr += 1}\n" input = env["rack.input"] - dig = Digest::SHA1.new - while buf = input.read(16384) - dig.update(buf) + dig = Digest::MD5.new + if buf = input.read(16384) + begin + dig.update(buf) + end while input.read(16384, buf) + buf.clear # remove this call if Ruby ever gets escape analysis + end + if env['HTTP_TRAILER'] =~ /\bContent-MD5\b/i + cmd5_b64 = env['HTTP_CONTENT_MD5'] or return [500, {}, ['No Content-MD5']] + cmd5_bin = cmd5_b64.unpack('m')[0] + return [500, {}, [ cmd5_b64 ] ] if cmd5_bin != dig.digest end - [ 200, {}, [ dig.hexdigest ] ] } [-- Attachment #22: 0021-tests-move-test_upload.rb-tests-to-t-integration.t.patch --] [-- Type: text/x-diff, Size: 12280 bytes --] From 181e4b5b6339fc5e9c3ad7d3690b736f6bd038aa Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:50 +0000 Subject: [PATCH 21/23] tests: move test_upload.rb tests to t/integration.t The overread tests are ported over, and checksumming alone is enough to guard against data corruption. Randomizing the size of `read' calls on the client side will shake out any boundary bugs on the server side. --- t/integration.t | 32 ++++- test/unit/test_upload.rb | 301 --------------------------------------- 2 files changed, 27 insertions(+), 306 deletions(-) delete mode 100644 test/unit/test_upload.rb diff --git a/t/integration.t b/t/integration.t index 38a9675..a568758 100644 --- a/t/integration.t +++ b/t/integration.t @@ -26,7 +26,6 @@ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!"; my %PUT = ( chunked_md5 => sub { my ($in, $out, $path, %opt) = @_; - my $bs = $opt{bs} // 16384; my $dig = Digest::MD5->new; print $out <<EOM; PUT $path HTTP/1.1\r @@ -36,7 +35,7 @@ Trailer: Content-MD5\r EOM my ($buf, $r); while (1) { - $r = read($in, $buf, $bs); + $r = read($in, $buf, 999 + int(rand(0xffff))); last if $r == 0; printf $out "%x\r\n", length($buf); print $out $buf, "\r\n"; @@ -46,15 +45,15 @@ EOM }, identity => sub { my ($in, $out, $path, %opt) = @_; - my $bs = $opt{bs} // 16384; my $clen = $opt{-s} // -s $in; print $out <<EOM; PUT $path HTTP/1.0\r Content-Length: $clen\r \r EOM - my ($buf, $r, $len); + my ($buf, $r, $len, $bs); while ($clen) { + $bs = 999 + int(rand(0xffff)); $len = $clen > $bs ? $bs : $clen; $r = read($in, $buf, $len); die 'premature EOF' if $r == 0; @@ -192,8 +191,10 @@ SKIP: { my ($sub, $path, %opt) = @_; seek($rh, 0, SEEK_SET); $c = tcp_start($srv); - $c->autoflush(0); + $c->autoflush($opt{sync} // 0); $PUT{$sub}->($rh, $c, $path, %opt); + defined($opt{overwrite}) and + print { $c } ('x' x $opt{overwrite}); $c->flush or die $!; ($status, $hdr) = slurp_hdr($c); is(readline($c), $blob_hash, "$sub $path"); @@ -205,6 +206,27 @@ SKIP: { $ck_hash->('chunked_md5', '/rack_input/size_first'); $ck_hash->('chunked_md5', '/rack_input/rewind_first'); + $ck_hash->('identity', '/rack_input', -s => $blob_size, sync => 1); + $ck_hash->('chunked_md5', '/rack_input', sync => 1); + + # ensure small overwrites don't get checksummed + $ck_hash->('identity', '/rack_input', -s => $blob_size, + overwrite => 1); # one extra byte + + # excessive overwrite truncated + $c = tcp_start($srv); + $c->autoflush(0); + print $c "PUT /rack_input HTTP/1.0\r\nContent-Length: 1\r\n\r\n"; + if (1) { + local $SIG{PIPE} = 'IGNORE'; + my $buf = "\0" x 8192; + my $n = 0; + my $end = time + 5; + $! = 0; + while (print $c $buf and time < $end) { ++$n } + ok($!, 'overwrite truncated') or diag "n=$n err=$! ".time; + } + undef $c; $curl // skip 'no curl found in PATH', 1; diff --git a/test/unit/test_upload.rb b/test/unit/test_upload.rb deleted file mode 100644 index 76e6c1c..0000000 --- a/test/unit/test_upload.rb +++ /dev/null @@ -1,301 +0,0 @@ -# -*- encoding: binary -*- - -# Copyright (c) 2009 Eric Wong -require './test/test_helper' -require 'digest/md5' - -include Unicorn - -class UploadTest < Test::Unit::TestCase - - def setup - @addr = ENV['UNICORN_TEST_ADDR'] || '127.0.0.1' - @port = unused_port - @hdr = {'Content-Type' => 'text/plain', 'Content-Length' => '0'} - @bs = 4096 - @count = 256 - @server = nil - - # we want random binary data to test 1.9 encoding-aware IO craziness - @random = File.open('/dev/urandom','rb') - @sha1 = Digest::SHA1.new - @sha1_app = lambda do |env| - input = env['rack.input'] - resp = {} - - @sha1.reset - while buf = input.read(@bs) - @sha1.update(buf) - end - resp[:sha1] = @sha1.hexdigest - - # rewind and read again - input.rewind - @sha1.reset - while buf = input.read(@bs) - @sha1.update(buf) - end - - if resp[:sha1] == @sha1.hexdigest - resp[:sysread_read_byte_match] = true - end - - if expect_size = env['HTTP_X_EXPECT_SIZE'] - if expect_size.to_i == input.size - resp[:expect_size_match] = true - end - end - resp[:size] = input.size - resp[:content_md5] = env['HTTP_CONTENT_MD5'] - - [ 200, @hdr.merge({'X-Resp' => resp.inspect}), [] ] - end - end - - def teardown - redirect_test_io { @server.stop(false) } if @server - @random.close - reset_sig_handlers - end - - def test_put - start_server(@sha1_app) - sock = tcp_socket(@addr, @port) - sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n") - @count.times do |i| - buf = @random.sysread(@bs) - @sha1.update(buf) - sock.syswrite(buf) - end - read = sock.read.split(/\r\n/) - assert_equal "HTTP/1.1 200 OK", read[0] - resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, '')) - assert_equal length, resp[:size] - assert_equal @sha1.hexdigest, resp[:sha1] - end - - def test_put_content_md5 - md5 = Digest::MD5.new - start_server(@sha1_app) - sock = tcp_socket(@addr, @port) - sock.syswrite("PUT / HTTP/1.0\r\nTransfer-Encoding: chunked\r\n" \ - "Trailer: Content-MD5\r\n\r\n") - @count.times do |i| - buf = @random.sysread(@bs) - @sha1.update(buf) - md5.update(buf) - sock.syswrite("#{'%x' % buf.size}\r\n") - sock.syswrite(buf << "\r\n") - end - sock.syswrite("0\r\n") - - content_md5 = [ md5.digest! ].pack('m').strip.freeze - sock.syswrite("Content-MD5: #{content_md5}\r\n\r\n") - read = sock.read.split(/\r\n/) - assert_equal "HTTP/1.1 200 OK", read[0] - resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, '')) - assert_equal length, resp[:size] - assert_equal @sha1.hexdigest, resp[:sha1] - assert_equal content_md5, resp[:content_md5] - end - - def test_put_trickle_small - @count, @bs = 2, 128 - start_server(@sha1_app) - assert_equal 256, length - sock = tcp_socket(@addr, @port) - hdr = "PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n" - @count.times do - buf = @random.sysread(@bs) - @sha1.update(buf) - hdr << buf - sock.syswrite(hdr) - hdr = '' - sleep 0.6 - end - read = sock.read.split(/\r\n/) - assert_equal "HTTP/1.1 200 OK", read[0] - resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, '')) - assert_equal length, resp[:size] - assert_equal @sha1.hexdigest, resp[:sha1] - end - - def test_put_keepalive_truncates_small_overwrite - start_server(@sha1_app) - sock = tcp_socket(@addr, @port) - to_upload = length + 1 - sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{to_upload}\r\n\r\n") - @count.times do - buf = @random.sysread(@bs) - @sha1.update(buf) - sock.syswrite(buf) - end - sock.syswrite('12345') # write 4 bytes more than we expected - @sha1.update('1') - - buf = sock.readpartial(4096) - while buf !~ /\r\n\r\n/ - buf << sock.readpartial(4096) - end - read = buf.split(/\r\n/) - assert_equal "HTTP/1.1 200 OK", read[0] - resp = eval(read.grep(/^X-Resp: /).first.sub!(/X-Resp: /, '')) - assert_equal to_upload, resp[:size] - assert_equal @sha1.hexdigest, resp[:sha1] - end - - def test_put_excessive_overwrite_closed - tmp = Tempfile.new('overwrite_check') - tmp.sync = true - start_server(lambda { |env| - nr = 0 - while buf = env['rack.input'].read(65536) - nr += buf.size - end - tmp.write(nr.to_s) - [ 200, @hdr, [] ] - }) - sock = tcp_socket(@addr, @port) - buf = ' ' * @bs - sock.syswrite("PUT / HTTP/1.0\r\nContent-Length: #{length}\r\n\r\n") - - @count.times { sock.syswrite(buf) } - assert_raise(Errno::ECONNRESET, Errno::EPIPE) do - ::Unicorn::Const::CHUNK_SIZE.times { sock.syswrite(buf) } - end - sock.gets - tmp.rewind - assert_equal length, tmp.read.to_i - end - - # Despite reading numerous articles and inspecting the 1.9.1-p0 C - # source, Eric Wong will never trust that we're always handling - # encoding-aware IO objects correctly. Thus this test uses shell - # utilities that should always operate on files/sockets on a - # byte-level. - def test_uncomfortable_with_onenine_encodings - # POSIX doesn't require all of these to be present on a system - which('curl') or return - which('sha1sum') or return - which('dd') or return - - start_server(@sha1_app) - - tmp = Tempfile.new('dd_dest') - assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}", - "bs=#{@bs}", "count=#{@count}"), - "dd #@random to #{tmp}") - sha1_re = %r!\b([a-f0-9]{40})\b! - sha1_out = `sha1sum #{tmp.path}` - assert $?.success?, 'sha1sum ran OK' - - assert_match(sha1_re, sha1_out) - sha1 = sha1_re.match(sha1_out)[1] - resp = `curl -isSfN -T#{tmp.path} http://#@addr:#@port/` - assert $?.success?, 'curl ran OK' - assert_match(%r!\b#{sha1}\b!, resp) - assert_match(/sysread_read_byte_match/, resp) - - # small StringIO path - assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}", - "bs=1024", "count=1"), - "dd #@random to #{tmp}") - sha1_re = %r!\b([a-f0-9]{40})\b! - sha1_out = `sha1sum #{tmp.path}` - assert $?.success?, 'sha1sum ran OK' - - assert_match(sha1_re, sha1_out) - sha1 = sha1_re.match(sha1_out)[1] - resp = `curl -isSfN -T#{tmp.path} http://#@addr:#@port/` - assert $?.success?, 'curl ran OK' - assert_match(%r!\b#{sha1}\b!, resp) - assert_match(/sysread_read_byte_match/, resp) - end - - def test_chunked_upload_via_curl - # POSIX doesn't require all of these to be present on a system - which('curl') or return - which('sha1sum') or return - which('dd') or return - - start_server(@sha1_app) - - tmp = Tempfile.new('dd_dest') - assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}", - "bs=#{@bs}", "count=#{@count}"), - "dd #@random to #{tmp}") - sha1_re = %r!\b([a-f0-9]{40})\b! - sha1_out = `sha1sum #{tmp.path}` - assert $?.success?, 'sha1sum ran OK' - - assert_match(sha1_re, sha1_out) - sha1 = sha1_re.match(sha1_out)[1] - cmd = "curl -H 'X-Expect-Size: #{tmp.size}' --tcp-nodelay \ - -isSf --no-buffer -T- " \ - "http://#@addr:#@port/" - resp = Tempfile.new('resp') - resp.sync = true - - rd, wr = IO.pipe.each do |io| - io.sync = io.close_on_exec = true - end - pid = spawn(*cmd, { 0 => rd, 1 => resp }) - rd.close - - tmp.rewind - @count.times { |i| - wr.write(tmp.read(@bs)) - sleep(rand / 10) if 0 == i % 8 - } - wr.close - pid, status = Process.waitpid2(pid) - - resp.rewind - resp = resp.read - assert status.success?, 'curl ran OK' - assert_match(%r!\b#{sha1}\b!, resp) - assert_match(/sysread_read_byte_match/, resp) - assert_match(/expect_size_match/, resp) - end - - def test_curl_chunked_small - # POSIX doesn't require all of these to be present on a system - which('curl') or return - which('sha1sum') or return - which('dd') or return - - start_server(@sha1_app) - - tmp = Tempfile.new('dd_dest') - # small StringIO path - assert(system("dd", "if=#{@random.path}", "of=#{tmp.path}", - "bs=1024", "count=1"), - "dd #@random to #{tmp}") - sha1_re = %r!\b([a-f0-9]{40})\b! - sha1_out = `sha1sum #{tmp.path}` - assert $?.success?, 'sha1sum ran OK' - - assert_match(sha1_re, sha1_out) - sha1 = sha1_re.match(sha1_out)[1] - resp = `curl -H 'X-Expect-Size: #{tmp.size}' --tcp-nodelay \ - -isSf --no-buffer -T- http://#@addr:#@port/ < #{tmp.path}` - assert $?.success?, 'curl ran OK' - assert_match(%r!\b#{sha1}\b!, resp) - assert_match(/sysread_read_byte_match/, resp) - assert_match(/expect_size_match/, resp) - end - - private - - def length - @bs * @count - end - - def start_server(app) - redirect_test_io do - @server = HttpServer.new(app, :listeners => [ "#{@addr}:#{@port}" ] ) - @server.start - end - end - -end [-- Attachment #23: 0022-drop-redundant-IO-close_on_exec-false-calls.patch --] [-- Type: text/x-diff, Size: 891 bytes --] From 841b9e756beb1aa00d0f89097a808adcbbf45397 Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:51 +0000 Subject: [PATCH 22/23] drop redundant IO#close_on_exec=false calls Passing the `{ FD => IO }' mapping to #spawn or #exec already ensures Ruby will clear FD_CLOEXEC on these FDs before execve(2). --- lib/unicorn/http_server.rb | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index 348e745..dd92b38 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -472,10 +472,7 @@ def worker_spawn(worker) def listener_sockets listener_fds = {} - LISTENERS.each do |sock| - sock.close_on_exec = false - listener_fds[sock.fileno] = sock - end + LISTENERS.each { |sock| listener_fds[sock.fileno] = sock } listener_fds end [-- Attachment #24: 0023-LISTEN_FDS-inherited-sockets-are-immortal-across-SIG.patch --] [-- Type: text/x-diff, Size: 3229 bytes --] From 6ff8785c9277c5978e6dc01cb1b3da25d6bae2db Mon Sep 17 00:00:00 2001 From: Eric Wong <BOFH@YHBT.net> Date: Mon, 5 Jun 2023 10:12:52 +0000 Subject: [PATCH 23/23] LISTEN_FDS-inherited sockets are immortal across SIGHUP When using systemd-style socket activation, consider the inherited socket immortal and do not drop it on SIGHUP. This means configs w/o any `listen' directives at all can continue to work after SIGHUP. I only noticed this while writing some tests in Perl 5 and the test suite is two lines shorter to test this feature :> --- lib/unicorn/http_server.rb | 7 ++++++- t/client_body_buffer_size.t | 1 - t/integration.t | 1 - 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index dd92b38..f1b4a54 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -77,6 +77,7 @@ def initialize(app, options = {}) options[:use_defaults] = true self.config = Unicorn::Configurator.new(options) self.listener_opts = {} + @immortal = [] # immortal inherited sockets from systemd # We use @self_pipe differently in the master and worker processes: # @@ -158,6 +159,7 @@ def listeners=(listeners) end set_names = listener_names(listeners) dead_names.concat(cur_names - set_names).uniq! + dead_names -= @immortal.map { |io| sock_name(io) } LISTENERS.delete_if do |io| if dead_names.include?(sock_name(io)) @@ -807,17 +809,20 @@ def inherit_listeners! # inherit sockets from parents, they need to be plain Socket objects # before they become Kgio::UNIXServer or Kgio::TCPServer inherited = ENV['UNICORN_FD'].to_s.split(',') + immortal = [] # emulate sd_listen_fds() for systemd sd_pid, sd_fds = ENV.values_at('LISTEN_PID', 'LISTEN_FDS') if sd_pid.to_i == $$ # n.b. $$ can never be zero # 3 = SD_LISTEN_FDS_START - inherited.concat((3...(3 + sd_fds.to_i)).to_a) + immortal = (3...(3 + sd_fds.to_i)).to_a + inherited.concat(immortal) end # to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS inherited.map! do |fd| io = Socket.for_fd(fd.to_i) + @immortal << io if immortal.include?(fd) io.autoclose = false io = server_cast(io) set_server_sockopt(io, listener_opts[sock_name(io)]) diff --git a/t/client_body_buffer_size.t b/t/client_body_buffer_size.t index b1a99f3..3067f28 100644 --- a/t/client_body_buffer_size.t +++ b/t/client_body_buffer_size.t @@ -36,7 +36,6 @@ POSIX::mkfifo($fifo, 0600) or die "mkfifo: $!"; seek($conf_fh, 0, SEEK_SET); truncate($conf_fh, 0); print $conf_fh <<EOM; -listen "$host_port" # TODO: remove this requirement for SIGHUP after_fork { |_,_| File.open('$fifo', 'w') { |fp| fp.write "pid=#\$\$" } } EOM $ar->do_kill('HUP'); diff --git a/t/integration.t b/t/integration.t index a568758..bb2ab51 100644 --- a/t/integration.t +++ b/t/integration.t @@ -17,7 +17,6 @@ my $u1 = "$tmpdir/u1"; print $conf_fh <<EOM; early_hints true listen "$u1" -listen "$host_port" # TODO: remove this requirement for SIGHUP EOM my $ar = unicorn(qw(-E none t/integration.ru -c), $conf, { 3 => $srv }); my $curl = which('curl');
<time.h> is still required for gmtime_r(3), and not all versions of <ruby.h> include <time.h>, already. Fixes: a6463151bd1db5b9 (httpdate: favor gettimeofday(2) over time(2) for correctness, 2023-06-01) --- ext/unicorn_http/httpdate.c | 1 + 1 file changed, 1 insertion(+) diff --git a/ext/unicorn_http/httpdate.c b/ext/unicorn_http/httpdate.c index 27a8f51..0faf5da 100644 --- a/ext/unicorn_http/httpdate.c +++ b/ext/unicorn_http/httpdate.c @@ -1,4 +1,5 @@ #include <ruby.h> +#include <time.h> #include <sys/time.h> #include <stdio.h>
While scanning the git@vger.kernel.org mailing list, I've learned time(2) may return the wrong value in the first 1 to 2.5 ms of every second. While I'm not sure if the Date: response header matters to anyone, returning the correct time seems prudent. Link: https://lore.kernel.org/git/20230320230507.3932018-1-gitster@pobox.com/ Link: https://inbox.sourceware.org/libc-alpha/20230306160321.2942372-1-adhemerval.zanella@linaro.org/T/ Link: https://sourceware.org/bugzilla/show_bug.cgi?id=30200 --- ext/unicorn_http/httpdate.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/ext/unicorn_http/httpdate.c b/ext/unicorn_http/httpdate.c index 3f512dd..27a8f51 100644 --- a/ext/unicorn_http/httpdate.c +++ b/ext/unicorn_http/httpdate.c @@ -1,5 +1,5 @@ #include <ruby.h> -#include <time.h> +#include <sys/time.h> #include <stdio.h> static const size_t buf_capa = sizeof("Thu, 01 Jan 1970 00:00:00 GMT"); @@ -43,13 +43,24 @@ static struct tm * my_gmtime_r(time_t *now, struct tm *tm) static VALUE httpdate(VALUE self) { static time_t last; - time_t now = time(NULL); /* not a syscall on modern 64-bit systems */ + struct timeval now; struct tm tm; - if (last == now) + /* + * Favor gettimeofday(2) over time(2), as the latter can return the + * wrong value in the first 1 .. 2.5 ms of every second(!) + * + * https://lore.kernel.org/git/20230320230507.3932018-1-gitster@pobox.com/ + * https://inbox.sourceware.org/libc-alpha/20230306160321.2942372-1-adhemerval.zanella@linaro.org/T/ + * https://sourceware.org/bugzilla/show_bug.cgi?id=30200 + */ + if (gettimeofday(&now, NULL)) + rb_sys_fail("gettimeofday"); + + if (last == now.tv_sec) return buf; - last = now; - gmtime_r(&now, &tm); + last = now.tv_sec; + gmtime_r(&now.tv_sec, &tm); /* we can make this thread-safe later if our Ruby loses the GVL */ snprintf(buf_ptr, buf_capa,
The actual `id_clear' declaration was removed last year, but I missed it's (unused) initialization :x Fixes: c56eb04d683e ("drop Ruby 1.9.3 support, require 2.0+ for now") --- ext/unicorn_http/unicorn_http.rl | 3 --- 1 file changed, 3 deletions(-) diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl index afdf680..fb5dcde 100644 --- a/ext/unicorn_http/unicorn_http.rl +++ b/ext/unicorn_http/unicorn_http.rl @@ -1033,9 +1033,6 @@ void Init_unicorn_http(void) id_set_backtrace = rb_intern("set_backtrace"); init_unicorn_httpdate(); -#ifndef HAVE_RB_HASH_CLEAR - id_clear = rb_intern("clear"); -#endif id_is_chunked_p = rb_intern("is_chunked?"); init_epollexclusive(mUnicorn);
Jeremy Evans <code@jeremyevans.net> wrote: > We deprecated Rack::Chunked in Rack 3.0 and plan to remove it in Rack > 3.1. I agree it would be best to deal with this now, I just wasn't sure > how you wanted to handle it. Your patch below to deal with it at the > server level looks good, though it doesn't appear to remove the Chunked > usage at line 69 of unicorn.rb. I recommend that also be removed. OK. Also added HEAD and STATUS_WITH_NO_ENTITY_BODY checks... No tests, yet; they'll be in Perl 5. (I started rewriting a bunch of tests in Perl5 last year since tests are where Ruby's yearly breaking changes are most unacceptable to me). No trailers for responses yet, either; I didn't realize Rack::Chunked added special support for that in 2020. I care deeply about trailers in requests, but never used them for responses. --------8<------- Subject: [PATCH v2] chunk unterminated HTTP/1.1 responses for Rack 3.1 Rack::Chunked will be gone in Rack 3.1, so provide a non-middleware fallback which takes advantage of IO#write supporting multiple arguments in Ruby 2.5+. We still need to support Ruby 2.4, at least, since Rack 3.0 does. So a new (GC-unfriendly) Unicorn::WriteSplat module now exists for Ruby <= 2.4 users. --- v2: remove Rack::Chunk load attempt fix arity check for Ruby <= 2.4 update docs + examples Interdiff: diff --git a/Documentation/unicorn.1 b/Documentation/unicorn.1 index d76d40f..b2c5e70 100644 --- a/Documentation/unicorn.1 +++ b/Documentation/unicorn.1 @@ -176,7 +176,7 @@ As of Unicorn 0.94.0, RACK_ENV is exported as a process\-wide environment variable as well. While not current a part of the Rack specification as of Rack 1.0.1, this has become a de facto standard in the Rack world. .PP -Note the Rack::ContentLength and Rack::Chunked middlewares are also +Note the Rack::ContentLength middleware is also loaded by "deployment" and "development", but no other values of RACK_ENV. If needed, they must be individually specified in the RACKUP_FILE, some frameworks do not require them. diff --git a/examples/echo.ru b/examples/echo.ru index 14908c5..e982180 100644 --- a/examples/echo.ru +++ b/examples/echo.ru @@ -19,7 +19,6 @@ def each(&block) end -use Rack::Chunked run lambda { |env| /\A100-continue\z/i =~ env['HTTP_EXPECT'] and return [100, {}, []] [ 200, { 'Content-Type' => 'application/octet-stream' }, diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl index c339024..afdf680 100644 --- a/ext/unicorn_http/unicorn_http.rl +++ b/ext/unicorn_http/unicorn_http.rl @@ -28,11 +28,15 @@ void init_unicorn_httpdate(void); #define UH_FL_TO_CLEAR 0x200 #define UH_FL_RESSTART 0x400 /* for check_client_connection */ #define UH_FL_HIJACK 0x800 -#define UH_FL_RES_CHUNK_OK (1U << 12) +#define UH_FL_RES_CHUNK_VER (1U << 12) +#define UH_FL_RES_CHUNK_METHOD (1U << 13) /* all of these flags need to be set for keepalive to be supported */ #define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER) +/* we can only chunk responses for non-HEAD HTTP/1.1 requests */ +#define UH_FL_RES_CHUNKABLE (UH_FL_RES_CHUNK_VER | UH_FL_RES_CHUNK_METHOD) + static unsigned int MAX_HEADER_LEN = 1024 * (80 + 32); /* same as Mongrel */ /* this is only intended for use with Rainbows! */ @@ -146,6 +150,9 @@ request_method(struct http_parser *hp, const char *ptr, size_t len) { VALUE v = rb_str_new(ptr, len); + if (len != 4 || memcmp(ptr, "HEAD", 4)) + HP_FL_SET(hp, RES_CHUNK_METHOD); + rb_hash_aset(hp->env, g_request_method, v); } @@ -159,7 +166,7 @@ http_version(struct http_parser *hp, const char *ptr, size_t len) if (CONST_MEM_EQ("HTTP/1.1", ptr, len)) { /* HTTP/1.1 implies keepalive unless "Connection: close" is set */ HP_FL_SET(hp, KAVERSION); - HP_FL_SET(hp, RES_CHUNK_OK); + HP_FL_SET(hp, RES_CHUNK_VER); v = g_http_11; } else if (CONST_MEM_EQ("HTTP/1.0", ptr, len)) { v = g_http_10; @@ -806,9 +813,9 @@ static VALUE HttpParser_keepalive(VALUE self) /* :nodoc: */ static VALUE chunkable_response_p(VALUE self) { - struct http_parser *hp = data_get(self); + const struct http_parser *hp = data_get(self); - return HP_FL_ALL(hp, RES_CHUNK_OK) ? Qtrue : Qfalse; + return HP_FL_ALL(hp, RES_CHUNKABLE) ? Qtrue : Qfalse; } /** diff --git a/lib/unicorn.rb b/lib/unicorn.rb index 8b1cda7..b817b77 100644 --- a/lib/unicorn.rb +++ b/lib/unicorn.rb @@ -66,7 +66,6 @@ def self.builder(ru, op) middleware = { # order matters ContentLength: nil, - Chunked: nil, CommonLogger: [ $stderr ], ShowExceptions: nil, Lint: nil, diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb index 342dd0b..0ed0ae3 100644 --- a/lib/unicorn/http_response.rb +++ b/lib/unicorn/http_response.rb @@ -12,6 +12,12 @@ module Unicorn::HttpResponse STATUS_CODES = defined?(Rack::Utils::HTTP_STATUS_CODES) ? Rack::Utils::HTTP_STATUS_CODES : {} + STATUS_WITH_NO_ENTITY_BODY = defined?( + Rack::Utils::STATUS_WITH_NO_ENTITY_BODY) ? + Rack::Utils::STATUS_WITH_NO_ENTITY_BODY : begin + warn 'Rack::Utils::STATUS_WITH_NO_ENTITY_BODY missing' + {} + end # internal API, code will always be common-enough-for-even-old-Rack def err_response(code, response_start_sent) @@ -40,7 +46,7 @@ def http_response_write(socket, status, headers, body, code = status.to_i msg = STATUS_CODES[code] start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze - term = false + term = STATUS_WITH_NO_ENTITY_BODY.include?(code) || false buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \ "Date: #{httpdate}\r\n" \ "Connection: close\r\n" diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb index 4ae4c85..c2ba75e 100644 --- a/lib/unicorn/socket_helper.rb +++ b/lib/unicorn/socket_helper.rb @@ -15,7 +15,7 @@ def kgio_tryaccept # :nodoc: end end - if IO.instance_method(:write).arity # Ruby <= 2.4 + if IO.instance_method(:write).arity == 1 # Ruby <= 2.4 require 'unicorn/write_splat' UNIXClient = Class.new(Kgio::Socket) # :nodoc: class UNIXSrv < Kgio::UNIXServer # :nodoc: diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb index cea9791..fe98fcc 100644 --- a/test/unit/test_server.rb +++ b/test/unit/test_server.rb @@ -196,7 +196,7 @@ def test_client_shutdown_writes # continue to process our request and never hit EOFError on our sock sock.shutdown(Socket::SHUT_WR) buf = sock.read - assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/).last + assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/, 2).last next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/")) assert_equal 'hello!\n', next_client lines = File.readlines("test_stderr.#$$.log") Documentation/unicorn.1 | 2 +- examples/echo.ru | 1 - ext/unicorn_http/unicorn_http.rl | 18 ++++++++++++++++++ lib/unicorn.rb | 5 ++--- lib/unicorn/http_response.rb | 27 ++++++++++++++++++++++++++- lib/unicorn/socket_helper.rb | 18 ++++++++++++++++-- lib/unicorn/write_splat.rb | 7 +++++++ test/unit/test_server.rb | 2 +- 8 files changed, 71 insertions(+), 9 deletions(-) create mode 100644 lib/unicorn/write_splat.rb diff --git a/Documentation/unicorn.1 b/Documentation/unicorn.1 index d76d40f..b2c5e70 100644 --- a/Documentation/unicorn.1 +++ b/Documentation/unicorn.1 @@ -176,7 +176,7 @@ As of Unicorn 0.94.0, RACK_ENV is exported as a process\-wide environment variable as well. While not current a part of the Rack specification as of Rack 1.0.1, this has become a de facto standard in the Rack world. .PP -Note the Rack::ContentLength and Rack::Chunked middlewares are also +Note the Rack::ContentLength middleware is also loaded by "deployment" and "development", but no other values of RACK_ENV. If needed, they must be individually specified in the RACKUP_FILE, some frameworks do not require them. diff --git a/examples/echo.ru b/examples/echo.ru index 14908c5..e982180 100644 --- a/examples/echo.ru +++ b/examples/echo.ru @@ -19,7 +19,6 @@ def each(&block) end -use Rack::Chunked run lambda { |env| /\A100-continue\z/i =~ env['HTTP_EXPECT'] and return [100, {}, []] [ 200, { 'Content-Type' => 'application/octet-stream' }, diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl index ba23438..afdf680 100644 --- a/ext/unicorn_http/unicorn_http.rl +++ b/ext/unicorn_http/unicorn_http.rl @@ -28,10 +28,15 @@ void init_unicorn_httpdate(void); #define UH_FL_TO_CLEAR 0x200 #define UH_FL_RESSTART 0x400 /* for check_client_connection */ #define UH_FL_HIJACK 0x800 +#define UH_FL_RES_CHUNK_VER (1U << 12) +#define UH_FL_RES_CHUNK_METHOD (1U << 13) /* all of these flags need to be set for keepalive to be supported */ #define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER) +/* we can only chunk responses for non-HEAD HTTP/1.1 requests */ +#define UH_FL_RES_CHUNKABLE (UH_FL_RES_CHUNK_VER | UH_FL_RES_CHUNK_METHOD) + static unsigned int MAX_HEADER_LEN = 1024 * (80 + 32); /* same as Mongrel */ /* this is only intended for use with Rainbows! */ @@ -145,6 +150,9 @@ request_method(struct http_parser *hp, const char *ptr, size_t len) { VALUE v = rb_str_new(ptr, len); + if (len != 4 || memcmp(ptr, "HEAD", 4)) + HP_FL_SET(hp, RES_CHUNK_METHOD); + rb_hash_aset(hp->env, g_request_method, v); } @@ -158,6 +166,7 @@ http_version(struct http_parser *hp, const char *ptr, size_t len) if (CONST_MEM_EQ("HTTP/1.1", ptr, len)) { /* HTTP/1.1 implies keepalive unless "Connection: close" is set */ HP_FL_SET(hp, KAVERSION); + HP_FL_SET(hp, RES_CHUNK_VER); v = g_http_11; } else if (CONST_MEM_EQ("HTTP/1.0", ptr, len)) { v = g_http_10; @@ -801,6 +810,14 @@ static VALUE HttpParser_keepalive(VALUE self) return HP_FL_ALL(hp, KEEPALIVE) ? Qtrue : Qfalse; } +/* :nodoc: */ +static VALUE chunkable_response_p(VALUE self) +{ + const struct http_parser *hp = data_get(self); + + return HP_FL_ALL(hp, RES_CHUNKABLE) ? Qtrue : Qfalse; +} + /** * call-seq: * parser.next? => true or false @@ -981,6 +998,7 @@ void Init_unicorn_http(void) rb_define_method(cHttpParser, "content_length", HttpParser_content_length, 0); rb_define_method(cHttpParser, "body_eof?", HttpParser_body_eof, 0); rb_define_method(cHttpParser, "keepalive?", HttpParser_keepalive, 0); + rb_define_method(cHttpParser, "chunkable_response?", chunkable_response_p, 0); rb_define_method(cHttpParser, "headers?", HttpParser_has_headers, 0); rb_define_method(cHttpParser, "next?", HttpParser_next, 0); rb_define_method(cHttpParser, "buf", HttpParser_buf, 0); diff --git a/lib/unicorn.rb b/lib/unicorn.rb index 1a50631..b817b77 100644 --- a/lib/unicorn.rb +++ b/lib/unicorn.rb @@ -66,7 +66,6 @@ def self.builder(ru, op) middleware = { # order matters ContentLength: nil, - Chunked: nil, CommonLogger: [ $stderr ], ShowExceptions: nil, Lint: nil, @@ -75,8 +74,8 @@ def self.builder(ru, op) # return value, matches rackup defaults based on env # Unicorn does not support persistent connections, but Rainbows! - # and Zbatery both do. Users accustomed to the Rack::Server default - # middlewares will need ContentLength/Chunked middlewares. + # does. Users accustomed to the Rack::Server default + # middlewares will need ContentLength middleware. case ENV["RACK_ENV"] when "development" when "deployment" diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb index 19469b4..0ed0ae3 100644 --- a/lib/unicorn/http_response.rb +++ b/lib/unicorn/http_response.rb @@ -12,6 +12,12 @@ module Unicorn::HttpResponse STATUS_CODES = defined?(Rack::Utils::HTTP_STATUS_CODES) ? Rack::Utils::HTTP_STATUS_CODES : {} + STATUS_WITH_NO_ENTITY_BODY = defined?( + Rack::Utils::STATUS_WITH_NO_ENTITY_BODY) ? + Rack::Utils::STATUS_WITH_NO_ENTITY_BODY : begin + warn 'Rack::Utils::STATUS_WITH_NO_ENTITY_BODY missing' + {} + end # internal API, code will always be common-enough-for-even-old-Rack def err_response(code, response_start_sent) @@ -35,11 +41,12 @@ def append_header(buf, key, value) def http_response_write(socket, status, headers, body, req = Unicorn::HttpRequest.new) hijack = nil - + do_chunk = false if headers code = status.to_i msg = STATUS_CODES[code] start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze + term = STATUS_WITH_NO_ENTITY_BODY.include?(code) || false buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \ "Date: #{httpdate}\r\n" \ "Connection: close\r\n" @@ -47,6 +54,12 @@ def http_response_write(socket, status, headers, body, case key when %r{\A(?:Date|Connection)\z}i next + when %r{\AContent-Length\z}i + append_header(buf, key, value) + term = true + when %r{\ATransfer-Encoding\z}i + append_header(buf, key, value) + term = true if /\bchunked\b/i === value # value may be Array :x when "rack.hijack" # This should only be hit under Rack >= 1.5, as this was an illegal # key in Rack < 1.5 @@ -55,12 +68,24 @@ def http_response_write(socket, status, headers, body, append_header(buf, key, value) end end + if !hijack && !term && req.chunkable_response? + do_chunk = true + buf << "Transfer-Encoding: chunked\r\n".freeze + end socket.write(buf << "\r\n".freeze) end if hijack req.hijacked! hijack.call(socket) + elsif do_chunk + begin + body.each do |b| + socket.write("#{b.bytesize.to_s(16)}\r\n", b, "\r\n".freeze) + end + ensure + socket.write("0\r\n\r\n".freeze) + end else body.each { |chunk| socket.write(chunk) } end diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb index 8a6f6ee..c2ba75e 100644 --- a/lib/unicorn/socket_helper.rb +++ b/lib/unicorn/socket_helper.rb @@ -15,6 +15,20 @@ def kgio_tryaccept # :nodoc: end end + if IO.instance_method(:write).arity == 1 # Ruby <= 2.4 + require 'unicorn/write_splat' + UNIXClient = Class.new(Kgio::Socket) # :nodoc: + class UNIXSrv < Kgio::UNIXServer # :nodoc: + include Unicorn::WriteSplat + def kgio_tryaccept # :nodoc: + super(UNIXClient) + end + end + TCPClient.__send__(:include, Unicorn::WriteSplat) + else # Ruby 2.5+ + UNIXSrv = Kgio::UNIXServer + end + module SocketHelper # internal interface @@ -135,7 +149,7 @@ def bind_listen(address = '0.0.0.0:8080', opt = {}) end old_umask = File.umask(opt[:umask] || 0) begin - Kgio::UNIXServer.new(address) + UNIXSrv.new(address) ensure File.umask(old_umask) end @@ -203,7 +217,7 @@ def server_cast(sock) Socket.unpack_sockaddr_in(sock.getsockname) TCPSrv.for_fd(sock.fileno) rescue ArgumentError - Kgio::UNIXServer.for_fd(sock.fileno) + UNIXSrv.for_fd(sock.fileno) end end diff --git a/lib/unicorn/write_splat.rb b/lib/unicorn/write_splat.rb new file mode 100644 index 0000000..7e6e363 --- /dev/null +++ b/lib/unicorn/write_splat.rb @@ -0,0 +1,7 @@ +# -*- encoding: binary -*- +# compatibility module for Ruby <= 2.4, remove when we go Ruby 2.5+ +module Unicorn::WriteSplat # :nodoc: + def write(*arg) # :nodoc: + super(arg.join('')) + end +end diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb index 98e85ab..fe98fcc 100644 --- a/test/unit/test_server.rb +++ b/test/unit/test_server.rb @@ -196,7 +196,7 @@ def test_client_shutdown_writes # continue to process our request and never hit EOFError on our sock sock.shutdown(Socket::SHUT_WR) buf = sock.read - assert_equal 'hello!\n', buf.split(/\r\n\r\n/).last + assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/, 2).last next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/")) assert_equal 'hello!\n', next_client lines = File.readlines("test_stderr.#$$.log")
On 06/02 12:00, Eric Wong wrote: > Jeremy Evans <code@jeremyevans.net> wrote: > > This takes Eric's patch from December 25, 2022, and includes all > > necessary test fixes to allow Unicorn tests to pass with both > > Rack 3 and Rack 2 (and probably Rack 1). It includes a test fix for > > newer curl versions and an OpenBSD test fix. > > > > Hopefully this is acceptable and Unicorn 6.2 can be released with Rack 3 > > support. If further fixes are needed, I'm happy to work on them. > > Isn't a chunk replacement needed for Rack 3.1, also? > I dunno if I missed anything else in Rack 3.x; and don't want to > make too many releases if we can do 3.0 and 3.1 in one go. We deprecated Rack::Chunked in Rack 3.0 and plan to remove it in Rack 3.1. I agree it would be best to deal with this now, I just wasn't sure how you wanted to handle it. Your patch below to deal with it at the server level looks good, though it doesn't appear to remove the Chunked usage at line 69 of unicorn.rb. I recommend that also be removed. Thanks, Jeremy > -------8<------- > Subject: [PATCH] chunk unterminated HTTP/1.1 responses > > Rack::Chunked will be gone in Rack 3.1, so provide a > non-middleware fallback which takes advantage of IO#write > supporting multiple arguments in Ruby 2.5+. > > We still need to support Ruby 2.4, at least, since Rack 3.0 > does. So a new (GC-unfriendly) Unicorn::WriteSplat module now > exists for Ruby <= 2.4 users. > --- > ext/unicorn_http/unicorn_http.rl | 11 +++++++++++ > lib/unicorn.rb | 4 ++-- > lib/unicorn/http_response.rb | 21 ++++++++++++++++++++- > lib/unicorn/socket_helper.rb | 18 ++++++++++++++++-- > lib/unicorn/write_splat.rb | 7 +++++++ > test/unit/test_server.rb | 2 +- > 6 files changed, 57 insertions(+), 6 deletions(-) > create mode 100644 lib/unicorn/write_splat.rb > > diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl > index ba23438..c339024 100644 > --- a/ext/unicorn_http/unicorn_http.rl > +++ b/ext/unicorn_http/unicorn_http.rl > @@ -28,6 +28,7 @@ void init_unicorn_httpdate(void); > #define UH_FL_TO_CLEAR 0x200 > #define UH_FL_RESSTART 0x400 /* for check_client_connection */ > #define UH_FL_HIJACK 0x800 > +#define UH_FL_RES_CHUNK_OK (1U << 12) > > /* all of these flags need to be set for keepalive to be supported */ > #define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER) > @@ -158,6 +159,7 @@ http_version(struct http_parser *hp, const char *ptr, size_t len) > if (CONST_MEM_EQ("HTTP/1.1", ptr, len)) { > /* HTTP/1.1 implies keepalive unless "Connection: close" is set */ > HP_FL_SET(hp, KAVERSION); > + HP_FL_SET(hp, RES_CHUNK_OK); > v = g_http_11; > } else if (CONST_MEM_EQ("HTTP/1.0", ptr, len)) { > v = g_http_10; > @@ -801,6 +803,14 @@ static VALUE HttpParser_keepalive(VALUE self) > return HP_FL_ALL(hp, KEEPALIVE) ? Qtrue : Qfalse; > } > > +/* :nodoc: */ > +static VALUE chunkable_response_p(VALUE self) > +{ > + struct http_parser *hp = data_get(self); > + > + return HP_FL_ALL(hp, RES_CHUNK_OK) ? Qtrue : Qfalse; > +} > + > /** > * call-seq: > * parser.next? => true or false > @@ -981,6 +991,7 @@ void Init_unicorn_http(void) > rb_define_method(cHttpParser, "content_length", HttpParser_content_length, 0); > rb_define_method(cHttpParser, "body_eof?", HttpParser_body_eof, 0); > rb_define_method(cHttpParser, "keepalive?", HttpParser_keepalive, 0); > + rb_define_method(cHttpParser, "chunkable_response?", chunkable_response_p, 0); > rb_define_method(cHttpParser, "headers?", HttpParser_has_headers, 0); > rb_define_method(cHttpParser, "next?", HttpParser_next, 0); > rb_define_method(cHttpParser, "buf", HttpParser_buf, 0); > diff --git a/lib/unicorn.rb b/lib/unicorn.rb > index 1a50631..8b1cda7 100644 > --- a/lib/unicorn.rb > +++ b/lib/unicorn.rb > @@ -75,8 +75,8 @@ def self.builder(ru, op) > > # return value, matches rackup defaults based on env > # Unicorn does not support persistent connections, but Rainbows! > - # and Zbatery both do. Users accustomed to the Rack::Server default > - # middlewares will need ContentLength/Chunked middlewares. > + # does. Users accustomed to the Rack::Server default > + # middlewares will need ContentLength middleware. > case ENV["RACK_ENV"] > when "development" > when "deployment" > diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb > index 19469b4..342dd0b 100644 > --- a/lib/unicorn/http_response.rb > +++ b/lib/unicorn/http_response.rb > @@ -35,11 +35,12 @@ def append_header(buf, key, value) > def http_response_write(socket, status, headers, body, > req = Unicorn::HttpRequest.new) > hijack = nil > - > + do_chunk = false > if headers > code = status.to_i > msg = STATUS_CODES[code] > start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze > + term = false > buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \ > "Date: #{httpdate}\r\n" \ > "Connection: close\r\n" > @@ -47,6 +48,12 @@ def http_response_write(socket, status, headers, body, > case key > when %r{\A(?:Date|Connection)\z}i > next > + when %r{\AContent-Length\z}i > + append_header(buf, key, value) > + term = true > + when %r{\ATransfer-Encoding\z}i > + append_header(buf, key, value) > + term = true if /\bchunked\b/i === value # value may be Array :x > when "rack.hijack" > # This should only be hit under Rack >= 1.5, as this was an illegal > # key in Rack < 1.5 > @@ -55,12 +62,24 @@ def http_response_write(socket, status, headers, body, > append_header(buf, key, value) > end > end > + if !hijack && !term && req.chunkable_response? > + do_chunk = true > + buf << "Transfer-Encoding: chunked\r\n".freeze > + end > socket.write(buf << "\r\n".freeze) > end > > if hijack > req.hijacked! > hijack.call(socket) > + elsif do_chunk > + begin > + body.each do |b| > + socket.write("#{b.bytesize.to_s(16)}\r\n", b, "\r\n".freeze) > + end > + ensure > + socket.write("0\r\n\r\n".freeze) > + end > else > body.each { |chunk| socket.write(chunk) } > end > diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb > index 8a6f6ee..4ae4c85 100644 > --- a/lib/unicorn/socket_helper.rb > +++ b/lib/unicorn/socket_helper.rb > @@ -15,6 +15,20 @@ def kgio_tryaccept # :nodoc: > end > end > > + if IO.instance_method(:write).arity # Ruby <= 2.4 > + require 'unicorn/write_splat' > + UNIXClient = Class.new(Kgio::Socket) # :nodoc: > + class UNIXSrv < Kgio::UNIXServer # :nodoc: > + include Unicorn::WriteSplat > + def kgio_tryaccept # :nodoc: > + super(UNIXClient) > + end > + end > + TCPClient.__send__(:include, Unicorn::WriteSplat) > + else # Ruby 2.5+ > + UNIXSrv = Kgio::UNIXServer > + end > + > module SocketHelper > > # internal interface > @@ -135,7 +149,7 @@ def bind_listen(address = '0.0.0.0:8080', opt = {}) > end > old_umask = File.umask(opt[:umask] || 0) > begin > - Kgio::UNIXServer.new(address) > + UNIXSrv.new(address) > ensure > File.umask(old_umask) > end > @@ -203,7 +217,7 @@ def server_cast(sock) > Socket.unpack_sockaddr_in(sock.getsockname) > TCPSrv.for_fd(sock.fileno) > rescue ArgumentError > - Kgio::UNIXServer.for_fd(sock.fileno) > + UNIXSrv.for_fd(sock.fileno) > end > end > > diff --git a/lib/unicorn/write_splat.rb b/lib/unicorn/write_splat.rb > new file mode 100644 > index 0000000..7e6e363 > --- /dev/null > +++ b/lib/unicorn/write_splat.rb > @@ -0,0 +1,7 @@ > +# -*- encoding: binary -*- > +# compatibility module for Ruby <= 2.4, remove when we go Ruby 2.5+ > +module Unicorn::WriteSplat # :nodoc: > + def write(*arg) # :nodoc: > + super(arg.join('')) > + end > +end > diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb > index 98e85ab..cea9791 100644 > --- a/test/unit/test_server.rb > +++ b/test/unit/test_server.rb > @@ -196,7 +196,7 @@ def test_client_shutdown_writes > # continue to process our request and never hit EOFError on our sock > sock.shutdown(Socket::SHUT_WR) > buf = sock.read > - assert_equal 'hello!\n', buf.split(/\r\n\r\n/).last > + assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/).last > next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/")) > assert_equal 'hello!\n', next_client > lines = File.readlines("test_stderr.#$$.log")
Eric Wong <bofh@yhbt.net> wrote:
> +++ b/lib/unicorn/socket_helper.rb
> @@ -15,6 +15,20 @@ def kgio_tryaccept # :nodoc:
> end
> end
>
> + if IO.instance_method(:write).arity # Ruby <= 2.4
Erm, that should be:
if IO.instance_method(:write).arity == 1 # Ruby <= 2.4
Jeremy Evans <code@jeremyevans.net> wrote: > This takes Eric's patch from December 25, 2022, and includes all > necessary test fixes to allow Unicorn tests to pass with both > Rack 3 and Rack 2 (and probably Rack 1). It includes a test fix for > newer curl versions and an OpenBSD test fix. > > Hopefully this is acceptable and Unicorn 6.2 can be released with Rack 3 > support. If further fixes are needed, I'm happy to work on them. Isn't a chunk replacement needed for Rack 3.1, also? I dunno if I missed anything else in Rack 3.x; and don't want to make too many releases if we can do 3.0 and 3.1 in one go. -------8<------- Subject: [PATCH] chunk unterminated HTTP/1.1 responses Rack::Chunked will be gone in Rack 3.1, so provide a non-middleware fallback which takes advantage of IO#write supporting multiple arguments in Ruby 2.5+. We still need to support Ruby 2.4, at least, since Rack 3.0 does. So a new (GC-unfriendly) Unicorn::WriteSplat module now exists for Ruby <= 2.4 users. --- ext/unicorn_http/unicorn_http.rl | 11 +++++++++++ lib/unicorn.rb | 4 ++-- lib/unicorn/http_response.rb | 21 ++++++++++++++++++++- lib/unicorn/socket_helper.rb | 18 ++++++++++++++++-- lib/unicorn/write_splat.rb | 7 +++++++ test/unit/test_server.rb | 2 +- 6 files changed, 57 insertions(+), 6 deletions(-) create mode 100644 lib/unicorn/write_splat.rb diff --git a/ext/unicorn_http/unicorn_http.rl b/ext/unicorn_http/unicorn_http.rl index ba23438..c339024 100644 --- a/ext/unicorn_http/unicorn_http.rl +++ b/ext/unicorn_http/unicorn_http.rl @@ -28,6 +28,7 @@ void init_unicorn_httpdate(void); #define UH_FL_TO_CLEAR 0x200 #define UH_FL_RESSTART 0x400 /* for check_client_connection */ #define UH_FL_HIJACK 0x800 +#define UH_FL_RES_CHUNK_OK (1U << 12) /* all of these flags need to be set for keepalive to be supported */ #define UH_FL_KEEPALIVE (UH_FL_KAVERSION | UH_FL_REQEOF | UH_FL_HASHEADER) @@ -158,6 +159,7 @@ http_version(struct http_parser *hp, const char *ptr, size_t len) if (CONST_MEM_EQ("HTTP/1.1", ptr, len)) { /* HTTP/1.1 implies keepalive unless "Connection: close" is set */ HP_FL_SET(hp, KAVERSION); + HP_FL_SET(hp, RES_CHUNK_OK); v = g_http_11; } else if (CONST_MEM_EQ("HTTP/1.0", ptr, len)) { v = g_http_10; @@ -801,6 +803,14 @@ static VALUE HttpParser_keepalive(VALUE self) return HP_FL_ALL(hp, KEEPALIVE) ? Qtrue : Qfalse; } +/* :nodoc: */ +static VALUE chunkable_response_p(VALUE self) +{ + struct http_parser *hp = data_get(self); + + return HP_FL_ALL(hp, RES_CHUNK_OK) ? Qtrue : Qfalse; +} + /** * call-seq: * parser.next? => true or false @@ -981,6 +991,7 @@ void Init_unicorn_http(void) rb_define_method(cHttpParser, "content_length", HttpParser_content_length, 0); rb_define_method(cHttpParser, "body_eof?", HttpParser_body_eof, 0); rb_define_method(cHttpParser, "keepalive?", HttpParser_keepalive, 0); + rb_define_method(cHttpParser, "chunkable_response?", chunkable_response_p, 0); rb_define_method(cHttpParser, "headers?", HttpParser_has_headers, 0); rb_define_method(cHttpParser, "next?", HttpParser_next, 0); rb_define_method(cHttpParser, "buf", HttpParser_buf, 0); diff --git a/lib/unicorn.rb b/lib/unicorn.rb index 1a50631..8b1cda7 100644 --- a/lib/unicorn.rb +++ b/lib/unicorn.rb @@ -75,8 +75,8 @@ def self.builder(ru, op) # return value, matches rackup defaults based on env # Unicorn does not support persistent connections, but Rainbows! - # and Zbatery both do. Users accustomed to the Rack::Server default - # middlewares will need ContentLength/Chunked middlewares. + # does. Users accustomed to the Rack::Server default + # middlewares will need ContentLength middleware. case ENV["RACK_ENV"] when "development" when "deployment" diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb index 19469b4..342dd0b 100644 --- a/lib/unicorn/http_response.rb +++ b/lib/unicorn/http_response.rb @@ -35,11 +35,12 @@ def append_header(buf, key, value) def http_response_write(socket, status, headers, body, req = Unicorn::HttpRequest.new) hijack = nil - + do_chunk = false if headers code = status.to_i msg = STATUS_CODES[code] start = req.response_start_sent ? ''.freeze : 'HTTP/1.1 '.freeze + term = false buf = "#{start}#{msg ? %Q(#{code} #{msg}) : status}\r\n" \ "Date: #{httpdate}\r\n" \ "Connection: close\r\n" @@ -47,6 +48,12 @@ def http_response_write(socket, status, headers, body, case key when %r{\A(?:Date|Connection)\z}i next + when %r{\AContent-Length\z}i + append_header(buf, key, value) + term = true + when %r{\ATransfer-Encoding\z}i + append_header(buf, key, value) + term = true if /\bchunked\b/i === value # value may be Array :x when "rack.hijack" # This should only be hit under Rack >= 1.5, as this was an illegal # key in Rack < 1.5 @@ -55,12 +62,24 @@ def http_response_write(socket, status, headers, body, append_header(buf, key, value) end end + if !hijack && !term && req.chunkable_response? + do_chunk = true + buf << "Transfer-Encoding: chunked\r\n".freeze + end socket.write(buf << "\r\n".freeze) end if hijack req.hijacked! hijack.call(socket) + elsif do_chunk + begin + body.each do |b| + socket.write("#{b.bytesize.to_s(16)}\r\n", b, "\r\n".freeze) + end + ensure + socket.write("0\r\n\r\n".freeze) + end else body.each { |chunk| socket.write(chunk) } end diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb index 8a6f6ee..4ae4c85 100644 --- a/lib/unicorn/socket_helper.rb +++ b/lib/unicorn/socket_helper.rb @@ -15,6 +15,20 @@ def kgio_tryaccept # :nodoc: end end + if IO.instance_method(:write).arity # Ruby <= 2.4 + require 'unicorn/write_splat' + UNIXClient = Class.new(Kgio::Socket) # :nodoc: + class UNIXSrv < Kgio::UNIXServer # :nodoc: + include Unicorn::WriteSplat + def kgio_tryaccept # :nodoc: + super(UNIXClient) + end + end + TCPClient.__send__(:include, Unicorn::WriteSplat) + else # Ruby 2.5+ + UNIXSrv = Kgio::UNIXServer + end + module SocketHelper # internal interface @@ -135,7 +149,7 @@ def bind_listen(address = '0.0.0.0:8080', opt = {}) end old_umask = File.umask(opt[:umask] || 0) begin - Kgio::UNIXServer.new(address) + UNIXSrv.new(address) ensure File.umask(old_umask) end @@ -203,7 +217,7 @@ def server_cast(sock) Socket.unpack_sockaddr_in(sock.getsockname) TCPSrv.for_fd(sock.fileno) rescue ArgumentError - Kgio::UNIXServer.for_fd(sock.fileno) + UNIXSrv.for_fd(sock.fileno) end end diff --git a/lib/unicorn/write_splat.rb b/lib/unicorn/write_splat.rb new file mode 100644 index 0000000..7e6e363 --- /dev/null +++ b/lib/unicorn/write_splat.rb @@ -0,0 +1,7 @@ +# -*- encoding: binary -*- +# compatibility module for Ruby <= 2.4, remove when we go Ruby 2.5+ +module Unicorn::WriteSplat # :nodoc: + def write(*arg) # :nodoc: + super(arg.join('')) + end +end diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb index 98e85ab..cea9791 100644 --- a/test/unit/test_server.rb +++ b/test/unit/test_server.rb @@ -196,7 +196,7 @@ def test_client_shutdown_writes # continue to process our request and never hit EOFError on our sock sock.shutdown(Socket::SHUT_WR) buf = sock.read - assert_equal 'hello!\n', buf.split(/\r\n\r\n/).last + assert_match %r{\bhello!\\n\b}, buf.split(/\r\n\r\n/).last next_client = Net::HTTP.get(URI.parse("http://127.0.0.1:#@port/")) assert_equal 'hello!\n', next_client lines = File.readlines("test_stderr.#$$.log")
This takes Eric's patch from December 25, 2022, and includes all necessary test fixes to allow Unicorn tests to pass with both Rack 3 and Rack 2 (and probably Rack 1). It includes a test fix for newer curl versions and an OpenBSD test fix. Hopefully this is acceptable and Unicorn 6.2 can be released with Rack 3 support. If further fixes are needed, I'm happy to work on them. Thanks, Jeremy --- From 6ba4e7234af9d6ea3e85e646f798b1fbf6799234 Mon Sep 17 00:00:00 2001 From: Jeremy Evans <code@jeremyevans.net> Date: Thu, 1 Jun 2023 08:55:14 -0700 Subject: [PATCH] Support Rack 3 and fix tests on Rack 3 Most changes are to the tests to avoid uppercase characters in header keys, which are no longer allowed in rack 3 (to allow for O(1) access). This also changes a few places where an array of headers was used to switch to a hash, as a hash is requierd in Rack 3. Newer versions of curl use a 000 http_code for invalid http codes, so switch from "42 -eq" to "500 -ne" in the test, as Rack::Lint will always raise a 500 error. There is one test that fails on OpenBSD when opening a fifo. This is unrelated to unicorn as far as I can see, so skip the remaining part of the test in that case on OpenBSD. Tests still pass on Rack 2, and presumably Rack 1 as well, though I didn't test Rack 1. Co-authored-by: Eric Wong <bofh@yhbt.net> --- lib/unicorn/http_response.rb | 19 +++++++++++++------ lib/unicorn/http_server.rb | 21 +++++---------------- t/heartbeat-timeout.ru | 2 +- t/rack-input-tests.ru | 2 +- t/t0300-no-default-middleware.sh | 2 +- t/t0301.ru | 4 ++-- t/write-on-close.ru | 2 +- test/exec/test_exec.rb | 20 +++++++++++++------- test/unit/test_ccc.rb | 2 +- test/unit/test_request.rb | 2 +- test/unit/test_server.rb | 8 ++++---- test/unit/test_signals.rb | 14 +++++++------- 12 files changed, 50 insertions(+), 48 deletions(-) diff --git a/lib/unicorn/http_response.rb b/lib/unicorn/http_response.rb index b23e521..19469b4 100644 --- a/lib/unicorn/http_response.rb +++ b/lib/unicorn/http_response.rb @@ -19,6 +19,18 @@ def err_response(code, response_start_sent) "#{code} #{STATUS_CODES[code]}\r\n\r\n" end + def append_header(buf, key, value) + case value + when Array # Rack 3 + value.each { |v| buf << "#{key}: #{v}\r\n" } + when /\n/ # Rack 2 + # avoiding blank, key-only cookies with /\n+/ + value.split(/\n+/).each { |v| buf << "#{key}: #{v}\r\n" } + else + buf << "#{key}: #{value}\r\n" + end + end + # writes the rack_response to socket as an HTTP response def http_response_write(socket, status, headers, body, req = Unicorn::HttpRequest.new) @@ -40,12 +52,7 @@ def http_response_write(socket, status, headers, body, # key in Rack < 1.5 hijack = value else - if value =~ /\n/ - # avoiding blank, key-only cookies with /\n+/ - value.split(/\n+/).each { |v| buf << "#{key}: #{v}\r\n" } - else - buf << "#{key}: #{value}\r\n" - end + append_header(buf, key, value) end end socket.write(buf << "\r\n".freeze) diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb index 3416808..cad515b 100644 --- a/lib/unicorn/http_server.rb +++ b/lib/unicorn/http_server.rb @@ -589,22 +589,11 @@ def handle_error(client, e) end def e103_response_write(client, headers) - response = if @request.response_start_sent - "103 Early Hints\r\n" - else - "HTTP/1.1 103 Early Hints\r\n" - end - - headers.each_pair do |k, vs| - next if !vs || vs.empty? - values = vs.to_s.split("\n".freeze) - values.each do |v| - response << "#{k}: #{v}\r\n" - end - end - response << "\r\n".freeze - response << "HTTP/1.1 ".freeze if @request.response_start_sent - client.write(response) + rss = @request.response_start_sent + buf = rss ? "103 Early Hints\r\n" : "HTTP/1.1 103 Early Hints\r\n" + headers.each { |key, value| append_header(buf, key, value) } + buf << (rss ? "\r\nHTTP/1.1 ".freeze : "\r\n".freeze) + client.write(buf) end def e100_response_write(client, env) diff --git a/t/heartbeat-timeout.ru b/t/heartbeat-timeout.ru index d9904e8..20a7938 100644 --- a/t/heartbeat-timeout.ru +++ b/t/heartbeat-timeout.ru @@ -1,5 +1,5 @@ use Rack::ContentLength -headers = { 'Content-Type' => 'text/plain' } +headers = { 'content-type' => 'text/plain' } run lambda { |env| case env['PATH_INFO'] when "/block-forever" diff --git a/t/rack-input-tests.ru b/t/rack-input-tests.ru index 8c35630..5459e85 100644 --- a/t/rack-input-tests.ru +++ b/t/rack-input-tests.ru @@ -16,6 +16,6 @@ end while input.read(rand(cap), buf) end - [ 200, {'Content-Type' => 'text/plain'}, [ digest.hexdigest << "\n" ] ] + [ 200, {'content-type' => 'text/plain'}, [ digest.hexdigest << "\n" ] ] end run app diff --git a/t/t0300-no-default-middleware.sh b/t/t0300-no-default-middleware.sh index 779dc02..00feacc 100644 --- a/t/t0300-no-default-middleware.sh +++ b/t/t0300-no-default-middleware.sh @@ -9,7 +9,7 @@ t_begin "setup and start" && { } t_begin "check exit status with Rack::Lint not present" && { - test 42 -eq "$(curl -sf -o/dev/null -w'%{http_code}' http://$listen/)" + test 500 -ne "$(curl -sf -o/dev/null -w'%{http_code}' http://$listen/)" } t_begin "killing succeeds" && { diff --git a/t/t0301.ru b/t/t0301.ru index 1ae8ea7..ce68213 100644 --- a/t/t0301.ru +++ b/t/t0301.ru @@ -6,8 +6,8 @@ "lint=#{caller.grep(%r{rack/lint\.rb})[0].split(':')[0]}\n" end h = { - 'Content-Length' => b.size.to_s, - 'Content-Type' => 'text/plain', + 'content-length' => b.size.to_s, + 'content-type' => 'text/plain', } [ 200, h, [ b ] ] end) diff --git a/t/write-on-close.ru b/t/write-on-close.ru index 54a2f2e..725c4d6 100644 --- a/t/write-on-close.ru +++ b/t/write-on-close.ru @@ -8,4 +8,4 @@ def close end end use Rack::ContentType, "text/plain" -run(lambda { |_| [ 200, [%w(Transfer-Encoding chunked)], WriteOnClose.new ] }) +run(lambda { |_| [ 200, { 'transfer-encoding' => 'chunked' }, WriteOnClose.new ] }) diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb index aacd917..2929b2e 100644 --- a/test/exec/test_exec.rb +++ b/test/exec/test_exec.rb @@ -25,20 +25,20 @@ class ExecTest < Test::Unit::TestCase HI = <<-EOS use Rack::ContentLength -run proc { |env| [ 200, { 'Content-Type' => 'text/plain' }, [ "HI\\n" ] ] } +run proc { |env| [ 200, { 'content-type' => 'text/plain' }, [ "HI\\n" ] ] } EOS SHOW_RACK_ENV = <<-EOS use Rack::ContentLength run proc { |env| - [ 200, { 'Content-Type' => 'text/plain' }, [ ENV['RACK_ENV'] ] ] + [ 200, { 'content-type' => 'text/plain' }, [ ENV['RACK_ENV'] ] ] } EOS HELLO = <<-EOS class Hello def call(env) - [ 200, { 'Content-Type' => 'text/plain' }, [ "HI\\n" ] ] + [ 200, { 'content-type' => 'text/plain' }, [ "HI\\n" ] ] end end EOS @@ -62,9 +62,9 @@ def call(env) a = ::File.stat(pwd) b = ::File.stat(Dir.pwd) if (a.ino == b.ino && a.dev == b.dev) - [ 200, { 'Content-Type' => 'text/plain' }, [ pwd ] ] + [ 200, { 'content-type' => 'text/plain' }, [ pwd ] ] else - [ 404, { 'Content-Type' => 'text/plain' }, [] ] + [ 404, { 'content-type' => 'text/plain' }, [] ] end } EOS @@ -255,7 +255,13 @@ def test_working_directory_controls_relative_paths end EOF pid = xfork { redirect_test_io { exec($unicorn_bin, "-c#{tmp.path}") } } - File.open("#{other.path}/fifo", "rb").close + begin + fifo = File.open("#{other.path}/fifo", "rb") + rescue Errno::EINTR + # OpenBSD raises Errno::EINTR when opening + return if RUBY_PLATFORM =~ /openbsd/ + end + fifo.close assert ! File.exist?("stderr_log_here") assert ! File.exist?("stdout_log_here") @@ -557,7 +563,7 @@ def test_unicorn_config_listen_with_options def test_unicorn_config_per_worker_listen port2 = unused_port pid_spit = 'use Rack::ContentLength;' \ - 'run proc { |e| [ 200, {"Content-Type"=>"text/plain"}, ["#$$\\n"] ] }' + 'run proc { |e| [ 200, {"content-type"=>"text/plain"}, ["#$$\\n"] ] }' File.open("config.ru", "wb") { |fp| fp.syswrite(pid_spit) } tmp = Tempfile.new('test.socket') File.unlink(tmp.path) diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb index 0dc72e8..f518230 100644 --- a/test/unit/test_ccc.rb +++ b/test/unit/test_ccc.rb @@ -29,7 +29,7 @@ def test_ccc_tcpi # will wake up when writer closes sleep_pipe[0].read if env['PATH_INFO'] == '/sleep' - [ 200, [ %w(Content-Length 0), %w(Content-Type text/plain) ], [] ] + [ 200, {'content-length'=>'0', 'content-type'=>'text/plain'}, [] ] end ENV['UNICORN_FD'] = srv.fileno.to_s opts = { diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb index 6cb0268..7f22b24 100644 --- a/test/unit/test_request.rb +++ b/test/unit/test_request.rb @@ -22,7 +22,7 @@ def kgio_addr def setup @request = HttpRequest.new @app = lambda do |env| - [ 200, { 'Content-Length' => '0', 'Content-Type' => 'text/plain' }, [] ] + [ 200, { 'content-length' => '0', 'content-type' => 'text/plain' }, [] ] end @lint = Rack::Lint.new(@app) end diff --git a/test/unit/test_server.rb b/test/unit/test_server.rb index bc9a222..98e85ab 100644 --- a/test/unit/test_server.rb +++ b/test/unit/test_server.rb @@ -16,7 +16,7 @@ class TestHandler def call(env) while env['rack.input'].read(4096) end - [200, { 'Content-Type' => 'text/plain' }, ['hello!\n']] + [200, { 'content-type' => 'text/plain' }, ['hello!\n']] rescue Unicorn::ClientShutdown, Unicorn::HttpParserError => e $stderr.syswrite("#{e.class}: #{e.message} #{e.backtrace.empty?}\n") raise e @@ -30,7 +30,7 @@ def call(env) env['rack.early_hints'].call( "Link" => "</style.css>; rel=preload; as=style\n</script.js>; rel=preload" ) - [200, { 'Content-Type' => 'text/plain' }, ['hello!\n']] + [200, { 'content-type' => 'text/plain' }, ['hello!\n']] end end @@ -45,7 +45,7 @@ def call(env) env["rack.after_reply"] << -> { @called = true } - [200, { 'Content-Type' => 'text/plain' }, ["after_reply_called: #{@called}"]] + [200, { 'content-type' => 'text/plain' }, ["after_reply_called: #{@called}"]] rescue Unicorn::ClientShutdown, Unicorn::HttpParserError => e $stderr.syswrite("#{e.class}: #{e.message} #{e.backtrace.empty?}\n") raise e @@ -81,7 +81,7 @@ def test_preload_app_config tmp.sysseek(0) tmp.truncate(0) tmp.syswrite($$) - lambda { |env| [ 200, { 'Content-Type' => 'text/plain' }, [ "#$$\n" ] ] } + lambda { |env| [ 200, { 'content-type' => 'text/plain' }, [ "#$$\n" ] ] } } redirect_test_io do @server = HttpServer.new(app, :listeners => [ "127.0.0.1:#@port"] ) diff --git a/test/unit/test_signals.rb b/test/unit/test_signals.rb index 56a7dfc..6c48754 100644 --- a/test/unit/test_signals.rb +++ b/test/unit/test_signals.rb @@ -47,7 +47,7 @@ def teardown def test_worker_dies_on_dead_master pid = fork { - app = lambda { |env| [ 200, {'X-Pid' => "#$$" }, [] ] } + app = lambda { |env| [ 200, {'x-pid' => "#$$" }, [] ] } opts = @server_opts.merge(:timeout => 3) redirect_test_io { HttpServer.new(app, opts).start.join } } @@ -56,7 +56,7 @@ def test_worker_dies_on_dead_master sock.syswrite("GET / HTTP/1.0\r\n\r\n") buf = sock.readpartial(4096) assert_nil sock.close - buf =~ /\bX-Pid: (\d+)\b/ or raise Exception + buf =~ /\bx-pid: (\d+)\b/ or raise Exception child = $1.to_i wait_master_ready("test_stderr.#{pid}.log") wait_workers_ready("test_stderr.#{pid}.log", 1) @@ -120,7 +120,7 @@ def test_timeout_slow_response def test_response_write app = lambda { |env| - [ 200, { 'Content-Type' => 'text/plain', 'X-Pid' => Process.pid.to_s }, + [ 200, { 'content-type' => 'text/plain', 'x-pid' => Process.pid.to_s }, Dd.new(@bs, @count) ] } redirect_test_io { @server = HttpServer.new(app, @server_opts).start } @@ -130,7 +130,7 @@ def test_response_write buf = '' header_len = pid = nil buf = sock.sysread(16384, buf) - pid = buf[/\r\nX-Pid: (\d+)\r\n/, 1].to_i + pid = buf[/\r\nx-pid: (\d+)\r\n/, 1].to_i header_len = buf[/\A(.+?\r\n\r\n)/m, 1].size assert pid > 0, "pid not positive: #{pid.inspect}" read = buf.size @@ -158,14 +158,14 @@ def test_request_read app = lambda { |env| while env['rack.input'].read(4096) end - [ 200, {'Content-Type'=>'text/plain', 'X-Pid'=>Process.pid.to_s}, [] ] + [ 200, {'content-type'=>'text/plain', 'x-pid'=>Process.pid.to_s}, [] ] } redirect_test_io { @server = HttpServer.new(app, @server_opts).start } wait_workers_ready("test_stderr.#{$$}.log", 1) sock = tcp_socket('127.0.0.1', @port) sock.syswrite("GET / HTTP/1.0\r\n\r\n") - pid = sock.sysread(4096)[/\r\nX-Pid: (\d+)\r\n/, 1].to_i + pid = sock.sysread(4096)[/\r\nx-pid: (\d+)\r\n/, 1].to_i assert_nil sock.close assert pid > 0, "pid not positive: #{pid.inspect}" @@ -182,7 +182,7 @@ def test_request_read redirect_test_io { @server.stop(true) } # can't check for == since pending signals get merged assert size_before < @tmp.stat.size - assert_equal pid, sock.sysread(4096)[/\r\nX-Pid: (\d+)\r\n/, 1].to_i + assert_equal pid, sock.sysread(4096)[/\r\nx-pid: (\d+)\r\n/, 1].to_i assert_nil sock.close end end -- 2.40.0
Eric Wong <e@80x24.org> wrote:
> FD=9 is the client socket, so it looks like you have a client
> that's opening a connection and not doing anything so recvfrom()
> fails and ppoll times out. You'd need to track down why you
> have a client opening a connection like that.
Btw, this wouldn't be a problem in production with nginx in
front of unicorn (which is the only recommended production
configuration).
nginx acts as a sponge to buffer the entire request (including
request body) since unicorn was never designed for slow clients,
malicious or not.