[PATCH 0/3] TCP_INFO check_client_connection followups

[PATCH 0/3] TCP_INFO check_client_connection followups

[Eric Wong]

This series goes on top of Simon's excellent work to get
a TCP_INFO-based check_client_connection working under Linux.

First off, add a test extracted from one of Simon's messages;
then revert the signature changes to existing methods to
avoid breaking tmm1/gctools, and finally add a more portable
fallback for Ruby 2.2+ users (tested on FreeBSD).

Further work will improve portability of raindrops for FreeBSD
(and maybe other *BSDs incidentally, too).  That will be sent to
raindrops-public@bogomips => https://bogomips.org/raindrops-public/

Eric Wong (3):
  new test for check_client_connection
  revert signature change to HttpServer#process_client
  support "struct tcp_info" on non-Linux and Ruby 2.2+

 lib/unicorn/http_request.rb  | 42 +++++++++++++++++++----
 lib/unicorn/http_server.rb   |  6 ++--
 lib/unicorn/oob_gc.rb        |  4 +--
 lib/unicorn/socket_helper.rb | 16 +++++++--
 test/unit/test_ccc.rb        | 81 ++++++++++++++++++++++++++++++++++++++++++++
 test/unit/test_request.rb    | 28 +++++++--------
 6 files changed, 150 insertions(+), 27 deletions(-)
 create mode 100644 test/unit/test_ccc.rb

Also pushed to the ccc-tcp-v3 branch:

The following changes since commit 73e1ce827faad59bfcaff0bc758c8255a5e4f747:

  t0011-active-unix-socket.sh: fix race condition in test (2017-03-01 00:24:04 +0000)

are available in the git repository at:

  git://bogomips.org/unicorn ccc-tcp-v3

for you to fetch changes up to 873218e317773462be2a72556d26dc4a723cc7be:

  support "struct tcp_info" on non-Linux and Ruby 2.2+ (2017-03-08 05:39:55 +0000)

----------------------------------------------------------------
Eric Wong (3):
      new test for check_client_connection
      revert signature change to HttpServer#process_client
      support "struct tcp_info" on non-Linux and Ruby 2.2+

Simon Eskildsen (1):
      check_client_connection: use tcp state on linux

 lib/unicorn/http_request.rb  | 73 ++++++++++++++++++++++++++++++++++++---
 lib/unicorn/socket_helper.rb | 16 +++++++--
 test/unit/test_ccc.rb        | 81 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 163 insertions(+), 7 deletions(-)
 create mode 100644 test/unit/test_ccc.rb

-- 
EW

[PATCH 1/3] new test for check_client_connection

[Eric Wong]

This was a bit tricky to test, but it's probably more reliable
now that we're relying on TCP_INFO.

Based on test by Simon Eskildsen <simon.eskildsen@shopify.com>:
  https://bogomips.org/unicorn-public/CAO3HKM49+aLD=KLij3zhJqkWnR7bCWVan0mOvxD85xfrW8RXOw@mail.gmail.com/
---
 test/unit/test_ccc.rb | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 81 insertions(+)
 create mode 100644 test/unit/test_ccc.rb

diff --git a/test/unit/test_ccc.rb b/test/unit/test_ccc.rb
new file mode 100644
index 0000000..22b1a9c
--- /dev/null
+++ b/test/unit/test_ccc.rb
@@ -0,0 +1,81 @@
+require 'socket'
+require 'unicorn'
+require 'io/wait'
+require 'tempfile'
+require 'test/unit'
+
+class TestCccTCPI < Test::Unit::TestCase
+  def test_ccc_tcpi
+    start_pid = $$
+    host = '127.0.0.1'
+    srv = TCPServer.new(host, 0)
+    port = srv.addr[1]
+    err = Tempfile.new('unicorn_ccc')
+    rd, wr = IO.pipe
+    pid = fork do
+      reqs = 0
+      rd.close
+      worker_pid = nil
+      app = lambda do |env|
+        worker_pid ||= begin
+          at_exit { wr.write(reqs.to_s) if worker_pid == $$ }
+          $$
+        end
+        reqs += 1
+        sleep(1) if env['PATH_INFO'] == '/sleep'
+        [ 200, [ %w(Content-Length 0),  %w(Content-Type text/plain) ], [] ]
+      end
+      ENV['UNICORN_FD'] = srv.fileno.to_s
+      opts = {
+        listeners: [ "#{host}:#{port}" ],
+        stderr_path: err.path,
+        check_client_connection: true,
+      }
+      uni = Unicorn::HttpServer.new(app, opts)
+      uni.start.join
+    end
+    wr.close
+
+    # make sure the server is running, at least
+    client = TCPSocket.new(host, port)
+    client.write("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
+    assert client.wait_readable(10), 'never got response from server'
+    res = client.read
+    assert_match %r{\AHTTP/1\.1 200}, res, 'got part of first response'
+    assert_match %r{\r\n\r\n\z}, res, 'got end of response, server is ready'
+    client.close
+
+    # start a slow request...
+    sleeper = TCPSocket.new(host, port)
+    sleeper.write("GET /sleep HTTP/1.1\r\nHost: example.com\r\n\r\n")
+
+    # and a bunch of aborted ones
+    nr = 100
+    nr.times do |i|
+      client = TCPSocket.new(host, port)
+      client.write("GET /collections/#{rand(10000)} HTTP/1.1\r\n" \
+                   "Host: example.com\r\n\r\n")
+      client.close
+    end
+    sleeper.close
+    kpid = pid
+    pid = nil
+    Process.kill(:QUIT, kpid)
+    _, status = Process.waitpid2(kpid)
+    assert status.success?
+    reqs = rd.read.to_i
+    warn "server got #{reqs} requests with #{nr} CCC aborted\n" if $DEBUG
+    assert_operator reqs, :<, nr
+    assert_operator reqs, :>=, 2, 'first 2 requests got through, at least'
+  ensure
+    return if start_pid != $$
+    srv.close if srv
+    if pid
+      Process.kill(:QUIT, pid)
+      _, status = Process.waitpid2(pid)
+      assert status.success?
+    end
+    err.close! if err
+    rd.close if rd
+  end
+end
-- 
EW

[PATCH 2/3] revert signature change to HttpServer#process_client

[Eric Wong]

We can force kgio_tryaccept to return an internal class
for TCP objects by subclassing Kgio::TCPServer.

This avoids breakage in any unfortunate projects which depend on
our undocumented internal APIs, such as gctools
<https://github.com/tmm1/gctools>

Cc: Aman Gupta <aman@tmm1.net>
---
 lib/unicorn/http_request.rb  | 10 +++++-----
 lib/unicorn/http_server.rb   |  6 +++---
 lib/unicorn/oob_gc.rb        |  4 ++--
 lib/unicorn/socket_helper.rb | 16 ++++++++++++++--
 test/unit/test_request.rb    | 28 ++++++++++++++--------------
 5 files changed, 38 insertions(+), 26 deletions(-)

diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 9acde50..68bde16 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -61,7 +61,7 @@ def self.check_client_connection=(bool)
   # returns an environment hash suitable for Rack if successful
   # This does minimal exception trapping and it is up to the caller
   # to handle any socket errors (e.g. user aborted upload).
-  def read(socket, listener)
+  def read(socket)
     clear
     e = env
 
@@ -82,7 +82,7 @@ def read(socket, listener)
       false until add_parse(socket.kgio_read!(16384))
     end
 
-    check_client_connection(socket, listener) if @@check_client_connection
+    check_client_connection(socket) if @@check_client_connection
 
     e['rack.input'] = 0 == content_length ?
                       NULL_IO : @@input_class.new(socket, self)
@@ -105,8 +105,8 @@ def hijacked?
   end
 
   if defined?(Raindrops::TCP_Info)
-    def check_client_connection(socket, listener) # :nodoc:
-      if Kgio::TCPServer === listener
+    def check_client_connection(socket) # :nodoc:
+      if Unicorn::TCPClient === socket
         @@tcp_info ||= Raindrops::TCP_Info.new(socket)
         @@tcp_info.get!(socket)
         raise Errno::EPIPE, "client closed connection".freeze,
@@ -127,7 +127,7 @@ def closed_state?(state) # :nodoc:
       end
     end
   else
-    def check_client_connection(socket, listener) # :nodoc:
+    def check_client_connection(socket) # :nodoc:
       write_http_header(socket)
     end
   end
diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 2aa1072..c2086cb 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -558,8 +558,8 @@ def e100_response_write(client, env)
 
   # once a client is accepted, it is processed in its entirety here
   # in 3 easy steps: read request, call app, write app response
-  def process_client(client, listener)
-    status, headers, body = @app.call(env = @request.read(client, listener))
+  def process_client(client)
+    status, headers, body = @app.call(env = @request.read(client))
 
     begin
       return if @request.hijacked?
@@ -655,7 +655,7 @@ def worker_loop(worker)
         # Unicorn::Worker#kgio_tryaccept is not like accept(2) at all,
         # but that will return false
         if client = sock.kgio_tryaccept
-          process_client(client, sock)
+          process_client(client)
           nr += 1
           worker.tick = time_now.to_i
         end
diff --git a/lib/unicorn/oob_gc.rb b/lib/unicorn/oob_gc.rb
index 74a1d51..5572e59 100644
--- a/lib/unicorn/oob_gc.rb
+++ b/lib/unicorn/oob_gc.rb
@@ -67,8 +67,8 @@ def self.new(app, interval = 5, path = %r{\A/})
 
   #:stopdoc:
   PATH_INFO = "PATH_INFO"
-  def process_client(client, listener)
-    super(client, listener) # Unicorn::HttpServer#process_client
+  def process_client(client)
+    super(client) # Unicorn::HttpServer#process_client
     if OOBGC_PATH =~ OOBGC_ENV[PATH_INFO] && ((@@nr -= 1) <= 0)
       @@nr = OOBGC_INTERVAL
       OOBGC_ENV.clear
diff --git a/lib/unicorn/socket_helper.rb b/lib/unicorn/socket_helper.rb
index df8315e..5371413 100644
--- a/lib/unicorn/socket_helper.rb
+++ b/lib/unicorn/socket_helper.rb
@@ -3,6 +3,18 @@
 require 'socket'
 
 module Unicorn
+
+  # Instead of using a generic Kgio::Socket for everything,
+  # tag TCP sockets so we can use TCP_INFO under Linux without
+  # incurring extra syscalls for Unix domain sockets.
+  # TODO: remove these when we remove kgio
+  TCPClient = Class.new(Kgio::Socket) # :nodoc:
+  class TCPSrv < Kgio::TCPServer # :nodoc:
+    def kgio_tryaccept # :nodoc:
+      super(TCPClient)
+    end
+  end
+
   module SocketHelper
 
     # internal interface
@@ -148,7 +160,7 @@ def new_tcp_server(addr, port, opt)
       end
       sock.bind(Socket.pack_sockaddr_in(port, addr))
       sock.autoclose = false
-      Kgio::TCPServer.for_fd(sock.fileno)
+      TCPSrv.for_fd(sock.fileno)
     end
 
     # returns rfc2732-style (e.g. "[::1]:666") addresses for IPv6
@@ -185,7 +197,7 @@ def sock_name(sock)
     def server_cast(sock)
       begin
         Socket.unpack_sockaddr_in(sock.getsockname)
-        Kgio::TCPServer.for_fd(sock.fileno)
+        TCPSrv.for_fd(sock.fileno)
       rescue ArgumentError
         Kgio::UNIXServer.for_fd(sock.fileno)
       end
diff --git a/test/unit/test_request.rb b/test/unit/test_request.rb
index dbe8af7..f0ccaf7 100644
--- a/test/unit/test_request.rb
+++ b/test/unit/test_request.rb
@@ -30,7 +30,7 @@ def setup
   def test_options
     client = MockRequest.new("OPTIONS * HTTP/1.1\r\n" \
                              "Host: foo\r\n\r\n")
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert_equal '', env['REQUEST_PATH']
     assert_equal '', env['PATH_INFO']
     assert_equal '*', env['REQUEST_URI']
@@ -40,7 +40,7 @@ def test_options
   def test_absolute_uri_with_query
     client = MockRequest.new("GET http://e:3/x?y=z HTTP/1.1\r\n" \
                              "Host: foo\r\n\r\n")
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert_equal '/x', env['REQUEST_PATH']
     assert_equal '/x', env['PATH_INFO']
     assert_equal 'y=z', env['QUERY_STRING']
@@ -50,7 +50,7 @@ def test_absolute_uri_with_query
   def test_absolute_uri_with_fragment
     client = MockRequest.new("GET http://e:3/x#frag HTTP/1.1\r\n" \
                              "Host: foo\r\n\r\n")
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert_equal '/x', env['REQUEST_PATH']
     assert_equal '/x', env['PATH_INFO']
     assert_equal '', env['QUERY_STRING']
@@ -61,7 +61,7 @@ def test_absolute_uri_with_fragment
   def test_absolute_uri_with_query_and_fragment
     client = MockRequest.new("GET http://e:3/x?a=b#frag HTTP/1.1\r\n" \
                              "Host: foo\r\n\r\n")
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert_equal '/x', env['REQUEST_PATH']
     assert_equal '/x', env['PATH_INFO']
     assert_equal 'a=b', env['QUERY_STRING']
@@ -73,7 +73,7 @@ def test_absolute_uri_unsupported_schemes
     %w(ssh+http://e/ ftp://e/x http+ssh://e/x).each do |abs_uri|
       client = MockRequest.new("GET #{abs_uri} HTTP/1.1\r\n" \
                                "Host: foo\r\n\r\n")
-      assert_raises(HttpParserError) { @request.read(client, nil) }
+      assert_raises(HttpParserError) { @request.read(client) }
     end
   end
 
@@ -81,7 +81,7 @@ def test_x_forwarded_proto_https
     client = MockRequest.new("GET / HTTP/1.1\r\n" \
                              "X-Forwarded-Proto: https\r\n" \
                              "Host: foo\r\n\r\n")
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert_equal "https", env['rack.url_scheme']
     res = @lint.call(env)
   end
@@ -90,7 +90,7 @@ def test_x_forwarded_proto_http
     client = MockRequest.new("GET / HTTP/1.1\r\n" \
                              "X-Forwarded-Proto: http\r\n" \
                              "Host: foo\r\n\r\n")
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert_equal "http", env['rack.url_scheme']
     res = @lint.call(env)
   end
@@ -99,14 +99,14 @@ def test_x_forwarded_proto_invalid
     client = MockRequest.new("GET / HTTP/1.1\r\n" \
                              "X-Forwarded-Proto: ftp\r\n" \
                              "Host: foo\r\n\r\n")
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert_equal "http", env['rack.url_scheme']
     res = @lint.call(env)
   end
 
   def test_rack_lint_get
     client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert_equal "http", env['rack.url_scheme']
     assert_equal '127.0.0.1', env['REMOTE_ADDR']
     res = @lint.call(env)
@@ -114,7 +114,7 @@ def test_rack_lint_get
 
   def test_no_content_stringio
     client = MockRequest.new("GET / HTTP/1.1\r\nHost: foo\r\n\r\n")
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert_equal StringIO, env['rack.input'].class
   end
 
@@ -122,7 +122,7 @@ def test_zero_content_stringio
     client = MockRequest.new("PUT / HTTP/1.1\r\n" \
                              "Content-Length: 0\r\n" \
                              "Host: foo\r\n\r\n")
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert_equal StringIO, env['rack.input'].class
   end
 
@@ -130,7 +130,7 @@ def test_real_content_not_stringio
     client = MockRequest.new("PUT / HTTP/1.1\r\n" \
                              "Content-Length: 1\r\n" \
                              "Host: foo\r\n\r\n")
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert_equal Unicorn::TeeInput, env['rack.input'].class
   end
 
@@ -141,7 +141,7 @@ def test_rack_lint_put
       "Content-Length: 5\r\n" \
       "\r\n" \
       "abcde")
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert ! env.include?(:http_body)
     res = @lint.call(env)
   end
@@ -167,7 +167,7 @@ def client.kgio_read!(*args)
       "\r\n")
     count.times { assert_equal bs, client.syswrite(buf) }
     assert_equal 0, client.sysseek(0)
-    env = @request.read(client, nil)
+    env = @request.read(client)
     assert ! env.include?(:http_body)
     assert_equal length, env['rack.input'].size
     count.times {
-- 
EW

[PATCH 3/3] support "struct tcp_info" on non-Linux and Ruby 2.2+

[Eric Wong]

Ruby 2.2+ can show "struct tcp_info" as a string via
Socket::Option#inspect, and we can attempt to parse it
out to extract the information we need.

Parsing this string is inefficient, but does not depend on the
ordering of the tcp_info struct.
---
 lib/unicorn/http_request.rb | 32 +++++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)

diff --git a/lib/unicorn/http_request.rb b/lib/unicorn/http_request.rb
index 68bde16..c08097c 100644
--- a/lib/unicorn/http_request.rb
+++ b/lib/unicorn/http_request.rb
@@ -29,6 +29,7 @@ class Unicorn::HttpParser
   EMPTY_ARRAY = [].freeze
   @@input_class = Unicorn::TeeInput
   @@check_client_connection = false
+  @@tcpi_inspect_ok = true
 
   def self.input_class
     @@input_class
@@ -127,8 +128,37 @@ def closed_state?(state) # :nodoc:
       end
     end
   else
+
+    # Ruby 2.2+ can show struct tcp_info as a string Socket::Option#inspect.
+    # Not that efficient, but probably still better than doing unnecessary
+    # work after a client gives up.
     def check_client_connection(socket) # :nodoc:
-      write_http_header(socket)
+      if Unicorn::TCPClient === socket && @@tcpi_inspect_ok
+        opt = socket.getsockopt(:IPPROTO_TCP, :TCP_INFO).inspect
+        if opt =~ /\bstate=(\S+)/
+          @@tcpi_inspect_ok = true
+          raise Errno::EPIPE, "client closed connection".freeze,
+                EMPTY_ARRAY if closed_state_str?($1)
+        else
+          @@tcpi_inspect_ok = false
+          write_http_header(socket)
+        end
+        opt.clear
+      else
+        write_http_header(socket)
+      end
+    end
+
+    def closed_state_str?(state)
+      case state
+      when 'ESTABLISHED'
+        false
+      # not a typo, ruby maps TCP_CLOSE (no 'D') to state=CLOSED (w/ 'D')
+      when 'CLOSE_WAIT', 'TIME_WAIT', 'CLOSED', 'LAST_ACK', 'CLOSING'
+        true
+      else
+        false
+      end
     end
   end
 
-- 
EW

Re: [PATCH 0/3] TCP_INFO check_client_connection followups

[Simon Eskildsen]

This looks great Eric. Thanks for taking this to the finish line!

On Wed, Mar 8, 2017 at 1:02 AM, Eric Wong <e@80x24.org> wrote:
> This series goes on top of Simon's excellent work to get
> a TCP_INFO-based check_client_connection working under Linux.
>
> First off, add a test extracted from one of Simon's messages;
> then revert the signature changes to existing methods to
> avoid breaking tmm1/gctools, and finally add a more portable
> fallback for Ruby 2.2+ users (tested on FreeBSD).
>
> Further work will improve portability of raindrops for FreeBSD
> (and maybe other *BSDs incidentally, too).  That will be sent to
> raindrops-public@bogomips => https://bogomips.org/raindrops-public/
>
> Eric Wong (3):
>   new test for check_client_connection
>   revert signature change to HttpServer#process_client
>   support "struct tcp_info" on non-Linux and Ruby 2.2+
>
>  lib/unicorn/http_request.rb  | 42 +++++++++++++++++++----
>  lib/unicorn/http_server.rb   |  6 ++--
>  lib/unicorn/oob_gc.rb        |  4 +--
>  lib/unicorn/socket_helper.rb | 16 +++++++--
>  test/unit/test_ccc.rb        | 81 ++++++++++++++++++++++++++++++++++++++++++++
>  test/unit/test_request.rb    | 28 +++++++--------
>  6 files changed, 150 insertions(+), 27 deletions(-)
>  create mode 100644 test/unit/test_ccc.rb
>
> Also pushed to the ccc-tcp-v3 branch:
>
> The following changes since commit 73e1ce827faad59bfcaff0bc758c8255a5e4f747:
>
>   t0011-active-unix-socket.sh: fix race condition in test (2017-03-01 00:24:04 +0000)
>
> are available in the git repository at:
>
>   git://bogomips.org/unicorn ccc-tcp-v3
>
> for you to fetch changes up to 873218e317773462be2a72556d26dc4a723cc7be:
>
>   support "struct tcp_info" on non-Linux and Ruby 2.2+ (2017-03-08 05:39:55 +0000)
>
> ----------------------------------------------------------------
> Eric Wong (3):
>       new test for check_client_connection
>       revert signature change to HttpServer#process_client
>       support "struct tcp_info" on non-Linux and Ruby 2.2+
>
> Simon Eskildsen (1):
>       check_client_connection: use tcp state on linux
>
>  lib/unicorn/http_request.rb  | 73 ++++++++++++++++++++++++++++++++++++---
>  lib/unicorn/socket_helper.rb | 16 +++++++--
>  test/unit/test_ccc.rb        | 81 ++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 163 insertions(+), 7 deletions(-)
>  create mode 100644 test/unit/test_ccc.rb
>
> --
> EW