From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS33070 50.56.128.0/17 X-Spam-Status: No, score=1.0 required=3.0 tests=AWL,MSGID_FROM_MTA_HEADER, RDNS_NONE shortcircuit=no autolearn=no version=3.3.2 Path: news.gmane.org!not-for-mail From: Lin Jen-Shin Newsgroups: gmane.comp.lang.ruby.rainbows.general Subject: [PATCH] Add support for EventMachineThread* Date: Tue, 17 Sep 2013 20:31:44 +0800 Message-ID: <1379421104-75563-1-git-send-email-godfat@godfat.org> NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1379421131 21827 80.91.229.3 (17 Sep 2013 12:32:11 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 17 Sep 2013 12:32:11 +0000 (UTC) Cc: Lin Jen-Shin To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Original-X-From: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Tue Sep 17 14:32:09 2013 Return-path: Envelope-to: gclrrg-rainbows-talk@m.gmane.org X-Original-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Delivered-To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=godfat.org; s=google; h=from:to:cc:subject:date:message-id; bh=0SiDyX5oySzED/ULpYqKmk2BVKzBMz64PaO4oO5t4uE=; b=QZUd769NpmilY7luzU7JXu32rb6l4B4ViCFQND875A4qISTSUYM8LcU2pyGbrQ8ikv 6QAmeDGXUdi5uhOThHTm2KdtAfRXj7R6aZBQoh/PBDdq3nJpEuHtxnrrEvDow39JW4Av 7rXpNRXNqhTOmZI/SJIeLONYPCKQ/4yeMGza8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=0SiDyX5oySzED/ULpYqKmk2BVKzBMz64PaO4oO5t4uE=; b=LtzEUbV+LWY0mtBwnZsv+GEP1SHSkKOBss9VZeNb81jUeTy8xsGCT6P9/o4fiyNaQ5 J8A62Uz03dGVZf6Rqp+pLdu4TkMYKrs6kz1Xzv7ZT/PzafpAcVPzjDqWuQLIepmAyJmb OwHyywzO9HeV0oHiveh5BVVoBmpYnnystN9r6Ewku0PEf+m8WjUrylKMyqQ9SbSJd+N+ vDgOfD8LldLLx0vV7nF1cLTNGb1sa5plGs+4xZ/9lUCM4+/SFZCS4Sz4viEnQyrygMZc +eVblV2yZm6jDv6KiW5Q4V/CNH8cEiR/zlD2zwgxBGwHJUQ5hbAPNbCxtF4vJ9Dhh2h2 jSBA== X-Gm-Message-State: ALoCoQlpLYb96d5QmIvVZz3GtqgM+woBMshpFaC8wpsdJbCU5UfLGtKxz0PsLMF8AfBkhfqnBO4e X-Received: by 10.68.2.166 with SMTP id 6mr7439986pbv.165.1379421117341; Tue, 17 Sep 2013 05:31:57 -0700 (PDT) X-Mailer: git-send-email 1.8.4 X-BeenThere: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org Errors-To: rainbows-talk-bounces-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org X-Broken-Reverse-DNS: no host name found for IP address 50.56.192.79 Xref: news.gmane.org gmane.comp.lang.ruby.rainbows.general:544 Archived-At: Received: from [50.56.192.79] (helo=rubyforge.org) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1VLuRq-0004MF-Kj for gclrrg-rainbows-talk@m.gmane.org; Tue, 17 Sep 2013 14:32:07 +0200 Received: from localhost.localdomain (localhost [127.0.0.1]) by rubyforge.org (Postfix) with ESMTP id 6663F2E1BD; Tue, 17 Sep 2013 12:32:05 +0000 (UTC) Received: from mail-pb0-f42.google.com (mail-pb0-f42.google.com [209.85.160.42]) by rubyforge.org (Postfix) with ESMTP id EBC2E2E188 for ; Tue, 17 Sep 2013 12:31:58 +0000 (UTC) Received: by mail-pb0-f42.google.com with SMTP id un15so5491950pbc.15 for ; Tue, 17 Sep 2013 05:31:57 -0700 (PDT) Received: from 220-135-38-235.hinet-ip.hinet.net (220-135-38-235.HINET-IP.hinet.net. [220.135.38.235]) by mx.google.com with ESMTPSA id h2sm8960832pbj.38.1969.12.31.16.00.00 (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 17 Sep 2013 05:31:56 -0700 (PDT) However, only EventMachineThreadDefer could pass all the tests *usually*. EventMachineThreadSpawn sometimes would fail, and EventMachineThreadPool would fail most of the times... I must have done something wrong with thread safety, but I can't tell from the code. Any comments would be much appreciated, Thanks! --- lib/rainbows.rb | 3 ++ lib/rainbows/event_machine.rb | 1 + lib/rainbows/event_machine/thread_client.rb | 42 +++++++++++++++++++++++ lib/rainbows/event_machine_thread_defer.rb | 13 +++++++ lib/rainbows/event_machine_thread_defer/client.rb | 9 +++++ lib/rainbows/event_machine_thread_pool.rb | 28 +++++++++++++++ lib/rainbows/event_machine_thread_pool/client.rb | 10 ++++++ lib/rainbows/event_machine_thread_spawn.rb | 6 ++++ lib/rainbows/event_machine_thread_spawn/client.rb | 12 +++++++ t/GNUmakefile | 13 +++---- t/simple-http_EventMachineThreadDefer.ru | 10 ++++++ t/simple-http_EventMachineThreadPool.ru | 10 ++++++ t/simple-http_EventMachineThreadSpawn.ru | 10 ++++++ t/t0023-sendfile-byte-range.sh | 4 ++- t/t0041-optional-pool-size.sh | 1 + t/t0044-autopush.sh | 4 ++- t/t0045-client_max_header_size.sh | 2 +- t/t0106-rack-input-keepalive.sh | 2 ++ t/t0113-rewindable-input-false.sh | 4 ++- t/t0114-rewindable-input-true.sh | 4 ++- t/test_isolate.rb | 2 +- 21 files changed, 176 insertions(+), 14 deletions(-) create mode 100644 lib/rainbows/event_machine/thread_client.rb create mode 100644 lib/rainbows/event_machine_thread_defer.rb create mode 100644 lib/rainbows/event_machine_thread_defer/client.rb create mode 100644 lib/rainbows/event_machine_thread_pool.rb create mode 100644 lib/rainbows/event_machine_thread_pool/client.rb create mode 100644 lib/rainbows/event_machine_thread_spawn.rb create mode 100644 lib/rainbows/event_machine_thread_spawn/client.rb create mode 100644 t/simple-http_EventMachineThreadDefer.ru create mode 100644 t/simple-http_EventMachineThreadPool.ru create mode 100644 t/simple-http_EventMachineThreadSpawn.ru diff --git a/lib/rainbows.rb b/lib/rainbows.rb index bfa1ba0..93c09fe 100644 --- a/lib/rainbows.rb +++ b/lib/rainbows.rb @@ -111,6 +111,9 @@ module Rainbows autoload :Epoll, "rainbows/epoll" autoload :XEpoll, "rainbows/xepoll" autoload :EventMachine, "rainbows/event_machine" + autoload :EventMachineThreadDefer, "rainbows/event_machine_thread_defer" + autoload :EventMachineThreadPool, "rainbows/event_machine_thread_pool" + autoload :EventMachineThreadSpawn, "rainbows/event_machine_thread_spawn" autoload :FiberSpawn, "rainbows/fiber_spawn" autoload :FiberPool, "rainbows/fiber_pool" autoload :ActorSpawn, "rainbows/actor_spawn" diff --git a/lib/rainbows/event_machine.rb b/lib/rainbows/event_machine.rb index b143b39..91a4883 100644 --- a/lib/rainbows/event_machine.rb +++ b/lib/rainbows/event_machine.rb @@ -47,6 +47,7 @@ module Rainbows::EventMachine autoload :ResponseChunkPipe, 'rainbows/event_machine/response_chunk_pipe' autoload :TryDefer, 'rainbows/event_machine/try_defer' autoload :Client, 'rainbows/event_machine/client' + autoload :ThreadClient, 'rainbows/event_machine/thread_client' include Rainbows::Base diff --git a/lib/rainbows/event_machine/thread_client.rb b/lib/rainbows/event_machine/thread_client.rb new file mode 100644 index 0000000..7ab2ef9 --- /dev/null +++ b/lib/rainbows/event_machine/thread_client.rb @@ -0,0 +1,42 @@ +# -*- encoding: binary -*- +# :enddoc: +class Rainbows::EventMachine::ThreadClient < + Rainbows::EventMachine::Client + + def app_call input + @deferred = true # we defer immediately + set_comm_inactivity_timeout 0 + @env[RACK_INPUT] = input + @env[REMOTE_ADDR] = @_io.kgio_addr + @env[ASYNC_CALLBACK] = method(:write_async_response) + @env[ASYNC_CLOSE] = EM::DefaultDeferrable.new + @hp.hijack_setup(@env, @_io) + app_dispatch # must be implemented by subclass + end + + # this is only called in the master thread + def response_write(response) + if @hp.hijacked? + @deferred = nil + hijacked + elsif nil == response[0] || -1 == response[0] + @deferred = true + else + @deferred = nil + ev_write_response(*response, @hp.next?) + end + rescue => e + @deferred = nil + handle_error(e) + end + + # fails-safe application dispatch, we absolutely cannot + # afford to fail or raise an exception (killing the thread) + # here because that could cause a deadlock and we'd leak FDs + def app_response + APP.call(@env.merge!(RACK_DEFAULTS)) + rescue => e + Rainbows::Error.app(e) # we guarantee this does not raise + [ 500, {}, [] ] + end +end diff --git a/lib/rainbows/event_machine_thread_defer.rb b/lib/rainbows/event_machine_thread_defer.rb new file mode 100644 index 0000000..e0b0cd1 --- /dev/null +++ b/lib/rainbows/event_machine_thread_defer.rb @@ -0,0 +1,13 @@ +# -*- encoding: binary -*- + +module Rainbows::EventMachineThreadDefer + autoload :Client, 'rainbows/event_machine_thread_defer/client' + extend Rainbows::PoolSize + include Rainbows::EventMachine + + def init_worker_process(worker) + EM.threadpool_size = Rainbows::O[:pool_size] + logger.info "EventMachineThreadDefer pool_size=#{Rainbows::O[:pool_size]}" + super + end +end diff --git a/lib/rainbows/event_machine_thread_defer/client.rb b/lib/rainbows/event_machine_thread_defer/client.rb new file mode 100644 index 0000000..b6b1e05 --- /dev/null +++ b/lib/rainbows/event_machine_thread_defer/client.rb @@ -0,0 +1,9 @@ +# -*- encoding: binary -*- +# :enddoc: +class Rainbows::EventMachineThreadDefer::Client < + Rainbows::EventMachine::ThreadClient + + def app_dispatch + EM.defer(method(:app_response), method(:response_write)) + end +end diff --git a/lib/rainbows/event_machine_thread_pool.rb b/lib/rainbows/event_machine_thread_pool.rb new file mode 100644 index 0000000..9bc7b50 --- /dev/null +++ b/lib/rainbows/event_machine_thread_pool.rb @@ -0,0 +1,28 @@ +# -*- encoding: binary -*- + +module Rainbows::EventMachineThreadPool + autoload :Client, 'rainbows/event_machine_thread_pool/client' + extend Rainbows::PoolSize + include Rainbows::EventMachine + + def init_worker_threads(queue) # :nodoc: + Rainbows::O[:pool_size].times.map do + Thread.new do + begin + client = queue.pop + response = client.app_response + EM.next_tick { client.response_write(response) } + rescue => e + Rainbows::Error.listen_loop(e) + end while Rainbows.alive + end + end + end + + def init_worker_process(worker) + queue = Client.const_set(:QUEUE, Queue.new) + threads = init_worker_threads(queue) + logger.info "EventMachineThreadPool pool_size=#{Rainbows::O[:pool_size]}" + super + end +end diff --git a/lib/rainbows/event_machine_thread_pool/client.rb b/lib/rainbows/event_machine_thread_pool/client.rb new file mode 100644 index 0000000..c55d4a0 --- /dev/null +++ b/lib/rainbows/event_machine_thread_pool/client.rb @@ -0,0 +1,10 @@ +# -*- encoding: binary -*- +# :enddoc: +class Rainbows::EventMachineThreadPool::Client < + Rainbows::EventMachine::ThreadClient + + # QUEUE constant will be set in worker_loop + def app_dispatch + QUEUE << self + end +end diff --git a/lib/rainbows/event_machine_thread_spawn.rb b/lib/rainbows/event_machine_thread_spawn.rb new file mode 100644 index 0000000..19129cc --- /dev/null +++ b/lib/rainbows/event_machine_thread_spawn.rb @@ -0,0 +1,6 @@ +# -*- encoding: binary -*- + +module Rainbows::EventMachineThreadSpawn + autoload :Client, 'rainbows/event_machine_thread_spawn/client' + include Rainbows::EventMachine +end diff --git a/lib/rainbows/event_machine_thread_spawn/client.rb b/lib/rainbows/event_machine_thread_spawn/client.rb new file mode 100644 index 0000000..23032ba --- /dev/null +++ b/lib/rainbows/event_machine_thread_spawn/client.rb @@ -0,0 +1,12 @@ +# -*- encoding: binary -*- +# :enddoc: +class Rainbows::EventMachineThreadSpawn::Client < + Rainbows::EventMachine::ThreadClient + + def app_dispatch + Thread.new do + response = app_response + EM.next_tick { response_write(response) } + end + end +end diff --git a/t/GNUmakefile b/t/GNUmakefile index 19aacbe..5979371 100644 --- a/t/GNUmakefile +++ b/t/GNUmakefile @@ -45,14 +45,11 @@ ifeq ($(RUBY_ENGINE),ruby) models += CoolioThreadPool models += CoolioThreadSpawn models += CoolioFiberSpawn - - # EventMachine 1.0.0 currently does not build on Ruby 2.0.0 - # NeverBlock depends on 2.0.0 - RBTWO := $(shell case $(RUBY_VERSION) in 2.0.*$(rp) echo true;;esac) - ifeq ($(RBTWO),) - models += EventMachine - models += NeverBlock - endif + models += EventMachine + models += EventMachineThreadDefer + models += EventMachineThreadPool + models += EventMachineThreadSpawn + models += NeverBlock endif endif diff --git a/t/simple-http_EventMachineThreadDefer.ru b/t/simple-http_EventMachineThreadDefer.ru new file mode 100644 index 0000000..4a9effa --- /dev/null +++ b/t/simple-http_EventMachineThreadDefer.ru @@ -0,0 +1,10 @@ +use Rack::ContentLength +use Rack::ContentType +run lambda { |env| + if env['rack.multithread'] == true && + env['rainbows.model'] == :EventMachineThreadDefer + [ 200, {}, [ env.inspect << "\n" ] ] + else + raise "rack.multithread is false" + end +} diff --git a/t/simple-http_EventMachineThreadPool.ru b/t/simple-http_EventMachineThreadPool.ru new file mode 100644 index 0000000..af4c7a0 --- /dev/null +++ b/t/simple-http_EventMachineThreadPool.ru @@ -0,0 +1,10 @@ +use Rack::ContentLength +use Rack::ContentType +run lambda { |env| + if env['rack.multithread'] == true && + env['rainbows.model'] == :EventMachineThreadPool + [ 200, {}, [ env.inspect << "\n" ] ] + else + raise "rack.multithread is false" + end +} diff --git a/t/simple-http_EventMachineThreadSpawn.ru b/t/simple-http_EventMachineThreadSpawn.ru new file mode 100644 index 0000000..7615093 --- /dev/null +++ b/t/simple-http_EventMachineThreadSpawn.ru @@ -0,0 +1,10 @@ +use Rack::ContentLength +use Rack::ContentType +run lambda { |env| + if env['rack.multithread'] == true && + env['rainbows.model'] == :EventMachineThreadSpawn + [ 200, {}, [ env.inspect << "\n" ] ] + else + raise "rack.multithread is false" + end +} diff --git a/t/t0023-sendfile-byte-range.sh b/t/t0023-sendfile-byte-range.sh index a5b6ab2..f91fd0e 100755 --- a/t/t0023-sendfile-byte-range.sh +++ b/t/t0023-sendfile-byte-range.sh @@ -10,7 +10,9 @@ ruby) ;; ;; esac -skip_models EventMachine NeverBlock +skip_models EventMachine EventMachineThreadDefer EventMachineThreadPool +skip_models EventMachineThreadSpawn +skip_models NeverBlock t_plan 13 "sendfile byte range response for $model" diff --git a/t/t0041-optional-pool-size.sh b/t/t0041-optional-pool-size.sh index f5f6400..b468d9d 100755 --- a/t/t0041-optional-pool-size.sh +++ b/t/t0041-optional-pool-size.sh @@ -2,6 +2,7 @@ . ./test-lib.sh case $model in +EventMachineThreadDefer|EventMachineThreadPool|\ NeverBlock|CoolioThreadPool|XEpollThreadPool) ;; *) t_info "skipping $model.$T since it doesn't support :pool_size" diff --git a/t/t0044-autopush.sh b/t/t0044-autopush.sh index 103f9fc..e5efb76 100644 --- a/t/t0044-autopush.sh +++ b/t/t0044-autopush.sh @@ -14,7 +14,9 @@ fi # these buffer internally in external libraries, so we can't detect when # to use TCP_CORK -skip_models EventMachine NeverBlock +skip_models EventMachine EventMachineThreadDefer EventMachineThreadPool +skip_models EventMachineThreadSpawn +skip_models NeverBlock skip_models StreamResponseEpoll skip_models Coolio CoolioThreadPool CoolioThreadSpawn skip_models Revactor Rev RevThreadPool RevThreadSpawn diff --git a/t/t0045-client_max_header_size.sh b/t/t0045-client_max_header_size.sh index cd8f1fe..320fe0c 100755 --- a/t/t0045-client_max_header_size.sh +++ b/t/t0045-client_max_header_size.sh @@ -60,7 +60,7 @@ t_begin "smallest HTTP/0.9 request works right" && { t_begin "HTTP/1.1 request fails" && { curl -vsSf http://$listen/ > $tmp 2>&1 && die "unexpected curl success" - grep '400$' $tmp + grep '400\( Bad Request\)\?$' $tmp } t_begin "increase client_max_header_size on reload" && { diff --git a/t/t0106-rack-input-keepalive.sh b/t/t0106-rack-input-keepalive.sh index 3862e16..b5f4f37 100755 --- a/t/t0106-rack-input-keepalive.sh +++ b/t/t0106-rack-input-keepalive.sh @@ -1,6 +1,8 @@ #!/bin/sh . ./test-lib.sh skip_models StreamResponseEpoll +skip_models EventMachineThreadDefer EventMachineThreadPool +skip_models EventMachineThreadSpawn t_plan 11 "rack.input pipelining test" t_begin "setup and startup" && { diff --git a/t/t0113-rewindable-input-false.sh b/t/t0113-rewindable-input-false.sh index 6eb2fda..0bba8a5 100755 --- a/t/t0113-rewindable-input-false.sh +++ b/t/t0113-rewindable-input-false.sh @@ -1,6 +1,8 @@ #!/bin/sh . ./test-lib.sh -skip_models EventMachine NeverBlock +skip_models EventMachine EventMachineThreadDefer EventMachineThreadPool +skip_models EventMachineThreadSpawn +skip_models NeverBlock skip_models Rev RevThreadSpawn RevThreadPool skip_models Coolio CoolioThreadSpawn CoolioThreadPool skip_models Epoll XEpoll diff --git a/t/t0114-rewindable-input-true.sh b/t/t0114-rewindable-input-true.sh index 9d256dc..742258d 100755 --- a/t/t0114-rewindable-input-true.sh +++ b/t/t0114-rewindable-input-true.sh @@ -1,6 +1,8 @@ #!/bin/sh . ./test-lib.sh -skip_models EventMachine NeverBlock +skip_models EventMachine EventMachineThreadDefer EventMachineThreadPool +skip_models EventMachineThreadSpawn +skip_models NeverBlock skip_models Rev RevThreadSpawn RevThreadPool skip_models Coolio CoolioThreadSpawn CoolioThreadPool skip_models Epoll XEpoll diff --git a/t/test_isolate.rb b/t/test_isolate.rb index f5f97b1..e86419a 100644 --- a/t/test_isolate.rb +++ b/t/test_isolate.rb @@ -26,7 +26,7 @@ Isolate.now!(opts) do gem 'sendfile', '1.1.0' gem 'cool.io', '1.1.0' - gem 'eventmachine', '1.0.0' + gem 'eventmachine', '1.0.3' gem 'sinatra', '1.3.3' gem 'async_sinatra', '1.0.0' -- 1.8.4 _______________________________________________ Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org http://rubyforge.org/mailman/listinfo/rainbows-talk Do not quote signatures (like this one) or top post when replying