From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS15169 209.85.128.0/17 X-Spam-Status: No, score=-1.0 required=3.0 tests=AWL,BAYES_00,URIBL_BLOCKED shortcircuit=no autolearn=ham version=3.3.2 X-Original-To: unicorn-public@bogomips.org Received: from mail-la0-f51.google.com (mail-la0-f51.google.com [209.85.215.51]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 31D711F420 for ; Tue, 22 Jul 2014 21:56:08 +0000 (UTC) Received: by mail-la0-f51.google.com with SMTP id el20so233762lab.38 for ; Tue, 22 Jul 2014 14:56:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=y7icOzXj+XBo4dn6cgKL7NXvZG0sn2FTMKpdBCXShH0=; b=ZkzBHQTUAT4eHJgtxP8g0NqTEiBezRMwTjbRBKk8IeNfPQ0YvYBE24uQAzzyKLs8Oh A996QLoOVu0S2/lJfZWNrk9m5AsD4m2fXHoBkXjDh9rYwvsn/ryKNgdEoLpo/6xJmbuW MxhRMVBuCs4HWV963oie6YCNQBFAnA3mXxvSjXOzfvg1w1dSJ0qaWT7MQ4IL6TIUvNzA OzTTNACIX6Ud2ryq8vGfj8YVXODYgWaasYOqvDUuL6ETxGHVYCp5cQquaugMx/kJ88b0 THaUcYwuNRoWJtjEmcGal1QWiGJIXRVKFK8Y1IBppwy2koiym4f05QYaYKlClxrMw83L qA+A== X-Gm-Message-State: ALoCoQkPloJWWZzQlvHSTP+EkqLBrSNYqPx16/xhP02l4hkxC3vY+MyJCDY32deOn1Pud27X5XXq MIME-Version: 1.0 X-Received: by 10.112.139.167 with SMTP id qz7mr36316691lbb.22.1406066165748; Tue, 22 Jul 2014 14:56:05 -0700 (PDT) Received: by 10.152.180.172 with HTTP; Tue, 22 Jul 2014 14:56:05 -0700 (PDT) In-Reply-To: <20140722205631.GA9106@dcvr.yhbt.net> References: <20140722205631.GA9106@dcvr.yhbt.net> Date: Tue, 22 Jul 2014 14:56:05 -0700 Message-ID: Subject: Re: High number of TCP retransmits on Solaris From: Paul Henry To: Eric Wong Cc: unicorn-public Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: PublicInbox::Filter 0.0.1 List-Id: Thanks so much! It turned out to be a separate issue entirely, one that appears only in Solaris. When you have a high rate of connections, sockets sit around in the TIME_WAIT state for a default of 60s. Basically, there's a bug in Solaris that causes a new connection to be routed (based on what chooses the ephemeral port) to an existing TIME_WAIT connection. When the new client sends a SYN, the server responds with an ACK that has an incorrect sequence number (intended for the old client). This causes the client to retransmit the SYN and then the server responds correctly. You can read more about this issue in detail here: http://lists.illumos.org/pipermail/developer/2011-April/001958.html Cheers, Paul On Tue, Jul 22, 2014 at 1:56 PM, Eric Wong wrote: > Paul Henry wrote: > > Hello! > > > > When using unicorn unicorn v4.8.2, we're seeing a high number of TCP > > retransmits at a high connection rate. > > > > Smartos version: 20131218T184706Z > > > > Rackup file used: > > > > > cat server.ru > > run -> (env) { [200, {"Content-Type" => "text/plain"}, > > [Time.now.iso8601(6)]] } > > > > To start unicorn: > > > bundle exec unicorn -l 9292 server.ru > > > > To benchmark unicorn: > > >> Benchmark.measure { 1000.times { time=Benchmark.measure { > open("http://:9292/api/v1/settings/time", > > "Host" => "api.wanelo.com")}.real; p (time*1000).round(2) if time > > 0.05 } } > > Thanks for including a small example to reproduce the problem on your > end. Is that open call is from "open-uri" in the stdlib? > (does it attempt persistent connections?) > > > After the total number of connections on the system goes above 8,000 > > (16,000 is the average number of connections), we start seeing delays of > > around 1.1 seconds. > > I was not able to reproduce the issue with two Linux machines over > my (pretty bad) 100Mbps LAN. > > I used the script at the bottom to create a lot of client connections > from my client machine to the machine running unicorn (but this is not > connecting to unicorn, but a scalable server (e.g. nginx) with infinite > persistence). > > > We don't see the issue over the local loopback interface, only over the > > net. When using Webrick, we also don't see this issue. > > What's strange is the issue does not manifest under Webrick for you. > Which version of Ruby is that Webrick from? > > strace-ing "rackup -s webrick server.ru" reveals several differences: > > 1) webrick uses a listen backlog of only 128 (unicorn uses 1024) > 2) webrick does not disable Nagle's algorithm. > 3) webrick does not set SO_KEEPALIVE (not really needed for unicorn) > > So perhaps you can try config like the following to more closely match > what webrick does: > > listen 9292, backlog: 128, tcp_nodelay: false > > On the other hand, maybe webrick is too slow. > > > Our tcp initial retransmit interval is 1 second. When the interval is > > reduced, the occasional latency goes down. We also see the retransmits in > > netstat, about 1 - 2 every second. > > > > Anything that we should look at next? > > Try the above config to minimize the differences between webrick > and unicorn. > > If that fails, perhaps disabling SO_KEEPALIVE will work, but I'm > a bit lost as I'm not familiar with SmartOS quirks. > (you'll need to comment it out in lib/unicorn/socket_helper.rb) > > Maybe try a little mock server like this, too (should be fastest :) > ---------------------------- hello_world.rb ---------------- > require 'socket' > s = TCPServer.new(host, port) > # start changing knobs here: > # s.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 1) > # s.setsockopt(:IPPROTOL_TCP, :TCP_NODELAY, 1) > res = "HTTP/1.0 200 OK\r\nContent-Length: 12\r\n\r\nhello world\n" > junk = "" > loop do > c = s.accept > c.readpartial(1024, junk) > c.write(res) > c.close > end > ----------------------------- many.rb -------------------------- > # opens a lot of idle connections, be careful :) > require 'socket' > pids = [] > host = '10.45.14.175' > port = 7500 # not unicorn > at_exit { pids.each { |pid| Process.kill(:TERM, pid) } } > > 24.times do > pid = fork do > keep = [] > begin > s = TCPSocket.new(host, port) > # put something in the socket buffers > s.write("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n") > keep << s > rescue => e > $stdout.syswrite("#$$ done (#{keep.size}): #{e.message}\n") > sleep > end while true > end > pids << pid > end > p Process.waitall > -- > EW >