handling Timeout.timeout with sockets

handling Timeout.timeout with sockets

[Mike Perham]

Users of dalli are seeing a very rare issue
(https://github.com/mperham/dalli/issues/146) where a Timeout will
cause the client to crash.  They noticed this reproduces the bug:

require 'timeout'
require 'dalli'

Dalli.logger.level = Logger::DEBUG
client = Dalli::Client.new(["127.0.0.1:11211"], :socket_timeout => 0.01)

begin
 Timeout.timeout 0.01 do
   while true
     client.set("test_123", {:test => "123"})
   end
 end
rescue Timeout::Error => e
 puts [e.class.name, e.message]
 puts e.backtrace.join("\n")
end

p client.get("test_123")

I've noticed that my code cannot rescue the Timeout::Error that is
raised and I expect that is by design.  The block should be timed out
after the interval and code within the block should not be able to
rescue and stop that.

However I'm wondering how my network client can recover from that
timeout?  Presumably there is now data to be read on the socket which,
due to the timeout, has not been read, meaning my protocol state
machine is now out of sync, thus leading to the crash.

Should I drain the socket somehow before each operation?  What is the
best way to do this?

mike

Re: handling Timeout.timeout with sockets

[Eric Wong]

Mike Perham <mperham@gmail.com> wrote:
> I've noticed that my code cannot rescue the Timeout::Error that is
> raised and I expect that is by design.

Yes, it's by design.  Reading timeout.rb (and its
"git log -p --full-diff" history) is enlightening, btw.   You'll also
get better performance if you explicitly specify a class for timeout
to raise (ref: https://bugs.ruby-lang.org/issues/5765)

> However I'm wondering how my network client can recover from that
> timeout?  Presumably there is now data to be read on the socket which,
> due to the timeout, has not been read, meaning my protocol state
> machine is now out of sync, thus leading to the crash.
> 
> Should I drain the socket somehow before each operation?  What is the
> best way to do this?

The easiest way is to open a new socket.

The other way is to _completely_ parse the old response (that timed out)
before reusing the socket.

Do _not_ rely on IO#nread in the io/wait module for this, it's
inherently racy: bytes may arrive after you call IO#nread.  Unless the
protocol changed recently, for memcached (and nearly everything
implemented on top of TCP), you must completely parse the response the
server gives you.