handling Timeout.timeout with sockets

[Mike Perham <mperham@gmail.com>]

Users of dalli are seeing a very rare issue
(https://github.com/mperham/dalli/issues/146) where a Timeout will
cause the client to crash.  They noticed this reproduces the bug:

require 'timeout'
require 'dalli'

Dalli.logger.level = Logger::DEBUG
client = Dalli::Client.new(["127.0.0.1:11211"], :socket_timeout => 0.01)

begin
 Timeout.timeout 0.01 do
   while true
     client.set("test_123", {:test => "123"})
   end
 end
rescue Timeout::Error => e
 puts [e.class.name, e.message]
 puts e.backtrace.join("\n")
end

p client.get("test_123")

I've noticed that my code cannot rescue the Timeout::Error that is
raised and I expect that is by design.  The block should be timed out
after the interval and code within the block should not be able to
rescue and stop that.

However I'm wondering how my network client can recover from that
timeout?  Presumably there is now data to be read on the socket which,
due to the timeout, has not been read, meaning my protocol state
machine is now out of sync, thus leading to the crash.

Should I drain the socket somehow before each operation?  What is the
best way to do this?

mike