Using Kgio::Socket.pair and OpenSSL::SSL::Context for a new TLS stack in EventMachine

Using Kgio::Socket.pair and OpenSSL::SSL::Context for a new TLS stack in EventMachine

[Iñaki Baz Castillo]

Hi, Ruby EventMachine has some drawbacks when using its SSL
stack(coded in C++), basically it consumes ***lot*** of memory as
showedin: https://github.com/eventmachine/eventmachine/issues/266
I've been suggested by the author of EM to create a TLS stack
usingRuby OpenSSL on top of a EM::Connection instance (which would be
apure TCP client/server connection rather than a TLS connection).
When acting as TCP server, an EM::Connection is created by EM for
eachincoming connection, and runs some callbacks:
- EM::Connection#initialize(), when the TCP connection begins.-
EM::Connection#post_init(), when the TCP connection has been
completed.- EM::Connection#receive_data(data), when data is received
from the client.- EM::Connection#unbind(cause=nil), when the TCP
connection isterminated locally or remotely.
So what I'm trying to achieve is using Kgio::Socket.pair() as a
bridgebetween the raw TLS data received from the client (after the
TCPconnection is done) and a OpenSSL::SSL::SSLSocket, something like:
--------------------------class MyTlsServer < EM::Connection
 @@ssl_context = ::OpenSSL::SSL::SSLContext.new
 def post_init   @sock1, @sock2 =
::Kgio::Socket.pair(::Kgio::Socket::AF_UNIX,::Kgio::Socket::SOCK_STREAM,
0)   @ssl = ::OpenSSL::SSL::SSLSocket.new @sock2, @@ssl_context
@ssl.sync_close = true end
 def receive_data data   # Write the received TLS raw data into @sock1, so
   # @ssl socket can read it from @sock2.   @sock1.write data
   # I should call @ssl.accept somewhere (just once per connection)
   # in order to perform the TLS handshake, but I don't know how,   #
it blocks forever.
   # No idea how to continue here. end
end--------------------------


Any suggestion about how could I continue it? it's being a hard
taskand I don't get it working in any way.
Really thanks a lot for any help.


-- 
Iñaki Baz Castillo
<ibc@aliax.net>

Re: Using Kgio::Socket.pair and OpenSSL::SSL::Context for a new TLS stack in EventMachine

[Iñaki Baz Castillo]

Sorry, I did copy&paste from a first mail (dropped by librelist as
usual) and pasting in Gmail is a pain. Let me try again:


Hi, Ruby EventMachine has some drawbacks when using its SSL stack
(coded in C++), basically it consumes ***lot*** of memory as showed
in:
 https://github.com/eventmachine/eventmachine/issues/266

I've been suggested by the author of EM to create a TLS stack using
Ruby OpenSSL on top of a EM::Connection instance (which would be a
pure TCP client/server connection rather than a TLS connection).

When acting as TCP server, an EM::Connection is created by EM for each
incoming connection, and runs some callbacks:

- EM::Connection#initialize(), when the TCP connection begins.
- EM::Connection#post_init(), when the TCP connection has been completed.
- EM::Connection#receive_data(data), when data is received from the client.
- EM::Connection#unbind(cause=nil), when the TCP connection is
terminated locally or remotely.

So what I'm trying to achieve is using Kgio::Socket.pair() as a bridge
between the raw TLS data received from the client (after the TCP
connection is done) and a OpenSSL::SSL::SSLSocket, something like:

--------------------------
class MyTlsServer < EM::Connection

  @@ssl_context = OpenSSL::SSL::SSLContext.new

  def post_init
    @sock1, @sock2 = Kgio::Socket.pair(Kgio::Socket::AF_UNIX,
Kgio::Socket::SOCK_STREAM, 0)
    @ssl = OpenSSL::SSL::SSLSocket.new @sock2, @@ssl_context
    @ssl.sync_close = true
  end

  def receive_data data
    # Write the received TLS raw data into @sock1, so @ssl
    # socket can read it from @sock2.
    @sock1.write data

    # I should call @ssl.accept somewhere (just once per connection)
    # in order to perform the TLS handshake, but I don't know how,
    # it blocks forever.

    # No idea how to continue here.
  end

end
--------------------------



Any suggestion about how could I continue it? it's being a hard task
and I don't get it working in any way.

Really thanks a lot for any help.


-- 
Iñaki Baz Castillo
<ibc@aliax.net>

Re: Re: Using Kgio::Socket.pair and OpenSSL::SSL::Context for a new TLS stack in EventMachine

[Eric Wong]

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> Hi, Ruby EventMachine has some drawbacks when using its SSL stack
> (coded in C++), basically it consumes ***lot*** of memory as showed
> in:
>  https://github.com/eventmachine/eventmachine/issues/266

So you guys tried disabling compression and the release buffers flag
with no effect?  That doesn't sound right, but I'm a poo-flinging monkey
when it comes to using OpenSSL...

This is with glibc malloc?  Can you try this with
MALLOC_MMAP_THRESHOLD_=131072 set?  (Or some lower value, say 16384 or
4096...)

> I've been suggested by the author of EM to create a TLS stack using
> Ruby OpenSSL on top of a EM::Connection instance (which would be a
> pure TCP client/server connection rather than a TLS connection).
>
> When acting as TCP server, an EM::Connection is created by EM for each
> incoming connection, and runs some callbacks:
> 
> - EM::Connection#initialize(), when the TCP connection begins.
> - EM::Connection#post_init(), when the TCP connection has been completed.
> - EM::Connection#receive_data(data), when data is received from the client.
> - EM::Connection#unbind(cause=nil), when the TCP connection is
> terminated locally or remotely.

I would probably go with EM.attach/EM.attach_fd and handle all the
read_nonblock/write_nonblock + buffering yourself in Ruby and
using the notify_readable/notify_writable functionality of EM.

I don't think you'll be able to use receive_data/send_data provided by
EM if you rely on Ruby OpenSSL (which is fine, use notify_*able).

> So what I'm trying to achieve is using Kgio::Socket.pair() as a bridge
> between the raw TLS data received from the client (after the TCP
> connection is done) and a OpenSSL::SSL::SSLSocket, something like:

I don't think socketpair is good for this, you'd basically be
reimplementing stud/stunnel/flipper inside your Ruby process:
double-copying everything through the kernel.

> Any suggestion about how could I continue it? it's being a hard task
> and I don't get it working in any way.

You could also look at the Flipper code inside kgio-monkey[1].  It's
basically a Ruby version of stud I wrote to learn/practice with
OpenSSL.

Flipper uses Kgio.poll (out of laziness), but it should be possible to
port/extend for use with EM/Coolio/SleepyPenguin::Epoll/IO.select.  I
use flipper for junk IMAP accounts and to look for bugs, but nothing
serious that I'd care to secure.

[1] - http://bogomips.org/kgio-monkey.git
      kgio-monkey is completely unaudited for security bugs, though

Re: Re: Using Kgio::Socket.pair and OpenSSL::SSL::Context for a new TLS stack in EventMachine

[Iñaki Baz Castillo]

Thanks a lot for your response Eric, I must analize all of it and will
come soon.

Really thanks a lot.


-- 
Iñaki Baz Castillo
<ibc@aliax.net>

Re: Re: Using Kgio::Socket.pair and OpenSSL::SSL::Context for a new TLS stack in EventMachine

[Iñaki Baz Castillo]

2011/12/9 Eric Wong <normalperson@yhbt.net>:
> Iñaki Baz Castillo <ibc@aliax.net> wrote:
>> Hi, Ruby EventMachine has some drawbacks when using its SSL stack
>> (coded in C++), basically it consumes ***lot*** of memory as showed
>> in:
>>  https://github.com/eventmachine/eventmachine/issues/266
>
> So you guys tried disabling compression and the release buffers flag
> with no effect?

I tryed disabling SSL compression and indeed it reduces a bit the
memory usage, but not enough. And if I create 500 persistent TLS
connections against a EM server, it consumes all the memory.


> This is with glibc malloc?  Can you try this with
> MALLOC_MMAP_THRESHOLD_=131072 set?  (Or some lower value, say 16384 or
> 4096...)

Where to put that? maybe on top of the rubymain.cpp file?




> I would probably go with EM.attach/EM.attach_fd and handle all the
> read_nonblock/write_nonblock + buffering yourself in Ruby and
> using the notify_readable/notify_writable functionality of EM.
>
> I don't think you'll be able to use receive_data/send_data provided by
> EM if you rely on Ruby OpenSSL (which is fine, use notify_*able).

Ok, the problem is still how to make OpenSSL to behave as it is
listening in a "real" TCP socket. I continue reading your suggestions.



>> So what I'm trying to achieve is using Kgio::Socket.pair() as a bridge
>> between the raw TLS data received from the client (after the TCP
>> connection is done) and a OpenSSL::SSL::SSLSocket, something like:
>
> I don't think socketpair is good for this, you'd basically be
> reimplementing stud/stunnel/flipper inside your Ruby process:
> double-copying everything through the kernel.

Right, discarded, ok.


>> Any suggestion about how could I continue it? it's being a hard task
>> and I don't get it working in any way.
>
> You could also look at the Flipper code inside kgio-monkey[1].  It's
> basically a Ruby version of stud I wrote to learn/practice with
> OpenSSL.

So do you mean creating the TLS sockets by myself and then using EM
attach/watch features?



> Flipper uses Kgio.poll (out of laziness), but it should be possible to
> port/extend for use with EM/Coolio/SleepyPenguin::Epoll/IO.select.  I
> use flipper for junk IMAP accounts and to look for bugs, but nothing
> serious that I'd care to secure.
>
> [1] - http://bogomips.org/kgio-monkey.git
>      kgio-monkey is completely unaudited for security bugs, though

mmm, really complex XD
I must spent long time checking it. Thanks a lot.

BTW: I didn't know about SleepyPenguin. Can it be used to build like a
EventMachine Ruby library?

Regards.



-- 
Iñaki Baz Castillo
<ibc@aliax.net>

Re: Re: Using Kgio::Socket.pair and OpenSSL::SSL::Context for a new TLS stack in EventMachine

[Eric Wong]

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> 2011/12/9 Eric Wong <normalperson@yhbt.net>:
> > Iñaki Baz Castillo <ibc@aliax.net> wrote:

> > This is with glibc malloc?  Can you try this with
> > MALLOC_MMAP_THRESHOLD_=131072 set?  (Or some lower value, say 16384 or
> > 4096...)
> 
> Where to put that? maybe on top of the rubymain.cpp file?

In your environment *before* exec-ing the Ruby process.  glibc reads
this early during initialization.

> >> Any suggestion about how could I continue it? it's being a hard task
> >> and I don't get it working in any way.
> >
> > You could also look at the Flipper code inside kgio-monkey[1].  It's
> > basically a Ruby version of stud I wrote to learn/practice with
> > OpenSSL.
> 
> So do you mean creating the TLS sockets by myself and then using EM
> attach/watch features?

Yes, you need to attach/watch on the bare TCP socket descriptor.
EM doesn't need to know about the TLS layer (and shouldn't).  You
handle all the read/write calls to the TLS layer yourself without
EM helping you with anything other than notification.

> > Flipper uses Kgio.poll (out of laziness), but it should be possible to
> > port/extend for use with EM/Coolio/SleepyPenguin::Epoll/IO.select.  I
> > use flipper for junk IMAP accounts and to look for bugs, but nothing
> > serious that I'd care to secure.
> >
> > [1] - http://bogomips.org/kgio-monkey.git
> >      kgio-monkey is completely unaudited for security bugs, though
> 
> mmm, really complex XD
> I must spent long time checking it. Thanks a lot.

You may want to get a feel for how Ruby + non-blocking OpenSSL sockets
work by reimplementing Flipper with IO.select/SSLSocket (and not rely
on kgio-monkey at all for now).

That might be a good learning exercise and eventually you can swap out
the IO.select part(s) for EM, and the agnostic forwarding logic for
application logic.

> BTW: I didn't know about SleepyPenguin. Can it be used to build like a
> EventMachine Ruby library?

Yes, but it'd be redundant.  For sleepy_penguin, I wanted the native
epoll API mapped to Ruby.  The portable EM/Coolio/libev/libevent
APIs cannot support concurrent epoll_ctl/epoll_wait calls to a single
epoll descriptor.  I think native kqueue can do this, too, but I know
it's impossible with poll/select.

Re: Re: Using Kgio::Socket.pair and OpenSSL::SSL::Context for a new TLS stack in EventMachine

[Iñaki Baz Castillo]

2011/12/22 Eric Wong <normalperson@yhbt.net>:
>> > This is with glibc malloc?  Can you try this with
>> > MALLOC_MMAP_THRESHOLD_=131072 set?  (Or some lower value, say 16384 or
>> > 4096...)
>>
>> Where to put that? maybe on top of the rubymain.cpp file?
>
> In your environment *before* exec-ing the Ruby process.  glibc reads
> this early during initialization.

Tested. Interesting, if I set MALLOC_MMAP_THRESHOLD_ to 4096 or 32XXX,
memory usage is the same, but after closing client connections, the EM
server again comes back to consume a few memory. If I don't set this
variable, the EM server memory usage is not decreased after client
disconnects all the connections.





-- 
Iñaki Baz Castillo
<ibc@aliax.net>