* leakage of sockets or activerecord connections
@ 2013-08-22 22:13 Corin Langosch
[not found] ` <52168D10.8080407-FIgL9nsKG9THeUWFKdsAYQC/G2K4zDHf@public.gmane.org>
0 siblings, 1 reply; 4+ messages in thread
From: Corin Langosch @ 2013-08-22 22:13 UTC (permalink / raw)
To: rainbows-talk-GrnCvJ7WPxnNLxjTenLetw
Hi,
I'm using rainbows to power my own small middleware. I doen't use rails (or any
other framework), only activerecord for database access. I chose
XEpollThreadSpawn, set worker_processes 1 and worker_connections 25. All classes
are eager loaded, no reloading of anything while the server is running. AR
connection pool size is set to 100.
Now it seems that every request opens a new connection and never frees/ closes
it. So after 100 requests I get an AR connection pool exception. I also see
exactly 100 postgresql clients connected. When I kill the server all clients get
disconnected.
I wonder if I have to setup and hooks (like in unicorn before_fork etc.)? In
fact I'd expect this happens automatically as the thread exits after the request
is completed?
Thanks,
Corin
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: leakage of sockets or activerecord connections
[not found] ` <52168D10.8080407-FIgL9nsKG9THeUWFKdsAYQC/G2K4zDHf@public.gmane.org>
@ 2013-08-22 22:45 ` Eric Wong
2013-08-22 22:46 ` Lin Jen-Shin (godfat)
1 sibling, 0 replies; 4+ messages in thread
From: Eric Wong @ 2013-08-22 22:45 UTC (permalink / raw)
To: Rainbows! list
Corin Langosch <info-FIgL9nsKG9THeUWFKdsAYQC/G2K4zDHf@public.gmane.org> wrote:
> I'm using rainbows to power my own small middleware. I doen't use
> rails (or any other framework), only activerecord for database
> access. I chose XEpollThreadSpawn, set worker_processes 1 and
> worker_connections 25. All classes are eager loaded, no reloading of
> anything while the server is running. AR connection pool size is set
> to 100.
I'm not sure how AR connection pool works (if it uses thread-local
variables for storing the connections). If it's using thread-locals,
it could be reliant on GC, and that would require:
a) GC to run frequently enough to reap connections
b) your Postgres bindings being GC-aware (and really being unreachable
in your VM)
> Now it seems that every request opens a new connection and never
> frees/ closes it. So after 100 requests I get an AR connection pool
> exception. I also see exactly 100 postgresql clients connected. When
> I kill the server all clients get disconnected.
>
> I wonder if I have to setup and hooks (like in unicorn before_fork
> etc.)? In fact I'd expect this happens automatically as the thread
> exits after the request is completed?
Can you reproduce the issue with XEpollThreadPool?
If it's using thread-local storage, you probably need to have a Rack
middleware push the connection back into the pool when it's done working
on a request.
Something like this in your config.ru:
class ReleaseConnections
def initialize(app)
@app = app
end
def call(env)
begin
@app.call(env)
ensure
Thread.current[:whatever_sockets].release_to_pool
end
end
end
use ReleaseConnections
...
run YourApp.new
In any case, I'd dig through the Rails/AR APIs to see how it works
and how to release resources back to the pool.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: leakage of sockets or activerecord connections
[not found] ` <52168D10.8080407-FIgL9nsKG9THeUWFKdsAYQC/G2K4zDHf@public.gmane.org>
2013-08-22 22:45 ` Eric Wong
@ 2013-08-22 22:46 ` Lin Jen-Shin (godfat)
[not found] ` <CAA2_N1soVYiNkRgZqa+mLO26KUzvTAXuk0gA9Qo7MPrZ0+bNXw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
1 sibling, 1 reply; 4+ messages in thread
From: Lin Jen-Shin (godfat) @ 2013-08-22 22:46 UTC (permalink / raw)
To: Rainbows! list
On Fri, Aug 23, 2013 at 6:13 AM, Corin Langosch <info-FIgL9nsKG9THeUWFKdsAYQC/G2K4zDHf@public.gmane.org> wrote:
> Hi,
>
> I wonder if I have to setup and hooks (like in unicorn before_fork etc.)? In
> fact I'd expect this happens automatically as the thread exits after the
> request is completed?
>
> Thanks,
> Corin
No, I think you should probably insert this middleware on top of your app:
ActiveRecord::ConnectionAdapters::ConnectionManagement
This is actually what Rails did for itself. Here's what it is doing:
class ConnectionManagement
def initialize(app)
@app = app
end
def call(env)
testing = env.key?('rack.test')
response = @app.call(env)
response[2] = ::Rack::BodyProxy.new(response[2]) do
ActiveRecord::Base.clear_active_connections! unless testing
end
response
rescue
ActiveRecord::Base.clear_active_connections! unless testing
raise
end
end
p.s. By pasting you this, now I understood why Rails works a bit
differently with threads in tests. Oh well. Maybe I should patch it.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: leakage of sockets or activerecord connections
[not found] ` <CAA2_N1soVYiNkRgZqa+mLO26KUzvTAXuk0gA9Qo7MPrZ0+bNXw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
@ 2013-08-27 12:31 ` Corin Langosch
0 siblings, 0 replies; 4+ messages in thread
From: Corin Langosch @ 2013-08-27 12:31 UTC (permalink / raw)
To: Rainbows! list
Am 23.08.2013 00:46, schrieb Lin Jen-Shin (godfat):
> No, I think you should probably insert this middleware on top of your app:
>
> ActiveRecord::ConnectionAdapters::ConnectionManagement
>
>
Thank you, works fine now! :)
Corin
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2013-08-27 12:38 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-08-22 22:13 leakage of sockets or activerecord connections Corin Langosch
[not found] ` <52168D10.8080407-FIgL9nsKG9THeUWFKdsAYQC/G2K4zDHf@public.gmane.org>
2013-08-22 22:45 ` Eric Wong
2013-08-22 22:46 ` Lin Jen-Shin (godfat)
[not found] ` <CAA2_N1soVYiNkRgZqa+mLO26KUzvTAXuk0gA9Qo7MPrZ0+bNXw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-27 12:31 ` Corin Langosch
Code repositories for project(s) associated with this public inbox
https://yhbt.net/rainbows.git/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).