background threads

background threads

[Alexandre Riveira]

Hello !

I'm trying to put in background threads in Rainbows. I created a 
config/initializers/background_thread.rb, where I

Thread.new do
# my background job
end

With that, I have problems to "reload" in rainbows

Thank you all for your help.


Alexandre Riveira
Brazil
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: background threads

[Eric Wong]

Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> Hello !
> 
> I'm trying to put in background threads in Rainbows. I created a
> config/initializers/background_thread.rb, where I
> 
> Thread.new do
> # my background job
> end
> 
> With that, I have problems to "reload" in rainbows

What kind of problems?

Is preload_app true or false?  (default is false)

preload_app being true will cause that background thread to get
started in the master process and not the workers.  You can
workaround it by starting th background thread in an after_fork
hook in the Rainbows!/unicorn config file:

http://unicorn.bogomips.org/Unicorn/Configurator.html#method-i-after_fork

> Thank you all for your help.

I'm often lazy when it comes to setting up hooks myself :x

So for background threads, I prefer to start it in the app stack
on the first request (use a mutex to protect against double-starts).
For rails, you can probably do it in a before_filter.

Raindrops::Watcher is a Rack app (no other framework, just rack)
which starts a background thread in the #call method:

	git clone git://bogomips.org/raindrops raindrops
	cat raindrops/lib/watcher.rb
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: background threads

[Alexandre Riveira]

Em 18-06-2012 21:55, Eric Wong escreveu:

What kind of problems?

The application consumes 100% cpu after reload


Is preload_app true or false?  (default is false)
preload_app is true


I changed  my code  as below  and got  a result in  small servers
  Thread.new(log) do |rotina_log|
     loop do
        log.info("rotina automatica") {"waiting process before end"}
        sleep(5)
        break unless File.exists? Dir.pwd + '/tmp/pids/rainbows.pid.oldbin'
     end

     loop do
       #rotina_log.info("rotina automatica") { "waiting 60 secs #{Process.pid}" }
       sleep(60)
       if $0.include? "old"
         rotina_log.info("rotina automatica") { "exit..... #{Process.pid}" }
         Thread.exit
       end
       begin
         #rotina_log.info("rotina automatica") { "exec #{Process.pid}" }
         empresas = Empresa::Base.find_all_by_cancelado(false)
         #sleep(180)
         empresas.invoke(:rotinas_automaticas, :usuario_id =>  Auth::Configuracao.instance.usuario_sistema_id)
         #rotina_log.info("rotina automatica") { "exec ok #{Process.pid}" }
       rescue Exception =>  e
         rotina_log.fatal("rotina automatica") { e.to_s + e.backtrace.inspect }
       end
     end
   end


Tanks,


Alexandre Riveira
Brazil

_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: background threads

[Eric Wong]

Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> Em 18-06-2012 21:55, Eric Wong escreveu:
> 
> What kind of problems?
> 
> The application consumes 100% cpu after reload
> 
> 
> Is preload_app true or false?  (default is false)
> preload_app is true

With preload_app true, the application worker won't be able to
communicate with the background thread after reload.

I'm not sure if your setup relies on threads communicating
directly with the application.

> I changed  my code  as below  and got  a result in  small servers
>  Thread.new(log) do |rotina_log|
>     loop do
>        log.info("rotina automatica") {"waiting process before end"}
>        sleep(5)
>        break unless File.exists? Dir.pwd + '/tmp/pids/rainbows.pid.oldbin'
>     end
> 
>     loop do
>       #rotina_log.info("rotina automatica") { "waiting 60 secs #{Process.pid}" }
>       sleep(60)
>       if $0.include? "old"
>         rotina_log.info("rotina automatica") { "exit..... #{Process.pid}" }
>         Thread.exit
>       end
>       begin
>         #rotina_log.info("rotina automatica") { "exec #{Process.pid}" }
>         empresas = Empresa::Base.find_all_by_cancelado(false)
>         #sleep(180)
>         empresas.invoke(:rotinas_automaticas, :usuario_id =>  Auth::Configuracao.instance.usuario_sistema_id)
>         #rotina_log.info("rotina automatica") { "exec ok #{Process.pid}" }
>       rescue Exception =>  e
>         rotina_log.fatal("rotina automatica") { e.to_s + e.backtrace.inspect }
>       end
>     end
>   end

Reading the code above (but not knowing your original code), I'm not
seeing any communcation between the threads.  It could probably be done
better as a standalone daemon or even a cronjob.  There's more setup
overhead, sure, but it should also be easier to debug/troubleshoot.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: background threads

[Alexandre Riveira]

Em 18-06-2012 21:55, Eric Wong escreveu:
> So for background threads, I prefer to start it in the app stack
> on the first request (use a mutex to protect against double-starts).
> For rails, you can probably do it in a before_filter.
Eric, como você garante a aplicação não criar uma thread em cada worker

Tanks

Alexandre Riveira
brazil
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: background threads

[Eric Wong]

Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> Em 18-06-2012 21:55, Eric Wong escreveu:
> >So for background threads, I prefer to start it in the app stack
> >on the first request (use a mutex to protect against double-starts).
> >For rails, you can probably do it in a before_filter.
> 
> Eric, how do you ensure the application does not create a thread for
> each worker

In the Raindrops::Watcher case, I don't.  Watcher works as a single
process and doesn't need/care about multiple background threads.

If you only want one background thread for all workers, it probably is
best to just use a separate daemon independent of the Rainbows! server.
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying

Re: background threads

[Eric Wong]

Alexandre Riveira <alexandre-VwDbj2YsoUp0ZRtCdD4y8VAUjnlXr6A1@public.gmane.org> wrote:
> Em 18-06-2012 22:42, Eric Wong escreveu:
> >If you only want one background thread for all workers, it
> >probably is best to just use a separate daemon independent of the
> >Rainbows! server.
> I want only one worker to run a background thread. I would like to
> ensure that only one worker thread will be in the background. Sure
> ... using rainbows :)

It's a bad idea, I think...

You can use process-shared locks Ruby provides File#flock.
It's also possible to get fcntl()-based locking to work with pack().

Maybe there's extensions to share locks with multiple processes
(e.g. POSIX semaphores, database-based locks in Redis/MySQL/Postgres).

...Or even use a really ugly method of using TCPServer to bind to a
pre-determined port (and bailing on Errno::EADDRINUSE) (no need to
accept connections on the TCPServer, just keep the object around
so it doesn't get close(2)-ed on GC.

But really, I think this type of background thread is a bad idea
for Rainbows! (or any HTTP server).
_______________________________________________
Rainbows! mailing list - rainbows-talk-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/rainbows-talk
Do not quote signatures (like this one) or top post when replying