Unicorn behaving irrationally in production after a day

Unicorn behaving irrationally in production after a day

[Bogdan Dumitru]

Hey guys,

I'm new to Unicorn and have been running into a weird issue. I'm currently running a EC2 cluster with each machine having nginx with 4 unicorn workers. I'm deploying using rubber (https://github.com/rubber/rubber), and the unicorn.rb config is basically the GitHub unicorn config (as detailed here: https://github.com/blog/517-unicorn).
Everything works fine when I deploy, but after half a day - 1 day the server behaves irrationally. The server is an api server for a mobile application so I can easily see some variables that, on request, have a certain value, but if I do some checking with the rails console (on the production machine) I get  a different value (the correct one). This seams to me like a weird object caching situation. It all gets solved if I restart unicorn.
I'm not sure what config information you guys might be interested in so I won't bloat this mail with anything but let me know if there's any questions I can answer.

I hope somebody can point me in the right direction because I can't really make heads or tails of it.
Cheers,
-b

PS: I've read about something similar happening when using memcached for object caching, in a previous version, but that isn't the case here.

_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn behaving irrationally in production after a day

[Daniel Condomitti]

Are you disconnecting from/reconnecting to your database and any cache
stores in the before and after_fork hooks? Sharing the socket across
workers can potentially cause issues like this.

On May 10, 2013, at 6:00 AM, Bogdan Dumitru <dumbogdan@gmail.com> wrote:

> Hey guys,
>
> I'm new to Unicorn and have been running into a weird issue. I'm currently running a EC2 cluster with each machine having nginx with 4 unicorn workers. I'm deploying using rubber (https://github.com/rubber/rubber), and the unicorn.rb config is basically the GitHub unicorn config (as detailed here: https://github.com/blog/517-unicorn).
> Everything works fine when I deploy, but after half a day - 1 day the server behaves irrationally. The server is an api server for a mobile application so I can easily see some variables that, on request, have a certain value, but if I do some checking with the rails console (on the production machine) I get  a different value (the correct one). This seams to me like a weird object caching situation. It all gets solved if I restart unicorn.
> I'm not sure what config information you guys might be interested in so I won't bloat this mail with anything but let me know if there's any questions I can answer.
>
> I hope somebody can point me in the right direction because I can't really make heads or tails of it.
> Cheers,
> -b
>
> PS: I've read about something similar happening when using memcached for object caching, in a previous version, but that isn't the case here.
>
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying

Re: Unicorn behaving irrationally in production after a day

[Aaron Suggs]

Unicorns are mythical creatures that always behave rationally. :)

This sounds like a bug in your application rather than unicorn. Are you using :memory_store as your Rails.cache? That would fit the symptoms you're describing, since the cache would not be shared across processes.

If so, the solution is to use a shared cache store, like memcache (or the file system cache while you're on a single server).

On May 10, 2013, at 9:00 AM, Bogdan Dumitru <dumbogdan@gmail.com> wrote:

> Hey guys,
> 
> I'm new to Unicorn and have been running into a weird issue. I'm currently running a EC2 cluster with each machine having nginx with 4 unicorn workers. I'm deploying using rubber (https://github.com/rubber/rubber), and the unicorn.rb config is basically the GitHub unicorn config (as detailed here: https://github.com/blog/517-unicorn).
> Everything works fine when I deploy, but after half a day - 1 day the server behaves irrationally. The server is an api server for a mobile application so I can easily see some variables that, on request, have a certain value, but if I do some checking with the rails console (on the production machine) I get  a different value (the correct one). This seams to me like a weird object caching situation. It all gets solved if I restart unicorn.
> I'm not sure what config information you guys might be interested in so I won't bloat this mail with anything but let me know if there's any questions I can answer.
> 
> I hope somebody can point me in the right direction because I can't really make heads or tails of it.
> Cheers,
> -b
> 
> PS: I've read about something similar happening when using memcached for object caching, in a previous version, but that isn't the case here.
> 
> _______________________________________________
> Unicorn mailing list - mongrel-unicorn@rubyforge.org
> http://rubyforge.org/mailman/listinfo/mongrel-unicorn
> Do not quote signatures (like this one) or top post when replying
_______________________________________________
Unicorn mailing list - mongrel-unicorn@rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying