Re: Rails 7 - Unicorn 6.1.0 -- reaped with preload_app

[Eric Wong <e@80x24.org>]

Cédric Carrard <cedriccarrard@gmail.com> wrote:
> Hello
> 
> Since the update to rails 7 and ruby 3.1.1 I have errors with unicorn
> 6.1.0 (we use docker-compose, nginx, sidekiq, postgresql and
> unicorn-worker-killer).

You might need something in your before_fork / after_fork hooks
for sidekiq or anything else that might make connections...

more below...

> 1:10280 configurator.rb:49] Unicorn -- reaped #<Process::Status: pid
> 415 exit 1> worker=15

Is there anything else?

> Here is the unicorn configuration:
> 
> require 'semantic_logger'
> 
> worker_processes(File.read('/proc/cpuinfo').scan(/^processor\s*:/).size
> * (ENV['WORKERS_PER_CPU'] || 2).to_i)

Sidenote: Etc.nprocessors is available since Ruby 2.2 and more
portable than reading /proc/cpuinfo.

I also recommend starting with 1-2 workers whenever you're
debugging to help narrow things down.  Some (most) FD
sharing problems appear when going to 1 -> 2, and 2 is usually
as good as dozens (until system memory/FD limits are hit)

> timeout 300
> preload_app true
> listen(ENV["PORT"] || 3000)
> logger SemanticLogger['Unicorn']
> 
> before_fork do |server, _worker|
>   Signal.trap 'TERM' do
>     server.logger.info 'Unicorn master intercepting TERM and sending
> myself QUIT instead'
>     Process.kill 'QUIT', Process.pid
>   end
> 
>   defined?(ActiveRecord::Base) && ActiveRecord::Base.connection.disconnect!

OK, I guess AR is for Postgres.  Any other things that need to
be handled here?  (e.g. sidekiq, sqlite, xapian, memcached, redis, ...).

"lsof -p $PID" on both master and any single worker can be used
to spot inadvertant FD sharing for things that may need to be
shared.

> end
> 
> after_fork do |server, _worker|
>   Signal.trap 'TERM' do
>     server.logger.info 'Unicorn worker intercepting TERM and doing
> nothing. Wait for master to send QUIT'
>   end
> 
>   SemanticLogger.reopen

I assume SemanticLogger is just a filesystem-based logger?
If it establishes a connection, maybe it needed to be
disconnected in before_fork.

>   defined?(ActiveRecord::Base) && ActiveRecord::Base.establish_connection
> end
> 
> If I remove the preload_app it works (but apparently it is recommended
> to leave this setting)

Recommendations depend on needs vs willingness to make things work :)
Yes, preload_app saves memory and improves startup time; but it
requires deeper app knowledge to use correctly (as you're
finding out).

> Do you have any idea where the problem could be?

Maybe the above helps, but it's really your codebase nobody in
here knows...

I have no idea what SemanticLogger is, but having fewer dependencies
can help you narrow things down.  In other words, try the normal
logger.

If you don't know the app very well (e.g. inherited from someone
else); I'd start by editing out large sections of the codebase
to narrow things down.