unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
From: "Jesper Rønn-Jensen" <jesperrr@gmail.com>
To: Eric Wong <e@80x24.org>
Cc: unicorn-public@bogomips.org
Subject: Re: [PATCH] `unicorn upgrade` script resilience against exceptions
Date: Tue, 7 Jun 2016 17:17:28 +0200	[thread overview]
Message-ID: <CAL-rKu6U=zY=1tMWbYZSwXS6k57F64JU8GZRFRmMFDB49bVmNA@mail.gmail.com> (raw)
In-Reply-To: <20160607134107.GA4500@dcvr.yhbt.net>

Thanks for input.

I will close this issue and wait for your patch later, then.

Will also close pull-request I put at github with reference to this thread.

PS. Thanks for tip on HTML vs Plain text mails. Didn't know Gmail
could send plain text :)



On Tue, Jun 7, 2016 at 3:41 PM, Eric Wong <e@80x24.org> wrote:
> Jesper Rønn-Jensen <jesperrr@gmail.com> wrote:
>> This is actually a change proposal "by accident". But first, some
>> background:
>
>         Please don't send HTML mail.  I guess the bounce message didn't
>         include the reason for the bounce, but should be fixed; now;
>         and the bounce will tell you to not send HTML.
>
>         Anyways if you stick to old-school Usenet/mailing list posting
>         conventions, I'm more than happy to help you :)
>
>> My server uses a standard version of this `init.sh` script. I found that
>> executing `unicorn upgrade` often gives me problems.
>
> Hm... I haven't looked at that script in a while given all
> the new-fangled init replacements...
>
>> For example, it breaks my Capistrano deploy script to use `unicorn:upgrade`
>> to deploy. This is a shame, since it's the correct
>> command for Unicorn to pick up any code changes. But unicorn upgrade in the
>> script fails with messages like:
>>
>> ```
>> sudo /etc/init.d/unicorn upgrade
>> Couldn't upgrade, starting 'export HOME=/home/deploy ; cd
>> /var/www/my_app/current && /home/deploy/.rvm/wrappers/my_app/bundle exec
>> unicorn -D -c /var/www/my_app/shared/config/unicorn.rb -E production'
>> instead
>> master failed to start, check stderr log for details
>> ```
>>
>> And the stderror log:
>> ```
>> E, [2016-06-07T09:03:27.517267 #27583] ERROR -- : reaped #<Process::Status:
>> pid 27621 exit 1> exec()-ed
>> /var/www/my_app/shared/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:195:in
>>     `pid=': Already running on PID:27583 (or
>> pid=/var/www/my_app/shared/tmp/pids/unicorn.pid is stale)
>>     (ArgumentError)
>> ```
>>
>> I am proposing this change because I misread the documentation for the
>> SIGNALS!
>> I thought that Unicorn itself sends a QUIT after all subprocesses has ended
>> via SIG USR2.
>>
>> The change is to remove the QUIT signal to the PID.oldbin process, and that
>> apparently stops all the failures I ran into. So I made this change on a
>> server, and it works!
>>
>> I just don't understand why, since reading the documentation again it says:
>> https://github.com/defunkt/unicorn/blob/d23d4713dc9ab9732c574f5aa34a4b6740b43164/SIGNALS#L35
>>
>> > * USR2 - reexecute the running binary.  A separate QUIT
>> >  should be sent to the original process once the child is verified to
>> >  be up and running.
>>
>> So clearly, I should send the QUIT signal.
>
> Yes.
>
>> My goal is to have a resilient, robust deployment, where unicorn picks up
>> any code changes.
>
> Unfortunately, PID files have always been racy with USR2.
> Nowadays systemd is fairly standardized and seems to work
> pretty well for managing sockets and services.
>
> I'm still not a fan of some systemd things, but I think the socket
> activation part is very nice.  Example @.service and .socket files
> are distributed nowadays:
>
>         https://unicorn.bogomips.org/examples/unicorn@.service
>         https://unicorn.bogomips.org/examples/unicorn.socket
>
>> Can you help me and enlighten me as to why this proposed change works?
>
> Anyways, things are racy in that script and increasing the
> "sleep 2" to a higher number will help if your system is really
> overloaded.
>
>> ---
>>  examples/init.sh | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/examples/init.sh b/examples/init.sh
>> index 1f0e035..cbadc11 100644
>> --- a/examples/init.sh
>> +++ b/examples/init.sh
>> @@ -45,7 +45,7 @@ restart|reload)
>>   $CMD
>>   ;;
>>  upgrade)
>> - if sig USR2 && sleep 2 && sig 0 && oldsig QUIT
>> + if sig USR2 && sleep 2 && sig 0
>>   then
>>   n=$TIMEOUT
>>   while test -s $old_pid && test $n -ge 0
>> --
>
> I actually wrote a better init script for a similar server,
> patch coming in a bit (or later, close to falling asleep).



-- 

Jesper Rønn-Jensen
Nine A/S
Mobile: +45 2989 1822
Blog http://justaddwater.dk/
jesperrr@gmail.com (Private e-mail and Google Talk IM)

  reply	other threads:[~2016-06-07 15:17 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <CAL-rKu6CZ731c=uHyZ8+Fg2fhC30e-3-J26XODacUK=YrfX+5Q@mail.gmail.com>
2016-06-07 13:41 ` [PATCH] `unicorn upgrade` script resilience against exceptions Eric Wong
2016-06-07 15:17   ` Jesper Rønn-Jensen [this message]
2016-06-07 21:13     ` [PATCH] examples/init.sh: update to reduce upgrade raciness Eric Wong
2016-06-15 20:21       ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://yhbt.net/unicorn/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAL-rKu6U=zY=1tMWbYZSwXS6k57F64JU8GZRFRmMFDB49bVmNA@mail.gmail.com' \
    --to=jesperrr@gmail.com \
    --cc=e@80x24.org \
    --cc=unicorn-public@bogomips.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).