unicorn Ruby/Rack server user+dev discussion/patches/pulls/bugs/help
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* Re: [DRE-maint] unicorn: native systemd service
  2015-06-30 17:30 20%             ` Eric Wong
@ 2015-07-08 13:08 24%               ` Christos Trochalakis
  0 siblings, 0 replies; 9+ results
From: Christos Trochalakis @ 2015-07-08 13:08 UTC (permalink / raw)
  To: Eric Wong; +Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public

>Christos Trochalakis <yatiohi@ideopolis.gr> wrote:
>> On Sat, Jun 27, 2015 at 03:05:01AM +0000, Eric Wong wrote:
>> >Is it possible to make systemd fire up two unicorn masters?
>> >That would be a nice feature to have with socket activation.
>>
>> That would be great! I am a bit surprised that specifying multiple
>> services ('Service=' option) is not supported.

Apparently I was wrong. I missed the 'Sockets=' option for Service units.

You can specify a list of socket units to be inherited (if active) when
the service is started. Note that this is not the reverse of Sockets'
'Service=' option: the service is not auto-started when there is an
incoming connnection on the relevant socket, you have to use 'Service='
for socket activation.

Here is a test config for 2 preforked unicorn masters using service
templates (I am using the latest unicorn with the systemd patch):

==> /srv/uni/config.ru <==
puts "initializing for 5sec"
sleep 5

app = proc do |env|
  [
    200,
    { 'Content-Type' => 'text/plain' },
    ["ppid:#{Process.ppid}\n"]
  ]
end

run app

==> /srv/uni/unicorn.conf.rb <==
worker_processes 2
working_directory "/srv/uni"

# Keep in sync with uni.socket file
listen 9000
listen 9001

==> /etc/systemd/system/uni@.service <==
[Unit]
Description=Unicorn Server %i
Wants=uni.socket
After=uni.socket

[Service]
ExecStart=/usr/local/bin/unicorn -c /srv/uni/unicorn.conf.rb -d
Sockets=uni.socket
KillSignal=SIGQUIT

[Install]
WantedBy=multi-user.target

==> /etc/systemd/system/uni.socket <==
[Unit]
Description=Unicorn Sockets

[Socket]
ListenStream=0.0.0.0:9000
ListenStream=0.0.0.0:9001
Service=uni@1.service

[Install]
WantedBy=sockets.target

==> end <==

systemctl daemon-reload
systemctl enable uni.socket
systemctl start uni.socket

systemctl enable uni@1 uni@2 # Make them start at boot
systemctl start uni@1 uni@2

Now you can start and stop uni@1 and uni@2 with zero latency and without losing
a connection. So we have a happy ending after all :)

While searching systemd's mailing list, I found a thread discussing about a
'Distribute=' option to sockets a few years back[0]. There is also a relevant
item in systemd's TODO list.

[0] http://thread.gmane.org/gmane.comp.sysutils.systemd.devel/14242/focus=14255

^ permalink raw reply	[relevance 24%]

* Re: [DRE-maint] unicorn: native systemd service
  2015-06-30  9:20 23%           ` Christos Trochalakis
@ 2015-06-30 17:30 20%             ` Eric Wong
  2015-07-08 13:08 24%               ` Christos Trochalakis
  0 siblings, 1 reply; 9+ results
From: Eric Wong @ 2015-06-30 17:30 UTC (permalink / raw)
  To: Christos Trochalakis
  Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public

Christos Trochalakis <yatiohi@ideopolis.gr> wrote:
> On Sat, Jun 27, 2015 at 03:05:01AM +0000, Eric Wong wrote:
> >Christos Trochalakis <yatiohi@ideopolis.gr> wrote:
> >>KillMode=mixed
> >
> >I don't think KillMode=mixed is necessary, here.  systemd can send
> >SIGQUIT to workers.
> >
> 
> Perhaps there is a race here, if a worker receives SIGQUIT first, and the
> master respawns a new worker before receiving/handling its own SIGQUIT.
> This is definitely a longshot, and will probably never happen, but
> even then every process will eventually quit gracefully.

Right. This race is possible, but workers will automatically die if
they don't have a master.
Easier just to kill the master, of course.

> >Is it possible to make systemd fire up two unicorn masters?
> >That would be a nice feature to have with socket activation.
> 
> That would be great! I am a bit surprised that specifying multiple
> services ('Service=' option) is not supported.

Can you get this feature added to systemd?

^ permalink raw reply	[relevance 20%]

* Re: [DRE-maint] unicorn: native systemd service
  2015-06-27  3:05 20%         ` Eric Wong
@ 2015-06-30  9:20 23%           ` Christos Trochalakis
  2015-06-30 17:30 20%             ` Eric Wong
  0 siblings, 1 reply; 9+ results
From: Christos Trochalakis @ 2015-06-30  9:20 UTC (permalink / raw)
  To: Eric Wong; +Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public

On Sat, Jun 27, 2015 at 03:05:01AM +0000, Eric Wong wrote:
>Christos Trochalakis <yatiohi@ideopolis.gr> wrote:
>> On Thu, Jun 25, 2015 at 11:26:26PM +0000, Eric Wong wrote:
>> >With socket activation, you should just be able to kill unicorn using
>> >SIGQUIT (just master, or even all workers) and restart without ever
>> >dropping a connection.  I do NOT suggest using SIGTERM for unicorn,
>> >since that'll cause the master to kill all workers ASAP.
>>
>> Yes, you are right socket activation is also an option! I have made some
>> experiments with a simple rack app to test it.
>>
>> systemd uses the LISTEN_FDS env variable that is an integer indicating the
>> number of inherited file descriptors. Those FDs have consecutive numbers
>> starting from `SD_LISTEN_FDS_START` which is `3` (man sd_listen_fds).
>>
>> So for example if LISTEN_FDS="2", UNICORN_FD should be "3,4". I used a
>> simple wrapper script for that. Here is the full configuration:
>
>OK, I'll probably add LISTEN_FDS and LISTEN_PID support to unicorn
>directly so the wrapper is unnecessary.
>
><snip>
>
>> KillMode=mixed
>
>I don't think KillMode=mixed is necessary, here.  systemd can send
>SIGQUIT to workers.
>

Perhaps there is a race here, if a worker receives SIGQUIT first, and the
master respawns a new worker before receiving/handling its own SIGQUIT.
This is definitely a longshot, and will probably never happen, but
even then every process will eventually quit gracefully.

><snip>
>
>> TCP socket options are not applied by unicorn on inherited sockets (TCPSocket
>> === sock is false). systemd socket files have support for most options now but
>> we might want unicorn to `setsockopt` them as well. For example,
>> 'DeferAcceptSec', 'KeepAliveIntervalSec', 'NoDelay' are supported since v216, so
>> they are not available in jessie (v215).
>
>They are now :)
>
>http://bogomips.org/unicorn-public/m/1435373879-4299-1-git-send-email-e@80x24.org.txt
>
>We don't have KeepAliveIntervalSec
>(aka TCP_KEEPINTVL) since it's not-portable and probably over
>Are you sure about that?

I added 'KeepAliveIntervalSec' by mistake. `KeepAlive`, a supported
option by unicorn, is included in jessie (systemd v215). Either way with
your last patch those options are enforced by unicorn.

>
>> socket activation is a really interesting setup, but personally I would not run
>> it with a large application. Clients would have to wait for the new master to
>> be up and running before a reply is returned, and that could take tenths of
>> seconds. The USR2 rexec solves that problem since both old and new workers are
>> accepting on the socket and we can kill the old ones when we are ready. In that
>> case the PIDFile trick is handy to support zero downtime restarts with no
>> latency.
>
>Is it possible to make systemd fire up two unicorn masters?
>That would be a nice feature to have with socket activation.

That would be great! I am a bit surprised that specifying multiple
services ('Service=' option) is not supported.



^ permalink raw reply	[relevance 23%]

* Re: [DRE-maint] unicorn: native systemd service
  2015-06-26 11:41 20%       ` Christos Trochalakis
@ 2015-06-27  3:05 20%         ` Eric Wong
  2015-06-30  9:20 23%           ` Christos Trochalakis
  0 siblings, 1 reply; 9+ results
From: Eric Wong @ 2015-06-27  3:05 UTC (permalink / raw)
  To: Christos Trochalakis
  Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public

Christos Trochalakis <yatiohi@ideopolis.gr> wrote:
> On Thu, Jun 25, 2015 at 11:26:26PM +0000, Eric Wong wrote:
> >With socket activation, you should just be able to kill unicorn using
> >SIGQUIT (just master, or even all workers) and restart without ever
> >dropping a connection.  I do NOT suggest using SIGTERM for unicorn,
> >since that'll cause the master to kill all workers ASAP.
> 
> Yes, you are right socket activation is also an option! I have made some
> experiments with a simple rack app to test it.
> 
> systemd uses the LISTEN_FDS env variable that is an integer indicating the
> number of inherited file descriptors. Those FDs have consecutive numbers
> starting from `SD_LISTEN_FDS_START` which is `3` (man sd_listen_fds).
> 
> So for example if LISTEN_FDS="2", UNICORN_FD should be "3,4". I used a
> simple wrapper script for that. Here is the full configuration:

OK, I'll probably add LISTEN_FDS and LISTEN_PID support to unicorn
directly so the wrapper is unnecessary.

<snip>

> KillMode=mixed

I don't think KillMode=mixed is necessary, here.  systemd can send
SIGQUIT to workers.

<snip>

> TCP socket options are not applied by unicorn on inherited sockets (TCPSocket
> === sock is false). systemd socket files have support for most options now but
> we might want unicorn to `setsockopt` them as well. For example,
> 'DeferAcceptSec', 'KeepAliveIntervalSec', 'NoDelay' are supported since v216, so
> they are not available in jessie (v215).

They are now :)

http://bogomips.org/unicorn-public/m/1435373879-4299-1-git-send-email-e@80x24.org.txt

We don't have KeepAliveIntervalSec
(aka TCP_KEEPINTVL) since it's not-portable and probably over
Are you sure about that?

> socket activation is a really interesting setup, but personally I would not run
> it with a large application. Clients would have to wait for the new master to
> be up and running before a reply is returned, and that could take tenths of
> seconds. The USR2 rexec solves that problem since both old and new workers are
> accepting on the socket and we can kill the old ones when we are ready. In that
> case the PIDFile trick is handy to support zero downtime restarts with no
> latency.

Is it possible to make systemd fire up two unicorn masters?
That would be a nice feature to have with socket activation.

^ permalink raw reply	[relevance 20%]

* Re: [DRE-maint] unicorn: native systemd service
  2015-06-25 23:26 20%     ` [DRE-maint] " Eric Wong
@ 2015-06-26 11:41 20%       ` Christos Trochalakis
  2015-06-27  3:05 20%         ` Eric Wong
  0 siblings, 1 reply; 9+ results
From: Christos Trochalakis @ 2015-06-26 11:41 UTC (permalink / raw)
  To: Eric Wong; +Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public

On Thu, Jun 25, 2015 at 11:26:26PM +0000, Eric Wong wrote:
>+Cc unicorn-public list
>Christos Trochalakis <christos@skroutz.gr> wrote:
>> Hello all,
>>
>> I have recently migrated our main ruby application to systemd implementing zero
>> downtime upgrades.
>>
>> systemd doesn't like replacing the binary on the fly. There is one exception to
>> this, services with PIDFile. When PIDFile is set, systemd reads it when the
>> main process exits and replaces the main process.  nginx also had this issue a
>> few months ago [0].
>>
>> So, in order to support zero-downtime upgrades we have to use a pid file.
>
>I don't think so.  You should be able to bind the listen socket in
>systemd and rely on the socket activation features (setting the
>UNICORN_FD environment variable to the created FD).
>
>You still need to have the matching "listen" directive in the unicorn
>config file so unicorn does not close it.
>
>With socket activation, you should just be able to kill unicorn using
>SIGQUIT (just master, or even all workers) and restart without ever
>dropping a connection.  I do NOT suggest using SIGTERM for unicorn,
>since that'll cause the master to kill all workers ASAP.
>

Yes, you are right socket activation is also an option! I have made some
experiments with a simple rack app to test it.

systemd uses the LISTEN_FDS env variable that is an integer indicating the
number of inherited file descriptors. Those FDs have consecutive numbers
starting from `SD_LISTEN_FDS_START` which is `3` (man sd_listen_fds).

So for example if LISTEN_FDS="2", UNICORN_FD should be "3,4". I used a
simple wrapper script for that. Here is the full configuration:

$ tail -n+1 /srv/uni/* /etc/systemd/system/uni.*

==> /srv/uni/config.ru <==
app = proc do |env|
  sleep 5
  [
    200,
    { 'Content-Type' => 'text/plain' },
    ["Socket Activated!\n", "pid:#{$$}\n", "ppid:#{Process.ppid}\n"]
  ]
end

run app

==> /srv/uni/unicorn.conf.rb <==
worker_processes 2
working_directory "/srv/uni"

# Keep in sync with uni.socket
listen 9000, :tcp_nopush => true
listen 9001, :tcp_nopush => true

==> /srv/uni/wrapper <==
#!/bin/bash

[ -z "$LISTEN_FDS" ] && exec $@

UNICORN_FD=""
for fd in `seq 3 $(($LISTEN_FDS+2))`; do
	UNICORN_FD="${UNICORN_FD}${fd},"
done
export UNICORN_FD

echo "wrapped fds: ${UNICORN_FD}"

exec $@

==> /etc/systemd/system/uni.service <==
[Unit]
Description=Unicorn Server

[Service]
ExecStart=/srv/uni/wrapper /usr/bin/unicorn -c /srv/uni/unicorn.conf.rb -d
KillSignal=SIGQUIT
KillMode=mixed

==> /etc/systemd/system/uni.socket <==
[Unit]
Description=Unicorn Socket

[Socket]
ListenStream=0.0.0.0:9000
ListenStream=0.0.0.0:9001

[Install]
WantedBy=sockets.target

Make sure to activate the systemd units:
chmod +x /srv/uni/wrapper
systemdctl daemon-reload
systemctl enable uni.socket
systemctl start  uni.socket

The application sleeps for 5secs before replying.

I run the following commands from 3 different terminals:

$ curl localhost:9000 [blocked for 5sec]
# systemctl stop uni.service [issues sigquit on the running unicorn, killing
                              the 2nd worker and waiting the 1st to finish]
$ curl localhost:9000 [blocked since there are no more workers to accept right now]

After the first request is served, unicorn dies and systemd respawns a new master.
The second request is accepted by the new master (notice the different ppid).

Some notes:

TCP socket options are not applied by unicorn on inherited sockets (TCPSocket
=== sock is false). systemd socket files have support for most options now but
we might want unicorn to `setsockopt` them as well. For example,
'DeferAcceptSec', 'KeepAliveIntervalSec', 'NoDelay' are supported since v216, so
they are not available in jessie (v215).

socket activation is a really interesting setup, but personally I would not run
it with a large application. Clients would have to wait for the new master to
be up and running before a reply is returned, and that could take tenths of
seconds. The USR2 rexec solves that problem since both old and new workers are
accepting on the socket and we can kill the old ones when we are ready. In that
case the PIDFile trick is handy to support zero downtime restarts with no
latency.


^ permalink raw reply	[relevance 20%]

* Re: unicorn: native systemd service
  2015-06-25 23:43 15%         ` Dmitry Smirnov
@ 2015-06-25 23:49 15%           ` Eric Wong
  0 siblings, 0 replies; 9+ results
From: Eric Wong @ 2015-06-25 23:49 UTC (permalink / raw)
  To: Dmitry Smirnov; +Cc: Hleb Valoshka, unicorn, unicorn-public

Dmitry Smirnov <onlyjob@debian.org> wrote:
> On Wed, 24 Jun 2015 23:26:49 Eric Wong wrote:
> > Dmitry: unicorn upstream here, is there anything in unicorn itself
> > can do to make systemd integration easier?
> 
> Thank you very much for keeping an eye on us and for all your help.
> I'm not sure if we need anything special -- let me experiment more with soft 
> restart and maybe I'll come back to you with questions. :)

No problem!  I basically live and breathe plain-text email,
so Debian is a good fit for me :)

> > I haven't had much interest or time to dig into systemd, but I'm willing
> > to put some effort into making unicorn work more smoothly with systemd
> > (and any other Free Software process managers/init systems).
> 
> Great. I'm entertaining idea of modifying init scripts to make soft restart 
> the default behaviour. Is there are any risks or drawbacks of always 
> attempting soft restart instead of normal restart? What do you think?

Probably not...  I almost regret making SIGTERM behave like nginx
instead of making SIGTERM identical to SIGQUIT.  OTOH, matching
nginx might make learning to use nginx (after learning unicorn signals)
easier.

I've certainly never used SIGTERM on unicorn or nginx myself.
Occasionally I've managed to break the RubyVM enough to require SIGKILL,
but I think using SIGTERM is unnecessary other than as "training wheels"
for nginx.

^ permalink raw reply	[relevance 15%]

* Re: unicorn: native systemd service
  2015-06-24 23:26 20%       ` unicorn: native systemd service Eric Wong
@ 2015-06-25 23:43 15%         ` Dmitry Smirnov
  2015-06-25 23:49 15%           ` Eric Wong
  0 siblings, 1 reply; 9+ results
From: Dmitry Smirnov @ 2015-06-25 23:43 UTC (permalink / raw)
  To: Eric Wong; +Cc: Hleb Valoshka, unicorn, unicorn-public

[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]

Hi Eric,

On Wed, 24 Jun 2015 23:26:49 Eric Wong wrote:
> Dmitry: unicorn upstream here, is there anything in unicorn itself
> can do to make systemd integration easier?

Thank you very much for keeping an eye on us and for all your help.
I'm not sure if we need anything special -- let me experiment more with soft 
restart and maybe I'll come back to you with questions. :)


> I haven't had much interest or time to dig into systemd, but I'm willing
> to put some effort into making unicorn work more smoothly with systemd
> (and any other Free Software process managers/init systems).

Great. I'm entertaining idea of modifying init scripts to make soft restart 
the default behaviour. Is there are any risks or drawbacks of always 
attempting soft restart instead of normal restart? What do you think?

-- 
Cheers,
 Dmitry Smirnov.

---

It is no use saying, 'We are doing our best.' You have got to succeed
in doing what is necessary.
        -- Winston Churchill


^ permalink raw reply	[relevance 15%]

* Re: [DRE-maint] unicorn: native systemd service
       [not found]       ` <20150625083118.GA22140@luke.ws.skroutz.gr>
@ 2015-06-25 23:26 20%     ` Eric Wong
  2015-06-26 11:41 20%       ` Christos Trochalakis
  0 siblings, 1 reply; 9+ results
From: Eric Wong @ 2015-06-25 23:26 UTC (permalink / raw)
  To: Christos Trochalakis
  Cc: Dmitry Smirnov, Hleb Valoshka, unicorn, unicorn-public

+Cc unicorn-public list 
Christos Trochalakis <christos@skroutz.gr> wrote:
> Hello all,
> 
> I have recently migrated our main ruby application to systemd implementing zero
> downtime upgrades.
> 
> systemd doesn't like replacing the binary on the fly. There is one exception to
> this, services with PIDFile. When PIDFile is set, systemd reads it when the
> main process exits and replaces the main process.  nginx also had this issue a
> few months ago [0].
> 
> So, in order to support zero-downtime upgrades we have to use a pid file.

I don't think so.  You should be able to bind the listen socket in
systemd and rely on the socket activation features (setting the
UNICORN_FD environment variable to the created FD).

You still need to have the matching "listen" directive in the unicorn
config file so unicorn does not close it.

With socket activation, you should just be able to kill unicorn using
SIGQUIT (just master, or even all workers) and restart without ever
dropping a connection.  I do NOT suggest using SIGTERM for unicorn,
since that'll cause the master to kill all workers ASAP.

If the master dies before the workers (e.g. from SIGKILL),
systemd ought to just ignore the workers.

I don't actually know the systemd config at all, but anybody familiar
with systemd socket activation should have no problems setting the
UNICORN_FD env.

^ permalink raw reply	[relevance 20%]

* Re: unicorn: native systemd service
       [not found]         ` <1810167.FXDESjsu8Z@debstor>
@ 2015-06-24 23:26 20%       ` Eric Wong
  2015-06-25 23:43 15%         ` Dmitry Smirnov
  0 siblings, 1 reply; 9+ results
From: Eric Wong @ 2015-06-24 23:26 UTC (permalink / raw)
  To: Dmitry Smirnov; +Cc: Hleb Valoshka, unicorn, unicorn-public

Adding unicorn-public@bogomips.org to Cc:

Those of you who haven't followed along on the unicorn@packages.debian.org
list can catch up here:
http://lists.alioth.debian.org/pipermail/pkg-ruby-extras-maintainers/2015-June/024539.html

Dmitry: unicorn upstream here, is there anything in unicorn itself
can do to make systemd integration easier?

I haven't had much interest or time to dig into systemd, but I'm willing
to put some effort into making unicorn work more smoothly with systemd
(and any other Free Software process managers/init systems).

So far unicorn.git has documented the UNICORN_FD environment variable in
the manpage in commit 548e1e67d314f6ebd17df37ece0ee20632462f6f [1]
and support SIGWINCH (kill children) in
commit a6077391bb62d0b13016084b0eea36b987afe8f0 [2]

I think that should be sufficient, but maybe there's more.
Thanks.

[1] http://bogomips.org/unicorn.git/patch/?id=548e1e67
[2] http://bogomips.org/unicorn.git/patch/?id=a6077391

^ permalink raw reply	[relevance 20%]

Results 1-9 of 9 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
     [not found]     <16714105.fRFKlev1fp@debstor>
     [not found]     ` <3137844.nsIDTWmvl4@debstor>
     [not found]       ` <CAAB-Kc=s-uSK83PcfDsybPRiYvTaLVxcu2x4eWDPaTf27AJ_XQ@mail.gmail.com>
     [not found]         ` <1810167.FXDESjsu8Z@debstor>
2015-06-24 23:26 20%       ` unicorn: native systemd service Eric Wong
2015-06-25 23:43 15%         ` Dmitry Smirnov
2015-06-25 23:49 15%           ` Eric Wong
     [not found]       ` <20150625083118.GA22140@luke.ws.skroutz.gr>
2015-06-25 23:26 20%     ` [DRE-maint] " Eric Wong
2015-06-26 11:41 20%       ` Christos Trochalakis
2015-06-27  3:05 20%         ` Eric Wong
2015-06-30  9:20 23%           ` Christos Trochalakis
2015-06-30 17:30 20%             ` Eric Wong
2015-07-08 13:08 24%               ` Christos Trochalakis

Code repositories for project(s) associated with this public inbox

	https://yhbt.net/unicorn.git/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).