raindrops tests fail on smp, fork is to blame

raindrops tests fail on smp, fork is to blame

[Hleb Valoshka]

Hello!

Raindrops build hangs on debian's build servers, but not on my local
box. This occurs when tests are run under ruby1.9.1. You can see
example of build log in [0]. It took a lot of time to understand that
the main difference between build and my box is number of CPUs:
debian's servers are smp, my kvm box had only 1 cpu.

Today I decided to start kvm with option -smp 2, and voila! build
hung. The more tests with fork() you have the higher probability build
will hang. When all tests are enabled, the probability of hang is
about 100% when tests are run by gem2deb (it loads all test files
simultaneously) and about 5-10% with "make test". When test_concurrect
from test_middleware.rb and test_incr_shared, test_incr_decr and
test_evaporate_with_fork from test_raindrops.rb are disabled build
runs without hangs.

Any ideas?

[0] https://buildd.debian.org/status/fetch.php?pkg=ruby-raindrops&arch=kfreebsd-amd64&ver=0.9.0-1&stamp=1340389966

Re: raindrops tests fail on smp, fork is to blame

[Eric Wong]

Hleb Valoshka <375gnu@gmail.com> wrote:
> Hello!
> 
> Raindrops build hangs on debian's build servers, but not on my local
> box. This occurs when tests are run under ruby1.9.1. You can see
> example of build log in [0]. It took a lot of time to understand that
> the main difference between build and my box is number of CPUs:
> debian's servers are smp, my kvm box had only 1 cpu.
> 
> Today I decided to start kvm with option -smp 2, and voila! build
> hung. The more tests with fork() you have the higher probability build
> will hang. When all tests are enabled, the probability of hang is
> about 100% when tests are run by gem2deb (it loads all test files
> simultaneously) and about 5-10% with "make test". When test_concurrect
> from test_middleware.rb and test_incr_shared, test_incr_decr and
> test_evaporate_with_fork from test_raindrops.rb are disabled build
> runs without hangs.
> 
> Any ideas?

This is only on GNU/kFreeBSD, right?

I managed to reproduce this on my older Debian GNU/kFreeBSD 6.0 KVM
instance, too.

I suspect it's a bad interaction with the internal timer thread of Ruby
1.9 and fork().  Probably some mutex is held when the fork() happens
and attempting to acquire it in the child process causes a deadlock.

Even without loading raindrops, the following code locks up for
me with ruby1.9.1 1.9.2.0-2 on my Debian GNU/kFreeBSD 6.0 instance:

  1000.times do
    pid = fork do
      1 + 1
    end
    Process.waitpid2(pid)
  end

Can you try it on 1.9.3-p194 on your newer wheezy/sid install?
I think some libc folks will need to be roped into this fix this.

Re: raindrops tests fail on smp, fork is to blame

[Hleb Valoshka]

On 6/23/12, Eric Wong <normalperson@yhbt.net> wrote:

> This is only on GNU/kFreeBSD, right?

Yes, only kFreeBSD, I didn't notice that I forgot to mention this fact.

> Can you try it on 1.9.3-p194 on your newer wheezy/sid install?

It locks always. Sometimes after 2 first forks, onсe just after the first fork.

> I think some libc folks will need to be roped into this fix this.

Okay, I'll send your code to gnu/kfreebsd guys to use it as test case.