Re: Some benchmarks

[Eric Wong <normalperson@yhbt.net>]

Iñaki Baz Castillo <ibc@aliax.net> wrote:
> Hi, I've done some benchamarks comparing FileUtils vs io_splice when
> copying files in my computer (AMD 64 Phenom II X4 965):

<snip>
> Test data:
> - Source file size:  1156222 bytes
> - Number of iterations:  1000
> Results:
> - FileUtils.cp:            12.707785606384277
> - FileUtils.copy_stream:   13.745135068893433
> - IO.splice:               9.723489761352539
> 
> 
> The script is below.
> 
> It's clear that using io_splice is good for big files (or lot of
> copies from same small source file).
> 
> I don't understand why in the last test (big file, 1000 copies)
> io_splice takes so long, maybe it takes more time initializing each
> object within the benchmark block?

It'ls likely the test wrote enough to force blocking writes to disk,
and your disk is now the bottleneck.  In that case, all the memory
tricks in the world won't help :)

Try it on a tmpfs mount.

Also, relying on GC to close file descriptors could be a small
performance problem given the number of iterations.

> Script:
> ---------------------------------------------------------
> #!/usr/bin/ruby
> 
> require "fileutils"
> require "benchmark"
> require "io/splice"
> 
> 
> SRC_FILE = ARGV[0]
> DST_FILE = ARGV[1]
> TIMES = ( ARGV[2] ? ARGV[2].to_i : 1 )
> 
> 
> puts "Test data:"
> puts "- Source file size:  #{File.size(SRC_FILE)} bytes"
> puts "- Number of iterations:  #{TIMES}"
> 
> 
> puts "Results:"
> 
> print "- FileUtils.cp:            "
> puts Benchmark.realtime {
>  TIMES.times do
>    FileUtils.cp SRC_FILE, DST_FILE
>  end
> }
> 
> 
> print "- FileUtils.copy_stream:   "
> puts Benchmark.realtime {
>  TIMES.times do
>    FileUtils.copy_stream SRC_FILE, DST_FILE
>  end
> }
> 
> 
> print "- IO.splice:               "
> puts Benchmark.realtime {
>  TIMES.times do |n|
>    source = File.open(SRC_FILE, 'rb')
>    dest = File.open(DST_FILE + "_#{n}", 'wb')
>    source_fd = source.fileno
>    dest_fd = dest.fileno
> 
>    # We use a pipe as a ring buffer in kernel space.
>    # pipes may store up to IO::Splice::PIPE_CAPA bytes
>    pipe = IO.pipe

You can reuse the pipe object through multiple runs assuming you drain
it properly.

>    rfd, wfd = pipe.map { |io| io.fileno }
> 
>    begin
>      nread = begin
>        # first pull as many bytes as possible into the pipe
>        IO.splice(source_fd, nil, wfd, nil, IO::Splice::PIPE_CAPA, 0)
>      rescue EOFError
>        break
>      end
> 
>      # now move the contents of the pipe buffer into the destination file
>      # the copied data never enters userspace
>      nwritten = IO.splice(rfd, nil, dest_fd, nil, nread, 0)
> 
>      nwritten == nread or
>        abort "short splice to destination file: #{nwritten} != #{nread}"
>    end while true
>  end
> }
> ---------------------------------------------------------
> 
> I've tryed to declare source, source_fd and pipe = IO.pipe before the
> benchmark block but then I get empty copied files (I need to declare
> all of them within the benchmark block). I assume the test script can
> be improved.

You need to rewind source if you don't specify an input offset for splice.
Likewise for the dest and destination offset.  The open+close is part of
the test for the FileUtils things, though, so I would just open close
(you were leaving the close up to GC, which usually fine for MRI but not
for benchmark purposes).