This releases fixes problems short reads when slurping files into memory. Thanks to Matthew Draper for this fix. There are also minor documentation updates.
* improved API support for uploading large files While we've always supported uploading large files, the (still-supported) existing APIs were somewhat awkward or required the file to exist on the file system. See MogileFS::NewFile for details and examples. * more informative exception messages for timed-out requests * :fail_timeout parameter, the timeout for retrying a failed tracker connection. This defaults to 5 seconds (same as previous versions where this was hard-coded. * :new_file_max_time parameter Controls the maximum of time spent creating and uploading a new file in MogileFS. This defaults to 1 hour (which matching the expiry time of a row in the MogileFS internal tempfile table). * store_file works on unlinked File/Tempfile objects * each_fid method in MogileFS::Admin fixed * stale_fid_checker example script added * mogstored_rack example split into a standalone RubyGem: http://bogomips.org/mogstored_rack/ * backend error constants are generated on const_missing, instead of when raised, making it easier to rescue exceptions we didn't explicitly enable * some internal cleanups and documentation improvements
Changes since 3.0.0-rc1:
* 1.8 copy_stream emulator respects creation umask
This matches the latest IO.copy_stream behavior.
[ruby-core:41308], r33851 in ruby/trunk
* higher timeouts for uploads to compensate for slow
(but not dead) servers.
Changes since v2.2.0:
=== client changes
* "store_file" now accepts any IO object capable of streaming
data (e.g. pipes and sockets). This uses chunked
Transfer-Encoding for PUTs, so backend storage nodes
will need to support this (latest mogstored does).
* "store_file" no longer uses an infinite timeout when
awaiting a response after a PUT, the new timeout for the
response is now calculated based on the time and size of
the PUT request.
* new commands: "file_debug" and "file_info" (new commands
in mogilefsd, be sure you have the latest version)
* "get_paths" takes optional ":pathcount" parameter
to control the number of returned paths.
* "get_file_data" supports offset and count for partial
transfer (requires support from storage node for
Range: requests, most HTTP servers are capable of this)
* IO.copy_stream is enabled by default under Ruby 1.9.3.
Expect performance improvements.
* "list_keys" with a passed block (for additional info) is
faster due to internal pipelining implementation and the
addition of "file_info" support.
* fixed handling of "+" in key/domain names (old bug)
* rare, truncated partial responses due to network/server
failure now raise MogileFS::InvalidResponseError
=== admin changes
Admin support is still a work-in-progress,
I usually just find myself using "mogadm" anyways.
* new admin commands: "replicate_now"
* "get_stats" no longer works on new mogilefsd versions
* get_domains handles "repl_policy" field correctly for classes
in MogileFS 2.x
* admin commands should convert all numeric fields to either
Integer or Float objects and not String representations
of numerics. Affected methods include:
get_hosts, get_devices, list_fids, each_fids, get_domains
=== miscellany
* we no longer add methods to standard Ruby classes
(at least we never /changed/ existing methods :P)
* {kgio}[http://bogomips.org/kgio] automatically used
if available, but not required to avoid exceptions
with non-blocking I/O
* dropped Ruby 1.8.6 support, 1.8.7 or later is required.
* Tested with MRI 1.8.7, 1.9.3 and Rubinius 1.2.4
* MogileFS::Mysql - deprecated, to be removed in 2012
* improved test suite
* some optional experimental features/changes,
see "git log" for details
* Added internal pipelining implementation, this is
not easy-to-use since the server can respond-out-of-order,
but still useful for things like a list_keys+file_info
loop.
=== client changes
* "store_file" now accepts any IO object capable of streaming
data (e.g. pipes and sockets). This uses chunked
Transfer-Encoding for PUTs, so backend storage nodes
will need to support this (latest mogstored does).
* "store_file" no longer uses an infinite timeout when
awaiting a response after a PUT, the new timeout for the
response is now calculated based on the time and size of
the PUT request.
* new commands: "file_debug" and "file_info" (new commands
in mogilefsd, be sure you have the latest version)
* "get_paths" takes optional ":pathcount" parameter
to control the number of returned paths.
* "get_file_data" supports offset and count for partial
transfer (requires support from storage node for
Range: requests, most HTTP servers are capable of this)
* IO.copy_stream is enabled by default under Ruby 1.9.3.
Expect performance improvements.
* "list_keys" with a passed block (for additional info) is
faster due to internal pipelining implementation and the
addition of "file_info" support.
* fixed handling of "+" in key/domain names (old bug)
* rare, truncated partial responses due to network/server
failure now raise MogileFS::InvalidResponseError
=== admin changes
Admin support is still a work-in-progress,
I usually just find myself using "mogadm" anyways.
* new admin commands: "replicate_now"
* "get_stats" no longer works on new mogilefsd versions
* get_domains handles "repl_policy" field correctly for classes
in MogileFS 2.x
* admin commands should convert all numeric fields to either
Integer or Float objects and not String representations
of numerics. Affected methods include:
get_hosts, get_devices, list_fids, each_fids, get_domains
=== miscellany
* we no longer add methods to standard Ruby classes
(at least we never /changed/ existing methods :P)
* {kgio}[http://bogomips.org/kgio] automatically used
if available, but not required to avoid exceptions
with non-blocking I/O
* dropped Ruby 1.8.6 support, 1.8.7 or later is required.
* Tested with MRI 1.8.7, 1.9.3 and Rubinius 1.2.4
* MogileFS::Mysql - deprecated, to be removed in 2012
* improved test suite
* some optional experimental features/changes,
see "git log" for details
* Added internal pipelining implementation, this is
not easy-to-use since the server can respond-out-of-order,
but still useful for things like a list_keys+file_info
loop.
= 2.2.0 * internal cleanups (no public API breakage) * refactor backend socket/connection handling for reliability There'll probably be more aggressive cleanups/refactoring in future releases if I have time.
= 2.1.0
* MySQL interface returns integer length and devcount (API change)
* Ensure store_{content,file} always returns size (API fix)
* Add get_uris API method
* Respect timeout when doing get_file_data
* MySQL interface filters out URLs for down/dead hosts/devices
* Really remove all NFS support
* get_file in slurp mode slurps all output correctly
Eric Wong (18):
tests: retry random ports correctly
test_mogilefs: fix race conditions
MySQL interface returns integer length and devcount
Respect timeout when doing get_file_data
Unify internal HTTP GET/HEAD methods
Close socket we create before raising exceptions
Compact get_paths output so we don't have nils
Really remove all NFS support
README: add a note about emailing me
Extra checking for full_timeout in sysread_full
Add get_uris API method
verify_uris: use write_nonblock instead of syswrite
README: add links to the repo.or.cz mirror
mysql: filter out URLs for down/dead hosts/devices
mog: small cleanup
Ensure store_{content,file} always returns size
GNUmakefile: better logging/output control
mogilefs-client 2.1.0
godfat (1):
call IO.select([sock]) if sock is not ready.
Retroactively tagging this old release since it was in SVN or Perforce...
initial
* verify_uris method made more robust * preliminary Ruby 1.9 compatibility, tests still need some work * allow store_content to be used with a streamable object of known length * add setup.rb for non-RubyGems users
initial
* Fix missing MogileFS::Util include for sysrwloop in MogileFS::MogileFS
* Fixed MogileFS#rename. Bug #14465 submitted by Justin Dossey. * Removed infinite loop in MogileFS::HTTPFile#store_file. Patch #13789 submitted by Andy Lo-A-Foe. * Made MogileFS#get_file_data timeout configurable. Bug #13490 submitted by Andy Lo-A-Foe. * Add MogileFS#size. Feature Request #14484 submitted by Justin Dossey. * Fix MogileFS#get_file_data to return the data for HTTP mode. Bug #7133 submitted by John Wanko. * New maintainer: Eric Wong * Add `mog' command-line tool as a demo/example * Lower memory consumption with large files * Allow get_file_data to accept a block for large files * Fix each_keys loop termination condition * Apply error handling patch from Matthew Willson. Bug #15987 * Merge large file patch from Andy Lo-A-Foe. Bug #13764
Originally generated with the Darkfish Rdoc Generator 2, modified by wrongdoc.