HACKING
History
LICENSE
NEWS
README
TODO
MogileFS
Admin
Backend
Client
MogileFS
NewFile
NewFile::Writer
ReadOnlyError
Ruby mogilefs-client 3.12.2 / 2020-02-06 09:02 UTC
Only one documentation change to update URLs.
bogomips.org is expiring and with the price of the .org TLD
inevitably going up, I won't be able to afford to pay
extortionists at ICANN/Ethos/PIR.
Ruby mogilefs-client 3.12.1 / 2019-11-11 22:09 UTC
Minor cleanups, mainly to fix annoying warnings in Ruby 2.7.0dev.
4 changes since v3.12.0 (2019-01-02):
pkg.mk: use dark216 olddoc theme for Earth Day 2019
mogilefs: actually use kwarg for Net::HTTP::Persistent.new
fix more indentation warnings
test_client: quiet warnings about redefining accessors
Ruby mogilefs-client 3.12.0 / 2019-01-02 22:49 UTC
Minor cleanups, mainly to fix indentation warnings in Ruby 2.6+
7 changes since v3.11.1 (2017-03-23):
pool: simplify empty class definition
new_file/stream: remove unused variable
mog: add updateclass command
socket/pure_ruby: favor String#clear if available
fix indentation warnings with ruby trunk
doc: https:// URLs instead of git://
update URLs to point to wiki
mogilefs-client 3.11.1 / 2017-03-23 02:04 UTC
This release updates the IO#write garbage workaround to be
enabled for Ruby 2.0 and 2.1 users; but avoids it on the
just-released Ruby 2.4.1; as that includes the backported bugfix
for https://bugs.ruby-lang.org/issues/13085
There's also some trivial doc and code aesthetics fixes.
6 changes since 3.11.0:
.olddoc.yml: add trailing slash on URL
socket_common: expand IO#write garbage workaround to 2.0 and 2.1
socket_common: limit garbage workaround to Ruby <= 2.4.1
socket_common: remove needless 'o' modifier for Regexp
socket_common: improve readability of case statement
GNUmakefile: fix incorrect comment
Ruby mogilefs-client 3.11.0 / 2017-01-31 18:32 UTC
This release provides compatibility with net-http-persistent 3.x
while retaining support for the (API-incompatible) 2.x series.
Users of mainline Ruby 2.2+ get a workaround for excessive
garbage for uploading files. This excessive garbage on IO#write
should be fixed when Ruby 2.5 gets released on Dec 25, 2017:
https://bugs.ruby-lang.org/issues/13085
There's also a few minor garbage reductions of small strings,
relying on the maintained cmogstored for Content-Range PUT
testing, and some minor build system and documentation updates.
12 changes since 3.10.0 (Aug 2016):
avoid pointless capture in regexp
use opt_str_freeze to avoid allocations in a few places
new_file/common: simplify empty class declarations
support net-http-persistent 3.x
test: switch to cmogstored for testing Content-Range PUTs
avoid excessive garbage on uploads with Ruby 2.2+
socket_common: limit IO#write garbage workaround to <= 2.4
freeze string literals in a few more places
update URLS to HTTPS
README: explain removal of NFS support
doc: remove private email address linkage
gemspec: remove olddoc development dependency
Ruby mogilefs-client 3.10.0 / 2016-08-31 07:32 UTC
A bunch of minor tweaks to reduce garbage and exceptions.
In addition to the existing :timeout and :fail_timeout options,
there is a new :connect_timeout directive for all connection
classes which only controls the time spent waiting for a TCP
connection. This defaults to the same value as the existing
:timeout option (3 seconds); but users are advised to lower
it to match ideal network conditions.
For users of Ruby 2.3+, this release takes advantage of
exception-less Socket#connect_nonblock using "exception: false".
Users of Ruby 2.1+ will also benefit from "exception: false"
usage for read_nonblock and write_nonblock calls.
kgio is no longer be useful with this release with Ruby 2.3+
This release also fixes a Ruby 1.8.7 compatibility bug for
non-kgio users. However, this may be the last 1.8.7-compatible
release. Fwiw, I wanted to remove Ruby 1.8 support around 5
years ago but several users were against it. Maybe nobody will
complain, this time...
18 changes since 3.9.0:
doc: avoid inadvertantly documenting the Process class
admin: simplify utilization conversion
more idiomatic comparisons with constants
bigfile/filter: only update MD5 if non-nil
bigfile: lazily require bigfile/filter
backend: simplify regexp
.olddoc.yml: add NNTP and mailing list archive URL
minor garbage reductions for newer Rubies
socket/pure_ruby: fix Ruby 1.8 compatibility
socket/pure_ruby: use `:exception=>false' on Ruby 2.1+
test_fresh: do not delete non-existent domain
admin: map unset reject_bad_md5 field to nil
socket/pure_ruby: connect with "exception:false" on Ruby 2.3+
implement :connect_timeout option
add .gitattributes for Ruby method detection
README: stop mentioning cgit
connect_timeout: match :timeout if unset
pkg.mk: use --local option for gem installation
Ruby mogilefs-client 3.9.0 / 2015-06-25 11:12 UTC
shortlog of changes since v3.8.0:
* test/test_fresh.rb: fix breakage from Ruby-trunk r50118
* use monotonic clock if possible on Ruby 2.1+
* avoid defineclass instructions for empty classes
* HACKING: minor documentation updates
* mogilefs/socket/pure_ruby: use IO#wait_*able
Ruby mogilefs-client 3.8.0 / 2015-02-10 00:36 UTC
Mostly documentation updates and test cleanups. Nothing interesting
for the regular client, but the admin client now shows the
checksum-related fields in get_devices.
Most notably, there's a public-inbox for those who don't want
to subscribe to the MogileFS list:
mogilefs-client-public@bogomips.org
Archives are at: http://bogomips.org/mogilefs-client-public/
There is no new mailing list subscription, it is only a
public-inbox. Cc-ing the MogileFS list at mogile@googlegroups.com
is encouraged if you do have a subscription.
shortlog since 3.7.1:
add mog-sync example script
admin (doc, get_hosts): display integers in results doc
update documentation and packaging
GNUmakefile: publish examples on the site, too
examples: add usage_fetcher example
TODO: remove Cool.io and EventMachine references
admin: flesh out get_devices
admin: reduce bytecode overhead of get_stats
tests: create fresh intances for all integration tests
include mogilefs/version.rb in the distro
test: fixup object lifetimes and teardown
new public-inbox: mogilefs-client-public@bogomips.org
Ruby mogilefs-client 3.7.1 / 2013-09-10 23:35 UTC
Only one bugfix:
new_file/stream: correct declare errors array correctly
This only triggered on rare syscall errors
(Errno::EMFILE/Errno::ENFILE)
Ruby mogilefs-client 3.7.0 / 2013-07-18 06:40 UTC
MogileFS::MogileFS#each_key now accepts optional :after and :limit args
This allows easy iteration while respecting :after and :limit,
giving each_key an interface consistent with each_file_info.
Ruby mogilefs-client 3.6.0 / 2013-02-21 20:33 UTC
We no longer send an empty zone= parameter in create_open
calls. Additionally, if we detect send() failure on a socket,
assume a tracker has been restarted and restart the request.
There are also some minor code cleanups.
mogilefs-client 3.5.0 / 2012-12-04 11:09 UTC
client changes:
* new_file gains :create_open_args and :create_close_args which
allows custom arguments to be passed to plugins.
(this matches the Perl client behavior)
* new_file also gains :info hash which can be populated with
information normally retrieved with file_info
* users with net-http-persistent installed will see a small speed
boost and reduction of TIME_WAIT sockets when dealing with
small data (useful for testing DB-intensive parts of MogileFS).
No changes for admin.
mogilefs-client 3.5.0-rc1 / 2012-10-31 19:53 UTC
client changes:
* new_file gains :create_open_args and :create_close_args which
allows custom arguments to be passed to plugins.
* new_file also gains :info hash which can be populated with
information normally retrieved with file_info
* users with net-http-persistent installed will see a small speed
boost and reduction of TIME_WAIT sockets when dealing with
small data (useful for testing DB-intensive parts of MogileFS).
No changes for admin.
Ruby mogilefs-client 3.4.0 / 2012-10-09 03:06 UTC
Admin speedups for get_domains, get_hosts, and get_devices.
Previous versions were completely unusable for parsing a list of
3000+ domains. Installations with thousands of hosts or devices
should see noticeable performance improvements.
The client interface gains the each_file_info iterator method.
This behaves like each_key, but gives access to the entire
response the file_info returns, including: checksum, devcount,
file size, class, and domain.
Ruby mogilefs-client 3.3.0 / 2012-08-11 11:12 UTC
No changes since 3.3.0-rc1.
We now correctly timeout requests to slow trackers and avoid
reusing the socket after timeouts (as old responses can arrive
late). A big thanks to David Rasch for helping with
timeout/idempotency issues for this release.
Minor documentation/packaging updates, it should be easier
to build a pre-release gem now (see HACKING doc).
Ruby mogilefs-client 3.3.0-rc1 / 2012-08-03 21:50 UTC
We now correctly timeout requests to slow trackers and avoid
reusing the socket after timeouts (as old responses can arrive
late). A big thanks to David Rasch for helping with
timeout/idempotency issues for this release.
Minor documentation/packaging updates, it should be easier
to build a pre-release gem now (see HACKING doc).
Ruby mogilefs-client 3.2.0 / 2012-06-29 22:28 UTC
Changes since 3.2.0-rc1 (MogileFS::Admin-only):
* admin supports "create_device" and "change_device_weight" commands
* admin casts "reject_bad_md5" field in "get_device" return
value to boolean (true/false).
* minor code simplifications to admin
Changes since 3.1.1
* "list_keys" and "exist?" client commands raise errors properly
on failure.
* backend connections no longer terminate on ERR responses, only
on socket/connection errors.
* support the "updateclass" client command. This is for updating
the class of a given key and not to be confused with the
"update_class" admin command.
* "new_file" checksum usage is now documented since MogileFS 2.60
includes official support for checksums
Ruby mogilefs-client 3.2.0-rc1 / 2012-06-15 01:05 UTC
* "list_keys" and "exist?" client commands raise errors properly
on failure.
* backend connections no longer terminate on ERR responses, only
on socket/connection errors.
* support the "updateclass" client command. This is for updating
the class of a given key and not to be confused with the
"update_class" admin command.
* "new_file" checksum usage is now documented since MogileFS 2.60
includes official support for checksums
Ruby mogilefs-client 3.1.1 / 2012-02-28 22:12 UTC
This releases fixes problems short reads when slurping files
into memory. Thanks to Matthew Draper for this fix.
There are also minor documentation updates.
Ruby mogilefs-client 3.1.0 / 2011-12-15 04:10 UTC
* improved API support for uploading large files
While we've always supported uploading large files, the
(still-supported) existing APIs were somewhat awkward or
required the file to exist on the file system. See
MogileFS::NewFile for details and examples.
* more informative exception messages for timed-out requests
* :fail_timeout parameter, the timeout for retrying a failed
tracker connection. This defaults to 5 seconds (same as
previous versions where this was hard-coded.
* :new_file_max_time parameter
Controls the maximum of time spent creating and uploading
a new file in MogileFS. This defaults to 1 hour (which
matching the expiry time of a row in the MogileFS internal
tempfile table).
* store_file works on unlinked File/Tempfile objects
* each_fid method in MogileFS::Admin fixed
* stale_fid_checker example script added
* mogstored_rack example split into a standalone RubyGem:
http://bogomips.org/mogstored_rack/
* backend error constants are generated on const_missing,
instead of when raised, making it easier to rescue exceptions
we didn't explicitly enable
* some internal cleanups and documentation improvements
Ruby mogilefs-client 3.0.0 / 2011-11-28 20:53 UTC
Changes since 3.0.0-rc1:
* 1.8 copy_stream emulator respects creation umask
This matches the latest IO.copy_stream behavior.
[ruby-core:41308], r33851 in ruby/trunk
* higher timeouts for uploads to compensate for slow
(but not dead) servers.
Changes since v2.2.0:
=== client changes
* "store_file" now accepts any IO object capable of streaming
data (e.g. pipes and sockets). This uses chunked
Transfer-Encoding for PUTs, so backend storage nodes
will need to support this (latest mogstored does).
* "store_file" no longer uses an infinite timeout when
awaiting a response after a PUT, the new timeout for the
response is now calculated based on the time and size of
the PUT request.
* new commands: "file_debug" and "file_info" (new commands
in mogilefsd, be sure you have the latest version)
* "get_paths" takes optional ":pathcount" parameter
to control the number of returned paths.
* "get_file_data" supports offset and count for partial
transfer (requires support from storage node for
Range: requests, most HTTP servers are capable of this)
* IO.copy_stream is enabled by default under Ruby 1.9.3.
Expect performance improvements.
* "list_keys" with a passed block (for additional info) is
faster due to internal pipelining implementation and the
addition of "file_info" support.
* fixed handling of "+" in key/domain names (old bug)
* rare, truncated partial responses due to network/server
failure now raise MogileFS::InvalidResponseError
=== admin changes
Admin support is still a work-in-progress,
I usually just find myself using "mogadm" anyways.
* new admin commands: "replicate_now"
* "get_stats" no longer works on new mogilefsd versions
* get_domains handles "repl_policy" field correctly for classes
in MogileFS 2.x
* admin commands should convert all numeric fields to either
Integer or Float objects and not String representations
of numerics. Affected methods include:
get_hosts, get_devices, list_fids, each_fids, get_domains
=== miscellany
* we no longer add methods to standard Ruby classes
(at least we never /changed/ existing methods :P)
* {kgio}[http://bogomips.org/kgio] automatically used
if available, but not required to avoid exceptions
with non-blocking I/O
* dropped Ruby 1.8.6 support, 1.8.7 or later is required.
* Tested with MRI 1.8.7, 1.9.3 and Rubinius 1.2.4
* MogileFS::Mysql - deprecated, to be removed in 2012
* improved test suite
* some optional experimental features/changes,
see "git log" for details
* Added internal pipelining implementation, this is
not easy-to-use since the server can respond-out-of-order,
but still useful for things like a list_keys+file_info
loop.
mogilefs-client 3.0.0-rc1 / 2011-11-21 02:48 UTC
=== client changes
* "store_file" now accepts any IO object capable of streaming
data (e.g. pipes and sockets). This uses chunked
Transfer-Encoding for PUTs, so backend storage nodes
will need to support this (latest mogstored does).
* "store_file" no longer uses an infinite timeout when
awaiting a response after a PUT, the new timeout for the
response is now calculated based on the time and size of
the PUT request.
* new commands: "file_debug" and "file_info" (new commands
in mogilefsd, be sure you have the latest version)
* "get_paths" takes optional ":pathcount" parameter
to control the number of returned paths.
* "get_file_data" supports offset and count for partial
transfer (requires support from storage node for
Range: requests, most HTTP servers are capable of this)
* IO.copy_stream is enabled by default under Ruby 1.9.3.
Expect performance improvements.
* "list_keys" with a passed block (for additional info) is
faster due to internal pipelining implementation and the
addition of "file_info" support.
* fixed handling of "+" in key/domain names (old bug)
* rare, truncated partial responses due to network/server
failure now raise MogileFS::InvalidResponseError
=== admin changes
Admin support is still a work-in-progress,
I usually just find myself using "mogadm" anyways.
* new admin commands: "replicate_now"
* "get_stats" no longer works on new mogilefsd versions
* get_domains handles "repl_policy" field correctly for classes
in MogileFS 2.x
* admin commands should convert all numeric fields to either
Integer or Float objects and not String representations
of numerics. Affected methods include:
get_hosts, get_devices, list_fids, each_fids, get_domains
=== miscellany
* we no longer add methods to standard Ruby classes
(at least we never /changed/ existing methods :P)
* {kgio}[http://bogomips.org/kgio] automatically used
if available, but not required to avoid exceptions
with non-blocking I/O
* dropped Ruby 1.8.6 support, 1.8.7 or later is required.
* Tested with MRI 1.8.7, 1.9.3 and Rubinius 1.2.4
* MogileFS::Mysql - deprecated, to be removed in 2012
* improved test suite
* some optional experimental features/changes,
see "git log" for details
* Added internal pipelining implementation, this is
not easy-to-use since the server can respond-out-of-order,
but still useful for things like a list_keys+file_info
loop.
mogilefs-client 2.2.0 / 2011-01-13 02:10 UTC
= 2.2.0
* internal cleanups (no public API breakage)
* refactor backend socket/connection handling for reliability
There'll probably be more aggressive cleanups/refactoring in
future releases if I have time.
mogilefs-client 2.1.0 / 2009-04-12 03:04 UTC
= 2.1.0
* MySQL interface returns integer length and devcount (API change)
* Ensure store_{content,file} always returns size (API fix)
* Add get_uris API method
* Respect timeout when doing get_file_data
* MySQL interface filters out URLs for down/dead hosts/devices
* Really remove all NFS support
* get_file in slurp mode slurps all output correctly
Eric Wong (18):
tests: retry random ports correctly
test_mogilefs: fix race conditions
MySQL interface returns integer length and devcount
Respect timeout when doing get_file_data
Unify internal HTTP GET/HEAD methods
Close socket we create before raising exceptions
Compact get_paths output so we don't have nils
Really remove all NFS support
README: add a note about emailing me
Extra checking for full_timeout in sysread_full
Add get_uris API method
verify_uris: use write_nonblock instead of syswrite
README: add links to the repo.or.cz mirror
mysql: filter out URLs for down/dead hosts/devices
mog: small cleanup
Ensure store_{content,file} always returns size
GNUmakefile: better logging/output control
mogilefs-client 2.1.0
godfat (1):
call IO.select([sock]) if sock is not ready.
mogilefs-client 1.2.1 / 2009-03-25 07:29 UTC
Retroactively tagging this old release since
it was in SVN or Perforce...
mogilefs-client 2.0.2 / 2009-03-02 21:56 UTC
initial
mogilefs-client v2.0.1 / 2009-02-25 08:13 UTC
* verify_uris method made more robust
* preliminary Ruby 1.9 compatibility, tests still need some work
* allow store_content to be used with a streamable object of known length
* add setup.rb for non-RubyGems users
mogilefs-client 2.0.0 / 2009-02-03 04:27 UTC
initial
v1.3.1 / 2008-10-03 03:01 UTC
* Fix missing MogileFS::Util include for sysrwloop in MogileFS::MogileFS
v1.3.0 / 2008-09-18 23:46 UTC
* Fixed MogileFS#rename. Bug #14465 submitted by Justin Dossey.
* Removed infinite loop in MogileFS::HTTPFile#store_file. Patch #13789
submitted by Andy Lo-A-Foe.
* Made MogileFS#get_file_data timeout configurable. Bug #13490 submitted by
Andy Lo-A-Foe.
* Add MogileFS#size. Feature Request #14484 submitted by Justin Dossey.
* Fix MogileFS#get_file_data to return the data for HTTP mode. Bug #7133
submitted by John Wanko.
* New maintainer: Eric Wong
* Add `mog' command-line tool as a demo/example
* Lower memory consumption with large files
* Allow get_file_data to accept a block for large files
* Fix each_keys loop termination condition
* Apply error handling patch from Matthew Willson. Bug #15987
* Merge large file patch from Andy Lo-A-Foe. Bug #13764
mail archives: https://yhbt.net/mogilefs-client-public/
http://ou63pmih66umazou.onion/mogilefs-client-public/
nntp://news.public-inbox.org/inbox.comp.file-systems.mogilefs.ruby
nntp://ou63pmih66umazou.onion/inbox.comp.file-systems.mogilefs.ruby
public: mogilefs-client-public@yhbt.net
source code: git clone https://yhbt.net/mogilefs-client.git
torsocks git clone http://ou63pmih66umazou.onion/mogilefs-client.git