about summary refs log tree commit homepage
DateCommit message (Collapse)
2011-12-08admin: fix off-by-one in each_fid method
list_fids was being used incorrectly :x
2011-12-08move Manifest.txt generation to Rake...
Bleh, I have revision control to tell me what's there, no need to maintain this file (or even manually generate it) myself
2011-12-08store_file works on unlinked Tempfile objects
Unlinking Tempfiles is good practice, so we should allow/encourage people to use unlinked Tempfiles with store_file.
2011-12-08new_file: add :largefile => :tempfile support
This is the most-compatible way to support largefiles with the new_file interface. This will unfortunately generate disk I/O, though... Also moving the "mog" util to use this with the "tee" subcommand since it already included its own private implementation of this before.
2011-12-08http_file: big_io support for unlinked open files
This makes it easier to use an unlinked Tempfile
2011-12-07add new_file :largefile => :content_range support
This is similar to the "largefile => 1" support in the Perl MogileFS::Client package. It requires net/http/persistent to avoid repeatedly setting up and tearing down a socket.
2011-12-07new_file gains a :largefile => :chunked option
This returns a new HTTPStream object that behaves like a writable IO object with the following methods: * write * print * printf * putc * puts * syswrite * << ..and also responds to IO.select (for writability)
2011-12-07test/fresh: unused variable warning fix
Might as well assert on it...
2011-12-07http_file: use lower keepalive times under Linux
TCP keepalives are inexpensive, so we can use them to monitor whether or not our connection is still alive while uploading. Remote servers make take an unpredictable amount of time to actually write out the data we've uploaded (and empty socket buffers to receive more), so it is extremely difficult to calculate an effective timeout for select() or poll().
2011-12-07test for client-side Content-MD5 callback on upload
Some applications will rely on this feature
2011-12-07Revert "http_file: disable MD5 checksums by default"
This reverts commit 11ee57e7aaa8766d13e29d3872bd78d50a3e5263. Bleh, checksums support is already off by default...
2011-12-06http_file: disable MD5 checksums by default
We won't force this onto the wire for trackers until upstream MogileFS decides to support it.
2011-12-06rely on SO_KEEPALIVE for upload timeouts
Users will be able to tweak these themselves in OS-dependent ways. It's probably better to rely on protocol-level timeouts when the HTTP server should be on a trusted network.
2011-12-06store_file/store_content: wire these up to new new_file opts
These allow us to specify Content-MD5 and checksums for use with the new and improved new_file interface without breaking existing apps.
2011-12-06new_file: backwards compatibility for old args
2011-12-06mog: "stat" shows checksum if it is available
Checksums are good, yes they are.
2011-12-06http_file: remove unused attrs
No need to clutter ourselves up with things we don't need.
2011-12-06"new_file" gets many options for Content-MD5 handling
This still needs docs and integration with store_content and store_file, but seems to be mostly working.
2011-12-06manifest updates for mogstored_rack tests
We need to test with this client, dependencies are hard :<
2011-12-06improve diagnostics on unreadable sockets with timeout
2011-12-06socket_common: small simplification in read() calculation
Minor cleanup, one less un-optimized method dispatch.
2011-12-06RequestTruncated exception includes timeout value used
It seems our timeout values are sometimes too low :<
2011-12-01remove mogstored_rack example
It's now a separate project: http://bogomips.org/mogstored_rack/
2011-11-28.document: add NEWS to rdoc
We generate it, might as well..
2011-11-28Ruby mogilefs-client 3.0.0 v3.0.0
Changes since 3.0.0-rc1: * 1.8 copy_stream emulator respects creation umask This matches the latest IO.copy_stream behavior. [ruby-core:41308], r33851 in ruby/trunk * higher timeouts for uploads to compensate for slow (but not dead) servers. Changes since v2.2.0: === client changes * "store_file" now accepts any IO object capable of streaming data (e.g. pipes and sockets). This uses chunked Transfer-Encoding for PUTs, so backend storage nodes will need to support this (latest mogstored does). * "store_file" no longer uses an infinite timeout when awaiting a response after a PUT, the new timeout for the response is now calculated based on the time and size of the PUT request. * new commands: "file_debug" and "file_info" (new commands in mogilefsd, be sure you have the latest version) * "get_paths" takes optional ":pathcount" parameter to control the number of returned paths. * "get_file_data" supports offset and count for partial transfer (requires support from storage node for Range: requests, most HTTP servers are capable of this) * IO.copy_stream is enabled by default under Ruby 1.9.3. Expect performance improvements. * "list_keys" with a passed block (for additional info) is faster due to internal pipelining implementation and the addition of "file_info" support. * fixed handling of "+" in key/domain names (old bug) * rare, truncated partial responses due to network/server failure now raise MogileFS::InvalidResponseError === admin changes Admin support is still a work-in-progress, I usually just find myself using "mogadm" anyways. * new admin commands: "replicate_now" * "get_stats" no longer works on new mogilefsd versions * get_domains handles "repl_policy" field correctly for classes in MogileFS 2.x * admin commands should convert all numeric fields to either Integer or Float objects and not String representations of numerics. Affected methods include: get_hosts, get_devices, list_fids, each_fids, get_domains === miscellany * we no longer add methods to standard Ruby classes (at least we never /changed/ existing methods :P) * {kgio}[http://bogomips.org/kgio] automatically used if available, but not required to avoid exceptions with non-blocking I/O * dropped Ruby 1.8.6 support, 1.8.7 or later is required. * Tested with MRI 1.8.7, 1.9.3 and Rubinius 1.2.4 * MogileFS::Mysql - deprecated, to be removed in 2012 * improved test suite * some optional experimental features/changes, see "git log" for details * Added internal pipelining implementation, this is not easy-to-use since the server can respond-out-of-order, but still useful for things like a list_keys+file_info loop.
2011-11-28test_mogilefs_integration: remove umask/permissions check on created file
It's not consistent between Ruby versions... oh well..
2011-11-28test_mogilefs: speling ficks
2011-11-26copy_stream emulator respects creation umask
This matches the latest IO.copy_stream behavior. [ruby-core:41308], r33851 in ruby/trunk
2011-11-25backend: small cleanup to use case statement
Easier for me to read, this way, and OK lines are usually more common than ERR lines :P
2011-11-24mogstored_rack: add ability to change :io_size and :open_flags
This allows folks to try IO::SYNC, and maybe even IO::DIRECT. Additionally, :io_size may be used to optimize IO::SYNC or make IO::DIRECT _work_. Removal of the "Tempfile" dependency means we no longer work reliably on NFS, but nobody sane puts MogileFS devices on NFS, anyways. (Actually, nobody sane uses NFS if atomicity is important :P)
2011-11-21http_file: timeout scaling for "write" method
Increase write timeouts as we become more vested in uploading to a socket. The remote server may take longer and longer to respond due to accumulated I/O delays if it is writing out to disk. We also don't need/want a "write" method in MogileFS::Socket, it's only for use with IO.copy_stream in HTTPFile.
2011-11-21mogilefs-client 3.0.0-rc1 v3.0.0-rc1
=== client changes * "store_file" now accepts any IO object capable of streaming data (e.g. pipes and sockets). This uses chunked Transfer-Encoding for PUTs, so backend storage nodes will need to support this (latest mogstored does). * "store_file" no longer uses an infinite timeout when awaiting a response after a PUT, the new timeout for the response is now calculated based on the time and size of the PUT request. * new commands: "file_debug" and "file_info" (new commands in mogilefsd, be sure you have the latest version) * "get_paths" takes optional ":pathcount" parameter to control the number of returned paths. * "get_file_data" supports offset and count for partial transfer (requires support from storage node for Range: requests, most HTTP servers are capable of this) * IO.copy_stream is enabled by default under Ruby 1.9.3. Expect performance improvements. * "list_keys" with a passed block (for additional info) is faster due to internal pipelining implementation and the addition of "file_info" support. * fixed handling of "+" in key/domain names (old bug) * rare, truncated partial responses due to network/server failure now raise MogileFS::InvalidResponseError === admin changes Admin support is still a work-in-progress, I usually just find myself using "mogadm" anyways. * new admin commands: "replicate_now" * "get_stats" no longer works on new mogilefsd versions * get_domains handles "repl_policy" field correctly for classes in MogileFS 2.x * admin commands should convert all numeric fields to either Integer or Float objects and not String representations of numerics. Affected methods include: get_hosts, get_devices, list_fids, each_fids, get_domains === miscellany * we no longer add methods to standard Ruby classes (at least we never /changed/ existing methods :P) * {kgio}[http://bogomips.org/kgio] automatically used if available, but not required to avoid exceptions with non-blocking I/O * dropped Ruby 1.8.6 support, 1.8.7 or later is required. * Tested with MRI 1.8.7, 1.9.3 and Rubinius 1.2.4 * MogileFS::Mysql - deprecated, to be removed in 2012 * improved test suite * some optional experimental features/changes, see "git log" for details * Added internal pipelining implementation, this is not easy-to-use since the server can respond-out-of-order, but still useful for things like a list_keys+file_info loop.
2011-11-21http_file: wait for the destination server to respond
Uploading large files can cause the destination to take equally long to respond, either because of extra I/O needed to verify or because we've hit fsync(2) or similar.
2011-11-21test/fresh: raise if mogadm fails to return writability
We don't want dependent tests to continue
2011-11-21mogstored_rack: fix Ruby 1.8 compatibility
Tempfile.open works differently, there.
2011-11-21test_mogilefs: fix bad test due to partial read
Ugh, shouldn't use sysread there
2011-11-20examples/mogstored_rack: fix temporary file path
We always want to use the destination directory as the temporary directory to avoid cross-device link/rename attempts.
2011-11-20add mogstored_rack example
This should allow any Rack server to become an HTTP server for mogstored. When using one of Unicorn/Rainbows!/Zbatery, the Content-MD5 HTTP Trailer will be supported. Otherwise, Content-MD5 can always be supported as a regular HTTP header (at the cost of requiring the client to read whatever they upload twice).
2011-11-20split out test for fresh mogilefsd instance testing
2011-11-20test_fresh: factor out mogstored setup
We'll support alternate mogstored setups...
2011-11-20test/exec: close-on-exec all Tempfiles
2011-11-19support MogileFS::MogileFS#get_file_data to dest path
Destination path will be created with 0600 permissions to be consistent with IO.copy_stream [ruby-core:41151]
2011-11-19test_admin: fix test due to broken mocks
integration tests > unit tests :P
2011-11-19mog: config parser handles "noclobber = true"
noclobber is false by default
2011-11-19mog: add --no-clobber/-n option for 'cp' and 'tee'
Overwriting data is bad, sometimes
2011-11-19add MogileFS::MogileFS#exist? method
Similar to File.exist? for checking the existence of a particular file.
2011-11-18admin (each_fid/list_fids): remove dependency on get_stats
The +to+ arg of list_fids is now actually +count+ internally because gaps may appear in an auto-incrementing FID range (due to deleted files). We also have a default +count+ of 100 (matches the internal MogileFS default).
2011-11-18backend: make url_decode easier to read (for me, at least)
I never learned to read Hash[] well. Also, String#freeze on keys to prevent Ruby from calling String#dup for us.
2011-11-18admin: add numeric conversions for consistency
Ruby isn't Perl, Ruby won't automatically coerce between Strings and numeric types (Integer and Floats)
2011-11-18admin (list_fids,each_fid): convert integer fields to integers
fid, devcount, length are all integers, so ensure they're actual Integer (Fixnum/Bignum) objects.