Date | Commit message (Collapse) |
|
list_fids was being used incorrectly :x
|
|
Bleh, I have revision control to tell me what's there, no need
to maintain this file (or even manually generate it) myself
|
|
Unlinking Tempfiles is good practice, so we should
allow/encourage people to use unlinked Tempfiles
with store_file.
|
|
This is the most-compatible way to support largefiles
with the new_file interface. This will unfortunately
generate disk I/O, though...
Also moving the "mog" util to use this with the "tee"
subcommand since it already included its own private
implementation of this before.
|
|
This makes it easier to use an unlinked Tempfile
|
|
This is similar to the "largefile => 1" support in the
Perl MogileFS::Client package. It requires net/http/persistent
to avoid repeatedly setting up and tearing down a socket.
|
|
This returns a new HTTPStream object that behaves
like a writable IO object with the following methods:
* write
* print
* printf
* putc
* puts
* syswrite
* <<
..and also responds to IO.select (for writability)
|
|
Might as well assert on it...
|
|
TCP keepalives are inexpensive, so we can use them to monitor
whether or not our connection is still alive while uploading.
Remote servers make take an unpredictable amount of time to
actually write out the data we've uploaded (and empty socket
buffers to receive more), so it is extremely difficult to
calculate an effective timeout for select() or poll().
|
|
Some applications will rely on this feature
|
|
This reverts commit 11ee57e7aaa8766d13e29d3872bd78d50a3e5263.
Bleh, checksums support is already off by default...
|
|
We won't force this onto the wire for trackers until upstream
MogileFS decides to support it.
|
|
Users will be able to tweak these themselves in OS-dependent
ways. It's probably better to rely on protocol-level timeouts
when the HTTP server should be on a trusted network.
|
|
These allow us to specify Content-MD5 and checksums for use
with the new and improved new_file interface without breaking
existing apps.
|
|
|
|
Checksums are good, yes they are.
|
|
No need to clutter ourselves up with things we don't need.
|
|
This still needs docs and integration with store_content and
store_file, but seems to be mostly working.
|
|
We need to test with this client, dependencies are hard :<
|
|
|
|
Minor cleanup, one less un-optimized method dispatch.
|
|
It seems our timeout values are sometimes too low :<
|
|
It's now a separate project:
http://bogomips.org/mogstored_rack/
|
|
We generate it, might as well..
|
|
Changes since 3.0.0-rc1:
* 1.8 copy_stream emulator respects creation umask
This matches the latest IO.copy_stream behavior.
[ruby-core:41308], r33851 in ruby/trunk
* higher timeouts for uploads to compensate for slow
(but not dead) servers.
Changes since v2.2.0:
=== client changes
* "store_file" now accepts any IO object capable of streaming
data (e.g. pipes and sockets). This uses chunked
Transfer-Encoding for PUTs, so backend storage nodes
will need to support this (latest mogstored does).
* "store_file" no longer uses an infinite timeout when
awaiting a response after a PUT, the new timeout for the
response is now calculated based on the time and size of
the PUT request.
* new commands: "file_debug" and "file_info" (new commands
in mogilefsd, be sure you have the latest version)
* "get_paths" takes optional ":pathcount" parameter
to control the number of returned paths.
* "get_file_data" supports offset and count for partial
transfer (requires support from storage node for
Range: requests, most HTTP servers are capable of this)
* IO.copy_stream is enabled by default under Ruby 1.9.3.
Expect performance improvements.
* "list_keys" with a passed block (for additional info) is
faster due to internal pipelining implementation and the
addition of "file_info" support.
* fixed handling of "+" in key/domain names (old bug)
* rare, truncated partial responses due to network/server
failure now raise MogileFS::InvalidResponseError
=== admin changes
Admin support is still a work-in-progress,
I usually just find myself using "mogadm" anyways.
* new admin commands: "replicate_now"
* "get_stats" no longer works on new mogilefsd versions
* get_domains handles "repl_policy" field correctly for classes
in MogileFS 2.x
* admin commands should convert all numeric fields to either
Integer or Float objects and not String representations
of numerics. Affected methods include:
get_hosts, get_devices, list_fids, each_fids, get_domains
=== miscellany
* we no longer add methods to standard Ruby classes
(at least we never /changed/ existing methods :P)
* {kgio}[http://bogomips.org/kgio] automatically used
if available, but not required to avoid exceptions
with non-blocking I/O
* dropped Ruby 1.8.6 support, 1.8.7 or later is required.
* Tested with MRI 1.8.7, 1.9.3 and Rubinius 1.2.4
* MogileFS::Mysql - deprecated, to be removed in 2012
* improved test suite
* some optional experimental features/changes,
see "git log" for details
* Added internal pipelining implementation, this is
not easy-to-use since the server can respond-out-of-order,
but still useful for things like a list_keys+file_info
loop.
|
|
It's not consistent between Ruby versions... oh well..
|
|
|
|
This matches the latest IO.copy_stream behavior.
[ruby-core:41308], r33851 in ruby/trunk
|
|
Easier for me to read, this way, and OK lines are usually more
common than ERR lines :P
|
|
This allows folks to try IO::SYNC, and maybe even IO::DIRECT.
Additionally, :io_size may be used to optimize IO::SYNC or
make IO::DIRECT _work_.
Removal of the "Tempfile" dependency means we no longer work
reliably on NFS, but nobody sane puts MogileFS devices on
NFS, anyways. (Actually, nobody sane uses NFS if atomicity
is important :P)
|
|
Increase write timeouts as we become more vested in uploading to
a socket. The remote server may take longer and longer to
respond due to accumulated I/O delays if it is writing out
to disk.
We also don't need/want a "write" method in MogileFS::Socket,
it's only for use with IO.copy_stream in HTTPFile.
|
|
=== client changes
* "store_file" now accepts any IO object capable of streaming
data (e.g. pipes and sockets). This uses chunked
Transfer-Encoding for PUTs, so backend storage nodes
will need to support this (latest mogstored does).
* "store_file" no longer uses an infinite timeout when
awaiting a response after a PUT, the new timeout for the
response is now calculated based on the time and size of
the PUT request.
* new commands: "file_debug" and "file_info" (new commands
in mogilefsd, be sure you have the latest version)
* "get_paths" takes optional ":pathcount" parameter
to control the number of returned paths.
* "get_file_data" supports offset and count for partial
transfer (requires support from storage node for
Range: requests, most HTTP servers are capable of this)
* IO.copy_stream is enabled by default under Ruby 1.9.3.
Expect performance improvements.
* "list_keys" with a passed block (for additional info) is
faster due to internal pipelining implementation and the
addition of "file_info" support.
* fixed handling of "+" in key/domain names (old bug)
* rare, truncated partial responses due to network/server
failure now raise MogileFS::InvalidResponseError
=== admin changes
Admin support is still a work-in-progress,
I usually just find myself using "mogadm" anyways.
* new admin commands: "replicate_now"
* "get_stats" no longer works on new mogilefsd versions
* get_domains handles "repl_policy" field correctly for classes
in MogileFS 2.x
* admin commands should convert all numeric fields to either
Integer or Float objects and not String representations
of numerics. Affected methods include:
get_hosts, get_devices, list_fids, each_fids, get_domains
=== miscellany
* we no longer add methods to standard Ruby classes
(at least we never /changed/ existing methods :P)
* {kgio}[http://bogomips.org/kgio] automatically used
if available, but not required to avoid exceptions
with non-blocking I/O
* dropped Ruby 1.8.6 support, 1.8.7 or later is required.
* Tested with MRI 1.8.7, 1.9.3 and Rubinius 1.2.4
* MogileFS::Mysql - deprecated, to be removed in 2012
* improved test suite
* some optional experimental features/changes,
see "git log" for details
* Added internal pipelining implementation, this is
not easy-to-use since the server can respond-out-of-order,
but still useful for things like a list_keys+file_info
loop.
|
|
Uploading large files can cause the destination to take
equally long to respond, either because of extra I/O
needed to verify or because we've hit fsync(2) or similar.
|
|
We don't want dependent tests to continue
|
|
Tempfile.open works differently, there.
|
|
Ugh, shouldn't use sysread there
|
|
We always want to use the destination directory as the
temporary directory to avoid cross-device link/rename
attempts.
|
|
This should allow any Rack server to become an HTTP server for
mogstored.
When using one of Unicorn/Rainbows!/Zbatery, the Content-MD5
HTTP Trailer will be supported. Otherwise, Content-MD5 can
always be supported as a regular HTTP header (at the cost of
requiring the client to read whatever they upload twice).
|
|
|
|
We'll support alternate mogstored setups...
|
|
|
|
Destination path will be created with 0600 permissions to
be consistent with IO.copy_stream [ruby-core:41151]
|
|
integration tests > unit tests :P
|
|
noclobber is false by default
|
|
Overwriting data is bad, sometimes
|
|
Similar to File.exist? for checking the existence of
a particular file.
|
|
The +to+ arg of list_fids is now actually +count+ internally
because gaps may appear in an auto-incrementing FID range
(due to deleted files). We also have a default +count+ of
100 (matches the internal MogileFS default).
|
|
I never learned to read Hash[] well. Also, String#freeze on
keys to prevent Ruby from calling String#dup for us.
|
|
Ruby isn't Perl, Ruby won't automatically coerce between
Strings and numeric types (Integer and Floats)
|
|
fid, devcount, length are all integers, so ensure they're
actual Integer (Fixnum/Bignum) objects.
|