ChangeLog from bogomips.org/mogilefs-client.git (v1.2.1..v3.1.1)
commit cea1724c77e4af0f2db2e694114aff9f51ff91f7
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Feb 28 22:10:15 2012 +0000
Ruby mogilefs-client 3.1.1
This releases fixes problems short reads when slurping files
into memory. Thanks to Matthew Draper for this fix.
There are also minor documentation updates.
commit 7d7d30329e806d38350cd8eba161d3157e5bbca4
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Feb 28 14:04:21 2012 -0800
doc: update to not refer to MogileFS 2.55 specifically
It's more future-proof, this way.
commit 6faffa9ba774bad500ca48cc52449dae0639227b
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Feb 28 14:02:54 2012 -0800
LICENSE: update copyright year + range
Not that it really matters given the MIT license...
commit d758083954ee480bde5b079cfd3f7b4fcf0f9bf7
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Feb 28 14:01:01 2012 -0800
doc: fixup mailto: links
It's better if email addresses are visible and not hidden behind
links for folks that don't integrate email clients into web
browsers.
commit 8a3f48341248b919b5c4816bf4f7079eb35d6882
Author: Matthew Draper <matthew@trebex.net>
Date: Tue Feb 28 23:39:09 2012 +1030
Don't stop reading too early.
Leave size unchanged, so it's available for easy comparison with
buf.bytesize.
[ew: added test case]
Acked-by: Eric Wong <normalperson@yhbt.net>
commit 55365e01dc0b2b5db01441004f4c6921d1fd04d6
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Feb 17 12:17:50 2012 -0800
fix documentation of :noverify for get_paths
:noverify always defaults to _true_ (meaning verification is
_off_ by default). This double negative is unfortunately
a confusing part of the existing API and MogileFS protocol.
commit 83305f7e52f95d9aea63de2afa0cf8f2616c2ced
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 15 05:17:23 2011 +0000
README: speling ficks
commit 6c05d25e740a5150e0c51b4a8d741b1be45ff78c
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 15 04:08:25 2011 +0000
Ruby mogilefs-client 3.1.0
* improved API support for uploading large files
While we've always supported uploading large files, the
(still-supported) existing APIs were somewhat awkward or
required the file to exist on the file system. See
MogileFS::NewFile for details and examples.
* more informative exception messages for timed-out requests
* :fail_timeout parameter, the timeout for retrying a failed
tracker connection. This defaults to 5 seconds (same as
previous versions where this was hard-coded.
* :new_file_max_time parameter
Controls the maximum of time spent creating and uploading
a new file in MogileFS. This defaults to 1 hour (which
matching the expiry time of a row in the MogileFS internal
tempfile table).
* store_file works on unlinked File/Tempfile objects
* each_fid method in MogileFS::Admin fixed
* stale_fid_checker example script added
* mogstored_rack example split into a standalone RubyGem:
http://bogomips.org/mogstored_rack/
* backend error constants are generated on const_missing,
instead of when raised, making it easier to rescue exceptions
we didn't explicitly enable
* some internal cleanups and documentation improvements
commit d199947566d5783ce2b56338d45d262712ff7151
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 15 04:03:33 2011 +0000
docs for various timeouts in MogileFS::MogileFS.new
We have lots of timeouts :x
commit 70a529d070920abf0989d3d89d775ea1aa774042
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Dec 12 09:22:35 2011 +0000
backend: rely on auto-generated exceptions in const_missing
Most of these exceptions are just too rare to be useful,
and code that cares for exceptions will rescue them
anyways.
commit c1cf4444b2ca9126e8450ae3224cd406f0e43624
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 14 17:37:23 2011 -0800
add configurable :fail_timeout for backends
This controls the retry timeout of a failed backend
commit 0b933fc83b8b519c8e587caa6606dd372dda18af
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Dec 12 07:56:00 2011 +0000
avoid relying on exceptions for list_keys
This will make debug output quieter.
commit 594295a29cbc33fc04b7c0b87ecf664f036819a9
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Dec 12 07:48:36 2011 +0000
simplify exist? implementation for avoiding exceptions
This adds an internal :ruby_no_raise flag to the backend
to avoid exceptions.
commit 8a54653184536b99bf74bcaedb8cf84ea0f4f693
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Dec 11 22:25:12 2011 +0000
net-http-persistent not required for Content-Range uploads
Performance with Content-Range uploads sucks either way, so it's
not noticeably worse off on a LAN /without/ persistent
connections.
commit b1e3b9f570c04100f73f63ceaff6ab07f938476a
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Dec 11 22:03:09 2011 +0000
examples/stale_fid_checker: --help + fail on older versions
each_fids was too buggy and likely to yield false positives
commit 81f4cf3eda8f42040149da4ad267adf6adcf3e83
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Dec 11 09:13:41 2011 +0000
History: point users towards NEWS doc
NEWS is more common in the wider Free Software community
commit 246bc4139262d4575b3a7b45e7a08bc93ff1a975
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Dec 11 09:05:08 2011 +0000
chunker: small cleanups to whitespace handling
The client-provided Content-MD5 may have leading/trailing
whitespace, so we shall strip that before comparing.
We also know any base64-encoded MD5 we generate will have only
trailing whitespace so we can get away with the (slightly)
faster and more-GC-friendly String#rstrip!
commit 3f342091cec9fcd5f99ba934444f17cd0c77de71
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Dec 11 08:47:39 2011 +0000
add :new_file_max_time parameter
This defaults to 1 hour (which matches the expiry time of a row
in the MogileFS internal tempfile table).
commit 9d5f3208dab86837106b56f105af8bbb121f438a
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Dec 11 08:33:00 2011 +0000
misc. documentation improvements
100% RDoc coverage (not that it's an indicator of _good_
documentation, but we're getting there)
commit 56e7c8eecf59b037a778ad1121b2ffd93258482f
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Dec 11 08:32:03 2011 +0000
examples: add stale_fid_checker script
This logic may go upstream into the tracker Fsck worker itself,
but until then, we can try this from the client side.
commit b69111a6573826d7c4fc3edd4fcc514d0eb425e9
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Dec 9 18:22:01 2011 -0800
finalize and document improved new_file API
commit 7f153071e40b1242ad0aa46c81ec0e927b9fc890
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Dec 9 16:23:48 2011 -0800
rename "Put" namespace to "NewFile"
This should make documentation easier
commit 5326214033b4f26ec426a0313c164f5de0225207
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Dec 9 23:50:05 2011 +0000
tests: speedup tests that require hitting mogstored
By pre-creating devN dirs, mogstored can create usage files
sooner and we won't have to wait 10s between "df" runs.
commit 83aa5dede6f0f9b04a631d0875d3e4d4f7927a8c
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Dec 9 11:35:25 2011 +0000
new_file_common: call "noop" before "create_close"
An upload could've taken a long time, ping and try to ensure
socket is valid to minimize (but not completely eliminate) the
chance create_close hits a stale socket (while reading the
response after writing to it) and becomes non-retryable. We
treat create_close specially as its less idempotent than any
other command (even other non-idempotent ones). There may be no
hope of retrying the upload at all if data was streamed and
calling create_close twice will hurt us...
commit 01c200204400c4373c5b5edeb11d1c70a8119795
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 18:09:38 2011 -0800
pool: shutdown backend when purging
If we started a connection, make sure it's stopped and the file
descriptor closed. Relying on GC to close file descriptors is
unreliable with some GC implementations.
commit 00d7a3cca62b19c34df66b337582a7c15c4b4848
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 17:59:26 2011 -0800
pool: O(1) object validity checking
Now that pools can be arbitrarily large, using an O(n) check
for object validity can be expensive.
commit 3e80aa8b62dd1b9e1d78e4c1c487f1ea5ee162cd
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 17:34:25 2011 -0800
pool: add @purge_threshold, @purge_keep accessors
Instead of hard coding the thresholds, allow
users to change accessors
commit f62e34251c1101b0fcdddea35dfa3f73c416a3ba
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 16:57:08 2011 -0800
reorganize largefile support code
Putting this into the MogileFS::Put::* namespace will
make it easier to document.
commit 50f9287610725257bf64781a2008398ff6ceeacb
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 15:53:09 2011 -0800
new_file(..., :largefile => :stream) avoids chunking
This allows us to stream files with a known Content-Length
into MogileFS. This can be useful for streaming an HTTP
download into MogileFS without:
* saving it to the filesystem
* relying on chunked encoding support on the server
commit a867de1e6bdf50bb7cb4316a7ecf3a9a38c9029b
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 21:06:51 2011 +0000
Rakefile: quiet down warnings for newer rake
commit 7a355fad7ef4932a160d2144b7e6a2c9f64c4c9c
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 21:01:02 2011 +0000
move Manifest.txt from git to .gitignore
It is auto-generated and goes into .gitignore
commit 4bba2b2ab0d6a00403afe8bc7575f74d45393bf9
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 21:00:20 2011 +0000
mog: config parser cleanup
commit e87d69883d050dfb24525a89bcd340250dad6991
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 20:49:57 2011 +0000
backend: auto-generate error constants on const_missing
This way, folks can rescue for exceptions we haven't defined,
yet.
commit 5c824e04d5de04b4d6c8fb58dfc60e4d3f6cd7c7
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 11:11:37 2011 +0000
admin: fix off-by-one in each_fid method
list_fids was being used incorrectly :x
commit ec4e5984a62cb95ef75f3004aa8932a9cc6ceb6c
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 09:19:17 2011 +0000
move Manifest.txt generation to Rake...
Bleh, I have revision control to tell me what's there, no need
to maintain this file (or even manually generate it) myself
commit 346b21f715e484bf9280d2d6b7ac2003b29a59ed
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 06:15:06 2011 +0000
store_file works on unlinked Tempfile objects
Unlinking Tempfiles is good practice, so we should
allow/encourage people to use unlinked Tempfiles
with store_file.
commit 83df70292440787ff32be893f7dd0fd2713bfdef
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 03:39:35 2011 +0000
new_file: add :largefile => :tempfile support
This is the most-compatible way to support largefiles
with the new_file interface. This will unfortunately
generate disk I/O, though...
Also moving the "mog" util to use this with the "tee"
subcommand since it already included its own private
implementation of this before.
commit 581148337d8345a09b32bf7c82f051eb438b6383
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 8 03:38:55 2011 +0000
http_file: big_io support for unlinked open files
This makes it easier to use an unlinked Tempfile
commit ad559b036966ba5797ac53a70e17342640d07952
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 7 11:09:37 2011 +0000
add new_file :largefile => :content_range support
This is similar to the "largefile => 1" support in the
Perl MogileFS::Client package. It requires net/http/persistent
to avoid repeatedly setting up and tearing down a socket.
commit 492bf768036b1c6287d013e37535618ebdeb8833
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 7 05:20:55 2011 +0000
new_file gains a :largefile => :chunked option
This returns a new HTTPStream object that behaves
like a writable IO object with the following methods:
* write
* print
* printf
* putc
* puts
* syswrite
* <<
..and also responds to IO.select (for writability)
commit 86318ca3e9ede3bac245a7bdcb31b9c3ad15ef67
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 7 05:18:39 2011 +0000
test/fresh: unused variable warning fix
Might as well assert on it...
commit 0bbbd8cfa2df665bc22a47831adf91668e391e3e
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 7 02:03:05 2011 +0000
http_file: use lower keepalive times under Linux
TCP keepalives are inexpensive, so we can use them to monitor
whether or not our connection is still alive while uploading.
Remote servers make take an unpredictable amount of time to
actually write out the data we've uploaded (and empty socket
buffers to receive more), so it is extremely difficult to
calculate an effective timeout for select() or poll().
commit da58f47f541cff65e814a614ecff722ba2a35bf8
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 7 01:29:43 2011 +0000
test for client-side Content-MD5 callback on upload
Some applications will rely on this feature
commit d7bec4c903bd08337b7dbf4611e618762050ed58
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 7 01:24:24 2011 +0000
Revert "http_file: disable MD5 checksums by default"
This reverts commit 11ee57e7aaa8766d13e29d3872bd78d50a3e5263.
Bleh, checksums support is already off by default...
commit 11ee57e7aaa8766d13e29d3872bd78d50a3e5263
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 6 17:06:08 2011 -0800
http_file: disable MD5 checksums by default
We won't force this onto the wire for trackers until upstream
MogileFS decides to support it.
commit d74fad403f23c1ee36b9fbf276d12b12ef5d92d1
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 6 13:25:22 2011 -0800
rely on SO_KEEPALIVE for upload timeouts
Users will be able to tweak these themselves in OS-dependent
ways. It's probably better to rely on protocol-level timeouts
when the HTTP server should be on a trusted network.
commit 296977523d04e184b954f5a00e9463dea0d8e8d6
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 6 20:54:07 2011 +0000
store_file/store_content: wire these up to new new_file opts
These allow us to specify Content-MD5 and checksums for use
with the new and improved new_file interface without breaking
existing apps.
commit 2c4dc76c6a4840865c83177ceeb217b859e20df8
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 6 20:54:06 2011 +0000
new_file: backwards compatibility for old args
commit db55f4b60ccb312881d84eea8c15daf115a7f1bd
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 6 20:54:05 2011 +0000
mog: "stat" shows checksum if it is available
Checksums are good, yes they are.
commit f5d512fe1b163e132d1698138cff83cb4a59658c
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 6 20:54:04 2011 +0000
http_file: remove unused attrs
No need to clutter ourselves up with things we don't need.
commit ad6113719d37824bcad6b8e78edafd838c9f8a78
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 6 20:54:03 2011 +0000
"new_file" gets many options for Content-MD5 handling
This still needs docs and integration with store_content and
store_file, but seems to be mostly working.
commit 652398d0def7291ea7a4e54c3e418d88038ed436
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 6 13:01:02 2011 -0800
manifest updates for mogstored_rack tests
We need to test with this client, dependencies are hard :<
commit 829d459f7335cb6eb6a2f6d24e0d6f984cef2d52
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 6 20:53:42 2011 +0000
improve diagnostics on unreadable sockets with timeout
commit 9ef18521688bb008e173bfcca57f4abcd012205f
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 6 20:54:09 2011 +0000
socket_common: small simplification in read() calculation
Minor cleanup, one less un-optimized method dispatch.
commit bafa6f41dbfda371e248656e50441bc7f6503826
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 6 20:54:08 2011 +0000
RequestTruncated exception includes timeout value used
It seems our timeout values are sometimes too low :<
commit d5f49b90ec7461b07e88eb2e13f9b510190be256
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 1 03:03:52 2011 +0000
remove mogstored_rack example
It's now a separate project:
http://bogomips.org/mogstored_rack/
commit c9f94ea9102fe7e88a6c2e4ae1065c31908f9eda
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 28 21:23:01 2011 +0000
.document: add NEWS to rdoc
We generate it, might as well..
commit bbd2522467c29a9e7d97870be2a0171fd8ee58dd
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 28 20:39:01 2011 +0000
Ruby mogilefs-client 3.0.0
Changes since 3.0.0-rc1:
* 1.8 copy_stream emulator respects creation umask
This matches the latest IO.copy_stream behavior.
[ruby-core:41308], r33851 in ruby/trunk
* higher timeouts for uploads to compensate for slow
(but not dead) servers.
Changes since v2.2.0:
=== client changes
* "store_file" now accepts any IO object capable of streaming
data (e.g. pipes and sockets). This uses chunked
Transfer-Encoding for PUTs, so backend storage nodes
will need to support this (latest mogstored does).
* "store_file" no longer uses an infinite timeout when
awaiting a response after a PUT, the new timeout for the
response is now calculated based on the time and size of
the PUT request.
* new commands: "file_debug" and "file_info" (new commands
in mogilefsd, be sure you have the latest version)
* "get_paths" takes optional ":pathcount" parameter
to control the number of returned paths.
* "get_file_data" supports offset and count for partial
transfer (requires support from storage node for
Range: requests, most HTTP servers are capable of this)
* IO.copy_stream is enabled by default under Ruby 1.9.3.
Expect performance improvements.
* "list_keys" with a passed block (for additional info) is
faster due to internal pipelining implementation and the
addition of "file_info" support.
* fixed handling of "+" in key/domain names (old bug)
* rare, truncated partial responses due to network/server
failure now raise MogileFS::InvalidResponseError
=== admin changes
Admin support is still a work-in-progress,
I usually just find myself using "mogadm" anyways.
* new admin commands: "replicate_now"
* "get_stats" no longer works on new mogilefsd versions
* get_domains handles "repl_policy" field correctly for classes
in MogileFS 2.x
* admin commands should convert all numeric fields to either
Integer or Float objects and not String representations
of numerics. Affected methods include:
get_hosts, get_devices, list_fids, each_fids, get_domains
=== miscellany
* we no longer add methods to standard Ruby classes
(at least we never /changed/ existing methods :P)
* {kgio}[http://bogomips.org/kgio] automatically used
if available, but not required to avoid exceptions
with non-blocking I/O
* dropped Ruby 1.8.6 support, 1.8.7 or later is required.
* Tested with MRI 1.8.7, 1.9.3 and Rubinius 1.2.4
* MogileFS::Mysql - deprecated, to be removed in 2012
* improved test suite
* some optional experimental features/changes,
see "git log" for details
* Added internal pipelining implementation, this is
not easy-to-use since the server can respond-out-of-order,
but still useful for things like a list_keys+file_info
loop.
commit f854b9b8a10649d1898b840a8659a69dfee693ab
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 28 20:49:01 2011 +0000
test_mogilefs_integration: remove umask/permissions check on created file
It's not consistent between Ruby versions... oh well..
commit 46e6a9058535b41a27aaac7ee98e84a335f0fade
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 28 20:48:24 2011 +0000
test_mogilefs: speling ficks
commit bc9dc23cc4552fe46f0473cf208169c41e6892aa
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 26 19:31:49 2011 +0000
copy_stream emulator respects creation umask
This matches the latest IO.copy_stream behavior.
[ruby-core:41308], r33851 in ruby/trunk
commit d806fa67899e697a2d399eff0553e9c3b60e484e
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 25 05:46:08 2011 +0000
backend: small cleanup to use case statement
Easier for me to read, this way, and OK lines are usually more
common than ERR lines :P
commit 0fce97034959a605e66ebd645bf4a1957bfed870
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 24 05:49:12 2011 +0000
mogstored_rack: add ability to change :io_size and :open_flags
This allows folks to try IO::SYNC, and maybe even IO::DIRECT.
Additionally, :io_size may be used to optimize IO::SYNC or
make IO::DIRECT _work_.
Removal of the "Tempfile" dependency means we no longer work
reliably on NFS, but nobody sane puts MogileFS devices on
NFS, anyways. (Actually, nobody sane uses NFS if atomicity
is important :P)
commit 6a1164a0ededd7f80ac3b493c78c0fa1b9b8b91b
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 21 20:39:59 2011 +0000
http_file: timeout scaling for "write" method
Increase write timeouts as we become more vested in uploading to
a socket. The remote server may take longer and longer to
respond due to accumulated I/O delays if it is writing out
to disk.
We also don't need/want a "write" method in MogileFS::Socket,
it's only for use with IO.copy_stream in HTTPFile.
commit 64b3335c8b41ca77395f7eb7bbd7d9b0a62bc8c7
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 21 01:02:15 2011 +0000
mogilefs-client 3.0.0-rc1
=== client changes
* "store_file" now accepts any IO object capable of streaming
data (e.g. pipes and sockets). This uses chunked
Transfer-Encoding for PUTs, so backend storage nodes
will need to support this (latest mogstored does).
* "store_file" no longer uses an infinite timeout when
awaiting a response after a PUT, the new timeout for the
response is now calculated based on the time and size of
the PUT request.
* new commands: "file_debug" and "file_info" (new commands
in mogilefsd, be sure you have the latest version)
* "get_paths" takes optional ":pathcount" parameter
to control the number of returned paths.
* "get_file_data" supports offset and count for partial
transfer (requires support from storage node for
Range: requests, most HTTP servers are capable of this)
* IO.copy_stream is enabled by default under Ruby 1.9.3.
Expect performance improvements.
* "list_keys" with a passed block (for additional info) is
faster due to internal pipelining implementation and the
addition of "file_info" support.
* fixed handling of "+" in key/domain names (old bug)
* rare, truncated partial responses due to network/server
failure now raise MogileFS::InvalidResponseError
=== admin changes
Admin support is still a work-in-progress,
I usually just find myself using "mogadm" anyways.
* new admin commands: "replicate_now"
* "get_stats" no longer works on new mogilefsd versions
* get_domains handles "repl_policy" field correctly for classes
in MogileFS 2.x
* admin commands should convert all numeric fields to either
Integer or Float objects and not String representations
of numerics. Affected methods include:
get_hosts, get_devices, list_fids, each_fids, get_domains
=== miscellany
* we no longer add methods to standard Ruby classes
(at least we never /changed/ existing methods :P)
* {kgio}[http://bogomips.org/kgio] automatically used
if available, but not required to avoid exceptions
with non-blocking I/O
* dropped Ruby 1.8.6 support, 1.8.7 or later is required.
* Tested with MRI 1.8.7, 1.9.3 and Rubinius 1.2.4
* MogileFS::Mysql - deprecated, to be removed in 2012
* improved test suite
* some optional experimental features/changes,
see "git log" for details
* Added internal pipelining implementation, this is
not easy-to-use since the server can respond-out-of-order,
but still useful for things like a list_keys+file_info
loop.
commit 126019445776ed08dd33d60e146a9d2e5786cf7c
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 21 00:06:23 2011 +0000
http_file: wait for the destination server to respond
Uploading large files can cause the destination to take
equally long to respond, either because of extra I/O
needed to verify or because we've hit fsync(2) or similar.
commit a292a04a7e8c7cf65b6f9a7462c97b10010e4877
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 21 00:05:56 2011 +0000
test/fresh: raise if mogadm fails to return writability
We don't want dependent tests to continue
commit fbf389f5e1bb0742726e8b3923074a72c410db10
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 21 00:05:33 2011 +0000
mogstored_rack: fix Ruby 1.8 compatibility
Tempfile.open works differently, there.
commit d3c08764deb3345c5bdc34b35d2dd2a2d8a4771e
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 21 00:05:02 2011 +0000
test_mogilefs: fix bad test due to partial read
Ugh, shouldn't use sysread there
commit a73444af13ce8ac9cd94347c627f8191b3a80634
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 20 11:16:30 2011 +0000
examples/mogstored_rack: fix temporary file path
We always want to use the destination directory as the
temporary directory to avoid cross-device link/rename
attempts.
commit 21fc20b7d798c3eab6155b24dcb58c95b53ef856
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 20 06:29:26 2011 +0000
add mogstored_rack example
This should allow any Rack server to become an HTTP server for
mogstored.
When using one of Unicorn/Rainbows!/Zbatery, the Content-MD5
HTTP Trailer will be supported. Otherwise, Content-MD5 can
always be supported as a regular HTTP header (at the cost of
requiring the client to read whatever they upload twice).
commit 6018860fe20c82daa131cb9e30dba228c862c540
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 20 03:09:33 2011 +0000
split out test for fresh mogilefsd instance testing
commit 9922005b9da00d4709202d87b434ca93c6b53a01
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 20 02:22:19 2011 +0000
test_fresh: factor out mogstored setup
We'll support alternate mogstored setups...
commit 7f721b1608882afa15c28be0ec6071fb1e4e5872
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 20 02:21:44 2011 +0000
test/exec: close-on-exec all Tempfiles
commit f8156e239ea4da4cbcd020f24bf06c4fb9e7cde7
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 19 08:45:07 2011 +0000
support MogileFS::MogileFS#get_file_data to dest path
Destination path will be created with 0600 permissions to
be consistent with IO.copy_stream [ruby-core:41151]
commit 381044eeda55a5ca7edcccd87d27c30a287723e1
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 19 00:18:31 2011 +0000
test_admin: fix test due to broken mocks
integration tests > unit tests :P
commit d2c175aaafa9a06c377de5df609759073d93e894
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 19 00:13:28 2011 +0000
mog: config parser handles "noclobber = true"
noclobber is false by default
commit 01f65edf9594fdd811c5671a429b98ffff6f6523
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 19 00:10:15 2011 +0000
mog: add --no-clobber/-n option for 'cp' and 'tee'
Overwriting data is bad, sometimes
commit cbce24009023b7cbd751c8c027281673d0cdaa27
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 19 00:00:07 2011 +0000
add MogileFS::MogileFS#exist? method
Similar to File.exist? for checking the existence of
a particular file.
commit 2ff4eacf708a3f2e42140a34568bf3792c063618
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 18 23:14:08 2011 +0000
admin (each_fid/list_fids): remove dependency on get_stats
The +to+ arg of list_fids is now actually +count+ internally
because gaps may appear in an auto-incrementing FID range
(due to deleted files). We also have a default +count+ of
100 (matches the internal MogileFS default).
commit e6edebe4288c98827e723d75fe9fea4c1a0b013f
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 18 22:49:55 2011 +0000
backend: make url_decode easier to read (for me, at least)
I never learned to read Hash[] well. Also, String#freeze on
keys to prevent Ruby from calling String#dup for us.
commit b4b3ec9ac07dad1aab6da1d3ba9c26c4b7959dc2
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 18 22:45:58 2011 +0000
admin: add numeric conversions for consistency
Ruby isn't Perl, Ruby won't automatically coerce between
Strings and numeric types (Integer and Floats)
commit 8d2e9c8a10cb6a53874e47a30b63e194f12d80fb
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 18 22:04:47 2011 +0000
admin (list_fids,each_fid): convert integer fields to integers
fid, devcount, length are all integers, so ensure they're
actual Integer (Fixnum/Bignum) objects.
commit 7498d798dd6b274383e9e0d6d40ed2b725129227
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 18 21:44:56 2011 +0000
admin: get_stats is deprecated and gone from latest MogileFS
Newer versions of mogilefsd do not support this command,
use mogstats(1) from MogileFS::Utils (on CPAN) instead.
commit 1d891c7d0c300444c8327222fdee95cd3174c667
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 18 10:50:25 2011 +0000
admin: add clear_cache command
Currently a no-op in MogileFS 2.55, but it was useful at one
point in the past and may be in the future.
commit 4074889c30690a3effef4fe49fa5a75f7d6c8e51
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 18 10:23:50 2011 +0000
admin: add support for replicate_now command
commit 0c86accfccef5637823574d575831c9b3c5a752f
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 18 03:03:35 2011 +0000
mogilefs: documentation update
* re-document exception classes
* latest upstream MogileFS is 2.55
commit 0f22b7030f5ba9fc033f19b850356d9e826eb792
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 18 03:00:20 2011 +0000
make MogileFS.io= an accessor
In case somebody wants to use this to override the
default class for copy_stream in Ruby. It's also
easier to read, this way.
commit b7d6538453082e92d4ab98cbbf9c9db7aa7c70a7
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 18 02:41:08 2011 +0000
TODO: add manpage item for "mog"
Maybe it'll actually get used.
commit d2c3306bf731e37b4ff05c7790df6434ee326afe
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 17 09:57:48 2011 +0000
mog: improve "stat" subcommand
* return error if there was a missing key
* use file_info to help us determine info
* show all URLs for a given key
commit 53279166865f7cd41976b7235713ce58c83e7dd1
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 17 09:49:51 2011 +0000
mog: fix shadow warnings
Found with "ruby -w" on 1.9.3
commit 31e37c1588c7413fb1d6e53f741a21542a20c8b2
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 17 09:45:02 2011 +0000
mog: add --version switch
It's sometimes handy to know which version we're running
commit 562e8b1a1e80e92f0b38891f5516efa221a1a6ce
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 17 09:42:51 2011 +0000
mog: fix unchunked tee
Also, just use aliases while we're at it
commit 255da707c4460a576b7ed7cd5da818a2f5efa1ef
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 17 09:41:02 2011 +0000
mog: "tee" subcommand does not require a --class
We can just use default
commit 5a8ed22d5ee2e101fdfd982589cf6dfe1fa1fa5e
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 15:46:37 2011 -0800
mysql: deprecate this interface
In hindsight, it's not appropriate and should be a separate
project...
commit 1d53980c8a23e15e5a9a91fcd265cc5a647d6544
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 15:00:03 2011 -0800
doc: update TODO
We have goals, yes we do
commit d3014a97fb97e290bcbefeeee3d015a04eb37e07
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 14:55:41 2011 -0800
doc: add file_info example
It can be useful, yes.
commit c5a7ef1be173875dafa4804af53e499688bd9452
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 14:52:09 2011 -0800
file_info_cleanup: nodoc this internal method
commit 31c413e69514a9b99cf96f3382fb03dee6552d6f
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 11:01:22 2011 +0000
http_file: use String#inspect on broken server response
If we have a broken server, it could've spewed anything, so
use #inspect to clean it up in case somebody put malicious
escape sequences in the server response.
commit 52a19f3c4ed9da852eac809276d7f64fe4defbef
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 10:57:43 2011 +0000
Revert "http_file: do not reopen opened file (on retry)"
As it turns out, the previous change was wrong since we have to
reopen it anyways as "upload" is called in a loop. Attempting
to rewind @active is still a good error-check, though.
This reverts commit 754b8077b55e37627c32d1339227001d509d8517.
commit 754b8077b55e37627c32d1339227001d509d8517
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 09:55:01 2011 +0000
http_file: do not reopen opened file (on retry)
Since we can either rewind it or we can't, we shouldn't
reopen already-opened files.
commit b8b0c8003ca16cc123c99e91fa65e8702a02a86f
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 09:54:22 2011 +0000
socket_test: ensure timed_write returns proper bytes
We rely on this since we alias "write" to this and
IO.copy_stream relies on the proper return value.
commit fb6a9f43728f46efee4e3f4aa6c689d260ae7387
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 09:53:26 2011 +0000
test_mogilefs: do not assume header + body is in one recv()
One can never guarantee this with TCP.
commit 41d9b886e9ae1bea35753d6f5de958154809330a
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 09:38:42 2011 +0000
mog: teach tee to optionally --chunk uploads
On some backend servers (latest Perlbal), chunked uploads work,
so we can stream uploads to them without buffering to a
Tempfile. Of course, this destroys our ability to retry...
commit 46f51f4eaaef4e8f7b36fe5f881551099d5113b8
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 09:24:05 2011 +0000
mog: fix "test -e" on missing files
commit 431de2b3f7b3b80a8f6da01549f475cff945f2bb
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 09:21:27 2011 +0000
list_keys_verbose: pipeline retries on all socket errors
We'll try to retry for all transient issues that aren't
our fault
commit 7faeab6c16cf127024d2cd8ce0de151e5375d199
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 14 09:16:23 2011 +0000
http_file: ensure regular paths are rewound, too
We could also reopen them, but it's harder to test FIFOs
for rewindability (and fail) that way.
commit a37a40ae2eed07439e23e3f4366816576ab801c9
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 13 07:47:16 2011 +0000
socket: alias write to timed_write
All of our writes must respect timeouts.
Return value must be the number bytes written.
commit 63b390fb6e6ea43f20c42865efa2b370f4fb0c55
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 13 06:53:21 2011 +0000
mog: small cleanups
* use warn instead of STDERR.puts
* prefer $std* to STD*, they're less ambiguous
* $stderr doesn't need binmode, we don't hit it directly
commit fab3a54359e7ddd67b96be010ce7fcdee31b23a6
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 13 06:45:50 2011 +0000
mog: ensure stdout/stderr are always sync=true
Luserspace buffering is often confusing to users reading
input, and performance-wise it's more often a waste
of memory bandwidth than it saves for syscall reductions,
especially since we usually do large transfers.
commit 7bedbf3ed5920a922da89874a2bd134fb1a82c83
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 12 12:28:49 2011 +0000
list_keys_verbose: retry on pipeline error
Sometimes a server will shut down on us in the
middle of a pipeline. That is bad.
commit 435d43d6b85481e7b678c3092dec7fcb246e30e8
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 12 11:57:17 2011 +0000
backend: document pipeline_dispatch behavior
It's tricky, but possible to get right (at least for list_keys)
commit d48dcc1d1e4679cf7dbd428fa4d2e7932f9839d3
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 12 09:22:56 2011 +0000
list_keys_verbose: preserve ordering with multiple queryworkers
When multiple queryworkers are used in the tracker, it's
possible to receive responses out-of-order. We need to
hold onto responses while the pipeline is being processed
and only make callbacks when they can be made in the
order the keys were given to us in.
commit ec7e00ad0a277c3b760d0f88482108bd0624224e
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 12 08:01:12 2011 +0000
fix misnamed exceptions
PipelineError was misnamed should be recoverable in user code.
MogileFS::Error should be raised for bugs (in our code or
user's code).
commit fd01fa4d894130e2e7dec48b3ea45846124cd15d
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 11 17:52:46 2011 -0800
list_keys: more accurate devcount file_info-less servers
get_paths defaults to pathcount=2, we want all of them,
so INT_MAX should be enough...
commit 8b377466612afc2fead11e462a26bcf12be86df4
Merge: 89e0eb4 beed7ec
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 11 17:48:42 2011 -0800
Merge remote-tracking branch 'origin/pipeline'
* origin/pipeline:
list_keys/each_key: better handling of verbose listings
pipeline: make errors easier to debug by including the request
initial pipeline API
commit 89e0eb49aa39b22b3de041dcc1db4ce6fe6c2a07
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 11 17:39:25 2011 -0800
auto-generate mogilefs/version from git
GIT-VERSION-GEN was ported from Bourne shell to Ruby.
commit b2c478eb88f442adb44b1fbe21ee25c1d368ae89
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 11 15:55:04 2011 -0800
socket/pure_ruby: reduce Errno::EAGAIN exceptions
Exceptions are expensive, and Errno::EAGAIN is especially so
in MRI 1.9.2+, so avoid generating too many of them. This
is the whole reason kgio exists, after all.
commit 890b3904c770955cdb0a70864060f99786a4f402
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 11 15:51:14 2011 -0800
test_fresh: factor out setup_mogilefs routine
We need to support plugins in the nearish future
commit beed7ecfbcb062283cd8842e03f7ccd57f477f49
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 11 23:26:23 2011 +0000
list_keys/each_key: better handling of verbose listings
This allows us to implement "mog ls -l" much more efficiently
commit d9b4aa1ad01a0506a544eaabe02a9852d01d5d4f
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 11 10:51:44 2011 +0000
pipeline: make errors easier to debug by including the request
Otherwise it's outright painful to figure out what failed in a
pipeline...
commit df3064c1cd63a9f626938f1dcde44f1909971a59
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 11 02:18:39 2011 -0800
initial pipeline API
This is useful for the "verbose" listing of keys since
we make a lot of file_info calls here. This API feels
very awkward, but I think it's unavoidable...
commit de7803e4c9732b5805679d4cada6370e26fd5d67
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 11 15:24:21 2011 -0800
test/test_backend: fix broken escaping test
This is a followup to commit
55de4a3375793fa31993a1e9b4be777007bd31b8
(url_unescape: fix ordering of "+" => " " of swap)
commit 3d067dcb24644bf78260569a9de6e6cdd60fd469
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 11 11:05:51 2011 +0000
Rakefile: packaging updates
commit 55de4a3375793fa31993a1e9b4be777007bd31b8
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 11 10:47:16 2011 +0000
url_unescape: fix ordering of "+" => " " of swap
Otherwise we'll be converting "%2B" into " " instead of
"+" when it appears in a file name.
commit 45f519f571cf88e1710e8175cff5742bed070ff9
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 16:48:28 2011 -0800
backend: factor out dispatch unlocked
We may use this for pipelining
commit 8a54ed8e52e428cb3790d71bed2b99ac52e0b991
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 16:04:00 2011 -0800
doc: misc updates
commit 8b1057fe8105f5672f89b87fffdee3a129e7764f
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 16:00:45 2011 -0800
admin: use Object#__send__ since Object#send can be overwritten
commit 3f222d41cbfe01671dce51e6329c2d949f30111a
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 15:59:36 2011 -0800
backend: avoid eval for defining error classes
commit fea4e19859025e458078f30d16be13bfe57dd8d2
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 15:56:46 2011 -0800
Revert "Compact get_paths output so we don't have nils"
This reverts commit f0ed5cb6ec6851a367175a93398e813e2d62667f.
Unnecessary paranoia, our Backend is more robust nowadays
by requiring CRLF before parsing
commit f570240f9fe1b44fcb72eb78d74cada0844ca39d
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 15:54:54 2011 -0800
mogilefs/mogilefs: make @domain an accessor
It makes sense for use with MogileFS::Pool (in case somebody
is pooling connections).
commit 4728ddc616e50727a2cf325ffb4cacb3f3163921
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 15:23:46 2011 -0800
get_paths defaults to noverify: true
This restores 2.x behavior and is faster and safer since
we're smart enough to deal with failover. Also dropping
default @zone support since it doesn't seem that useful.
commit 7e24f6dc9f5aa90f123dc0e9ee10d22d57a6dee1
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 12:37:14 2011 -0800
test_http_reader: fix stupid/broken assertion
I shouldn't be allowed to code.
commit fb77320dfa382190e1f724a7c9f00dd629dfa954
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 10:58:22 2011 +0000
test/test_http_reader: fix unused var warning
commit c6ae612f3022e4f010dc8ecce7b19e11e6b12f93
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 10:16:23 2011 +0000
documentation/packaging updates
Trying to find a happy medium within Hoe while keeping
my preference for gmake and forcing wrongdoc on readers:
JavaScript and GUIs all suck :P
commit 7b4f4a079e1fa86aab41b7d8238ca07593159eb9
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 09:22:49 2011 +0000
backend: more descriptive error for backends being unreachable
Each backend could have a different story to tell
commit febf8fc2aab898218333e245bab4bda972de5692
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 05:28:08 2011 +0000
ensure get_file_data users notice truncated responses
We don't want to silently truncate data on our users,
that would be bad.
commit 9c0a7c670494fb55fb8ef98838fa0f53aab38d0d
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 01:24:10 2011 +0000
socket/pure_ruby: swap IO#wait for IO#select
The return value of IO#wait is strange and confusing,
relying on FIONREAD is a waste of time anyways.
commit 38a38766deef994577cfb89e7a1b5a30e8bcd7eb
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 01:16:59 2011 +0000
test: set Thread.abort_on_exception= true
Should make tests easier to fix and debug.
commit 39202060afca2be96c1d01dd83b738de0bd2cc7f
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 10 01:16:20 2011 +0000
backend: fix idempotent retry logic
We don't want to blindly retry on invalid keys and such,
only on invalid (truncated) responses, timeouts, and
syscall errors.
commit 265f3d32126cf2a31587a6021494f528bbdf8171
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 9 10:06:39 2011 +0000
fix warnings, oops
Found with 1.9.3
commit 4b8bff9f4c90e6eb442c82b6f125279600311bf4
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 9 09:06:19 2011 +0000
backend: automatically retry on idempotent commands
Read-only commands to the MogileFS tracker may be safely
retried if a request is sent but no response is received.
commit e0711f60d8cdd49e543764ce2252d77163cf6fd5
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 9 02:06:50 2011 +0000
avoid NilMethodError if get_file_data is passed an invalid key
It should raise on an unknown key, of course.
commit bbad43ec56c978c2d62c8f429354b04d20683c99
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 23:33:28 2011 +0000
Rakefile: remove ZenTest test dependency
I don't use any of its tools
commit 42449ebe6d56ef76249c52267fe5e786edccbba5
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 22:46:22 2011 +0000
get_file_data supports offset and count
Just like IO.copy_stream
commit d9d627d79f5beb38b048ce17c88b779eda1a75b8
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 22:46:08 2011 +0000
update Manifest (ugh)
commit 60a31e86ee57d5d1f3b25bde6d2ea1e718f0353b
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 22:11:21 2011 +0000
get_paths: expand and improve tests
get_paths may take a Hash for its optional arguments and
now supports the optional :pathcount argument.
There may now be a default @zone for MogileFS::MogileFS
objects as well (specified via :zone).
commit 8bc17c29ef5024802fc7207023cdaa7f1b5eda5f
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 21:21:05 2011 +0000
get_uri: object allocation reduction
Using Array#map! instead of Array#map can save us at
least one object allocation.
commit 5bd27a2ff2d5b32a4d6aa3115fda63127164d1e3
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 10:23:21 2011 +0000
test/aggregate: not all Ruby implementations support "-n" well
At least my year-old Rubinius installation does not
commit 4403d6936f72d8676f04799c419ca7f556b47bd1
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 10:21:04 2011 +0000
test/exec: remove "uuid" gem dependency on 1.8
Just borrow the code from 1.9.3
commit 5f7f02420d2df6a29115dc9dcf31587290d0658f
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 10:09:27 2011 +0000
copy_stream (1.8) fix close issue
Not sure exactly what was causing it...
commit 79d83988cb08ff646fcf7950c74e366ae6ebc290
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 10:08:38 2011 +0000
bigfile: explicitly close HTTPReader socket
No need to unnecessarily trigger GC nor hit EMFILE/ENFILE
on VMs that rarely GC IO objects...
commit 0d908e6ba1af6e0cc229dc4d66a2a0e97c07164c
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 09:44:15 2011 +0000
http_reader: remove support for overriding http_method
GET is all we need and we can save some code this way.
If we ever need to use HEAD much again, we can use
net/http/persistent since our internal HTTP classes
are only optimized for large responses.
commit 9be9c8b2ca91938025d9066ebde5c832da04f0ef
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 09:40:16 2011 +0000
split deprecated paths_size to its own file
This is only needed for users on old MogileFS servers
commit b07078b19046b485d758ebe46eeadbecf3b61db1
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 09:24:39 2011 +0000
http_* shorter error message for failures
Since we'll have multiple tries, try to limit errors.
commit d62ee4e837b8cbad55354ee3a9e92ee68decb90e
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 09:16:58 2011 +0000
backend: revert 1.8 splat breakage
Oh well, not a big savings there anyways unlike the inner loop.
commit c16b4bcc3fe68de8ee3c615c5e2bfda80f900b9a
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 09:11:19 2011 +0000
get_file_data: avoid exposing users to copy_stream invocation
We'll now accept an optional argument which can be passed to
IO.copy_stream directly. This should make life easier on users
so they won't be exposed to our internals to make efficient
copies of large files.
commit 100124a50620271e9c7317c28efacdee3cc568e3
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 08:50:11 2011 +0000
backend: small garbage reduction for url_decode
Obviously I could go farther, but not at the expense of
readability. There are C libraries I could use, but MogileFS
may move to a JSON-based protocol in the future anyways...
commit 9bdb7b8ca1016010277a8c5fcc96a97ee82b3558
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 08:35:18 2011 +0000
size/list_keys improvements
We can use the file_info command to get things faster, now.
commit 145655bde64def9e6149137bbb577778f501d890
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 07:55:25 2011 +0000
test/integration: skip dummy test with minitest
setup is expensive with integration test since we wait for the
monitor
commit a3dcec56ef17d2c509b185d5ec97af773b508e27
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 07:00:16 2011 +0000
tests: migrate tests for modifying classes to test_fresh
classids can get recycled, it seems.
commit efc339b43a272b92468629f7f833e1941052db2c
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 05:15:10 2011 +0000
mog: "ls -l" shows full key name
commit 27afdca06806f0c77be3625055539ab3331f5b3e
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 03:26:13 2011 +0000
Fix Ruby 1.8 compatibility
Our custom copy_stream needs to flush data like it does
under 1.9. 1.8 also can't do a blocking open(2) on a FIFO
in a native thread so we'll fork() instead.
commit 1a5b3487e5a38a4897a6461b3907cd6bde501e0c
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 02:45:40 2011 +0000
add support for the file_debug command
This was added in MogileFS 2.45
commit fd5098ba0825d5ee603265a40e8c62b97191b7b6
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 01:53:06 2011 +0000
add file_info command support and test
This is a command added in MogileFS 2.45
commit 088d88332564a5d09cf8a57695a53453ab59f686
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 01:07:24 2011 +0000
tests: include test for opening FIFO paths
It's conceivable we'd need it.
commit e814b15f1386e2ea53bc018aaa92aecb147400ae
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 8 00:52:30 2011 +0000
Disable retries when storing non-rewindable data
If a user tries to pipe something to us and we can't
rewind on failure, propagate that error all the way
up to avoid risking a corrupted upload.
commit 9bcff8f08dc084b880ef978c890a2706a1bbb304
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 7 23:17:41 2011 +0000
http_reader: fix usage with pure sockets
We won't redefine the "new" singleton method since that
conflicts with existing usage.
commit a603d3231683096085841ae6f0ba96eeb27171a3
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 7 22:51:02 2011 +0000
avoid circular requires
Ruby 1.9.3 considers them harmful
commit fc76cbe64b3f5a3e641d32bff68af919df7b3d9e
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 7 22:21:19 2011 +0000
cleanup unused variable warnings
commit 6aed1bd0229cf31a3aeb9c08d86ae851c8d538c6
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 7 09:43:37 2011 +0000
fix return value for chunked Util::StoreContent users
Using unknown sizes with StoreContent is now supported
(but you're probably better off using a pipe or just
and object that acts like an IO)
commit a919e84b911511ed641b3edcf7b77687abc07f44
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 7 09:36:59 2011 +0000
http_file: remove unnecessary check
There's only one place where we close the socket
commit ee85cf3816c795bf193e644f97da5274edbdf9c7
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 7 09:33:12 2011 +0000
store_file works on IO#read-like methods, too
The readpartial is not in the Rack spec for rack.input
objects, but something like IO#read is.
commit 61f9e531398aeaf617d038bea4494c143ee8b5af
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 7 09:27:16 2011 +0000
flesh out pipe streaming + chunk for uploads
Of course the backend server needs to support chunking,
but the latest Perlbal does.
commit 4f927415dc2a8e8dcaeda510d351192e3cc88747
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 7 08:44:55 2011 +0000
tests: more tests for admin stuff
We'll need to setup a fresh MogileFS instance for some tests.
commit 5a4650a07f394268884a8dad58fac42f9a9288c5
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 7 07:37:56 2011 +0000
tests: more + improved integration tests
We'll also be testing some admin commands in there.
commit c4e2b81f2493f0a0013ba03c11b74a61f3ca56f6
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Nov 7 05:32:51 2011 +0000
admin: allow changing replpolicy instead of just mindevcount
This retains backwards compatibility and users of MogileFS 2.x
to change replpolicy along with mindevcount.
commit e3cbb44e8ef085421e2e001ba1382259242ce793
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 6 08:10:36 2011 +0000
admin: update rdoc for deleting a domain
commit 60243808b59cb0bad8721a59970a46ddd74e68e0
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 6 07:47:32 2011 +0000
delete returns true on success
Not some random hash from the parser.
commit bbe2702bb95a8fd11c1b8fb87f758a946876f58e
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 6 07:46:38 2011 +0000
admin: get_domains includes replpolicy
We'll need to evaluate dropping support for older MogileFS.
They probably didn't have replpolicy back in the day.
commit 347c41d307f85b1264fbf1c06fff9e5c8cb2e091
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 6 04:39:08 2011 +0000
tests: integration tests for mogtool --bigfile
We need to test deprecated behavior to not break
expectations for large, archived files in the future.
commit c66266416da2e05889ed93997f2c8a7cba216c56
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 6 04:38:21 2011 +0000
bigfile: fix retry on rebalanced keys
commit 5f5e9071e51134b2028c1b2f99655be66debc2a4
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Nov 6 04:38:05 2011 +0000
Manifest: update
commit fec183631eee46ce3f00605142c6dfb122797e27
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 5 12:33:53 2011 +0000
return file size properly when storing file
commit f54e49512910f2958141a51cb685009ed39b2906
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 5 03:06:00 2011 +0000
support chunked uploads via PUT
This can be useful for streaming to a backend
(this feature needs tests)
commit 283050fc283868f79796f05901bd9149713ae282
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 5 02:34:35 2011 +0000
redo IO.copy_stream usage
Avoid deepening stack depth and make it easier to migrate
fully to 1.9 in the future (dropping 1.8 support).
commit f057a82730d252c509f8c590430dbf4507d17601
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 5 02:11:03 2011 +0000
backend: require CRLF on every line we parse
Otherwise IO#gets and MogileFS::Socket#timed_gets could
return a truncated response before EOF and we would never
know the response was truncated.
commit 972bad366ae58ec726329a45cbf908e16e1889df
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 5 02:02:18 2011 +0000
make EmptyPathError more descriptive
commit d6b2624ed1f4502d499dad93ee44a716b5c56f08
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 5 01:59:27 2011 +0000
rename httpfile => http_file
Should be easier to read this way
commit 37c6ee5d64d2cbdc739b4f4afcf55b3762234392
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Nov 5 01:54:14 2011 +0000
refactor httpfile and remove layering violation
Splitting calls to backend.create_open + create_close between
files is also extremely confusing and error-prone.
Hopefully nobody actually depends on some attributes
we've removed.
commit 8fefb8c07fdb69eb80c5aefbb0cdd74388cf908f
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 4 23:50:17 2011 +0000
new HTTPReader implementation replaces the http_read_sock method
This is cleaner and replaces redundant code where we would retry
paths. MogileFS::MogileFS#size now raises on error instead of
returning nil.
commit afe10667b8deeb60e3f70d2ddc7a20ae7e0bc072
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 4 02:15:15 2011 +0000
remove the network module
It's only used internally by the deprecated bigfile
implementation, and we don't even need it for that.
commit 8a6680f0209bce12d56ae2ee2613448783f3d55a
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 3 00:18:38 2011 +0000
doc: tag files containing internal implementation details
We don't want our users to rely on them
commit 1c78c6d067b27ccd763bd2732f57771261e047f3
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Nov 3 00:06:08 2011 +0000
bigfile: documentation update
commit 7f8687c9496317a5ffd0da9f4489d59ff8376899
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 2 23:21:54 2011 +0000
util: remove Socket.mogilefs_init method
It's no longer used anywhere
commit 93242dbc664409e8bd78f021ba7b527f340fa1d7
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 2 23:20:58 2011 +0000
remove sysrwloop for copy_stream
We're trying to use as much as we can from Ruby 1.9
commit c5bb0bffe17e7fa924fcca2f5dc85271ac466cea
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 2 23:20:38 2011 +0000
test/kgio: disable of MOGILEFS_CLIENT_PURE is set
commit 77b927de0c7c0f0e5e96691f2c9099e307c3e1f3
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 2 23:00:28 2011 +0000
mog: remove sysrwloop dependency for "tee"
We can do this in a way that's compatible with IO.copy_stream
commit 100fcbe4b368cd6510e65af5b773ab9d70ef42d6
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 2 22:59:12 2011 +0000
mogilefs: don't pass class argument if it's "default"
mogilefsd may not like that
commit 443d7bdf4c1fb3b950cd05ffb475aef12d50c4b6
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 2 22:33:01 2011 +0000
bigfile: make parse_info public and require arg
commit 909a2ae12d11593d59fccf30b7c2143cc9f441ab
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 2 20:36:47 2011 +0000
bigfile: better factor this for use with IO.copy_stream
syswrloop and it's filter arg is going away, but we're
stuck supporting bigfiles in read-only mode for eternity.
commit 46355ddfecb4e210b4dd3b6f91dde330e8d509ff
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 2 02:40:03 2011 +0000
remove the rest of the Socket monkeypatches
No more mucking with existing Ruby classes \o/
commit af2d9c971cee5a6896fa3da44adff72d5f2ae44a
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 2 02:14:46 2011 +0000
util: remove syswrite_full method
It's no longer used
commit 4b8cc437c502e5fda86ac5935baf69ec5a585e74
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Nov 2 02:06:19 2011 +0000
make http_read_sock and friends use new socket API
This will need further refactoring
commit ec29cbf28b2fa3a0155462f8ee927265f9fc3d94
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 1 23:38:31 2011 +0000
use IO.copy_stream if available for making uploads with HTTPFile
commit 60a29ca15a1d90cb7ef011719111e3d6d5691614
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 1 23:30:41 2011 +0000
httpfile: remove silly syswrite_full usage
Since Sockets default to IO#sync=true, IO#write is
just like syswrite_full and there's no reason to
continue using it.
commit 0bffe57f784b40dc355c2eca82d1d7a8cab09093
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Nov 1 20:08:25 2011 +0000
backend: rely on Array#shuffle instead sort_by { rand }
This is easier-to-read and more correct. This is not available
in 1.8.6, but we already broke 1.8.6 compatibility with
String#bytesize.
commit 6897db699255ce1fbd31ffb304db357351f5509a
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Oct 31 23:45:25 2011 +0000
wire up new socket class to backend
This should allow us to swap in kgio more easily.
commit 7b50489a1bca82ccb0b2e374f3d4f95bb651be5a
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 21 08:03:58 2011 +0000
introduce MogileFS::Socket class
More work towards _not_ monkey-patching core classes.
commit d367d4c5645c946505d39c57b4f619a5c3123a10
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Oct 22 02:47:43 2011 +0000
test_backend: remove TempServer usage
This needs to be less implementation detail-oriented
commit c35ea4f30221e18fa5d504a444a40f611e21c291
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Oct 22 01:41:09 2011 +0000
httpfile: remove comment to NFSFile
NFS support is gone
commit 095967f8a960ec1720ea6aaa2dbed1063ba74ba8
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Oct 22 01:39:43 2011 +0000
README.txt: shorten bogomips.org URLs
bogomips.org went on a URL diet earlier this year.
commit 582463d9826b8b07d01846d8fdcf2beba45df1f9
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Oct 20 09:16:51 2011 +0000
util: small ivar reduction
commit 0d7f4e94223bb7413b98f39ac03ddc2970f2bbb3
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Oct 20 09:03:23 2011 +0000
remove Socket#mogilefs_peername
One tiny step in removing core class changes.
commit bd4e832d294635594c2af4da7a6b05fb884b2ce2
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Oct 20 06:35:38 2011 +0000
tests: do not assume "." is in $LOAD_PATH anymore
Ruby 1.9.2+ removes "." from $LOAD_PATH
commit 7e72f6a063703ff065588fa869c3dbad29829e56
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Jan 12 18:08:18 2011 -0800
mogilefs-client 2.2.0
= 2.2.0
* internal cleanups (no public API breakage)
* refactor backend socket/connection handling for reliability
There'll probably be more aggressive cleanups/refactoring in
future releases if I have time.
commit 60c22ca86f8fcea8b51c90d5cb759cc2f29cac59
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Jan 12 17:49:32 2011 -0800
refactor backend socket/connection handling
We now take steps to ensure the socket we wrote to is always the same as
the socket we'll read from. And if a read ever fails we'll close the
socket to avoid having responses. We've also switched from Socket#send
to IO#write as IO#write guarantees write-in-full behavior as long as the
socket is alive (MogileFS queries are small)
The public MogileFS::Backend#shutdown method is now thread-safe,
too.
commit 79718dd2b5d7fa05974388d409dec44a298ebefe
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Feb 19 15:50:19 2010 -0800
bump VERSION to 3.0.0pre
The 3.x.x series will probably be incompatible with 2.x.x...
commit 3f714b9b60879e4e8ce9ed43c62ea9339a09ffe5
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Feb 19 15:47:40 2010 -0800
remove support for TCP_CORK
It's needless optimization on a LAN (most places where MogileFS
is deployed) and the extra method calls in Ruby seem to force
more overhead in to the application. So stop doing it.
commit da9f66a8c1cd76de75bf35a441d27492a29a9bf2
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Feb 19 15:45:46 2010 -0800
avoid unnecessary IO#sync = true
Sockets have sync=true by default, and there's no need for it
when reading files
commit 5a9a0fb45eb191262b4b09975ebf502438272b27
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Nov 20 14:38:33 2009 -0800
set encoding: binary for all files
We won't trust Ruby 1.9 String weirdness since data storage
is locale-agnostic
commit b34465591ba83034985dc4be5fa3516ae059f408
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Jun 26 11:40:37 2009 -0700
backend: cleaner newline handling of error strings
IO#gets respects "$/" which may be altered outside of our
control. So force gets to use "\n" which will also match "\r\n"
(in case the mogilefs protocol changes subtly in the future).
There's also no point in exposing the trailing "\r\n" from the
network protocol to our users.
commit 19ba6f2602ffc5e54fc91e22d28f6231290c15bc
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Jun 26 11:12:14 2009 -0700
GNUmakefile: spaces for decls to avoid confusing make
Otherwise it could be confused as belonging to the
task defined above the declarations.
commit 3299ed2412a2b27d88e04ce7c40416371507f8ec
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Apr 11 19:54:47 2009 -0700
mogilefs-client 2.1.0
commit 127799e5ba632d2d4cc0ac4c1a48cdf0143eb75f
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Apr 11 19:52:36 2009 -0700
GNUmakefile: better logging/output control
Running "make V=1" displays test results verbosely.
commit ba24d98b90b17ca59e9d274b7f5925e237418b4a
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Apr 10 18:09:43 2009 -0700
Ensure store_{content,file} always returns size
It's useful to know the size of the file we're storing.
commit 33ce9880cff9eaf8b7ad7558357cd0fcaed8144a
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Apr 1 16:03:39 2009 -0700
mog: small cleanup
Tempfiles already have their own finalizer and we
ensure they're closed, too.
commit 539f39680436dfebe6de9e5c8b905428af438468
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Mar 27 14:18:53 2009 -0700
mysql: filter out URLs for down/dead hosts/devices
Clients won't be able to make use of them if they're down or
dead. We trust the MogileFS metadata in MySQL to be synced
correctly with the actual device/host states.
commit 4610f9a59b22993785689f292ab23d309a5ce048
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Mar 25 00:27:24 2009 -0700
README: add links to the repo.or.cz mirror
These have been mirrored to there forever, but
never got linked correctly.
commit 20963ad8c801701208f2496ba62fd4c57bcc97e0
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Mar 24 23:07:51 2009 -0700
verify_uris: use write_nonblock instead of syswrite
This really shouldn't make a difference since the smallest
buffer size for a socket I've seen is 4096 and MogileFS
URIs cannot get that long.
commit 40e82ab559dee657a5884c77a46e85613856c9d8
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Mar 24 23:02:14 2009 -0700
Add get_uris API method
This is just like get_paths, but integrates better into Ruby
applications that use the parsed-out URI to do operations
directly on the URIs.
commit 65f9a224dfda3ac8b0ee9290b456d23ece32a053
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Mar 24 22:34:14 2009 -0700
Extra checking for full_timeout in sysread_full
full_timeout could be set if timeout is set. Don't
croak on a NilClass error if we screw it up. Also,
correctly detect readiness when checking the return
value of IO.select.
commit 76076d42db5cf16317823131f47570c8f80e22a1
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Mar 24 22:16:07 2009 -0700
README: add a note about emailing me
Yes, I'm a doof when it comes to using web pages.
commit fdd00effb846e77807751562c11d78b4240dd93d
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Mar 24 22:09:08 2009 -0700
Really remove all NFS support
This cleans up some of the internal HTTP handling code a bit,
too; and does a better job of closing sockets than it did
previously.
commit f0ed5cb6ec6851a367175a93398e813e2d62667f
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Mar 24 20:16:37 2009 -0700
Compact get_paths output so we don't have nils
I'm not sure if mogilefsd ever returned broken output for this,
but just in case we compact the output so users don't have to
worry about them.
commit b7880146d13c8ed837ffa92102f1a907dd0d21ee
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Mar 24 20:08:08 2009 -0700
Close socket we create before raising exceptions
This can potentially fix some file descriptor leaks (but there
may not be any leaks because of GC which can mysteriously close
file descriptors as I've discovered).
Better safe than sorry...
commit 9ae7c33f4465d3d44b5ae31d9f454f89e5748fb3
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Mar 24 19:43:48 2009 -0700
Unify internal HTTP GET/HEAD methods
This takes advantage of the (ugly) new mogilefs_size Socket
attribute to avoid duplicating Content-Length parsing code.
I really wish Net::HTTP in Ruby was actually usable...
commit c07be896889f13235bda1bd32983261e73513a70
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Mar 24 18:56:50 2009 -0700
Respect timeout when doing get_file_data
This adds a sysread_full utility method with configurable
timeouts. Individual reads can be timed out as well as
the entire sysread_full call.
commit a69925d4b4671fe65e54e34a009e17fb1fa47700
Author: godfat <godfat@godfat.org>
Date: Wed Mar 11 16:19:43 2009 +0800
call IO.select([sock]) if sock is not ready.
commit cb0eb2a46cae96a602d0c78033a3e1154e165986
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Mar 24 17:14:18 2009 -0700
MySQL interface returns integer length and devcount
This is an API change.
Additionally ensure the normal MogileFS::MogileFS API
doesn't break here.
commit b9722d72614a0198b7db6257981d71b9d62b43f4
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Mar 18 03:40:17 2009 -0700
test_mogilefs: fix race conditions
commit 878e3c83595c1ad44e648904987c4d917baa06b2
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Mar 18 03:39:35 2009 -0700
tests: retry random ports correctly
Oops
commit 1959ed17631f7a2b044f609ba147845cdf1307bb
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Mar 2 13:54:52 2009 -0800
mogilefs-client 2.0.2
commit 41d028768fbeb49a31542cb0eaa6ec396d3baf5a
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Mar 2 13:28:31 2009 -0800
Tasks to enforce permissions are set correctly
commit 72ed3d1243037d5c625be6b23fb0098250f5f6dd
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Feb 26 14:21:14 2009 -0800
Update Manifest.txt with setup.rb
commit 918d58b902614ad2c6eede674382e25557276481
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Feb 26 14:14:40 2009 -0800
backend: don't blindly add "Error" to exception class names
We were getting classes with "SizeVerifyErrorError", which
just looks stupid. This was because MogileFS puts
the "_error" suffix into error messages it sends us and
we were blindly appending Error to the end. Avoid doing
that, even if it's somewhat inconsistent since "ErrorError"
looks really stupid IMNSHO...
Note: this is a minor API change
commit d6eff4e796bbb2c135f6e15d7c8580e16c6cbb5f
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Feb 26 13:55:56 2009 -0800
backend: raise exceptions with the error message
Getting to lasterrstr from several levels up the stack is
annoying otherwise...
commit bc7e7cd06f1e2b4eacbf5b7595af6938c2332adc
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Feb 25 00:22:48 2009 -0800
README.txt: update documentation URL
The old one has become invalid.
commit cfdf498adee9ec21420728abd59ce01c181b0d5f
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Feb 25 00:10:33 2009 -0800
mogilefs-client 2.0.1
commit 8d6ced74c2538cfa3836ec14668abf02e712507e
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Feb 24 23:42:14 2009 -0800
network: make verify_uris method more robust
IO.select can raise errors (IOError most notably) on bad
descriptors. Ensure we can detect and delete them from our
waiting sets if IO.select raises something.
commit 6cf59a3231bd53a1bfe91df31e900e4349a4746e
Author: Jacob Burkhart <jacob@brontes3d.com>
Date: Mon Feb 23 13:01:55 2009 -0500
Allow 'store_content' to be used with a streamable object
New way to call 'store_content' with a
MogileFS::Util::StoreContent allows you to roll your own method
of streaming data to mogile on an upload (instead of using a
string or file)
[ew: this still requires a known content length beforehand]
[ew: applied with --whitespace=strip, rewritten subject,
80-column wrapping]
Signed-off-by: Eric Wong <normalperson@yhbt.net>
commit aaf794c5455b17ab8f0a15ccd81f7496bcc0b8d5
Author: Jacob Burkhart <jacob@brontes3d.com>
Date: Mon Feb 23 12:22:00 2009 -0500
fix tests
Signed-off-by: Eric Wong <normalperson@yhbt.net>
commit d49d49152ddace8258b9aeb3930e209edb3cef68
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Feb 23 11:49:08 2009 -0800
Add setup.rb
I'm tired of the overhead of loading rubygems
commit ee21c71551d220a0fc89111dbd5b13a71d54ff3f
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Feb 7 20:18:31 2009 -0800
test: workaround some concurrency issues in tests
At least processes are easier to debug than threads...
commit 6f103af69968237f1dbd33fac40975dd4db8c701
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Feb 7 19:52:11 2009 -0800
backend: Allow TEST_DEAD_PORT to be set
When running tests in parallel, releasing a listener when
killing the process that owns it can cause another test to grab
it. Allow setting TEST_DEAD_PORT in env to avoid this race
condition.
commit 75115f3296d31e5e9f10645f216d3d7c1e9024d1
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Feb 7 18:56:14 2009 -0800
test_util: cleanup forked servers
Not sure why the atexit statement doesn't kick in in test/setup;
but I'm too lazy to look into it at the moment. Also remove a
"require 'tempfile'" while we're at it.
commit b384c7dc1f0ce052f518fe1b94a4bf5588328359
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Feb 7 18:50:47 2009 -0800
Rewrite tests to not use threads (1.9)
Kernel threads in 1.9 are excruciatingly painful to debug
(much more so than userspace threads in 1.8). I'm not in
a masochistic days...
commit cdc5fc07a34bd17f2162dcb171928f49af55ead7
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Feb 7 17:22:19 2009 -0800
Force binary encoding on all objects
File and StringIO objects need to be opened in binary mode,
otherwise they take the default encoding format. Thankfully,
Sockets and Tempfile objects seem to be binary by default as of
1.9.1; but it really is a mess to have to deal with FS
abstractions that try to deal with encoding crap behind your
back...
commit ffc4ec4baaae5020c98dc545c6c6ade0336a3311
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Feb 7 16:38:41 2009 -0800
test_backend: 1.9 compatibility fixes
Not sure how hosts was ever allowed to be a String, but sort_by
does not work on Strings in 1.9 so this was a bug before, too.
Also I'm not sure how the accept_nr variable gets shared between
threads; better just to use sockets for here...
commit fb8874fbf719264064bf9761476de4c16d7f8c89
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Feb 7 16:36:28 2009 -0800
backend: Ruby 1.9-compatible url_escape
Use String#ord instead of String#[] when it is available.
commit db52b53fad2f6ef88cc6105d7b1280a4f4293edf
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Feb 7 12:17:13 2009 -0800
test/setup: don't blindly undefine default_test
the "default_test" method may not exist in the version
of Ruby we're using (1.9.1p0)
commit 78f7849963ca41d315e029a895b8b919e4981b1b
Author: Eric Wong <normalperson@yhbt.net>
Date: Sat Feb 7 12:13:22 2009 -0800
MogileFS::Pool#use always returns nil
We don't want to return the used object back to the caller.
Noticed during 1.9.1 compatibility testing.
commit 7b1e64a80c7554ad30f68e977993742a8d14350d
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Feb 2 20:23:02 2009 -0800
mogilefs-client 2.0.0
commit 72bf21fe39651881d06c77042e299a0aaaeb4bba
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Feb 2 17:05:39 2009 -0800
backend: cleanup and simplify slightly
Again, eliminate unnecessary return statements and
simplify hash generation.
commit a24cd5c8e29161c5dbf707f65ec0d64b2a75dc1b
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Jan 28 16:02:26 2009 -0800
MogileFS::Mysql: refresh devices/domains on cache miss
Domains and devices may be added/changed at any time,
so there's a risk of cache misses in case something
got added before our cache refresh interval. Always
retry if we have a cache miss.
commit e482442c5c46ab67d5db5206e8dea66d2e5c4dc9
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Jan 23 19:25:56 2009 -0800
GNUmakefile: add flog/flay targets
Some static code analysis tools by Seattle.rb folks
I'm avoiding sticking these tasks in the Rakefile since they're
too slow (and I like GNU Make more than I like Ruby *g*)
commit 929184e57d73ba29b6a97c52c22e3281a0810d2f
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Jan 23 18:45:39 2009 -0800
Remove unnecessary statements and temp variables
Last I checked, the trailing "return" is not optimized away by
MRI 1.8. Additionally, remove some useless temporary variables.
commit 69dd8a992379439f20e1aa0ab564f18285b3a515
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Jan 23 18:24:02 2009 -0800
documentation updates
commit b4ca5228ec024bdcbb98fcdb883738941fe0d1fd
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Jan 23 17:44:26 2009 -0800
allow using the default class to store files
Don't specify an empty class (e.g. "class="), instead
just omit the parameter entirely if it is nil.
commit b294f73c1686be08b9a2509c74ad4fc28151050a
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jan 6 02:48:47 2009 -0800
Enable test for db_backend
Due to the unfortunate name, it never got run.
Oops!
commit 6d7fc201b307aca412a7a6129d5f2583cfdfade5
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jan 6 02:36:01 2009 -0800
Manifest: add test/test_util.rb
commit d3f4b01e9b15dc8f7dd71973d07380b98caa6d2e
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jan 6 02:32:23 2009 -0800
test_backend: shorten timeout test
Since the test server sleeps forever, there's no harm in having
a shorter timeout so our tests run faster (over twice as fast
with `make -j').
commit 805d5cc33362cf92196b4ec1edf33b01b5c02746
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jan 6 02:30:56 2009 -0800
test_backend: race condition caused occasional test failures
Oops, there may be more of these...
commit db750a6eeff7010cedb31c99f8aeeb439541c9ef
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jan 6 02:05:58 2009 -0800
syswrite_full: allow an optional timeout to be specified
This makes it easier to use when making requests.
commit e18ff3c8c759c88aca53163e2f5b15c5436280ec
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jan 6 02:05:57 2009 -0800
Use syswrite_full where we previously used syswrite
This is a bit safer since we'll retry on short writes
instead of silently failing...
commit 8f33fa98734a3b2d44c81e63ff77851887d736d3
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Jan 6 02:05:56 2009 -0800
Use syswrite_full when PUT-ing smaller files
Socket#send will return with short writes and IO.write just
sucks, so expose our existing syswrite_full routine to the
public.
While we're at it, add a unit test to try to ensure that we're
correctly doing syswrite_full here. System-specific behavior
dealing with buffer sizes is not easy to test...
commit 27092c172aaf828740f85d45a410a18c346db8f8
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jan 5 19:05:38 2009 -0800
Mysql: fix get_paths
Oops, this method needs to bypass transformations we
in the normal MogileFS::MogileFS code path.
commit 14da9651b3227bea0dcc01eaf09364d8d1ebc274
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jan 5 13:40:31 2009 -0800
Remove all trailing whitespace
This got noticed when I imported this project into another
project vendor/ subdirectory (no, I've decided to not use
submodules for that project).
commit fdefa0ac60dda5d96abf279d5a4df4521325d838
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jan 5 13:26:57 2009 -0800
Update Manifest, include Make target for generating it
commit 12d8afa7539915ca448e362ee6edc39e49d9a07b
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Jan 5 13:20:24 2009 -0800
mog: fix fallback to MOG_HOSTS env if MOG_TRACKERS is empty
commit c0450c6888c6234bfbd324672ddcedb33901d21e
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 30 18:38:48 2008 -0800
Don't let size(key) get tricked by non-200 OK responses
We never checked the HTTP status code when making the HEAD
request. All this HTTP stuff should probably be moved to
HTTPFile
commit dda46f246f93ac31512582767c402957fb2fbaa1
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 30 18:03:06 2008 -0800
Cleanup and document http_get_sock
Correctly fail when we get non-200 HTTP responses and retry on
the next URI.
commit e07b59855a61a4053810f72e5a9f2000d54964e5
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 30 14:39:31 2008 -0800
new MogileFS::Network module, move verify_uris there
This will include networks and optimal URI selection support
commit dfa94f76c60e54376372e181a25c080c611bb323
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Dec 29 16:50:54 2008 -0800
History: note that 2.0.0 is still a work-in-progress
commit 91276ae9131c405050b1f31af0680475043a9e0b
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 24 16:53:22 2008 -0800
sysrwloop: make sure descriptors are flushed and synced
"io.sync = true" does not appear to flush file pointers in MRI
1.8, we need to explicitly flush it first.
commit 0d3607a455141a428b105b39e3be1c263cc48e8a
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 24 16:47:45 2008 -0800
GNUmakefile: allow "make test/test*.rb" to work
Also, remove the PID uniqueness thing, it's easy enough
to just create a new directory if we want to run tests
in parallel... Additionally, add ignores in for the
log files.
commit ab87a8472ee42785364ab1ee84ea00e06cb592d8
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 24 03:22:57 2008 -0800
test_mogilefs: fix broken test case that was timing out
And incorrectly nested assertions in the tests made
sure the errors were never caught, too.
Now the unit tests run in about 3s for me
Damn Ruby conventions and 2-space indents...
commit c07ee9fdcc00e5af18993b3cda9d17f91351d09e
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 24 03:09:50 2008 -0800
GNUmakefile: remove dependency on bash3
-o pipefail was only useful when I was experimenting
with unbuffering output to the terminal. I've decided
to just forgo terminal spewing in the tests, and the
lone use of cat(1) is highly unlikely to fail..
commit 28aab06e493363b143476d44ec3d39e8403cd28e
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 24 00:13:48 2008 -0800
test_mogilefs: allow NR_CHUNKS to be overridable via env
This way tests can run quicker by default but we
can still use larger ones to test things.
commit e6ff1c564a5420d7ea855fc7312caaa9e280f921
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 23 20:44:27 2008 -0800
shorten MogileFS__DbBackend test name
commit a326d26e281e440375925970912c6a9e6b07b7a8
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 23 20:27:08 2008 -0800
HTTPFile: handle multiple device failover correctly
Previously, when we got multiple destinations to
upload to and one of them failed, we failed to
correctly retry the next destination. This will
set the correct devid and URL.
commit 5ba07ec56c5c8f89ef9390f649f12fdfcf611b49
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 23 19:09:40 2008 -0800
MogileFS::MogileFS: simplify get_paths
We dropped NFS support, so this can be simplified further.
commit d4a3c3a63b20ea6a863e70c026c7b08d1c70d52b
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 23 18:22:15 2008 -0800
Add GNU Makefile for running tests in a parallel
I'm getting a 40% decreas with my current set of tests
on my dual core machine.
commit 8ddfde0f598d0f7ddebf2f81293475e16125f538
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 23 18:14:03 2008 -0800
test/setup: remove rubygems and zentest assertions
I don't think we ever used the assertions from ZenTest
and rubygems increases load time on my box.
commit cd821f79aff72385412c4338a2883d5cad3a8d76
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 23 17:06:13 2008 -0800
test_client: make this work outside of "rake test"
I prefer running my tests directly from the command-line
ruby -I lib/ test/test_client.rb
commit a784189571101463dd4291207f989147e98d81f4
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 23 16:56:49 2008 -0800
backend: remove pointless connect_to method
We used to mock it for the tests, but we use real sockets now.
commit f806d7c3adbdc5062666df0336f5da6d0dd72dab
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 23 16:56:07 2008 -0800
test/setup: EACCES, not EACCES_S_
Also, fix up a warning about @thr being uninitialized
that showed up in testing.
commit 9d2810194a64427b90a4e226fbd0dfb16c9bc095
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 23 16:45:48 2008 -0800
Make sure select(2) calls are IO.select
We don't mock `select' anywhere, and this makes us safer
in case we're included in another module.
commit 819027c713930d8234abdc633344922382958bed
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 23 16:43:29 2008 -0800
backend: remember to require 'thread' here
It was accidenally removed during the TCPSocket removal
commit a6cebdeeb3f89b8097abbe4a5fddd798e16c8758
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Dec 22 20:56:57 2008 -0800
test/setup: TempServer saner port acquiry
Originally I had a process-based port thing stolen from Mongrel
(and subsequently screwed up). Just use a random port that's
not in use right now. Additionlly, retry if we get EACCESS in
case we're on a strange system where binding certain ports are
not allowed.
commit dd2089b2646e77fef396e2e62ad313851bfbf04e
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Dec 22 20:44:45 2008 -0800
Remove NFS support
MogileFS 2.x upstream no longer supports it, and it's
become a maintenance burden and NFS is a horrible thing
anyways.
Attempting to use this with servers that support NFS will result
in MogileFS::UnsupportedPathError being raised.
commit 8e44957300037c8bfd029fb21b665ceb98e7817a
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Dec 22 20:08:56 2008 -0800
HTTPFile: rename bigfile accessor to big_io
"bigfile" is used by the large file support, too.
"big_io" also allows an IO object to be sent to us, as well
(and not a pathname)
commit 16cfeeb035c0c0362e8f4fb1707f9006573defc1
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Dec 22 19:00:13 2008 -0800
MogileFS::HTTPFile: support TCP_CORK on Linux
Since we use TCP_NODELAY with all sockets, the bigfile code path
here is less efficient because it sends the headers separately.
So use TCP_CORK for this part of the code to avoid sending
partial frames.
Additionally rewrite this to use send(2) which is more suitable
for sockets.
Since we didn't have any previously, add a test for uploading
large files.
commit 46b2c5ff196c9194abae8954c9efdc899b1f1ff8
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Dec 22 18:45:08 2008 -0800
Create all sockets with TCP_NODELAY
This should reduce latency. We send reasonably-sized
chunks whenever sending large requests anyways, so
throughput shouldn't be noticeably affected.
commit 4a5794447a682b8034b2381054df28bd6296c4ee
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Dec 22 18:28:27 2008 -0800
Replace TCPSocket + timeout code with Socket + IO.select
This removes the dependency on unsafe methods used in the
Timeout class.
Charles makes some good points here:
http://blog.headius.com/2008/02/rubys-threadraise-threadkill-timeoutrb.html
And even matz agrees:
http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/294446
Of course, I strongly dislike any unnecessary use of threads,
and implementations using native threads to do timeouts makes me
even more uncomfortable.
commit b7ba273a043633c01d9ae8cda8ae9db4cc5dcf07
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Dec 22 17:37:02 2008 -0800
test: remove FakeSocket code
These tests forced us to use TCPSocket, which wasn't friendly to
sockets created with Socket.new (for asynchronous connect(2)
calls).
No changes to the core MogileFS components have been
made. I want to verify the new TempServer implementation
performs correctly with the current code.
commit df2af32f01bc995e9f311b51d3351ee042e23483
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 18 20:52:24 2008 -0800
Allow MogileFS::MogileFS to use :db_backend => MogileFS::Mysql
This should complete the integration of the read-only Mysql
backend into MogileFS::MogileFS.
commit bd42a70bb299e088b1c244c3ffe46695393521c5
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 18 20:31:35 2008 -0800
MogileFS::Mysql: sleep() is MogileFS::Backend-compatible
Keep methods in the Mysql backend as compatible with the
standard Backend as possible.
commit fa796bf60396430a3de4ce0409d7f1edf663856f
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 18 20:19:15 2008 -0800
test: move mocks from test_mysql to test/setup
This allows us to reuse this code in another test
commit 1b2589208ef8765ab17af06be3e12164ab7b1492
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 18 20:14:23 2008 -0800
test_mysql: remove unnecessary requires
Not sure how they got there, probably just some junk leftover
from copy & pasting...
commit 0cdb20bfb76653a74d44c434e243029117a7a48d
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 18 19:57:58 2008 -0800
MogileFS::Mysql: start making this a backend for MogileFS::MogileFS
This will allow us to avoid redundant code and modules,
allowing existing applications to more easily use the
direct MySQL interface as a drop-in replacement for
read-only MogileFS::MogileFS applications.
commit 19e15f7b029b3c3c5e9eeed948a37e2316ed5975
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 18 19:30:35 2008 -0800
MogileFS::Mysql: list_keys returns nil if nothing was found
This is to be compatible with MogileFS::MogileFS
commit 61e2ccc21b8d94087a11efced682461a63e49b09
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 18 19:09:57 2008 -0800
Add some documentation and comments to new code
Otherwise I'll forget what I've been doing, too.
commit 37a0f630737d57ca42a8a8019b7c4351387459a9
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 18 18:09:29 2008 -0800
MogileFS::Bigfile: remove Thread.abort_on_exception setting
This was leftover from testing. As it affects global
application state behind the user's back, it should not
be set here.
commit 8e271d4f2a29bf1935cc2b3ecfecd0f444b933ba
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 18 17:53:48 2008 -0800
MogileFS::Mysql: read-only methods for MogileFS::MogileFS compat
These methods help allow MogileFS::Mysql objects to be used as a
drop-in replacement for MogileFS::MogileFS in read-only
situations.
commit 6f0a6159d8335cc6a08f118048f7c8175807c7c1
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 18 17:24:46 2008 -0800
Add test for the Mysql interface
This doesn't require a Mysql server to be setup, limiting the
effectiveness of it a bit, but should be enough to ensure
refactorings don't break things.
commit daab68d1da073648098be96fbb468315cec36b48
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Dec 18 15:22:13 2008 -0800
MogileFS::Mysql: small cleanups to make testing easier
The get_paths change should be a small optimization, too.
commit 0255844ed2495aa7fa7366fb439176174413472a
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Dec 12 11:38:29 2008 -0800
MogileFS::Mysql: more flexible Mysql driver handling
Remove hard dependency on mysqlplus driver. We can detect
whether the driver supports the c_async_query method or not.
This also allows more flexible connection setup and management.
commit cb85f82e640762aea81b7ad2b8aec1efed0aa572
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Dec 2 16:33:05 2008 -0800
initial bigfile read support
Needs more tests, but it seems to work...
I seem to have discovered a bug in mogtool which causes it to
generate incorrect MD5 checksums when the --gzip flag is used
(and --gzip actually just does zlib deflate, not something that
gzip(1) can actually decrypt).
So right now MD5 checksums are only verified on non-zlib-deflated
files.
commit 83dfe9644cbbc83b0f3bd05537874eecc8ad0a8c
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Dec 3 14:47:22 2008 -0800
MogileFS::Util: avoid calling syswrite on zero-length buffer
This bug can't have a noticeable effect since writing a
zero-length buffer would result in nothing being written, but
this is more correct and can save a method dispatch and syscall.
commit 02393ab1f3517436f1b8adee3c97e2b71a1f41f4
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 31 16:40:45 2008 -0700
sysrwloop: safely handle non-blocking sockets
Non-blocking sockets can bail out with EAGAIN (and any socket
also with EINTR), so retry if we get those calls.
commit 4f57bb12f080de0ba7314319b1f1434e0c2e40a8
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 31 14:54:14 2008 -0700
mog: fix 'tee' command so that it actually outputs to stdout
However, if the stdout stream is pointing to /dev/null, then
just avoid writing anything to save the function call overhead
entirely.
commit 746c453e030190392a0743e8b935a38eab899a7f
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Sep 18 14:00:59 2008 -0700
Allow sysrwloop to take an optional filter Proc
This will let us run our copies through MD5 and
decompression inline.
commit b36b892c735407bba34502e71e95d7481fc3ad79
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Sep 17 18:34:15 2008 -0700
Move http_get_sock into its own function
We'll be reusing it in the big file module
commit 03fe01aa98e00ccf12f58e5bdb69d742e054fa36
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 24 14:25:34 2008 -0700
mysql: documentation for public functions
commit 363775c63c03f45175136bc514eba3ea143a5f09
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 24 14:15:11 2008 -0700
add .gitignore
commit 1c9463b944ac44277598955a2ce331679afff5cd
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 24 14:13:58 2008 -0700
History: add note about MogileFS::Mysql
commit 0bdf69f413f64cbf52a15540cf0441930bf81688
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 24 14:13:06 2008 -0700
LICENSE: add Eric Wong to copyright
commit 5b073c6f5878a1a6b582ab007b6ffa237f6317bb
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 24 14:12:54 2008 -0700
README: formatting changes, remove trailing whitespace
commit f6cbfa23e03c815b7957156a6f3e6c5e33c8ce2c
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 24 14:11:15 2008 -0700
README: note that :root is only required for NFS
commit 97b29016c18b8bd1b5df7a3e218ba9a9d2a1b96c
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 24 13:40:57 2008 -0700
README: add git repository links, update Warning section
commit 47ee90f6c2190f4754d445e75b5e16c766e4da56
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 24 13:31:09 2008 -0700
mysql: allow specifying connect,read,write timeouts
The Mysql driver allows these timeouts to be specified,
so allow the following parameters to be passed to the
initializer:
:connect_timeout
:read_timeout
:write_timeout
These must be integer values (representing seconds)
and will be passed straight through to the underlying
Mysql driver (which will hopefully support sub-second
timeouts at some point...).
All timeouts will default to 1 (second) here.
commit 6b94f724d090d352951f92fceda1b9cd47a997c7
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Oct 21 19:06:34 2008 -0700
Mysql: size() method returns an integer
Returning a string here does not make sense, and we
should be consistent with the MogileFS::MogileFS
interface.
commit 877338e485c614fc7539a47cab7ae2bf8597382d
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Oct 21 18:55:29 2008 -0700
MySQL: switch to async interface
This lets us use multiple green threads safely, but means
we now require the async mysqlplus driver available here:
http://github.com/oldmoe/mysqlplus/tree/master
commit f0df7a9b9b73cb39d445400cf1869de021e516a9
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 17 20:55:20 2008 -0700
MogileFS::Backend: timeout code allows retrying other backends
This allows other trackers to be used if the current
one dies or croaks while we're connected. This also
allows us to better handle spurious wakeups, if we get
any...
commit fb3536089dc7f663458b90f311ab704cfb7b768f
Author: Eric Wong <normalperson@yhbt.net>
Date: Fri Oct 17 14:38:27 2008 -0700
Add MogileFS::Mysql class, allowing read-only access to slaves
This allows for full read redundancy if the master MySQL instance
or trackers go down. This can potentially offer higher
performance; too; as it eliminates trackers from the call
path.
commit 39c131182f33c02eb2883e115a845f0d17062edd
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Oct 16 17:29:53 2008 -0700
Add verify_uris utility method
This lets us do client-side verification of URIs, taking load
off mogilefsd processes and being more reliable (since it'll be
us, the clients, talking to the backend storage nodes to
retrieve/send data and not the mogilefsd processes).
commit f0601cab1c309bf9d213ac8b392f359f2ea87e1d
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 21:06:30 2008 -0700
mog: try more verbose error handling
commit fb3d01d38c5b2b4a839ab0d1da97fe2c0ccc5b8c
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 18:56:01 2008 -0700
Start working on v2.0.0 entry and notes
commit 2aa8badf3b7e1d664e137901ce6bd6b61e793ca6
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 20:06:18 2008 -0700
Avoid double-close on bigfiles
This was leading to ugly "no_temp_file" errors that
got converted to exceptions.
commit 2e9e8cc2c0ea08e090018891dd8538ec675c0825
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 18:51:11 2008 -0700
Backend: use the consistent shutdown() method to close our socket
This always closes it and ignores any exceptions that may
occur (since it's going away anyways). This also sets
the instance variable to nil for easy checking.
commit a9566b561662aa0925eff49adad9813e5ffff179
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 18:47:12 2008 -0700
list_keys trap exception in this method
commit 968cbe34ece986d4d0f072117cf5e278823dfae9
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 18:34:32 2008 -0700
mog: retry on Backend::NoDevicesError
I seem to hit this quite often under heavy load...
commit a62053e547279b59fd2ba3bcbe0a84583ecf23b7
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 18:06:40 2008 -0700
Replace the last of the ad-hoc RuntimeError exceptions
The ArgumentErrors happen at initialization time, so
I'll keep those as-is
commit 20c337eb0c043e0033b794df93b51627f13b729e
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 17:50:22 2008 -0700
mog: increase timeout for interactive use
The default 3 second timeout is too low sometimes on an
overloaded system. Since this is intended for interactive
use, be more patient...
commit c02257508fd7a8e8529b4b37634323fc1136fee9
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 17:45:18 2008 -0700
Backend: convert socket/invalid response errors to exceptions
commit c22186b461c7f73f0a083dae3d40eebab09d7139
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 17:27:57 2008 -0700
HTTPFile: use a standard set of exceptions here, too
Again, easier to trap under MogileFS::Error is better
commit 84c9d66d180aa392a8150b61236ee1577601158c
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 16:46:53 2008 -0700
HTTPFile: remove scary message about this not working
It surely does work for me..
commit 2c09ff42b2e96f28dda283730bdb9deb68946843
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 16:30:15 2008 -0700
Raise MogileFS::ReadOnly error for readonly instances
Also easier to trap and deal with
commit 0d4068e5e890aad959419ad932a162ab170868c5
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 16:24:12 2008 -0700
Use exceptions for all errors sent to us from the backend
commit fb9f44979ee98b7d90c0501986e75d633ec555f0
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 15:27:29 2008 -0700
Backend: automatically create exceptions based on errors we get
This allows us to raise exceptions based on the errors
we get upstream can are easier to trap an deal with
in applications
commit ccdf900ef999019a2a4b04eefb3ec34b4c0eae15
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Oct 15 14:35:13 2008 -0700
create MogileFS::Error class to base our other errors off of
This will allow users to trap MogileFS::Error to trap errors
specific to our library.
commit 35644223ceba73228b8563aef43e75def3da2ecf
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Oct 7 13:43:29 2008 -0700
remove 'open-uri' dependency
The open-uri code path doesn't scale to large files at
all, so there's no point in depending on it as the
TCPSocket code path works just as well.
commit cbbf0f96d36922d237563bc71378902f0d342f36
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Oct 7 13:05:42 2008 -0700
MogileFS::MogileFS:: add paths_size() API call
This lets clients issue a single get_paths() call and
to both retrieve paths and file sizes.
commit ffe803c5fe97890c0b44d78ffb5251e842d2cc13
Author: Eric Wong <normalperson@yhbt.net>
Date: Tue Oct 7 13:01:02 2008 -0700
Use TCPSocket instead of Net::HTTP for HEAD requests
Net::HTTP doesn't save us any lines of code here and it uses up
unnecessary memory and allocations, slightly increasing latency.
Also, while we're at it, more gracefully handle size() failures
if nodes are down.
commit 971b5ccb0708a4cc06772c5e7c2be379b3ff4d3d
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Oct 2 19:59:51 2008 -0700
Version 1.3.1
commit 53a15da3ce5f8bd3f00dcf30dac395423f6cb644
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Oct 2 19:56:56 2008 -0700
Remember to include MogileFS::Util in MogileFS::MogileFS
Tests passed, but certain applications based around this
did not...
commit 242c6ff266959b581ff9bb44abbf897feae92444
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Sep 18 16:42:57 2008 -0700
Version 1.3.0
commit 29c83a1ffe5d28a3f5f71f3fa4c0683bd5d2eb5a
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Sep 18 16:39:56 2008 -0700
Update maintainer and release info
commit f10e8899a1cc969ed2e7afd0b0b0a211ba6b70de
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Sep 18 16:38:29 2008 -0700
Fix ZenTest dependency and make it a dev-only dep
commit 9c866e3c6a1115f5e16aaa6612790905f6cf6391
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Sep 18 16:29:59 2008 -0700
mog: add `test' command, currently only supports "test -e"
Eventually this will have more switches for testing
various bigfile options.
commit 7f93c3f711785cc1215d17a9ab412998260745f0
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Sep 17 16:48:16 2008 -0700
mog: ensure the tempfile is killed at death
commit 71a440d04e6c9496062ed123708b71c051a22290
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Sep 11 18:59:45 2008 -0700
mog: handle SIGPIPE and SIGINT gracefully
SIGPIPE is very handy for scripting, and I hit SIGINT
in the console pretty often, too; so don't spew to
STDERR when we catch these signals.
commit c06411e24f7323688e9036db138caf307ad025f0
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Sep 8 17:36:23 2008 -0700
Add `mog' command-line tool as an example/frontend
This supports several UNIX-like subcommands:
cp FILE KEY - copy a file to a given key
cat KEY(s) - cat any number of keys to STDOUT
ls PREFIX - list keys matching PREFIX (not globbing)
rm KEY(s) - remove keys
mv FROMKEY TOKEY - rename a key
stat KEY(s) - show various information, including URLs and Size
tee KEY - read input from STDIN and write it to key
(due to the limitations of HTTP servers
and clients this is not streamed)
commit 6c8f2db31097998aaab21c05cab53b50bfc243c5
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Sep 4 18:53:14 2008 -0700
Lower the large-file threshold to 64K
Yes, I'm quite miserly when it comes to memory usage. Since the
file is already on disk, just read it incrementally and stream
it out to avoid having to deal with potential memory exhaustion
issues on busy systems. There's also no benefit to slurping
256MB and anything above 64K leads to diminishing returns on
most systems I've seen.
commit 673861cd6fcb8f563de9c6300252d5ff290adee1
Author: Eric Wong <normalperson@yhbt.net>
Date: Thu Sep 4 18:47:39 2008 -0700
mogilefs: only timeout on connect/header reading
Reading the rest of the body can take a long time for big files
(we expect that for big blocks) and the timeout is stupid in
that case.
commit 71c71da5df7f680bd7dbdf4c28c7da3c4ee3ed45
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Sep 8 17:29:17 2008 -0700
Allow get_file_data to accept a block for large files
This block will be passed the IO object for reading the file.
This is to prevent the client from having to slurp an entire
large file into memory all at once.
commit 99735076974cad8c5485b949179701f9d36b0699
Author: Eric Wong <normalperson@yhbt.net>
Date: Mon Sep 8 17:26:02 2008 -0700
Add MogileFS::Util for a sysrwloop() function
Enable it in the HTTPFile PUT code. We'll also use
it for get_file_data when handling large files next.
commit efaeed97b61f59d493ed8942d711fc705e0e9fb2
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Sep 3 20:40:43 2008 -0700
Fix each_keys loop termination condition
The underlying list_keys function can return nil,
so don't try to run .empty? on nil.
commit 651eeeacec1e9c0d59652c3a0d90ee83ddea46a8
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Sep 3 15:36:25 2008 -0700
HTTPFile: use IO.sysread for better performance
IO.sysread is more GC-friendly than IO.read because it does not
have to allocate additional userspace buffers. Userspace read
buffering is redundant nowadays with modern kernels (especially
in Linux 2.6 and almost always a performance hit).
Additionally, use an underdocumented feature of both IO.sysread
and IO.read that allows it to reuse the existing buffer.
Further reducing garbage collector overhead for large files.
commit 4c1f1924271dc828bb5eb22461b2b02390b11f6f
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Sep 3 15:24:13 2008 -0700
HTTPFile: more reasonable chunk size for big files
16M for a chunk is a huge amount of memory to slurp at once.
64K is much more reasonable and chunk sizes above this lead
to dimishing returns in performance.
commit 5939e0855ff1e84021cbfd4a72a6b6d9827a25df
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Sep 3 15:12:36 2008 -0700
Apply error handling patch from Matthew Willson
Ref: http://rubyforge.org/tracker/index.php?func=detail&aid=15987&group_id=1513&atid=5923
> Submitted By: Matthew Willson
> Summary: Errors on subsequent requests after client times out
> waiting for a response from tracker on a previous request (it
> leaves the socket open)
>
> Detailed description
>
> The summary says it all really.
>
> Once in a while, the tracker will time out responding to, say, a
> create_open or a create_close command, raising one of these:
>
> MogileFS::UnreadableSocketError: 146.101.142.132:7001 never became readable
> from /usr/lib/ruby/gems/1.8/gems/mogilefs-client-1.2.1/lib/mogilefs/backend.rb:158:in `readable?'
> from /usr/lib/ruby/gems/1.8/gems/mogilefs-client-1.2.1/lib/mogilefs/backend.rb:122:in `do_request'
> from /usr/lib/ruby/1.8/thread.rb:135:in `synchronize'
> from /usr/lib/ruby/gems/1.8/gems/mogilefs-client-1.2.1/lib/mogilefs/backend.rb:108:in `do_request'
> from /usr/lib/ruby/gems/1.8/gems/mogilefs-client-1.2.1/lib/mogilefs/backend.rb:16:in `create_open'
> from /usr/lib/ruby/gems/1.8/gems/mogilefs-client-1.2.1/lib/mogilefs/mogilefs.rb:108:in `new_file'
> from /usr/lib/ruby/gems/1.8/gems/mogilefs-client-1.2.1/lib/mogilefs/mogilefs.rb:163:in `store_content'
> from ./script/../config/../config/../app/models/mogile_backed.rb:59:in `store_in_mogile'
> from ./script/../config/../config/../app/models/image.rb:139:in `data'
> from ./script/../config/../config/../app/models/mogile_backed.rb:59:in `store_in_mogile'
> from (irb):15
> from (irb):15
>
> The problem is that, if you code catches this error and carries
> on using the same client object for a subsequent request, the
> 'OK' response which we timed out waiting for, will eventually
> arrive, and sit in the socket's read buffer. It will then be
> read and treated as the response to some unconnected subsequent
> command, resulting in a variety of seemingly intermittent and
> confusing errors.
>
> I've attached a patch for this against 1.2.1, which simply
> closes the socket whenever it times out waiting for a reponse.
> The next request will then open a new socket as required.
>
> Also included a quick fix/improvement to error reporting in one
> case, which helped me to track the problem down.
commit b0b702171b01e391d9fe29eb4374d971c2f5bc83
Author: Eric Wong <normalperson@yhbt.net>
Date: Wed Sep 3 15:05:56 2008 -0700
merge large file patch from Andy La Foe
Merging involved "Content-length" => "Content-Length"
capitalizing 'L' as per p4#3627 (aka SVN r433)
Ref: http://rubyforge.org/tracker/index.php?func=detail&aid=13764&group_id=1513&atid=5923
> Submitted By: Andy Lo-A-Foe (arloafoe)
> Category: mogilefs-client
> Summary:
> Store very large files (> 256M) without running out of memory in store_file
>
> Detailed description
>
> This is a patch to the MogileFS::store_file mechanism in order to
> support very large filee stores using HTTPFile. We sometimes have to
> store files of up to 1GB in size. Using chunking is not really an option
> since it has proven to be very unreliable (mogtool) and there is no
> support for it in the current version of this client. This patch
> basically reads 16M chunks at a time and writes them to the tracker
> socket instead of trying to stuff the while file in the StringIO and
> running out of memory. It's probably very rough and the get_file_data
> symmetry patch is not there yet. Feedback appreciated.
commit 2c1c7312f96fd2688366c5637e749c668c0dceb7
Author: zenspider <zenspider@d2e05cf2-00e0-46e5-a3de-bbee4d6b9404>
Date: Fri Mar 21 22:22:07 2008 +0000
Credit for bug report.
From p4 revision #3630
git-svn-id: http://seattlerb.rubyforge.org/svn/mogilefs-client/dev@434 d2e05cf2-00e0-46e5-a3de-bbee4d6b9404
commit 5432c0e4d6b05bcc49c3a7c48eeff11f657beeca
Author: zenspider <zenspider@d2e05cf2-00e0-46e5-a3de-bbee4d6b9404>
Date: Fri Mar 21 22:22:00 2008 +0000
Fixed MogileFS#rename.
Removed infinite loop in MogileFS::HTTPFile#store_file.
Made MogileFS#get_file_data timeout configurable.
Add MogileFS#size.
From p4 revision #3627
git-svn-id: http://seattlerb.rubyforge.org/svn/mogilefs-client/dev@433 d2e05cf2-00e0-46e5-a3de-bbee4d6b9404
commit e20f64cd7ba2deb249b9b51ce9154ba5337ddf08
Author: zenspider <zenspider@d2e05cf2-00e0-46e5-a3de-bbee4d6b9404>
Date: Fri Mar 21 22:15:07 2008 +0000
Fixed bug #7273 in HTTP mode of client where data would not get returned. Submitted by Matthew Willson.
From p4 revision #3337
git-svn-id: http://seattlerb.rubyforge.org/svn/mogilefs-client/dev@377 d2e05cf2-00e0-46e5-a3de-bbee4d6b9404
commit d7aba2f7ca1c1282b5c9afaabe2082b4d1a85170
Author: zenspider <zenspider@d2e05cf2-00e0-46e5-a3de-bbee4d6b9404>
Date: Fri Mar 21 22:15:02 2008 +0000
Import mogilefs-client converted to Hoe.
From p4 revision #3336
git-svn-id: http://seattlerb.rubyforge.org/svn/mogilefs-client/dev@376 d2e05cf2-00e0-46e5-a3de-bbee4d6b9404
Originally generated with the Darkfish Rdoc Generator 2, modified by wrongdoc.