Date | Commit message (Collapse) |
|
client changes:
* new_file gains :create_open_args and :create_close_args which
allows custom arguments to be passed to plugins.
(this matches the Perl client behavior)
* new_file also gains :info hash which can be populated with
information normally retrieved with file_info
* users with net-http-persistent installed will see a small speed
boost and reduction of TIME_WAIT sockets when dealing with
small data (useful for testing DB-intensive parts of MogileFS).
No changes for admin.
|
|
Avoid needless encoding and data transfer to the tracker.
|
|
client changes:
* new_file gains :create_open_args and :create_close_args which
allows custom arguments to be passed to plugins.
* new_file also gains :info hash which can be populated with
information normally retrieved with file_info
* users with net-http-persistent installed will see a small speed
boost and reduction of TIME_WAIT sockets when dealing with
small data (useful for testing DB-intensive parts of MogileFS).
No changes for admin.
|
|
We now have separate Net::HTTP::Persistent instances
between clients that may have different timeouts and
also between GET and PUT requests. This hurts our
ability to reuse sockets, but correctness is probably
more important.
|
|
Less noise should be better.
|
|
Implementation details should remain private.
|
|
This lets us send unsupported/new arguments to plugins
and matches the functionality of the Perl client library.
|
|
It's possible for networks and servers to break up
even small HTTP headers. We also better enforce the
timeout if we too too long to write the request.
|
|
Tests fail anyways on uncaught exceptions, so
assert_nothing_raised only makes it harder to debug the problem
by swallowing the backtrace.
|
|
We don't need to set socket keepalive until we've successfully
written the request out and are awaiting a response.
|
|
Given StringIO objects are already in memory, NHP can make
small uploads which fit into memory faster. Large uploads
(using big_io or :largefile => :stream still go through
IO.copy_stream for now)
|
|
This allows clients to avoid calling #file_info or #get_uris
immediate after uploading a file to MogileFS. This can speed
things up for cache-using clients with write-through caching.
|
|
Admin speedups for get_domains, get_hosts, and get_devices.
Previous versions were completely unusable for parsing a list of
3000+ domains. Installations with thousands of hosts or devices
should see noticeable performance improvements.
The client interface gains the each_file_info iterator method.
This behaves like each_key, but gives access to the entire
response the file_info returns, including: checksum, devcount,
file size, class, and domain.
|
|
Array#pop is faster than Array#shift in most Ruby versions
as the latter may require memmove() of all elements.
Additionally, ensure ordering is correct if a backend
chokes up (for each_file_info, too).
|
|
This allows fast listing of keys and metadata (length,
checksum, devcount, class).
|
|
Blank observed_state (for dead devices) are mapped to nil.
|
|
By avoiding #grep, installations with thousands of hosts/devices
should be sped up considerably. list_fids should be roughly
twice as fast.
This is a followup to commit a309f22f835afe0e6be0e4e2f1a13eaead7434f2
|
|
On an installation with 3000+ domains, get_domains gets
bottlenecked by calling #grep on the keys of a hash. Allow
specifying a "want" directive to filter in data we expect
(instead of grepping blindly)
|
|
No changes since 3.3.0-rc1.
We now correctly timeout requests to slow trackers and avoid
reusing the socket after timeouts (as old responses can arrive
late). A big thanks to David Rasch for helping with
timeout/idempotency issues for this release.
Minor documentation/packaging updates, it should be easier
to build a pre-release gem now (see HACKING doc).
|
|
We now correctly timeout requests to slow trackers and avoid
reusing the socket after timeouts (as old responses can arrive
late). A big thanks to David Rasch for helping with
timeout/idempotency issues for this release.
Minor documentation/packaging updates, it should be easier
to build a pre-release gem now (see HACKING doc).
|
|
In case users are unfamiliar with Hoe.
|
|
Don't require users to have wrongdoc installed to generate
a prerelease gem
|
|
Hoe#url is deprecated and Hoe#urls replaces it
|
|
This test needs to keep the socket alive on the server to ensure
the client can drop the connection.
We also need to ensure the test fails if the following change is
made:
--- a/lib/mogilefs/backend.rb
+++ b/lib/mogilefs/backend.rb
@@ -255,7 +255,7 @@ def do_request(cmd, args, idempotent = false)
end
shutdown_unlocked(true)
rescue MogileFS::UnreadableSocketError, MogileFS::Timeout
- shutdown_unlocked(true)
+ # shutdown_unlocked(true)
rescue
# we DO NOT want the response we timed out waiting for, to crop up later
# on, on the same socket, intersperesed with a subsequent request! we
Thanks to David Rasch for inspiring this change.
|
|
While retrying idempotent requests (even on timeouts) would
prevent stale sockets from being noticed, it is better to kill
the socket and immediately propagate the timeout error to the
user. Retrying in a timeout may cause a request/response to
take longer (perhaps _much_ longer) than the timeout configured
by the user.
|
|
* backend/do_request: when a request times out for a slow server,
we now continue retries until we get a socket error, and close the
connection afterward so we don't get interspersed responses
* test: added a test for slow servers
Signed-off-by: Eric Wong <normalperson@yhbt.net>
|
|
Retrying to open a source file that cannot be opened
(repeatedly) and finally raising NoStorageNodesError is
confusing to users.
|
|
Changes since 3.2.0-rc1 (MogileFS::Admin-only):
* admin supports "create_device" and "change_device_weight" commands
* admin casts "reject_bad_md5" field in "get_device" return
value to boolean (true/false).
* minor code simplifications to admin
Changes since 3.1.1
* "list_keys" and "exist?" client commands raise errors properly
on failure.
* backend connections no longer terminate on ERR responses, only
on socket/connection errors.
* support the "updateclass" client command. This is for updating
the class of a given key and not to be confused with the
"update_class" admin command.
* "new_file" checksum usage is now documented since MogileFS 2.60
includes official support for checksums
|
|
assert_include? isn't a part of test/unit in 1.8.7
|
|
This makes things easier-to-read, and admin functions aren't
performance critical so we won't worry about it.
|
|
This is a boolean value on the server and we always try to
make our return values Ruby-friendly
|
|
This feature uses the "set_weight" command to change
device weights and has a part of the MogileFS protocol
for many years, now.
|
|
The ability to create devices from the Ruby API might be
useful to somebody, especially when writing test cases.
|
|
* "list_keys" and "exist?" client commands raise errors properly
on failure.
* backend connections no longer terminate on ERR responses, only
on socket/connection errors.
* support the "updateclass" client command. This is for updating
the class of a given key and not to be confused with the
"update_class" admin command.
* "new_file" checksum usage is now documented since MogileFS 2.60
includes official support for checksums
|
|
The server sending us a properly-formed "ERR" response is a big
difference than an actual socket or connection error.
|
|
Specifying an invalid domain will raise
MogileFS::Backend::UnregDomain error instead of merely
returning `false'. Only checks for keys in the correct
domain (but non-existent keys) return false.
|
|
list_keys needs to raise UnregDomainError on invalid domains.
This regression was introduced in
commit 0b933fc83b8b519c8e587caa6606dd372dda18af
|
|
Avoid taking sides or showing preference for particular text
editors. Text editors are _very_ personal preferences, and
explicitly supporting/acknowledging one would set precedence for
explicitly supporting/acknowledging others. This would lead
down the slippery slope of having modelines for every single
text editor in every single file we have.
For this same reason, we do not (and never will) expose
editor-specific suffixes in .gitignore.
Full disclosure: I'm a vim user myself
|
|
Now that checksum support is officially a part of MogileFS, we
can document (and thus encourage) it without risking
compatibility issues.
|
|
This changes the class associated with +key+.
This is _not_ the same as the "update_class" admin command
which actually modifies the class itself.
|
|
These files should be included in release gem/tarballs, users
should not require web acccess to access documentation.
|
|
Not everybody uses RubyGems.
|
|
This releases fixes problems short reads when slurping files
into memory. Thanks to Matthew Draper for this fix.
There are also minor documentation updates.
|
|
It's more future-proof, this way.
|
|
Not that it really matters given the MIT license...
|
|
It's better if email addresses are visible and not hidden behind
links for folks that don't integrate email clients into web
browsers.
|
|
Leave size unchanged, so it's available for easy comparison with
buf.bytesize.
[ew: added test case]
Acked-by: Eric Wong <normalperson@yhbt.net>
|
|
:noverify always defaults to _true_ (meaning verification is
_off_ by default). This double negative is unfortunately
a confusing part of the existing API and MogileFS protocol.
|
|
|
|
* improved API support for uploading large files
While we've always supported uploading large files, the
(still-supported) existing APIs were somewhat awkward or
required the file to exist on the file system. See
MogileFS::NewFile for details and examples.
* more informative exception messages for timed-out requests
* :fail_timeout parameter, the timeout for retrying a failed
tracker connection. This defaults to 5 seconds (same as
previous versions where this was hard-coded.
* :new_file_max_time parameter
Controls the maximum of time spent creating and uploading
a new file in MogileFS. This defaults to 1 hour (which
matching the expiry time of a row in the MogileFS internal
tempfile table).
* store_file works on unlinked File/Tempfile objects
* each_fid method in MogileFS::Admin fixed
* stale_fid_checker example script added
* mogstored_rack example split into a standalone RubyGem:
http://bogomips.org/mogstored_rack/
* backend error constants are generated on const_missing,
instead of when raised, making it easier to rescue exceptions
we didn't explicitly enable
* some internal cleanups and documentation improvements
|