| Date | Commit message (Collapse) |
|---|
|
MogileFS::MogileFS#each_key now accepts optional :after and :limit args
This allows easy iteration while respecting :after and :limit,
giving each_key an interface consistent with each_file_info.
|
|
This allows easy iteration while respecting :after and :limit.
This gives each_key an interface consistent with each_file_info.
|
|
We no longer send an empty zone= parameter in create_open
calls. Additionally, if we detect send() failure on a socket,
assume a tracker has been restarted and restart the request.
There are also some minor code cleanups.
|
|
Note: this mysql component is still largely unmaintained.
|
|
This bug was only introduced in
commit 3c73f6b434a4c226c09e31c6ba3b034cb314b3fb
|
|
In single tracker configurations, a restarted tracker may cause
send()/write() failure on the TCP socket. Retry immediately in
this case, since there's no danger even for non-idempotent
tracker requests.
|
|
It's a small waste and adds to visual noise.
|
|
IO.copy_stream does not raise EOFError when specified
copy length is less than input size. Thus we must rely
on the MogileFS tracker to validate the input size.
|
|
client changes:
* new_file gains :create_open_args and :create_close_args which
allows custom arguments to be passed to plugins.
(this matches the Perl client behavior)
* new_file also gains :info hash which can be populated with
information normally retrieved with file_info
* users with net-http-persistent installed will see a small speed
boost and reduction of TIME_WAIT sockets when dealing with
small data (useful for testing DB-intensive parts of MogileFS).
No changes for admin.
|
|
Avoid needless encoding and data transfer to the tracker.
|
|
client changes:
* new_file gains :create_open_args and :create_close_args which
allows custom arguments to be passed to plugins.
* new_file also gains :info hash which can be populated with
information normally retrieved with file_info
* users with net-http-persistent installed will see a small speed
boost and reduction of TIME_WAIT sockets when dealing with
small data (useful for testing DB-intensive parts of MogileFS).
No changes for admin.
|
|
We now have separate Net::HTTP::Persistent instances
between clients that may have different timeouts and
also between GET and PUT requests. This hurts our
ability to reuse sockets, but correctness is probably
more important.
|
|
Less noise should be better.
|
|
Implementation details should remain private.
|
|
This lets us send unsupported/new arguments to plugins
and matches the functionality of the Perl client library.
|
|
It's possible for networks and servers to break up
even small HTTP headers. We also better enforce the
timeout if we too too long to write the request.
|
|
Tests fail anyways on uncaught exceptions, so
assert_nothing_raised only makes it harder to debug the problem
by swallowing the backtrace.
|
|
We don't need to set socket keepalive until we've successfully
written the request out and are awaiting a response.
|
|
Given StringIO objects are already in memory, NHP can make
small uploads which fit into memory faster. Large uploads
(using big_io or :largefile => :stream still go through
IO.copy_stream for now)
|
|
This allows clients to avoid calling #file_info or #get_uris
immediate after uploading a file to MogileFS. This can speed
things up for cache-using clients with write-through caching.
|
|
Admin speedups for get_domains, get_hosts, and get_devices.
Previous versions were completely unusable for parsing a list of
3000+ domains. Installations with thousands of hosts or devices
should see noticeable performance improvements.
The client interface gains the each_file_info iterator method.
This behaves like each_key, but gives access to the entire
response the file_info returns, including: checksum, devcount,
file size, class, and domain.
|
|
Array#pop is faster than Array#shift in most Ruby versions
as the latter may require memmove() of all elements.
Additionally, ensure ordering is correct if a backend
chokes up (for each_file_info, too).
|
|
This allows fast listing of keys and metadata (length,
checksum, devcount, class).
|
|
Blank observed_state (for dead devices) are mapped to nil.
|
|
By avoiding #grep, installations with thousands of hosts/devices
should be sped up considerably. list_fids should be roughly
twice as fast.
This is a followup to commit a309f22f835afe0e6be0e4e2f1a13eaead7434f2
|
|
On an installation with 3000+ domains, get_domains gets
bottlenecked by calling #grep on the keys of a hash. Allow
specifying a "want" directive to filter in data we expect
(instead of grepping blindly)
|
|
No changes since 3.3.0-rc1.
We now correctly timeout requests to slow trackers and avoid
reusing the socket after timeouts (as old responses can arrive
late). A big thanks to David Rasch for helping with
timeout/idempotency issues for this release.
Minor documentation/packaging updates, it should be easier
to build a pre-release gem now (see HACKING doc).
|
|
We now correctly timeout requests to slow trackers and avoid
reusing the socket after timeouts (as old responses can arrive
late). A big thanks to David Rasch for helping with
timeout/idempotency issues for this release.
Minor documentation/packaging updates, it should be easier
to build a pre-release gem now (see HACKING doc).
|
|
In case users are unfamiliar with Hoe.
|
|
Don't require users to have wrongdoc installed to generate
a prerelease gem
|
|
Hoe#url is deprecated and Hoe#urls replaces it
|
|
This test needs to keep the socket alive on the server to ensure
the client can drop the connection.
We also need to ensure the test fails if the following change is
made:
--- a/lib/mogilefs/backend.rb
+++ b/lib/mogilefs/backend.rb
@@ -255,7 +255,7 @@ def do_request(cmd, args, idempotent = false)
end
shutdown_unlocked(true)
rescue MogileFS::UnreadableSocketError, MogileFS::Timeout
- shutdown_unlocked(true)
+ # shutdown_unlocked(true)
rescue
# we DO NOT want the response we timed out waiting for, to crop up later
# on, on the same socket, intersperesed with a subsequent request! we
Thanks to David Rasch for inspiring this change.
|
|
While retrying idempotent requests (even on timeouts) would
prevent stale sockets from being noticed, it is better to kill
the socket and immediately propagate the timeout error to the
user. Retrying in a timeout may cause a request/response to
take longer (perhaps _much_ longer) than the timeout configured
by the user.
|
|
* backend/do_request: when a request times out for a slow server,
we now continue retries until we get a socket error, and close the
connection afterward so we don't get interspersed responses
* test: added a test for slow servers
Signed-off-by: Eric Wong <normalperson@yhbt.net>
|
|
Retrying to open a source file that cannot be opened
(repeatedly) and finally raising NoStorageNodesError is
confusing to users.
|
|
Changes since 3.2.0-rc1 (MogileFS::Admin-only):
* admin supports "create_device" and "change_device_weight" commands
* admin casts "reject_bad_md5" field in "get_device" return
value to boolean (true/false).
* minor code simplifications to admin
Changes since 3.1.1
* "list_keys" and "exist?" client commands raise errors properly
on failure.
* backend connections no longer terminate on ERR responses, only
on socket/connection errors.
* support the "updateclass" client command. This is for updating
the class of a given key and not to be confused with the
"update_class" admin command.
* "new_file" checksum usage is now documented since MogileFS 2.60
includes official support for checksums
|
|
assert_include? isn't a part of test/unit in 1.8.7
|
|
This makes things easier-to-read, and admin functions aren't
performance critical so we won't worry about it.
|
|
This is a boolean value on the server and we always try to
make our return values Ruby-friendly
|
|
This feature uses the "set_weight" command to change
device weights and has a part of the MogileFS protocol
for many years, now.
|
|
The ability to create devices from the Ruby API might be
useful to somebody, especially when writing test cases.
|
|
* "list_keys" and "exist?" client commands raise errors properly
on failure.
* backend connections no longer terminate on ERR responses, only
on socket/connection errors.
* support the "updateclass" client command. This is for updating
the class of a given key and not to be confused with the
"update_class" admin command.
* "new_file" checksum usage is now documented since MogileFS 2.60
includes official support for checksums
|
|
The server sending us a properly-formed "ERR" response is a big
difference than an actual socket or connection error.
|
|
Specifying an invalid domain will raise
MogileFS::Backend::UnregDomain error instead of merely
returning `false'. Only checks for keys in the correct
domain (but non-existent keys) return false.
|
|
list_keys needs to raise UnregDomainError on invalid domains.
This regression was introduced in
commit 0b933fc83b8b519c8e587caa6606dd372dda18af
|
|
Avoid taking sides or showing preference for particular text
editors. Text editors are _very_ personal preferences, and
explicitly supporting/acknowledging one would set precedence for
explicitly supporting/acknowledging others. This would lead
down the slippery slope of having modelines for every single
text editor in every single file we have.
For this same reason, we do not (and never will) expose
editor-specific suffixes in .gitignore.
Full disclosure: I'm a vim user myself
|
|
Now that checksum support is officially a part of MogileFS, we
can document (and thus encourage) it without risking
compatibility issues.
|
|
This changes the class associated with +key+.
This is _not_ the same as the "update_class" admin command
which actually modifies the class itself.
|
|
These files should be included in release gem/tarballs, users
should not require web acccess to access documentation.
|
|
Not everybody uses RubyGems.
|