| Date | Commit message (Collapse) |
|---|
|
Mostly documentation updates and test cleanups. Nothing interesting
for the regular client, but the admin client now shows the
checksum-related fields in get_devices.
Most notably, there's a public-inbox for those who don't want
to subscribe to the MogileFS list:
mogilefs-client-public@bogomips.org
Archives are at: http://bogomips.org/mogilefs-client-public/
There is no new mailing list subscription, it is only a
public-inbox. Cc-ing the MogileFS list at mogile@googlegroups.com
is encouraged if you do have a subscription.
shortlog since 3.7.1:
add mog-sync example script
admin (doc, get_hosts): display integers in results doc
update documentation and packaging
GNUmakefile: publish examples on the site, too
examples: add usage_fetcher example
TODO: remove Cool.io and EventMachine references
admin: flesh out get_devices
admin: reduce bytecode overhead of get_stats
tests: create fresh intances for all integration tests
include mogilefs/version.rb in the distro
test: fixup object lifetimes and teardown
new public-inbox: mogilefs-client-public@bogomips.org
|
|
Probably not worth being a full-blown mailing list since
mogile@googlegroups.com exists, but it's nice to have a public place
where people can post without subscribing to anything.
|
|
This should prevent stray processes from being leftover
after tests are run as well as cleaning up tmpdirs
more thoroughly.
|
|
|
|
This should make it easier to fully test on machines without
access to an existing MogileFS instance. You'll still
need mogilefsd and mogstored available, however, but these
tests are skipped on machines without them.
|
|
This is a deprecated call anyways and no longer supported
by modern versions of the server, so avoid wasting RAM with it.
We will remove this in 4.x
|
|
This adds the reject_bad_md5 and utilization fields.
While we're at it, be more explicit with mapping and
avoid creating an unnecessary hash.
|
|
Cool.io is unmaintained and EM seems to be, too.
|
|
This was written a while ago for a large cluster, so I
figured I might as well include it here.
|
|
|
|
The documentation (and website) uses olddoc, now,
allowing faster load times and more consistent
user experience across different browsers, particularly
text-only ones.
The packaging is also updated to be consistent with
the rest of the projects I deal with.
|
|
Our code integerizes certain results nowadays, so update
the documentation to match.
|
|
This allows verbatim copying between two domains on different
(or the same) MogileFS instance.
It is idempotent and may be used like "rsync -a" for periodic
syncing.
It is also multi-threaded with configurable concurrency for
metadata vs file transfer checks.
Usage: mog-sync.rb SRC_TRACKER_LIST/SRC_DOMAIN DST_TRACKER_LIST/DST_DOMAIN
-j, --metadata-jobs JOBS Number of metadata jobs to run in parallel
-J, --copy-jobs JOBS Number of copy jobs to run in parallel
-h, --help Show this help message.
--get-file-data-timeout SECONDS
--new-file-max-time SECONDS
--fail-timeout SECONDS
--timeout SECONDS
-v, --verbose
-d, --delete
-n, --dry-run
-p, --prefix STRING
--src-class STRING
--dst-class STRING
--after STRING
--max-size STRING
-F, --clobber-missing-checksum
|
|
Only one bugfix:
new_file/stream: correct declare errors array correctly
This only triggered on rare syscall errors
(Errno::EMFILE/Errno::ENFILE)
|
|
This caused NameError failures when we ran out of file descriptors.
|
|
MogileFS::MogileFS#each_key now accepts optional :after and :limit args
This allows easy iteration while respecting :after and :limit,
giving each_key an interface consistent with each_file_info.
|
|
This allows easy iteration while respecting :after and :limit.
This gives each_key an interface consistent with each_file_info.
|
|
We no longer send an empty zone= parameter in create_open
calls. Additionally, if we detect send() failure on a socket,
assume a tracker has been restarted and restart the request.
There are also some minor code cleanups.
|
|
Note: this mysql component is still largely unmaintained.
|
|
This bug was only introduced in
commit 3c73f6b434a4c226c09e31c6ba3b034cb314b3fb
|
|
In single tracker configurations, a restarted tracker may cause
send()/write() failure on the TCP socket. Retry immediately in
this case, since there's no danger even for non-idempotent
tracker requests.
|
|
It's a small waste and adds to visual noise.
|
|
IO.copy_stream does not raise EOFError when specified
copy length is less than input size. Thus we must rely
on the MogileFS tracker to validate the input size.
|
|
client changes:
* new_file gains :create_open_args and :create_close_args which
allows custom arguments to be passed to plugins.
(this matches the Perl client behavior)
* new_file also gains :info hash which can be populated with
information normally retrieved with file_info
* users with net-http-persistent installed will see a small speed
boost and reduction of TIME_WAIT sockets when dealing with
small data (useful for testing DB-intensive parts of MogileFS).
No changes for admin.
|
|
Avoid needless encoding and data transfer to the tracker.
|
|
client changes:
* new_file gains :create_open_args and :create_close_args which
allows custom arguments to be passed to plugins.
* new_file also gains :info hash which can be populated with
information normally retrieved with file_info
* users with net-http-persistent installed will see a small speed
boost and reduction of TIME_WAIT sockets when dealing with
small data (useful for testing DB-intensive parts of MogileFS).
No changes for admin.
|
|
We now have separate Net::HTTP::Persistent instances
between clients that may have different timeouts and
also between GET and PUT requests. This hurts our
ability to reuse sockets, but correctness is probably
more important.
|
|
Less noise should be better.
|
|
Implementation details should remain private.
|
|
This lets us send unsupported/new arguments to plugins
and matches the functionality of the Perl client library.
|
|
It's possible for networks and servers to break up
even small HTTP headers. We also better enforce the
timeout if we too too long to write the request.
|
|
Tests fail anyways on uncaught exceptions, so
assert_nothing_raised only makes it harder to debug the problem
by swallowing the backtrace.
|
|
We don't need to set socket keepalive until we've successfully
written the request out and are awaiting a response.
|
|
Given StringIO objects are already in memory, NHP can make
small uploads which fit into memory faster. Large uploads
(using big_io or :largefile => :stream still go through
IO.copy_stream for now)
|
|
This allows clients to avoid calling #file_info or #get_uris
immediate after uploading a file to MogileFS. This can speed
things up for cache-using clients with write-through caching.
|
|
Admin speedups for get_domains, get_hosts, and get_devices.
Previous versions were completely unusable for parsing a list of
3000+ domains. Installations with thousands of hosts or devices
should see noticeable performance improvements.
The client interface gains the each_file_info iterator method.
This behaves like each_key, but gives access to the entire
response the file_info returns, including: checksum, devcount,
file size, class, and domain.
|
|
Array#pop is faster than Array#shift in most Ruby versions
as the latter may require memmove() of all elements.
Additionally, ensure ordering is correct if a backend
chokes up (for each_file_info, too).
|
|
This allows fast listing of keys and metadata (length,
checksum, devcount, class).
|
|
Blank observed_state (for dead devices) are mapped to nil.
|
|
By avoiding #grep, installations with thousands of hosts/devices
should be sped up considerably. list_fids should be roughly
twice as fast.
This is a followup to commit a309f22f835afe0e6be0e4e2f1a13eaead7434f2
|
|
On an installation with 3000+ domains, get_domains gets
bottlenecked by calling #grep on the keys of a hash. Allow
specifying a "want" directive to filter in data we expect
(instead of grepping blindly)
|
|
No changes since 3.3.0-rc1.
We now correctly timeout requests to slow trackers and avoid
reusing the socket after timeouts (as old responses can arrive
late). A big thanks to David Rasch for helping with
timeout/idempotency issues for this release.
Minor documentation/packaging updates, it should be easier
to build a pre-release gem now (see HACKING doc).
|
|
We now correctly timeout requests to slow trackers and avoid
reusing the socket after timeouts (as old responses can arrive
late). A big thanks to David Rasch for helping with
timeout/idempotency issues for this release.
Minor documentation/packaging updates, it should be easier
to build a pre-release gem now (see HACKING doc).
|
|
In case users are unfamiliar with Hoe.
|
|
Don't require users to have wrongdoc installed to generate
a prerelease gem
|
|
Hoe#url is deprecated and Hoe#urls replaces it
|
|
This test needs to keep the socket alive on the server to ensure
the client can drop the connection.
We also need to ensure the test fails if the following change is
made:
--- a/lib/mogilefs/backend.rb
+++ b/lib/mogilefs/backend.rb
@@ -255,7 +255,7 @@ def do_request(cmd, args, idempotent = false)
end
shutdown_unlocked(true)
rescue MogileFS::UnreadableSocketError, MogileFS::Timeout
- shutdown_unlocked(true)
+ # shutdown_unlocked(true)
rescue
# we DO NOT want the response we timed out waiting for, to crop up later
# on, on the same socket, intersperesed with a subsequent request! we
Thanks to David Rasch for inspiring this change.
|
|
While retrying idempotent requests (even on timeouts) would
prevent stale sockets from being noticed, it is better to kill
the socket and immediately propagate the timeout error to the
user. Retrying in a timeout may cause a request/response to
take longer (perhaps _much_ longer) than the timeout configured
by the user.
|
|
* backend/do_request: when a request times out for a slow server,
we now continue retries until we get a socket error, and close the
connection afterward so we don't get interspersed responses
* test: added a test for slow servers
Signed-off-by: Eric Wong <normalperson@yhbt.net>
|
|
Retrying to open a source file that cannot be opened
(repeatedly) and finally raising NoStorageNodesError is
confusing to users.
|