| Date | Commit message (Collapse) |
|---|
|
While retrying idempotent requests (even on timeouts) would
prevent stale sockets from being noticed, it is better to kill
the socket and immediately propagate the timeout error to the
user. Retrying in a timeout may cause a request/response to
take longer (perhaps _much_ longer) than the timeout configured
by the user.
|
|
* backend/do_request: when a request times out for a slow server,
we now continue retries until we get a socket error, and close the
connection afterward so we don't get interspersed responses
* test: added a test for slow servers
Signed-off-by: Eric Wong <normalperson@yhbt.net>
|
|
Changes since 3.2.0-rc1 (MogileFS::Admin-only):
* admin supports "create_device" and "change_device_weight" commands
* admin casts "reject_bad_md5" field in "get_device" return
value to boolean (true/false).
* minor code simplifications to admin
Changes since 3.1.1
* "list_keys" and "exist?" client commands raise errors properly
on failure.
* backend connections no longer terminate on ERR responses, only
on socket/connection errors.
* support the "updateclass" client command. This is for updating
the class of a given key and not to be confused with the
"update_class" admin command.
* "new_file" checksum usage is now documented since MogileFS 2.60
includes official support for checksums
|
|
assert_include? isn't a part of test/unit in 1.8.7
|
|
This makes things easier-to-read, and admin functions aren't
performance critical so we won't worry about it.
|
|
This is a boolean value on the server and we always try to
make our return values Ruby-friendly
|
|
This feature uses the "set_weight" command to change
device weights and has a part of the MogileFS protocol
for many years, now.
|
|
The ability to create devices from the Ruby API might be
useful to somebody, especially when writing test cases.
|
|
* "list_keys" and "exist?" client commands raise errors properly
on failure.
* backend connections no longer terminate on ERR responses, only
on socket/connection errors.
* support the "updateclass" client command. This is for updating
the class of a given key and not to be confused with the
"update_class" admin command.
* "new_file" checksum usage is now documented since MogileFS 2.60
includes official support for checksums
|
|
The server sending us a properly-formed "ERR" response is a big
difference than an actual socket or connection error.
|
|
Specifying an invalid domain will raise
MogileFS::Backend::UnregDomain error instead of merely
returning `false'. Only checks for keys in the correct
domain (but non-existent keys) return false.
|
|
list_keys needs to raise UnregDomainError on invalid domains.
This regression was introduced in
commit 0b933fc83b8b519c8e587caa6606dd372dda18af
|
|
Avoid taking sides or showing preference for particular text
editors. Text editors are _very_ personal preferences, and
explicitly supporting/acknowledging one would set precedence for
explicitly supporting/acknowledging others. This would lead
down the slippery slope of having modelines for every single
text editor in every single file we have.
For this same reason, we do not (and never will) expose
editor-specific suffixes in .gitignore.
Full disclosure: I'm a vim user myself
|
|
Now that checksum support is officially a part of MogileFS, we
can document (and thus encourage) it without risking
compatibility issues.
|
|
This changes the class associated with +key+.
This is _not_ the same as the "update_class" admin command
which actually modifies the class itself.
|
|
These files should be included in release gem/tarballs, users
should not require web acccess to access documentation.
|
|
Not everybody uses RubyGems.
|
|
This releases fixes problems short reads when slurping files
into memory. Thanks to Matthew Draper for this fix.
There are also minor documentation updates.
|
|
It's more future-proof, this way.
|
|
Not that it really matters given the MIT license...
|
|
It's better if email addresses are visible and not hidden behind
links for folks that don't integrate email clients into web
browsers.
|
|
Leave size unchanged, so it's available for easy comparison with
buf.bytesize.
[ew: added test case]
Acked-by: Eric Wong <normalperson@yhbt.net>
|
|
:noverify always defaults to _true_ (meaning verification is
_off_ by default). This double negative is unfortunately
a confusing part of the existing API and MogileFS protocol.
|
|
|
|
* improved API support for uploading large files
While we've always supported uploading large files, the
(still-supported) existing APIs were somewhat awkward or
required the file to exist on the file system. See
MogileFS::NewFile for details and examples.
* more informative exception messages for timed-out requests
* :fail_timeout parameter, the timeout for retrying a failed
tracker connection. This defaults to 5 seconds (same as
previous versions where this was hard-coded.
* :new_file_max_time parameter
Controls the maximum of time spent creating and uploading
a new file in MogileFS. This defaults to 1 hour (which
matching the expiry time of a row in the MogileFS internal
tempfile table).
* store_file works on unlinked File/Tempfile objects
* each_fid method in MogileFS::Admin fixed
* stale_fid_checker example script added
* mogstored_rack example split into a standalone RubyGem:
http://bogomips.org/mogstored_rack/
* backend error constants are generated on const_missing,
instead of when raised, making it easier to rescue exceptions
we didn't explicitly enable
* some internal cleanups and documentation improvements
|
|
We have lots of timeouts :x
|
|
Most of these exceptions are just too rare to be useful,
and code that cares for exceptions will rescue them
anyways.
|
|
This controls the retry timeout of a failed backend
|
|
This will make debug output quieter.
|
|
This adds an internal :ruby_no_raise flag to the backend
to avoid exceptions.
|
|
Performance with Content-Range uploads sucks either way, so it's
not noticeably worse off on a LAN /without/ persistent
connections.
|
|
each_fids was too buggy and likely to yield false positives
|
|
NEWS is more common in the wider Free Software community
|
|
The client-provided Content-MD5 may have leading/trailing
whitespace, so we shall strip that before comparing.
We also know any base64-encoded MD5 we generate will have only
trailing whitespace so we can get away with the (slightly)
faster and more-GC-friendly String#rstrip!
|
|
This defaults to 1 hour (which matches the expiry time of a row
in the MogileFS internal tempfile table).
|
|
100% RDoc coverage (not that it's an indicator of _good_
documentation, but we're getting there)
|
|
This logic may go upstream into the tracker Fsck worker itself,
but until then, we can try this from the client side.
|
|
|
|
This should make documentation easier
|
|
By pre-creating devN dirs, mogstored can create usage files
sooner and we won't have to wait 10s between "df" runs.
|
|
An upload could've taken a long time, ping and try to ensure
socket is valid to minimize (but not completely eliminate) the
chance create_close hits a stale socket (while reading the
response after writing to it) and becomes non-retryable. We
treat create_close specially as its less idempotent than any
other command (even other non-idempotent ones). There may be no
hope of retrying the upload at all if data was streamed and
calling create_close twice will hurt us...
|
|
If we started a connection, make sure it's stopped and the file
descriptor closed. Relying on GC to close file descriptors is
unreliable with some GC implementations.
|
|
Now that pools can be arbitrarily large, using an O(n) check
for object validity can be expensive.
|
|
Instead of hard coding the thresholds, allow
users to change accessors
|
|
Putting this into the MogileFS::Put::* namespace will
make it easier to document.
|
|
This allows us to stream files with a known Content-Length
into MogileFS. This can be useful for streaming an HTTP
download into MogileFS without:
* saving it to the filesystem
* relying on chunked encoding support on the server
|
|
|
|
It is auto-generated and goes into .gitignore
|
|
|
|
This way, folks can rescue for exceptions we haven't defined,
yet.
|