| Date | Commit message (Collapse) |
|---|
|
This releases fixes problems short reads when slurping files
into memory. Thanks to Matthew Draper for this fix.
There are also minor documentation updates.
|
|
It's more future-proof, this way.
|
|
Not that it really matters given the MIT license...
|
|
It's better if email addresses are visible and not hidden behind
links for folks that don't integrate email clients into web
browsers.
|
|
Leave size unchanged, so it's available for easy comparison with
buf.bytesize.
[ew: added test case]
Acked-by: Eric Wong <normalperson@yhbt.net>
|
|
:noverify always defaults to _true_ (meaning verification is
_off_ by default). This double negative is unfortunately
a confusing part of the existing API and MogileFS protocol.
|
|
|
|
* improved API support for uploading large files
While we've always supported uploading large files, the
(still-supported) existing APIs were somewhat awkward or
required the file to exist on the file system. See
MogileFS::NewFile for details and examples.
* more informative exception messages for timed-out requests
* :fail_timeout parameter, the timeout for retrying a failed
tracker connection. This defaults to 5 seconds (same as
previous versions where this was hard-coded.
* :new_file_max_time parameter
Controls the maximum of time spent creating and uploading
a new file in MogileFS. This defaults to 1 hour (which
matching the expiry time of a row in the MogileFS internal
tempfile table).
* store_file works on unlinked File/Tempfile objects
* each_fid method in MogileFS::Admin fixed
* stale_fid_checker example script added
* mogstored_rack example split into a standalone RubyGem:
http://bogomips.org/mogstored_rack/
* backend error constants are generated on const_missing,
instead of when raised, making it easier to rescue exceptions
we didn't explicitly enable
* some internal cleanups and documentation improvements
|
|
We have lots of timeouts :x
|
|
Most of these exceptions are just too rare to be useful,
and code that cares for exceptions will rescue them
anyways.
|
|
This controls the retry timeout of a failed backend
|
|
This will make debug output quieter.
|
|
This adds an internal :ruby_no_raise flag to the backend
to avoid exceptions.
|
|
Performance with Content-Range uploads sucks either way, so it's
not noticeably worse off on a LAN /without/ persistent
connections.
|
|
each_fids was too buggy and likely to yield false positives
|
|
NEWS is more common in the wider Free Software community
|
|
The client-provided Content-MD5 may have leading/trailing
whitespace, so we shall strip that before comparing.
We also know any base64-encoded MD5 we generate will have only
trailing whitespace so we can get away with the (slightly)
faster and more-GC-friendly String#rstrip!
|
|
This defaults to 1 hour (which matches the expiry time of a row
in the MogileFS internal tempfile table).
|
|
100% RDoc coverage (not that it's an indicator of _good_
documentation, but we're getting there)
|
|
This logic may go upstream into the tracker Fsck worker itself,
but until then, we can try this from the client side.
|
|
|
|
This should make documentation easier
|
|
By pre-creating devN dirs, mogstored can create usage files
sooner and we won't have to wait 10s between "df" runs.
|
|
An upload could've taken a long time, ping and try to ensure
socket is valid to minimize (but not completely eliminate) the
chance create_close hits a stale socket (while reading the
response after writing to it) and becomes non-retryable. We
treat create_close specially as its less idempotent than any
other command (even other non-idempotent ones). There may be no
hope of retrying the upload at all if data was streamed and
calling create_close twice will hurt us...
|
|
If we started a connection, make sure it's stopped and the file
descriptor closed. Relying on GC to close file descriptors is
unreliable with some GC implementations.
|
|
Now that pools can be arbitrarily large, using an O(n) check
for object validity can be expensive.
|
|
Instead of hard coding the thresholds, allow
users to change accessors
|
|
Putting this into the MogileFS::Put::* namespace will
make it easier to document.
|
|
This allows us to stream files with a known Content-Length
into MogileFS. This can be useful for streaming an HTTP
download into MogileFS without:
* saving it to the filesystem
* relying on chunked encoding support on the server
|
|
|
|
It is auto-generated and goes into .gitignore
|
|
|
|
This way, folks can rescue for exceptions we haven't defined,
yet.
|
|
list_fids was being used incorrectly :x
|
|
Bleh, I have revision control to tell me what's there, no need
to maintain this file (or even manually generate it) myself
|
|
Unlinking Tempfiles is good practice, so we should
allow/encourage people to use unlinked Tempfiles
with store_file.
|
|
This is the most-compatible way to support largefiles
with the new_file interface. This will unfortunately
generate disk I/O, though...
Also moving the "mog" util to use this with the "tee"
subcommand since it already included its own private
implementation of this before.
|
|
This makes it easier to use an unlinked Tempfile
|
|
This is similar to the "largefile => 1" support in the
Perl MogileFS::Client package. It requires net/http/persistent
to avoid repeatedly setting up and tearing down a socket.
|
|
This returns a new HTTPStream object that behaves
like a writable IO object with the following methods:
* write
* print
* printf
* putc
* puts
* syswrite
* <<
..and also responds to IO.select (for writability)
|
|
Might as well assert on it...
|
|
TCP keepalives are inexpensive, so we can use them to monitor
whether or not our connection is still alive while uploading.
Remote servers make take an unpredictable amount of time to
actually write out the data we've uploaded (and empty socket
buffers to receive more), so it is extremely difficult to
calculate an effective timeout for select() or poll().
|
|
Some applications will rely on this feature
|
|
This reverts commit 11ee57e7aaa8766d13e29d3872bd78d50a3e5263.
Bleh, checksums support is already off by default...
|
|
We won't force this onto the wire for trackers until upstream
MogileFS decides to support it.
|
|
Users will be able to tweak these themselves in OS-dependent
ways. It's probably better to rely on protocol-level timeouts
when the HTTP server should be on a trusted network.
|
|
These allow us to specify Content-MD5 and checksums for use
with the new and improved new_file interface without breaking
existing apps.
|
|
|
|
Checksums are good, yes they are.
|
|
No need to clutter ourselves up with things we don't need.
|