| Date | Commit message (Collapse) |
|---|
|
This release fixes a segfault on some systems/toolchains where
our per-thread stack size was too small. Given the prevalance
of 64-bit systems nowadays, using a small stack is unlikely to
yield any benefits.
Users on 32-bit systems who wish to continue with a minimal
stack should use "ulimit -s" in startup scripts or configure
their process manager appropriately (e.g. setting the
"LimitSTACK" directive in described in systemd.exec(5)).
Thanks to Xiao Yu <xyu@automattic.com> for reporting and
testing at our public mailbox:
https://yhbt.net/cmogstored-public/CABfxMcW+kb5gwq3pSB_89P49EVv+4UkJXz+mUPQTy19AdrwbAg@mail.gmail.com/T/
|
|
As compilers and system C libraries change, the using a
non-default stack size is too risky and can lead to
difficult-to-diagnose problems.
Using the default stack size seems to solve the segfaults
at http_close reported by Xiao Yu <xyu@automattic.com>.
Users on modern 64-bit systems were unlikely to find any benefit
in using a small stack size with this code base.
Users on 32-bit systems who wish to continue with a minimal
stack should use "ulimit -s" in startup scripts or configure
their process manager appropriately (e.g. setting the
"LimitSTACK" directive in described in systemd.exec(5)).
Reported-and-tested-by: Xiao Yu <xyu@automattic.com>
Link: https://yhbt.net/cmogstored-public/CABfxMcW+kb5gwq3pSB_89P49EVv+4UkJXz+mUPQTy19AdrwbAg@mail.gmail.com/T/
|
|
devXXX/usage files are emitted properly for systems where
the mount point can't be resolved. This is needed for
multi-device filesystems such as btrfs.
PUT and DELETE requests now update the in-memory representation
of these devXXX/usage files, since the 10s update interval may
be too low for high-traffic situations.
Their is a new "USAGE FILES" section in the manpage documenting
the changes from 1.7.0 and this release.
Our public mail archives are now available over IMAPS.
gnulib is updated to 4e082bffbcc46e68 in the tarball.
|
|
public-inbox.org started supporting IMAP + IMAPS a few months
ago, and has always supported Tor .onions via NNTP. Non-TLS is
still supported for older systems and users with oppressive
firewalls.
|
|
commit 4e082bffbcc46e68644ae0d59b4f09bf2b5feb84
("sys_random: Work around an uClibc bug.")
|
|
Under heavy write traffic, free space changes constantly, and
the periodic updates every 10 (or MOG_DISK_USAGE_INTERVAL)
seconds can be too far behind. Since we keep the usage file
contents in-memory now for out-of-FD situations, we can update
that without incurring extra VFS traffic.
v2: We no longer try to use fstatvfs(2) and instead pay the cost of
an extra name lookups and just update all usage files. This was
necessary since calculating free space while a file is still
open can take a long time on some FSes and we need to send the
HTTP response back ASAP to avoid timeouts on the client-side.
This avoids contention in the request worker threads and the
mostly idle main thread to do more work.
|
|
And fix formatting of the SIGNALS section while we're at it.
|
|
LUKS + btrfs on Linux gives an .st_rdev value of `0', so we
can't reliably figure out what the "device:" field in
/devXX/usage should be without parsing /proc/partitions.
Since MogileFS::Worker::Monitor only cares about the "used:" and
"total:" fields, we'll just emit "(?)" in the device field. The
effort of parsing /proc/partitions to correctly display a field
that our only known consumer won't use is a waste of our time.
|
|
Improve RFC 7230 conformance w.r.t Content-Length and
Transfer-Encoding handling in PUT requests. We now
favor "Transfer-Encoding: chunked" if a Content-Length
header is also present. Furthermore, we no longer
accept Transfer-Encoding values aside from chunk,
since we don't support gzip/compress/deflate as described
in RFC 7230.
|
|
RFC 7230 is actually explicit about favoring the
"Transfer-Encoding: chunked" over a Content-Length header
when a client specifies both.
|
|
RFC 7230 3.3.3, point 3 states:
> If a Transfer-Encoding header field
> is present in a request and the chunked transfer coding is not
> the final encoding, the message body length cannot be determined
> reliably; the server MUST respond with the 400 (Bad Request)
> status code and then close the connection.
And no MogileFS client is known to send "gzip", "deflate", or
"compress" as part of the Transfer-Encoding, so we'll only
accept "chunked".
|
|
|
|
s/bogomips.org/yhbt.net/ in all documentation, due to
bogomips.org expiring. The tarball is also updated with
the latest gnulib changes.
|
|
|
|
bogomips.org is due to expire, soon, and I'm not willing to pay
extortionist fees to Ethos Capital/PIR/ICANN to keep a .org. So
it's at yhbt.net, for now, but it will change again to
whatever's affordable... Identity is overrated.
Tor users can use .onions and kick ICANN to the curb:
torsocks w3m http://cmogstored.ou63pmih66umazou.onion/
torsocks git clone http://ou63pmih66umazou.onion/cmogstored.git/
torsocks w3m http://ou63pmih66umazou.onion/cmogstored-public/
|
|
|
|
The Linux kernel bugfix should hit mainline and stable kernels,
soon. But there's no reason for us to be caring if errno is
EINTR or not...
cf. https://lore.kernel.org/lkml/20190427093319.sgicqik2oqkez3wk@dcvr/
https://lore.kernel.org/lkml/20190507043954.9020-1-deepa.kernel@gmail.com/
There are also some minor build/test updates since v1.7.0 (2018-12-18):
test/mgmt-usage.rb: fix mismatched indentation warning
add .gitattributes for Ruby files
test/mgmt_auto_adjust.rb: improve diagnostic messages
.gitignore: add extra ignores for gnulib in Debian 9
notify.c: workaround epoll_pwait bug in current Linux 5.0/5.1
doc: remove mailing list subscription info
|
|
|
|
Mail subscriber lists are centralized data which is not commonly
forkable or reproducible. Mail archives are more important.
|
|
The bugfix should hit mainline and stable kernels, soon; but
there's no reason for us to be caring if errno is EINTR, or
not...
https://lore.kernel.org/lkml/20190427093319.sgicqik2oqkez3wk@dcvr/
https://lore.kernel.org/lkml/20190507043954.9020-1-deepa.kernel@gmail.com/
|
|
I usually use gnulib.git, but not everybody does and it's worth
cleaning things up a bit for this common case.
Tested with gnulib 20140202+stable-2+deb9u1 in Debian 9 (stretch).
Further updates may be needed for other common distros which
package gnulib.
|
|
Chasing down a regression in Linux 5.0:
https://lkml.kernel.org/r/20190427093319.sgicqik2oqkez3wk@dcvr
|
|
This hopefully makes our other changes easier-to-read.
|
|
Newer versions of Ruby warn on it
|
|
The big feature in this release is "devNNN/usage" are served
from memory, allowing up-to-date usage information even
unwritable/unreadable filesystems.
This can also be used to reduce spinups and wear on HDDs.
"devNNN/usage" files are still updated on the FS by default for
compatibility with existing HTTP servers, but admins may wish
to disable updates to them by removing all permissions from
the "usage" files:
chmod 0000 $MOG_DOCROOT/dev*/usage
Filesystem errors from the sendfile(2) syscalls are also
logged to syslog. There's also a bugfix for zombies for
libkqueue-on-epoll users, but that doesn't affect native
kqueue users on *BSDs.
And the usual round of gnulib, minor doc and style updates.
18 changes since v1.6.0:
cmogstored.h: remove unused mog_file.mmptr member
doc: documentation for ioq
doc: further comment updates around ioq
build-aux/txt2pre: support '=' in URLs
test/inherit: fix ambiguous parenthese warning
test/inherit: stop testing Ruby itself
doc: update URLs to HTTPS
compat_sendfile: ensure this works without an offset
doc/queues.txt: add key point about only retrieving ONE event
fix trace.h dependency on probes.h
update to gnulib.git 90f289f249a266b1afb9c63e182f5d979d17df5f
http_get.c: log filesystem-level errors from sendfile
serve /dev*/usage requests from memory
doc: URL updates to reduce redirects and favor HTTPS
test/inherit.rb: fix syntax error under Ruby 1.8
update copyrights for 2018 and use SPDX for "GPL-3.0+"
selfwake: enable self-pipe with kqueue
http_parser: workaround parsing OOM in Ragel 6.10
|
|
Noticed in FreeBSD 11.2 where Ragel 6.10 was OOM-ing, this
doesn't affect Ragel 6.9.
TODO: make sure this is fixed upstream in Ragel.
|
|
This was causing my libkqueue build to stall on Linux where
epoll_pwait exists.
We actually favor kqueue in the code for testing purposes,
so we need to enable the self-wake pipe when using libkqueue
if epoll_pwait is detected.
|
|
copyrights updated by "update-copyright" in gnulib:
git ls-files | UPDATE_COPYRIGHT_HOLDER='all contributors' \
UPDATE_COPYRIGHT_USE_INTERVALS=2 \
xargs /path/to/gnulib/build-aux/update-copyright
While we're at it, SPDX seems to be the accepted way to identify
licenses nowadays, so lets use it.
git ls-files | xargs perl -i -p -e \
's,GPLv3 or later.*,GPL-3.0+ <https://www.gnu.org/licenses/gpl-3.0.txt>,g'
|
|
Not sure if it's worth supporting 1.8, anymore, but parts
of the Ruby VM test and benchmark suite still remain
1.8-compatible...
|
|
HTTPS is usually more secure and redirects slow readers down.
|
|
Filesystems may become unwritable and out-of-date "usage" files
will cause trackers to see out-of-date information.
We still write to the filesystem by default for compatibility
with existing HTTP servers. However, giving the "usage" file a
0000 mode will prevent cmogstored from overwriting it. This
allows admins to also reduce wear on storage devices:
chmod 0000 $mogroot/dev*/usage
|
|
Socket errors are too common to log (especially from malicous
clients), but filesystem errors are rare and important.
|
|
|
|
I haven't tested with systemtap, lately; maybe something
bit rotted.
|
|
This had become such second nature to me that I forgot to
document it :x
|
|
While we never call sendfile without an offset, some projects
may copy our code and want to use it without an offset.
|
|
HTTPS seems to be working well for the rest of bogomips.org
with Let's Encrypt, so lets use it and hope it protects some
users from snooping.
|
|
TCPSocket.new raises exceptions on failure.
|
|
Who tests the tests?
|
|
We'll just have everything on r******** soon, I hope :p
|
|
Let's not forget about this queue, it is a useful design.
|
|
It's a queue that looks like a semaphore, so document it
in doc/queues.txt and provide pointers to perhaps-forgotten
documentation.
|
|
This was intended for zero-copy PUT support, but that is
probably not worth it due to checksumming and the general
unpredictability of mmap/munmap performance, especially
on non-Linux systems.
|
|
There are minor robustness fixes on handling errors when
allocating memory or spawn failures on otherwise-hosed systems.
These bugfixes will not affect real users unless the system
is already hosed or in badly overtaxed, so there's no real
need to upgrade.
There are minor portability improvements and I now test under
FreeBSD 10.x.
The iostat test cases are relaxed a bit to account for
virtualized devices (as iostat is less useful with modern
17 changes since 1.5.0 (Nov 2015):
Rakefile: add missing <div> for Atom feed
test/pwrite-wrap: remove unused variable and comment
test/pwrite_wrap: squelch unnecessary output
test/pwrite_wrap: reduce space overhead required
update copyrights for 2016
build-aux/txt2pre: drop CGI.pm requirement
stdin is always redirected to /dev/null
minor vfork/fork safety fixes
process: try to handle OOM gracefully
http_put: gracefully handle path allocation errors
iostat_process: declare environ extern
test/mgmt: relax checks for iostat mapping
gnulib copyright update for 2016
upgrade: avoid syslog call if execve fails
rely on gnulib for environ portability
INSTALL: update latest Debian stable version to 8.x
README: stop mentioning cgit
|
|
I do not expect to run it much longer since it contains CSS
and renders poorly without it.
|
|
Debian 8.x (jessie) was released over a year ago :x
|
|
This avoids warnings on my GNU system while still working on
FreeBSD.
|
|
We cannot safely call syslog on all platforms under vfork;
but we have normal exit handling to tell us of the presence
of execve errors, just not which.
|
|
|
|
In the age of virtualized devices and fast solid-state storage,
iostat information isn't as useful at it was a decade ago and
probably less useful in tests. So relax the tests.
|