cmogstored news

cmogstored news
---------------

1.8.1 / 2021-02-13 02:25 UTC
----------------------------

This release fixes a segfault on some systems/toolchains where
our per-thread stack size was too small. Given the prevalance
of 64-bit systems nowadays, using a small stack is unlikely to
yield any benefits.

Users on 32-bit systems who wish to continue with a minimal
stack should use "ulimit -s" in startup scripts or configure
their process manager appropriately (e.g. setting the
"LimitSTACK" directive in described in systemd.exec(5)).

Thanks to Xiao Yu <xyu@automattic.com> for reporting and
testing at our public mailbox:
https://yhbt.net/cmogstored-public/CABfxMcW+kb5gwq3pSB_89P49EVv+4UkJXz+mUPQTy19AdrwbAg@mail.gmail.com/T/

1.8.0 / 2020-08-13 20:54 UTC
----------------------------

devXXX/usage files are emitted properly for systems where
the mount point can't be resolved. This is needed for
multi-device filesystems such as btrfs.

PUT and DELETE requests now update the in-memory representation
of these devXXX/usage files, since the 10s update interval may
be too low for high-traffic situations.

Their is a new "USAGE FILES" section in the manpage documenting
the changes from 1.7.0 and this release.

Our public mail archives are now available over IMAPS.
gnulib is updated to 4e082bffbcc46e68 in the tarball.

1.7.3 / 2020-03-22 00:12 UTC
----------------------------

Improve RFC 7230 conformance w.r.t Content-Length and
Transfer-Encoding handling in PUT requests. We now
favor "Transfer-Encoding: chunked" if a Content-Length
header is also present. Furthermore, we no longer
accept Transfer-Encoding values aside from chunk,
since we don't support gzip/compress/deflate as described
in RFC 7230.

1.7.2 / 2020-02-19 01:30 UTC
----------------------------

s/bogomips.org/yhbt.net/ in all documentation, due to
bogomips.org expiring. The tarball is also updated with
the latest gnulib changes.

1.7.1 / 2019-05-12 00:46 UTC
----------------------------

The Linux kernel bugfix should hit mainline and stable kernels,
soon. But there's no reason for us to be caring if errno is
EINTR or not...

cf. https://lore.kernel.org/lkml/20190427093319.sgicqik2oqkez3wk@dcvr/
https://lore.kernel.org/lkml/20190507043954.9020-1-deepa.kernel@gmail.com/

There are also some minor build/test updates since v1.7.0 (2018-12-18):

test/mgmt-usage.rb: fix mismatched indentation warning
add .gitattributes for Ruby files
test/mgmt_auto_adjust.rb: improve diagnostic messages
.gitignore: add extra ignores for gnulib in Debian 9
notify.c: workaround epoll_pwait bug in current Linux 5.0/5.1
doc: remove mailing list subscription info

1.7.0 / 2018-12-18 04:01 UTC
----------------------------

The big feature in this release is "devNNN/usage" are served
from memory, allowing up-to-date usage information even
unwritable/unreadable filesystems.

This can also be used to reduce spinups and wear on HDDs.

"devNNN/usage" files are still updated on the FS by default for
compatibility with existing HTTP servers, but admins may wish
to disable updates to them by removing all permissions from
the "usage" files:

chmod 0000 $MOG_DOCROOT/dev*/usage

Filesystem errors from the sendfile(2) syscalls are also
logged to syslog. There's also a bugfix for zombies for
libkqueue-on-epoll users, but that doesn't affect native
kqueue users on *BSDs.

And the usual round of gnulib, minor doc and style updates.

18 changes since v1.6.0:

cmogstored.h: remove unused mog_file.mmptr member
doc: documentation for ioq
doc: further comment updates around ioq
build-aux/txt2pre: support '=' in URLs
test/inherit: fix ambiguous parenthese warning
test/inherit: stop testing Ruby itself
doc: update URLs to HTTPS
compat_sendfile: ensure this works without an offset
doc/queues.txt: add key point about only retrieving ONE event
fix trace.h dependency on probes.h
update to gnulib.git 90f289f249a266b1afb9c63e182f5d979d17df5f
http_get.c: log filesystem-level errors from sendfile
serve /dev*/usage requests from memory
doc: URL updates to reduce redirects and favor HTTPS
test/inherit.rb: fix syntax error under Ruby 1.8
update copyrights for 2018 and use SPDX for "GPL-3.0+"
selfwake: enable self-pipe with kqueue
http_parser: workaround parsing OOM in Ragel 6.10

1.6.0 / 2016-08-31 03:14 UTC
----------------------------

There are minor robustness fixes on handling errors when
allocating memory or spawn failures on otherwise-hosed systems.
These bugfixes will not affect real users unless the system
is already hosed or in badly overtaxed, so there's no real
need to upgrade.

There are minor portability improvements and I now test under
FreeBSD 10.x.

The iostat test cases are relaxed a bit to account for
virtualized devices (as iostat is less useful with modern

17 changes since 1.5.0 (Nov 2015):
Rakefile: add missing <div> for Atom feed
test/pwrite-wrap: remove unused variable and comment
test/pwrite_wrap: squelch unnecessary output
test/pwrite_wrap: reduce space overhead required
update copyrights for 2016
build-aux/txt2pre: drop CGI.pm requirement
stdin is always redirected to /dev/null
minor vfork/fork safety fixes
process: try to handle OOM gracefully
http_put: gracefully handle path allocation errors
iostat_process: declare environ extern
test/mgmt: relax checks for iostat mapping
gnulib copyright update for 2016
upgrade: avoid syslog call if execve fails
rely on gnulib for environ portability
INSTALL: update latest Debian stable version to 8.x
README: stop mentioning cgit

1.5.0 / 2015-11-21 01:33 UTC
----------------------------

A bunch of minor changes; most notable is systemd-style socket
activation support. This was easy-to-add since we've always had
socket activation support for nginx-style SIGUSR2 upgrades.

This places no link or runtime dependency on libsystemd, so the
LISTEN_FDS and LISTEN_PID environment variables may be used in other
init systems as well. While I have my own reservations about
systemd itself, I also strongly believe in using socket activation
to prevent downtime. Existing behavior with CMOGSTORED_FD
(used for SIGUSR2 upgrades) is now documented in the manpage and
will always supported.

We've also added vfork support for Linux systems, allowing
faster spawning of iostat if malloc is using too much memory.

Behavior changes:

Bad Range: headers return 416 responses in more cases for invalid
ranges (e.g. miscalculated ranges such as "1--1", while
completely wrong ones (lacking a "bytes=" prefix) are ignored
entirely as in nginx.

Bugfixes:

There are also some cleanups to avoid dying on OOM in more places
on weird systems which trigger OOM. More work on this is ongoing.

Also updates to the latest gnulib.git
commit 71d39c1644762745b94e9449c45bfd716a79a5eb
("autoupdate") along with a change which fixes a memory leak when
people build from cmogstored.git using gnulib
commit c6148bca89e9465fd6ba3a10d273ec4cb58c2dbe
or later ("mountlist: add me_mntroot field on Linux machines").

This memory leak did not affect any released tarballs of cmogstored.
Note, users building from git (as opposed to the tarball) will
need gnulib commit 41d1b6c42641a5b9e21486ca2074198ee7909bd7
("mountlist: add support for deallocating returned list entries")
or later (from July 2013).

There are also various documentation updates and our mailing
list is now readable over NNTP:

nntp://news.public-inbox.org/inbox.comp.file-systems.mogilefs.cmogstored

1.5.0rc1 / 2015-11-11 21:24 UTC
-------------------------------

A bunch of minor changes; most notable is systemd-style socket
activation support. This was easy-to-add since we've always had
socket activation support for nginx-style SIGUSR2 upgrades.

Behavior changes:

Bad Range: headers return 416 responses in more cases for invalid
ranges (e.g. miscalculated ranges such as "1--1", while
completely wrong ones (lacking a "bytes=" prefix)) are ignored
entirely as in nginx.

Bugfixes:

There are also some cleanups to avoid dying on OOM in more places
on weird systems which trigger OOM. More work on this is ongoing.

Also updates to the latest gnulib.git
commit f197c2c9e5e0d12c373f26d5b3211809457bc972
("intprops: new public macro EXPR_SIGNED")
along with a change which fixes a memory leak when people
build from cmogstored.git using gnulib
commit c6148bca89e9465fd6ba3a10d273ec4cb58c2dbe
or later ("mountlist: add me_mntroot field on Linux machines").
This memory leak did not affect any released tarballs of cmogstored.

shortlog of changes since 1.4.3:

doc: use "builder" RubyGem to generate Atom feed
dev.c: fail gracefully on out-of-memory errors
do not die on OOM when for mgmt paths
HACKING: update URLs to reduce redirects
http: return 416 errors in more cases for bad Ranges
update .gitignores for latest autotools + gnulib
Rakefile: remove text-only part from the Atom feed
support systemd-style socket activation via environment
set TCP listener options on inherited sockets
doc: add example systemd config files
use free_mount_entry from gnulib instead of rolling our own
fix tmpdir dependency for slow Ruby tests
doc: publish examples directory to website

1.4.3 / 2015-03-09 22:52 UTC
----------------------------

For all platforms, the startup device scanning thread at startup
may not handle EINTR properly. This bug only manifested at
startup and does not affect running instances. However, this
bug is also readily apparent on newer versions of FreeBSD
which support the ppoll function call.

Thanks to Mykola Golub <trociny@FreeBSD.org> for the bug report
which led to this release.

For systems lacking epoll_pwait (older GNU/Linux, all *BSDs),
there is also a bugfix for systems which experience signal spam
leading to errno clobbering in the main thread. This bug was
only only noticed due to a bug report against Ruby:

https://bugs.ruby-lang.org/issues/10866

There is no need to upgrade if 1.4.1 is already running well
on modern GNU/Linux systems capable of epoll_pwait. But then
again nginx-style SIGUSR2 upgrades are transparent to clients.

shortlog since 1.4.2:

Makefile.am: fix publish rule for website
Fix assertion failure during startup
avoid relying on ppoll as a cancellation point
preserve errno when inside sig handler for self-pipe

1.4.2 / 2015-03-06 02:18 UTC
----------------------------

* Makefile.am: gzip README and associated data
* manpage: update contact and copyright information
* update copyrights to 2014 (and all contributors)
* doc/design.txt: add a few more notes on compromises
* http_dav: log 500 errors from DELETE requests
* tapset/http_access_log: note CLF differences
* copyright updates for 2015

1.4.1 / 2014-09-07 02:21 UTC
----------------------------

The PHP PECL MogileFS extension uses neon to handle WebDAV operations,
and neon seems to send (valid but unfortunate) headers with empty
string values. Thanks to Patrice Damezin at Skyrock.com for reporting
this bug.

There's also a few minor cleanups. The latest 2.6.34 stable kernel
release no longer requires our EPOLL_CTL_MOD race workaround. There
are also some test suite updates for future releases of Ruby.
Bigger changes coming later this year...

There's also a new public mailing list at:

cmogstored-public@bogomips.org

No subscription will ever be necessary to post.
Subscription is optional via:

cmogstored-public+subscribe@bogomips.org

Archives are available at http://bogomips.org/cmogstored-public/

Eric Wong (11):
minor cleanups for functions which do not return
remove old fsck_queue declarations
svc_dev: calling free does not need the lock
test/mgmt: lengthen test for iostat watch
test/http: plug race condition in FIFO test
test/http_chunked_put: test for gigantic trailer
update address to public mailing list
Rakefile: remove freecode/freshmeat references
Rakefile: shorten ChangeLog dump
queue_epoll: disable buggy epoll workaround for 2.6.34.15+
http_common: correctly handle empty header values

1.4.0 / 2014-02-24 03:01 UTC
----------------------------

bsd_sendfile is now supported on Debian GNU/kFreeBSD systems.
This release also fixes a compability bug with Perl mogstored config
files where "daemonize = (0|1)" was not supported properly.

Eric Wong (3):
check for sys/sendfile.h header instead of __linux__
allow bsd_sendfile with freebsd-glue on Debian/kFreeBSD
support "daemonize = 0|1" in the config file

1.3.3 / 2014-02-09 04:28 UTC
----------------------------

This release fixes build problems with Debian GNU/kFreeBSD support
(turns out it's been broken for over a year and nobody noticed :x).
There are also build system upgrades for automake 1.14 and test case
cleanups, but no changes to any of the core code. No changes nor
need to upgrade if you're on anything other than Debian GNU/kFreeBSD.

1.3.2 / 2013-12-10 22:10 UTC
----------------------------

This release speeds up graceful shutdown on busy systems such
as FreeBSD. There is also a minor resource savings for users
of the undocumented --worker-processes switch. There are also
some minor memory error fixes for test cases (which did not
affect the daemon itself).

Upgrading is optional unless you are affected by these fixes.

Note: GNU/Linux users are encouraged to read the manpage update
regarding glibc malloc arenas

Eric Wong (9):
selfwake: do share pipe descriptors with workers
test/chunk-parser-1: fix uninitialized file structures
test: fix valgrind warnings in test-only C code
doc: refer to malloc-related environment variables
thrpool: sleep instead of yield when poking thread
test/mgmt-usage: relax regexp for ZFS
m4/.gitignore: bump for newer gnulib
doc: fix wording in manpage
doc: fix link to MogileFS homepage

1.3.1 / 2013-10-12 21:45 UTC
----------------------------

This release fixes a bug which only affects users of the
undocumented multi-process configuration feature
(which is also multi-threaded).

* avoid use-after-free with multi-process setups

readdir on the same DIR pointer is undefined if DIR was inherited by
multiple children. Using the reentrant readdir_r would not have
helped, since the underlying file descriptor and kernel file handle
were still shared (and we need rewinddir, too).

This readdir usage bug existed in cmogstored since the earliest
releases, but was harmless until the cmogstored 1.3 series.

This misuse of readdir lead to hitting a leftover call to free().
So this bug only manifested since
commit 1fab1e7a7f03f3bc0abb1b5181117f2d4605ce3b
(svc: implement top-level by_mog_devid hash)

Fortunately, these bugs only affect users of the undocumented
multi-process feature (not just multi-threaded).

1.3.0 / 2013-09-30 08:51 UTC
----------------------------

There are no changes from 1.3.0rc2.

For the most part, cmogstored 1.2.2 works well, but 1.3 contains some
fairly major changes and improvements.

cmogstored CPU usage may be higher than other servers because it's
designed to use whatever resources it has at its disposal to
distribute load to different storage devices. cmogstored 1.3
continues this, but it should be safer to lower thread counts
without hurting performance too much for non-dedicated servers.

cmogstored 1.3 contains improvements for storage hosts at the
extremes ends of the performance scale. For large machines with many
cores, memory/thread usage is reduced because we had too many acceptor
threads. There are more improvements for smaller machines, especially
those with slow/imbalanced drive speeds and few CPUs. Some of the
improvements came from my testing with ancient single-core machines,
others came from testing on 24-core machines :)

Major features in 1.3:

ioq - a I/O queues for all MogileFS requests
--------------------------------------------

The new I/O queue (ioq) implements the equivalent of AIO channels
functionality from Perlbal/mogstored. This feature prevents a
failing/overloaded disk from monopolizing all the threads in the system.

Since cmogstored uses threads directly (and not AIO), the common
(uncontended case) behaves like a successful sem_wait with POSIX
semaphores. Queueing+rescheduling only occurs in the contended case
(unlike with AIO-style APIs, where request are always queued). I
experimented with, but did not use POSIX semaphores as contention would
still starve the thread pool.

Unlike the old fsck_queue, ioq is based on the MogileFS devid in the URL
and not the st_dev ID of the actual underlying file. This is less
correct from a systems perspective, but should make no difference for
normal production deployments (which are expected to use one MogileFS
devid for each st_dev ID) and has several advantages:

1) testing/mock deploys of this feature with mock deploys is easier

2) we do not require any additional filesystem syscall (open/*stat)
to look up the ioq based on st_dev, so we can use ioq to avoid
stalls from slow open/openat/stat/fstatat/unlink/unlinkat syscalls.

Otherwise, the implementation of this very closely resembles the old
fsck queue implementation, but is generic across HTTP and sidechannel
clients. The existing fsck queue functionality is now implemented using
ioq. Thus, fsck queue functionality is mapped by the MogileFS devid and
not the system st_dev ID as a result of this change.

One benefit of this feature is the ability to run fewer aio_threads
safely without worrying about cross-device contention on machines with
limited resources or few disks (or not solely dedicated to MogileFS
storage).

The capacity of these I/O queues is automatically scaled to the number
of available aio_threads, so they can change dynamically while your
admin is tuning "SERVER aio_threads = XX"

However, on a dedicated storage node, running many aio_threads (as is
the default) should still be beneficial. Having more threads can keep
the internal I/O queues of the kernel and storage hardware more
populated and can improve throughput.

thread shutdown fixes (epoll)
-----------------------------

Our previous reliance on pthreads cancellation primitives left us open
to a small race condition where I/O events (from epoll) could be lost
during graceful shutdown or thread reduction via
"SERVER aio_threads = XX". We no longer rely on pthreads cancellation
for stopping threads and instead implement explicit check points for
epoll.

This did not affect kqueue users, but the code is simpler and more
consistent across epoll/kqueue implementations.

Graceful shutdown improvements
------------------------------

The addition of our I/O queueing and use of our custom thread shutdown
API also allowed us to improve the responsiveness and fairness when the
process enters graceful shutdown mode. This improves fairness and
avoids client-side timeouts when large PUT requests are being issued
over a fast network to slow disks during graceful shutdown.

Currently, graceful shutdown remains single-threaded, but we will likely
become multi-threaded in the future (like normal runtime).

Miscellaneous fixes and improvements
------------------------------------

Further improved matching for (Linux) device-mapper setups where the
same device (not symlinks) appears multiple times in /dev

aio_threads count is automatically updated when new devices are
added/removed. This is currently synced to MOG_DISK_USAGE_INTERVAL, but
will use inotify (or the kqueue equivalent) in the future.

HTTP read buffers grow monotonically (up to 64K) and always use aligned
memory. This allows deployments which pass large HTTP headers do not
trigger unnecessary reallocations. Deployments which use small HTTP
headers should notice no memory increase.

Acceptor threads are now limited to two per process instead of being
scaled to CPU count. This avoids excessive threads/memory usage and
contention of kernel-level mutexes for large multi-core machines.

The gnulib version used for building the tarball is now included in the
tarball for ease-of-reproducibility.

Additional tests for uncommon error conditions using the fault-injection
capabilities of GNU ld.

The "shutdown" command over the sidechannel is more responsive for epoll
users.

Improved reporting of failed requests during PUT requests. Again, I run
MogileFS instances on some of the most horrible networks on the planet[2]

fix LIB_CLOCK_GETTIME linkage on some toolchains.

"SERVER mogstored.persist_client = (0|1)" over the sidechannel is supported
for compatibility with Perlbal/mogstored

The Status: header is no longer returned on HTTP responses. All known
MogileFS clients parse the HTTP status response correctly without the
need for the Status: header. Neither Perlbal nor nginx set the Status:
header on responses, so this is unlikely to introduce incompatibilities.
The Status: header was originally inherited from HTTP servers which had
to deal with a much larger range of (non-compliant) clients.

1.3.0rc2 / 2013-09-03 09:05 UTC
-------------------------------

SystemTap support is mostly fleshed out. There are some bundled awk
scripts which should make better sense of the all.stp which logs just
about everything.

Raising aio_threads now correctly increases ioq capacity. This
regression was only introduced in the 1.3.0 rc series, as ioq
was not in 1.2.x.

1.3.0rc1 / 2013-07-14 02:32 UTC
-------------------------------

For the most part, cmogstored 1.2.2 works well, but 1.3 contains some
fairly major changes and improvements.

cmogstored CPU usage may be higher than other servers because it's
designed to use whatever resources it has at its disposal to distribute
load to different storage devices. cmogstored 1.3 will continue this,
but it should be safer to lower thread counts without hurting
performance too much for non-dedicated servers.

Unfortunately, the minor, Linux-only bug affecting 1.2.2 for (uncommon)
thread shutdowns required some fairly intrusive changes to fix, so I'm
not sure if releasing a 1.2.3 is worth it. If you're happy with 1.2.x,
I recommend marking the host down via mogadm before lowering
"SERVER aio_threads = XX" or sending SIGQUIT to cmogstored. But
I think thread shutdown is uncommon enough to not affect normal
deployments.

cmogstored 1.3 will contain improvements for storage hosts at the
extremes ends of the performance scale. For large machines with many
cores, memory/thread usage is reduced because we had too many acceptor
threads. There are more improvements for smaller machines, especially
those with slow/imbalanced drive speeds and few CPUs. Some of the
improvements came from my testing with ancient single-core machines,
others came from testing on 24-core machines :)

The SystemTap tracing work is still in-progress (although the 1.3 cycle
was originally intended to focus on this :x). I expect the remaining
changes to be non-intrusive and will work on them through the RC cycle.