| Date | Commit message (Collapse) |
|---|
|
This avoids warnings on my GNU system while still working on
FreeBSD.
|
|
We cannot safely call syslog on all platforms under vfork;
but we have normal exit handling to tell us of the presence
of execve errors, just not which.
|
|
|
|
In the age of virtualized devices and fast solid-state storage,
iostat information isn't as useful at it was a decade ago and
probably less useful in tests. So relax the tests.
|
|
This is necessary for FreeBSD and probably other non-GNU systems.
|
|
Failing to allocate memory should be a temporary error and be
non-fatal.
|
|
If we fail to register a process, it is not fatal since
a process is already running. However, we may not know
about when to restart it when it dies.
|
|
In case "/bin/sh" or "/dev/null" becomes unavailable during the
lifetime of cmogstored, we will no longer crash when attempting
to (re)start iostat. However, your system is probably hosed
anyways if "/bin/sh" or "/dev/null" become unavailable.
This also fixes a bug where we would leak the iostat pipe if
either fork/vfork fails.
We also close an innocuous race condition where the child
might toggle flags in the parent process and trigger an
extra wakeup.
Finally, we use sigprocmask in the child in case pthread_sigmask
does not not work on some systems after forking. This is likely
only a cosmetic change.
|
|
There is no reason for stdin to ever be connected to a terminal,
ensure we have a consistent stdin for iostat processes and the
like.
|
|
CGI.pm is no longer in the main Perl distro, so depending
on it is not worth the effort for a few lines.
|
|
git ls-files | UPDATE_COPYRIGHT_HOLDER='all contributors' \
UPDATE_COPYRIGHT_USE_INTERVALS=2 \
xargs /path/to/gnulib/build-aux/update-copyright
|
|
It's probably overkill to use 100G of space, even
if its sparse.
|
|
Oops, leftover from development many years ago.
|
|
They were blindly copied and s/search/replace/-ed from epoll-wrap.c
|
|
Apparently this is needed for proper XHTML rendering in iceweasel?
|
|
A bunch of minor changes; most notable is systemd-style socket
activation support. This was easy-to-add since we've always had
socket activation support for nginx-style SIGUSR2 upgrades.
This places no link or runtime dependency on libsystemd, so the
LISTEN_FDS and LISTEN_PID environment variables may be used in other
init systems as well. While I have my own reservations about
systemd itself, I also strongly believe in using socket activation
to prevent downtime. Existing behavior with CMOGSTORED_FD
(used for SIGUSR2 upgrades) is now documented in the manpage and
will always supported.
We've also added vfork support for Linux systems, allowing
faster spawning of iostat if malloc is using too much memory.
Behavior changes:
Bad Range: headers return 416 responses in more cases for invalid
ranges (e.g. miscalculated ranges such as "1--1", while
completely wrong ones (lacking a "bytes=" prefix) are ignored
entirely as in nginx.
Bugfixes:
There are also some cleanups to avoid dying on OOM in more places
on weird systems which trigger OOM. More work on this is ongoing.
Also updates to the latest gnulib.git
commit 71d39c1644762745b94e9449c45bfd716a79a5eb
("autoupdate") along with a change which fixes a memory leak when
people build from cmogstored.git using gnulib
commit c6148bca89e9465fd6ba3a10d273ec4cb58c2dbe
or later ("mountlist: add me_mntroot field on Linux machines").
This memory leak did not affect any released tarballs of cmogstored.
Note, users building from git (as opposed to the tarball) will
need gnulib commit 41d1b6c42641a5b9e21486ca2074198ee7909bd7
("mountlist: add support for deallocating returned list entries")
or later (from July 2013).
There are also various documentation updates and our mailing
list is now readable over NNTP:
nntp://news.public-inbox.org/inbox.comp.file-systems.mogilefs.cmogstored
|
|
gnulib commit 41d1b6c42641a5b9e21486ca2074198ee7909bd7
("mountlist: add support for deallocating returned list entries")
or later (from July 2013) is needed for free_mount_entry support
introduced in our commit 1225f9ce4c32b3bba61ce92a487d99260a001995
("use free_mount_entry from gnulib instead of rolling our own").
|
|
Oops.
|
|
Sometimes people will forget to install the manpage, make sure
it's online in plain-text or HTML format.
|
|
Generate pre-formatted HTML which gives us a consistent visual style
with our mailing list archives and enhance linkability. <a>, <pre>,
and <title> are among the few useful HTML tags I'll use :P
Drop the AUTHORS file, it's pointless maintenance task and users can
just look at git history instead (and honestly, I have zero interest in
recognition; I only use my real name to deter GPL violations).
|
|
Most notably, our mailing list is now available over NNTP.
Stop advertising ssoma since it's too much to expect users would
be willing to install and use yet another new tool when
NNTP is already standardized and our NNTP server is pretty
efficient.
|
|
Given the prevalance of gigantic VM footprints due to current glibc
malloc and our potentially large number of threads, vfork can speed
up fork used for spawning iostat and SIGUSR2 upgrades.
vfork only pauses the spawning thread, so it will not affect other
I/O threads used in cmogstored; only the non-performance-critical
master thread.
Swapping 'fork()' for 'vfork()' in the following C test program
should show a large speedup under Linux.
Changing FILL to increase or decrease memory usage will respectively
decrease or increase performance improvement gain from vfork over
fork..
-----------------------------8<-------------------------
/* gcc -o x x.c -Wall -O2 -lpthread && ./x */
#include <sys/types.h>
#include <sys/time.h>
#include <unistd.h>
#include <pthread.h>
#include <poll.h>
#include <stdio.h>
#include <sys/wait.h>
#include <stdlib.h>
#include <string.h>
#define FILL (1024 * 1024)
static void *thfunc(void *p)
{
void *ptr = malloc(FILL);
memset(ptr, 1, FILL);
poll(0, 0, -1);
return 0;
}
int main(void)
{
long i;
void *ptr = malloc(FILL);
memset(ptr, 1, FILL);
for (i = 0; i < 100; i++) {
pthread_t th;
pthread_create(&th, 0, thfunc, (void *)i);
}
poll(0, 0, 1000);
for (i = 0; i < 100; i++) {
/* swapping fork with vfork increases performance on Linux */
pid_t pid = fork();
if (pid < 0) {
fprintf(stderr, "ERROR: forking %m\n");
return 1;
}
if (pid == 0) {
char *argv[] = { "/bin/true", 0 };
char *env[] = { 0 };
execve(argv[0], argv, env);
return 1;
} else {
int s;
waitpid(pid, &s, 0);
}
}
return 0;
}
|
|
This has always been supported internally, and we can't stop
supporting it since we'll be supporting upgrades from old versions
indefinitely. So document it, as it has some minor advantages over
the LISTEN_{FDS,PID} environment handling of systemd.
|
|
A bunch of minor changes; most notable is systemd-style socket
activation support. This was easy-to-add since we've always had
socket activation support for nginx-style SIGUSR2 upgrades.
This places no link or runtime dependency on libsystemd, so the
LISTEN_FDS and LISTEN_PID environment variables may be used in other
init systems as well. While I have my own reservations about
systemd itself, I also strongly believe in using socket activation
to prevent downtime.
Behavior changes:
Bad Range: headers return 416 responses in more cases for invalid
ranges (e.g. miscalculated ranges such as "1--1", while
completely wrong ones (lacking a "bytes=" prefix)) are ignored
entirely as in nginx.
Bugfixes:
There are also some cleanups to avoid dying on OOM in more places
on weird systems which trigger OOM. More work on this is ongoing.
Also updates to the latest gnulib.git
commit f197c2c9e5e0d12c373f26d5b3211809457bc972
("intprops: new public macro EXPR_SIGNED")
along with a change which fixes a memory leak when people
build from cmogstored.git using gnulib
commit c6148bca89e9465fd6ba3a10d273ec4cb58c2dbe
or later ("mountlist: add me_mntroot field on Linux machines").
This memory leak did not affect any released tarballs of cmogstored.
shortlog of changes since 1.4.3:
doc: use "builder" RubyGem to generate Atom feed
dev.c: fail gracefully on out-of-memory errors
do not die on OOM when for mgmt paths
HACKING: update URLs to reduce redirects
http: return 416 errors in more cases for bad Ranges
update .gitignores for latest autotools + gnulib
Rakefile: remove text-only part from the Atom feed
support systemd-style socket activation via environment
set TCP listener options on inherited sockets
doc: add example systemd config files
use free_mount_entry from gnulib instead of rolling our own
fix tmpdir dependency for slow Ruby tests
doc: publish examples directory to website
|
|
This might improve visibility of these scripts for use with systemd.
|
|
.slowrb tests have a different suffix and the test dependencies
need to be split out separately.
|
|
gnulib.git added the me_mntroot element in
commit c6148bca89e9465fd6ba3a10d273ec4cb58c2dbe,
so we would leak memory during filesystem refreshes as a result :x
Use the gnulib-provided API (free_mount_entry) instead of freeing
elements ourselves.
|
|
Since we'll support systemd, it's not a bad idea to include
reasonable example files for users.
|
|
systemd users may not set the correct TCP socket options for
us, so be sure to set TCP_NODELAY, SO_KEEPALIVE, and use
a sufficiently large listen backlog to avoid hurting performance
for users who bind sockets outside of cmogstored.
|
|
While I have my reservations about systemd, socket activation alone
is a good idea and we already have existing infrastructure for
supporting it in SIGUSR2 upgrades.
We are intentionally avoiding linkage to libsystemd to avoid dealing
with ABI compatibility issues between old and new systems. This
also allows us to integrate more easily with non-systemd systems
which use the same environment variables as systemd.
|
|
The pre-formatted HTML is readable as raw XML, and feed readers
tend to have no problem rendering the HTML, so there's no point
in nearly doubling our bandwidth usage on the text-only part
given we're already serving XML.
While we're at it, disable XML indentation to avoid wasting space;
it doesn't significantly hamper readability, either.
|
|
Tested on automake 1:1.14.1-4 on Debian jessie,
and automake 1:1.11.6-1 on Debian wheezy.
gnulib was tested on
commit 36d982f39b683d0266b9c6ff1e01cbfc94bd97f6
("test-timespec: fix typo in previous change") from
git://git.savannah.gnu.org/gnulib.git
|
|
For completely unparseable Range: headers, we'll ignore them
entirely as nginx does. However, if /bytes=/ is matched, we'll
start returning 416 errors instead of 400.
|
|
The ragel link no longer worked, actually...
|
|
This also makes trywrite OOM-aware and will simulate a write error
on allocation.
|
|
The rest of cmogstored shall be updated to fail gracefully on OOM
in due time. It may take a while, since not many systems encounter
this, but we shall become more robust as time goes on.
|
|
Nokogiri takes too long to build and install due to the C extension
and bundled library. Prefer a widely-used pure-Ruby gem instead.
|
|
For all platforms, the startup device scanning thread at startup
may not handle EINTR properly. This bug only manifested at
startup and does not affect running instances. However, this
bug is also readily apparent on newer versions of FreeBSD
which support the ppoll function call.
Thanks to Mykola Golub <trociny@FreeBSD.org> for the bug report
which led to this release.
For systems lacking epoll_pwait (older GNU/Linux, all *BSDs),
there is also a bugfix for systems which experience signal spam
leading to errno clobbering in the main thread. This bug was
only only noticed due to a bug report against Ruby:
https://bugs.ruby-lang.org/issues/10866
There is no need to upgrade if 1.4.1 is already running well
on modern GNU/Linux systems capable of epoll_pwait. But then
again nginx-style SIGUSR2 upgrades are transparent to clients.
shortlog since 1.4.2:
Makefile.am: fix publish rule for website
Fix assertion failure during startup
avoid relying on ppoll as a cancellation point
preserve errno when inside sig handler for self-pipe
|
|
We must not clobber errno of the main thread inside signal
handler in case write fails.
This bug only affects systems without epoll_pwait where the
self-pipe is required, so it does not affect modern GNU/Linux
systems; but does affect FreeBSD systems and anybody else
relying on kqueue.
Thanks to Steven Stewart-Gallus for a Ruby bug report which
inspired this fix: https://bugs.ruby-lang.org/issues/10866
Cc: Mykola Golub <trociny@FreeBSD.org>
Cc: Steven Stewart-Gallus <sstewartgallus00@mylangara.bc.ca>
|
|
While glibc supports ppoll, ppoll is not standardized and
apparently is not a cancellation point in some versions FreeBSD
based on Mykola Golub's bug report in
<20150309151851.GC2195@gmail.com>
Reported-by: Mykola Golub <trociny@FreeBSD.org>
|
|
During the initial device scan, it is possible for the waiter to be
interrupted while awaiting cancellation. We must account for this
on all platforms regardless of whether pselect or ppoll is used.
Reported-by: Mykola Golub <trociny@FreeBSD.org>
|
|
Oops, we cannot have zero-byte gzipped files :x
|
|
* Makefile.am: gzip README and associated data
* manpage: update contact and copyright information
* update copyrights to 2014 (and all contributors)
* doc/design.txt: add a few more notes on compromises
* http_dav: log 500 errors from DELETE requests
* tapset/http_access_log: note CLF differences
* copyright updates for 2015
|
|
Via update-copyright in gnulib, also added a few copyrights
to non-trivial files.
git ls-files | UPDATE_COPYRIGHT_HOLDER='all contributors' \
UPDATE_COPYRIGHT_USE_INTERVALS=2 \
xargs /path/to/gnulib/build-aux/update-copyright
|
|
We have two differences from CLF, note them correctly.
|
|
Errors on failed unlink can be a prelude to a bigger problem, so
log it locally ourselves even if the tracker will notice it.
This commit was tested manually by setting up cmogstored to point to a
read-only mount point on my system and attempting a DELETE request on
it.
|
|
In case I forget, writing this down while my mind is on
the subject for other projects.
|
|
In the future, we can use the update-copyright tool from gnulib:
git ls-files | UPDATE_COPYRIGHT_HOLDER='all contributors' \
UPDATE_COPYRIGHT_USE_INTERVALS=2 \
xargs /path/to/gnulib/build-aux/update-copyright
This project (nor any project I manage) has or ever will have have
copyright assignment. All contributors retain copyrights to their
contributions.
|
|
I'll continue accepting email to my private address,
but public email is preferred as it is easier for others
to find messages well as making it easier to credit
bug reporters.
|
|
Speeds up site loading when combined with things like
try_gzip_static in nginx.
|