| Date | Commit message (Collapse) |
|---|
|
Instead of a malloc where arguments can silently overflow,
use calloc directly to handle the overflow and zeroing for
us.
|
|
We should warn gracefully when we hit IPv7+ or whatever...
|
|
While no user is likely to have enough listeners to trigger an
overflow, just use ruby_xrealloc2 to be safe since it's already
provided by Ruby (and AFAIK reallocarray(3) isn't standardized).
|
|
Given the history of Ruby removing public C APIs, get ahead
of potential incompatibilities by switching to an externally
maintained unordered hash table.
khashl is a newer, more memory-efficient evolution of the
khash hash table adopted by the git SCM and this will be
my first (and likely not last) time using it in a public
codebase.
|
|
OBJ_FREEZE before calling rb_hash_aset was actually preventing
deduplication since Ruby 2.6. This introduces a performance
regression for Ruby 2.5 users, but I expect the majority of
users are on newer versions (I'm on 2.7, which is still ancient).
|
|
Once again Ruby seems ready to introduce more incompatibilities
and force busywork upon maintainers[1]. In order to avoid
incompatibilities in the future, I used the following Perl
script to prepend `frozen_string_literal: false' to every
Ruby file:
use v5.12;
use autodie;
my $usage = 'perl /path/to/script <LIST_OF_RB_FILES>';
my $fsl = "# frozen_string_literal: false\n";
for my $f (@ARGV) {
open my $fh, '<', $f;
my $s = do { local $/; <$fh> } // die "read($f): $!";
next if $s =~ /^#\s*frozen_string_literal:/sm;
# fsl must be after encoding: line if it exists:
if ($s =~ s/^([ \t]*\#[ \t\-\*\#]+encoding:[^\n]+\n)/$1$fsl/sm
# or after the shebang
|| $s =~ s/^(#![^\n]+\n)/$1$fsl/
# or after embedded switches in rackup files:
|| ($f =~ /\.ru$/ &&
$s =~ s/^(#\\[^\n]+\n)/$1$fsl/)
# or prepend as a last resort:
|| (substr($s, 0, 0) = $fsl)) {
open $fh, '>', $f;
print $fh $s;
close $fh;
}
}
Somebody interested will have to go through every Ruby source
file and enable frozen_string_literal once they've thoroughly
verified it's safe to do so.
[1] https://bugs.ruby-lang.org/issues/20205
|
|
Compiler optimization isn't useful when doing portability checks
for any of the things we care about.
|
|
I just debugged an issue with our system, I was witnessing the
number of file descriptor in our process grow at an alarming rate
which I mapped to our use of raindrops to report utilisation.
For various reasons we don’t call raindrops from a Rack middleware
but have one process that monitor the socket continuously, and
share that data with the workers.
Since we call tcp_listener_stats every seconds in a process
that doesn't do much else, GC very rarely triggers if at all
which cause `InetDiagSocket` instances to accumulate very
quickly.
Each of those instances holds a file descriptor.
Looking at the raindrops implementation it seems to assume
the GC will take care of regularly closing these sockets, but
I think it’s a bit too bold of an assumption.
[ew: don't close user-passed sockets on exception]
Acked-by: Eric Wong <e@80x24.org>
|
|
alloca makes stack usage unpredictable and life difficult for
static analysis tools and compilers. The 46 bytes of
INET6_ADDRSTRLEN is fine to keep on stack, but page size can be
several MB large in some architectures (but typically 4K on
common architectures).
Thus we handle page size-ed allocations via `rb_str_tmp_new'.
`rb_str_tmp_new' has been in public Ruby headers since the 1.9
days and used by the core `zlib', `digest', and `zlib'
extensions, so it should be safe to use (and `rb_str_resize' is
used in many more C extensions).
|
|
This is less code and hopefully smaller binaries.
`rb_io_check_closed' has been in Ruby since the pre-CVS of
decades ago, and it doesn't matter if it's removed or not
in the future since Ruby 3.1+ doesn't see this code path
and calls `rb_io_descriptor' directly.
|
|
Calling `#to_io' is only necessary when we're handling an argument
from user code where the user could pass a non-IO object.
`#to_io' calls are a waste of time when we create the IO object
ourselves (in `Raindrops::InetDiagSock.new').
This allows us to define the `my_fileno' macro for Ruby 3.1+
users to call the new `rb_io_descriptor' function directly
without an extra C stack frame. This also allows us to get
rid of nesting CPP directives inside C functions which (IMHO)
improves readability.
Furthermore, any necessary #to_io calls using `rb_convert_type'
can be replaced with `rb_io_get_io' to decrease code size.
`rb_io_get_io' has been in ruby/io.h since Ruby 1.9.2 and
there's no expectation that it'd be deprecated since it only
deals with opaque `VALUE' types.
|
|
Prefer to use `rb_io_descriptor` in my_fileno if it is available. This
function was introduced in Ruby 3.1. Ruby 3.3 will deprecate access to
all internal fields of `rb_io_t`.
This change is necessary for compatibility with the current
ruby.git master:
https://bugs.ruby-lang.org/issues/19057#note-17
|
|
This project has been 1.9.3+ only for a long time (and likely
going 2.x+-only), so kill some code we don't need.
|
|
SO_REUSEPORT was introduced in 2013, see https://lwn.net/Articles/542629/.
However, the current logic for calculating socket queue backlog stats
is from 2011, predating the advent of SO_REUSEPORT.
The current strategy was thus written before the notion that the mapping of
INET_ADDR:socket could potentially be 1:N instead of always 1:1.
This causes raindrops to provide invalid socket backlog values when
SO_REUSEPORT is used, which Unicorn supports since 2013. The current behaviour
will set the queue metric to the queue depth of the last inet_diag_msg
it processes matching the INET_ADDR. In practice, this will result in
the backlog being off by a factor N, assuming relatively even
distribution of requests to sockets, which the kernel should guarantee
through consistent hashing.
The fix here is to accumulate the socket queue depth as we iterate
over the the socket diagnostics, rather than reset it each time.
I have a provided a test, but it only checks the queues, not the
accept metrics, as those are not affected by this bug, and it is
not possible to know which of the listeners will be dispatched the
request by the kernel, and thus which should call accept.
|
|
Currently, all memory used by Raindrops is mapped as MAP_ANONYMOUS. This
means that although Raindrops counters can be shared between processes
that have forked from each other, it is not possible to share the
counter values with another, unrelated process.
This patch adds support for backing the Raindrops mapping with a file
descriptor obtained from an IO object. The #initialize API has been
enhanced with two new keyword options:
Raindrops.new(size, io: nil, zero: false)
If an instance of IO is provided, then the underlying file descriptor
for that IO will be used to back the memory mapping Raindrops creates.
An unrelated process can then open the same file, and read the counters;
either by mmap'ing the file itself (or using Raindrops to do so), or by
making ordinary seek()/read() calls if performance is not a concern.
Note that the provided IO object _must_ implement #truncate; this is
used to set the size of the file to be right-sized for the memory
mapping that is created.
If the zero argument is passed as true, then the mapping will be zero'd
by Raindrops as part of its initialization. If it's false, then the
Raindrops counters existing in the file will be preserved. This allows
counter values to be persisted (although note that Raindrops makes no
attempt to msync the values, so they are not durable to e.g. system
crashes).
Counter values can easily be shared between processes
in-memory only without touching the disk by passing in a File on a
tmpfs as the io object.
|
|
With GC.compact in Ruby 3.x, Ruby-defined constants need to be
explicitly marked to prevent movement.
Link: https://yhbt.net/kgio-public/CAAvYYt5Z5f2rMuXO5DMpR1-6uRvu_gXKDvqcyoZ+oNcLiTH39g@mail.gmail.com/T/
|
|
The expiration for bogomips.org is coming up and I'm not
keen on paying or supporting extortionists.
Not wanting to be beholden to ICANN or any powerful
organizations, .onion sites are available to Tor users:
http://raindrops.ou63pmih66umazou.onion/
http://ou63pmih66umazou.onion/raindrops.git/
http://ou63pmih66umazou.onion/raindrops-public/
(the demo is not yet available via .onion, yet, could be a bit)
|
|
Oops, but I guess nobody uses i386 anymore...
|
|
ruby.h already pulls in extconf.h and has done so since Ruby 1.8.
|
|
Drop vestigial Ruby 1.8 bits, and start using
rb_thread_call_without_gvl on modern Rubies.
|
|
This is part of the stable API, so we shall document it for
others to use.
|
|
* origin/freebsd:
define Raindrops::TCP hash for TCP states
tcp_info: support this struct under FreeBSD
|
|
errno is in the thread-specific section and it is slightly
cheaper to read it once rather than twice. Recent versions
of mainline Ruby itself follows the same pattern.
|
|
getnameinfo is overkill for NI_NUMERICHOST + NI_NUMERICSERV usage,
and has a more complex and error-prone API than using inet_ntop
and snprintf.
|
|
FreeBSD not only uses different values than Linux for TCP
states, but different names, too. To ease writing portable code
between the OSes, do more CPP metaprogramming via extconf.rb
and define a common hash supported on both OSes.
Putting all this in a hash allows for easy dumping and mapping
in an OS-neutral way, since the actual TCP states are
OS-independent.
|
|
Of course these fields are not portable between Linux and FreeBSD,
but they should remain ABI-compatible for future versions of each OS.
Tested on FreeBSD 10.3-RELEASE i386
TCP state names will be another problem...
|
|
This provides some extra type safety if combined with other
C extensions, as well as allowing us to account for memory usage of
the HTTP parser in ObjectSpace.
This requires Ruby 1.9.3+ and has remained a stable API since
then. This will become officially supported when Ruby 2.3.0 is
released later this month.
This API has only been documented in doc/extension.rdoc (formerly
README.EXT) in the Ruby source tree since April 2015, r50318
|
|
While #get! is the same as the #initialize method,
the former is public and called explicitly by folks
wishing to reduce allocation overhead.
|
|
Defining the "Raindrops" class explicitly helps RDoc find
subclasses for documentation, and ought to reduce the binary
size slightly due to the removal of rb_intern calls.
Furthermore, use "Socket" to ensure the base class for
Raindrops::InetDiagSocket is documented properly in RDoc.
|
|
While raindrops.bogomips.org exists, having extra subjectAltName
entries is bloating the certificate. This will make it easier
to mirror the homepage on Tor hidden services.
|
|
This helps the compiler detect bugs and
quiets down a -Wsuggest-attribute=format warning
|
|
With invalid addresses specified which give no currently-bound
address, we must avoid leaving placeholders ('true' objects)
in our results.
Clean up some shadowing "cur" while we're at it.
|
|
Neither the GCC nor libatomic_ops URLs are valid, anymore.
Update them to the latest versions and ensure the URLs themselves
are visible in documentation as links to external sites.
Additionally, the current cgit installation on bogomips.org will
soon be replaced by a CSS-free web-based viewer.
Also, correct the reference to "LGPL-2.1+" while we're at it
and add references to the nntp:// mail archives.
|
|
The linux/tcp.h header may not exist on alternative libc
implementations such as musl.
Noticed-by: Doug Forster <doug.forster@gmail.com>
|
|
POSIX and glibc 2.2+ declare the hostlen and servlen
args of getnameinfo(3) to be socklen_t, not size_t,
so favor socklen_t for those calculations.
While we're at it, nlmsg_len is u32, too, so cast it
as such to avoid the warning.
Tested on clang version 3.5-1ubuntu1 on x86-64
|
|
A reviewer may wonder why fprintf is chosen instead of rb_warn,
so make it clear we're outside of the GVL when spewing the
warning message and cannot use most rb_* functions.
|
|
Existing subscribers on librelist will need to resubscribe since
there's no published subscriber lists anywhere.
The public-inbox + mlmmj setup on bogomips.org allows posting
without subscription and offers downloadable archives via git.
The lack of rsyncable archives on librelist nowadays and
subscription-required nature of librelist are points against it.
Repliers should Cc: all recipients (using the reply-all
function of their mail client) since many readers are not
subscribed.
This project has never accepted or encouraged HTML email, but
librelist accepted it. The bogomips.org mail server is
configured to treat HTML mail as spam, so do not send HTML mail
if you expect a response.
Users who wish to subscribe may send a message to:
raindrops-public+subscribe@bogomips.org
Similarly, they may unsubscribe via:
raindrops-public+unsubscribe@bogomips.org
HTTP archives are available via:
http://bogomips.org/raindrops-public/
ssoma users may also use: git://bogomips.org/raindrops-public
(see README change)
Old messages to the librelist addresses will continue to
get routed to the new mailing list.
ref: http://public-inbox.org/
|
|
Clarify the code so memory leak alarms in reviewers eyes do not go off.
|
|
Do not define or use rb_thread_blocking_region if
rb_thread_io_blocking_region is available.
rb_thread_blocking_region is gone entirely in Ruby trunk.
|
|
Scoped ipv6 addresses are defined in rfc4007.
Ruby doesn't support them yet and it's unknown whether it will
(see http://bugs.ruby-lang.org/issues/8464).
So we just remove scope ids.
Tested with MRI and Rubinius.
|
|
RARRAY_PTR is expensive with GCs in Ruby 2.1.0dev and Rubinius, so
use rb_ary_entry for non-performance critical paths. Eventually,
RARRAY_AREF/RARRAY_ASET may be common, but for now, using
rb_ary_entry should require the least cognitive overhead for a
developer.
|
|
__sync_lock_test_and_set might not be the correct check after all,
but __sync_val_compare_and_swap probably is. At least this is
what Ruby 2.0.0 uses to check and attempt -march=i486 on.
|
|
Using an extra 4 bytes for the listener_stats should not
significantly increase space usage, and it has the side benefit
of making our code slightly smaller.
$ ~/linux/scripts/bloat-o-meter before.so after.so
add/remove: 0/0 grow/shrink: 1/2 up/down: 14/-32 (-18)
function old new delta
tcp_stats 392 406 +14
st_to_hash 195 187 -8
diag 763 739 -24
|
|
This initialization was unnecessary and avoids the following
warning with -Wmissing-braces on gcc 4.7.2-5 on Debian testing:
linux_inet_diag.c: In function ‘stats_for’:
linux_inet_diag.c:192:8: warning: missing braces around initializer [-Wmissing-braces]
linux_inet_diag.c:192:8: warning: (near initialization for ‘sa.ss’) [-Wmissing-braces]
|
|
This allows reusing existing Linux::TCP_Info objects to avoid
generating garbage.
|
|
The runnable CPUs of a process may change over the lifetime of
the process. So favor the count of configured processor count
since that is more likely to be stable.
We do not currently do not have special handling for
hot-plugging/removal of CPUs on systems that may load raindrops
in a single CPU state.
|
|
This is exported (visibly) under Ruby 1.9.3 but not
in headers, so it was causing warnings.
|
|
Attempting to test for CMPXCHG on x86 should allow this check to
fail on i386 systems. We also won't need try_run as a result,
enabling cross-compilation. The configure.in check in Ruby
1.9.3 does something similar and that's far more widely used
than raindrops is.
|
|
A non-profit TLD makes more sense for a Free Software project.
|
|
This appears to cause __BSD_VISIBLE to not be defined,
which is required for MAP_ANON to be visible in
sys/mman.h
Thanks for Aleksandar Simic for the hint and Troex Nevelin
for the bug report!
|