about summary refs log tree commit homepage
DateCommit message (Collapse)
2024-03-25linux_inet_diag: avoid errors for users compiling w/o assertions HEAD master
We should warn gracefully when we hit IPv7+ or whatever...
2024-03-25khashl: use ruby_xrealloc2 to avoid overflow
While no user is likely to have enough listeners to trigger an overflow, just use ruby_xrealloc2 to be safe since it's already provided by Ruby (and AFAIK reallocarray(3) isn't standardized).
2024-03-25use switch to khashl for hash table outside of GVL
Given the history of Ruby removing public C APIs, get ahead of potential incompatibilities by switching to an externally maintained unordered hash table. khashl is a newer, more memory-efficient evolution of the khash hash table adopted by the git SCM and this will be my first (and likely not last) time using it in a public codebase.
2024-03-25linux_inet_diag: remove needless OBJ_FREEZE calls
OBJ_FREEZE before calling rb_hash_aset was actually preventing deduplication since Ruby 2.6. This introduces a performance regression for Ruby 2.5 users, but I expect the majority of users are on newer versions (I'm on 2.7, which is still ancient).
2024-03-23treewide: future-proof frozen_string_literal changes
Once again Ruby seems ready to introduce more incompatibilities and force busywork upon maintainers[1]. In order to avoid incompatibilities in the future, I used the following Perl script to prepend `frozen_string_literal: false' to every Ruby file: use v5.12; use autodie; my $usage = 'perl /path/to/script <LIST_OF_RB_FILES>'; my $fsl = "# frozen_string_literal: false\n"; for my $f (@ARGV) { open my $fh, '<', $f; my $s = do { local $/; <$fh> } // die "read($f): $!"; next if $s =~ /^#\s*frozen_string_literal:/sm; # fsl must be after encoding: line if it exists: if ($s =~ s/^([ \t]*\#[ \t\-\*\#]+encoding:[^\n]+\n)/$1$fsl/sm # or after the shebang || $s =~ s/^(#![^\n]+\n)/$1$fsl/ # or after embedded switches in rackup files: || ($f =~ /\.ru$/ && $s =~ s/^(#\\[^\n]+\n)/$1$fsl/) # or prepend as a last resort: || (substr($s, 0, 0) = $fsl)) { open $fh, '>', $f; print $fh $s; close $fh; } } Somebody interested will have to go through every Ruby source file and enable frozen_string_literal once they've thoroughly verified it's safe to do so. [1] https://bugs.ruby-lang.org/issues/20205
2024-01-15gemspec: support rack 3.x
Hopefully this doesn't cause more breakage and we'll never actually need a rack 4
2024-01-15extconf: disable optimization to speed up checks by ~3%
Compiler optimization isn't useful when doing portability checks for any of the things we care about.
2024-01-15test/test_last_data_recv: don't require posix_mq
posix_mq is rarely installed, so don't force users to have it in order to test or develop raindrops.
2024-01-15tests: support running tests without rack||aggregate
Diskspace and bandwidth are expensive, and we can make rack+aggregate optional in tests, too.
2024-01-15middleware: reuse inet_diag netlink socket
Eric Wong <bofh@yhbt.net> wrote: > I'll squash this in for fork+preload safety. I forget there are multithreaded servers using this :x ------8<----- Subject: [PATCH v3] middleware: reuse inet_diag netlink socket No point in constantly allocating and deallocating FDs (and Ruby IO objects) when reusing them is supported. However, we must guard it against parallel callers since linux_inet_diag.c::diag releases the GVL while calling sendmsg(2) and recvmsg(2), so it's possible two Ruby threads can end up crossing streams if the middlware is used with a multi-threaded server. AFAIK, Raindrops::Middleware isn't a heavily-trafficked endpoint, so saving FDs and avoiding thread specific data is preferable for memory-constrained systems I use.
2023-12-29tcp_listener_stats: always eagerly close sockets
I just debugged an issue with our system, I was witnessing the number of file descriptor in our process grow at an alarming rate which I mapped to our use of raindrops to report utilisation. For various reasons we don’t call raindrops from a Rack middleware but have one process that monitor the socket continuously, and share that data with the workers. Since we call tcp_listener_stats every seconds in a process that doesn't do much else, GC very rarely triggers if at all which cause `InetDiagSocket` instances to accumulate very quickly. Each of those instances holds a file descriptor. Looking at the raindrops implementation it seems to assume the GC will take care of regularly closing these sockets, but I think it’s a bit too bold of an assumption. [ew: don't close user-passed sockets on exception] Acked-by: Eric Wong <e@80x24.org>
2023-09-10aggregate/last_data_recv: support Socket#accept{,_nonblock}
Socket#accept and Socket#accept_nonblock return an Addrinfo object in addition to a client socket. This allows web servers to avoid having to make getpeername(2) syscalls to get the same information.
2023-06-26test_linux_reuseport_tcp_listen_stats: skip w/o unicorn
We don't list `unicorn' as a development dependency since unicorn currently depends on this project. While unicorn might drop us as a dependency, don't waste disk space and bandwidth of potential raindrops hackers who don't have unicorn.
2023-06-26linux_inet_diag: get rid of alloca usage
alloca makes stack usage unpredictable and life difficult for static analysis tools and compilers. The 46 bytes of INET6_ADDRSTRLEN is fine to keep on stack, but page size can be several MB large in some architectures (but typically 4K on common architectures). Thus we handle page size-ed allocations via `rb_str_tmp_new'. `rb_str_tmp_new' has been in public Ruby headers since the 1.9 days and used by the core `zlib', `digest', and `zlib' extensions, so it should be safe to use (and `rb_str_resize' is used in many more C extensions).
2023-06-26my_fileno: use rb_io_check_closed for Ruby <3.1
This is less code and hopefully smaller binaries. `rb_io_check_closed' has been in Ruby since the pre-CVS of decades ago, and it doesn't matter if it's removed or not in the future since Ruby 3.1+ doesn't see this code path and calls `rb_io_descriptor' directly.
2023-06-26avoid unnecessary #to_io calls
Calling `#to_io' is only necessary when we're handling an argument from user code where the user could pass a non-IO object. `#to_io' calls are a waste of time when we create the IO object ourselves (in `Raindrops::InetDiagSock.new'). This allows us to define the `my_fileno' macro for Ruby 3.1+ users to call the new `rb_io_descriptor' function directly without an extra C stack frame. This also allows us to get rid of nesting CPP directives inside C functions which (IMHO) improves readability. Furthermore, any necessary #to_io calls using `rb_convert_type' can be replaced with `rb_io_get_io' to decrease code size. `rb_io_get_io' has been in ruby/io.h since Ruby 1.9.2 and there's no expectation that it'd be deprecated since it only deals with opaque `VALUE' types.
2023-06-11Improve compatibility with Ruby 3.3+
Prefer to use `rb_io_descriptor` in my_fileno if it is available. This function was introduced in Ruby 3.1. Ruby 3.3 will deprecate access to all internal fields of `rb_io_t`. This change is necessary for compatibility with the current ruby.git master: https://bugs.ruby-lang.org/issues/19057#note-17
2023-06-06my_fileno: drop Ruby 1.8 support and simplify
This project has been 1.9.3+ only for a long time (and likely going 2.x+-only), so kill some code we don't need.
2023-02-25raindrops 0.20.1 v0.20.1
Stats for SO_REUSEPORT sockets are now handled properly. Thanks to Dale Hamel for the patches. Dale Hamel (2): Fix queue stats for sockets with SO_REUSEPORT Fix off by one error in test
2023-02-24Fix off by one error in test
It doesn't affect the test outcome, but it is not the intended behaviour of the test. We listen once to find out what port to use, then should bind with SO_REUSEPORT N-1 times, but we are doing it N times, so we have one extra listener.
2023-02-24Fix queue stats for sockets with SO_REUSEPORT
SO_REUSEPORT was introduced in 2013, see https://lwn.net/Articles/542629/. However, the current logic for calculating socket queue backlog stats is from 2011, predating the advent of SO_REUSEPORT. The current strategy was thus written before the notion that the mapping of INET_ADDR:socket could potentially be 1:N instead of always 1:1. This causes raindrops to provide invalid socket backlog values when SO_REUSEPORT is used, which Unicorn supports since 2013. The current behaviour will set the queue metric to the queue depth of the last inet_diag_msg it processes matching the INET_ADDR. In practice, this will result in the backlog being off by a factor N, assuming relatively even distribution of requests to sockets, which the kernel should guarantee through consistent hashing. The fix here is to accumulate the socket queue depth as we iterate over the the socket diagnostics, rather than reset it each time. I have a provided a test, but it only checks the queues, not the accept metrics, as those are not affected by this bug, and it is not possible to know which of the listeners will be dispatched the request by the kernel, and thus which should call accept.
2021-12-06raindrops 0.20.0 v0.20.0
Raindrops may now use file-backed mmap() rather than anonymous memory. Thanks to KJ Tsanaktsidis for the patch: https://yhbt.net/raindrops-public/20211125065618.3432-1-ktsanaktsidis@zendesk.com/T/ The documentation is also updated to note our mail archives are now available via IMAP(S).
2021-11-30doc: update with IMAP(S) URLs and improve descriptions
Our server on yhbt.net runs an IMAP(S) which is accessible over Tor, too. We no longer advertise a "mailing list" since the subscriber info is centralized and not easily reproducible by third-parties. Thus we focus on making our archives easily mirror-able via IMAP(S), NNTP(S) and HTTP(S).
2021-11-26Allow Raindrops objects to be backed by a file
Currently, all memory used by Raindrops is mapped as MAP_ANONYMOUS. This means that although Raindrops counters can be shared between processes that have forked from each other, it is not possible to share the counter values with another, unrelated process. This patch adds support for backing the Raindrops mapping with a file descriptor obtained from an IO object. The #initialize API has been enhanced with two new keyword options: Raindrops.new(size, io: nil, zero: false) If an instance of IO is provided, then the underlying file descriptor for that IO will be used to back the memory mapping Raindrops creates. An unrelated process can then open the same file, and read the counters; either by mmap'ing the file itself (or using Raindrops to do so), or by making ordinary seek()/read() calls if performance is not a concern. Note that the provided IO object _must_ implement #truncate; this is used to set the size of the file to be right-sized for the memory mapping that is created. If the zero argument is passed as true, then the mapping will be zero'd by Raindrops as part of its initialization. If it's false, then the Raindrops counters existing in the file will be preserved. This allows counter values to be persisted (although note that Raindrops makes no attempt to msync the values, so they are not durable to e.g. system crashes). Counter values can easily be shared between processes in-memory only without touching the disk by passing in a File on a tmpfs as the io object.
2021-05-25raindrops 0.19.2 v0.19.2
This release fixes compatibility with GC.compact on Ruby 3.x when using ListenStats on Linux. The listener stats functionality is rarely used and does not affect most users who just have raindrops installed for shared atomic counters.
2021-05-25pkg.mk: remove index.html and created.rid from homepage
index.html and README.html are identical, so get rid of it. created.rid has no business being public, anyways. Our homepage is just a directory listing, nowadays, to improve discoverability of non-HTML docs.
2021-05-25doc: use Tor v3 .onion URLs
Tor is dropping v2 .onion in favor of more secure (but less readable) v3 .onion URLs. Zooko's triangle once again :/ (no, we won't support planet-destroying proof-of-work schemes)
2021-05-25mark ListenStats in C ext for GC.compact
With GC.compact in Ruby 3.x, Ruby-defined constants need to be explicitly marked to prevent movement. Link: https://yhbt.net/kgio-public/CAAvYYt5Z5f2rMuXO5DMpR1-6uRvu_gXKDvqcyoZ+oNcLiTH39g@mail.gmail.com/T/
2020-01-08raindrops 0.19.1 v0.19.1
This release fixes some warnings on newer Rubies. We're no longer on bogomips.org since it's due for expiry and I can't pay extortionists for a .org, so s/bogomips.org/yhbt.net/ for now, and be prepared to move again when extortionists move onto extorting the .net TLD. doc: switch homepage to dark216 ext/raindrops/extconf: fix cflags reset on ancient CC fixes for newer rubies replace bogomips.org with yhbt.net
2020-01-06replace bogomips.org with yhbt.net
The expiration for bogomips.org is coming up and I'm not keen on paying or supporting extortionists. Not wanting to be beholden to ICANN or any powerful organizations, .onion sites are available to Tor users: http://raindrops.ou63pmih66umazou.onion/ http://ou63pmih66umazou.onion/raindrops.git/ http://ou63pmih66umazou.onion/raindrops-public/ (the demo is not yet available via .onion, yet, could be a bit)
2020-01-06fixes for newer rubies
Newer rubies have more warnings
2020-01-06ext/raindrops/extconf: fix cflags reset on ancient CC
Oops, but I guess nobody uses i386 anymore...
2019-04-22doc: switch homepage to dark216
Earth Day! https://80x24.org/olddoc-public/20190422014906.4253-1-e@80x24.org/
2017-08-09raindrops 0.19.0 - Rack 2.x middleware compatibility v0.19.0
This release fixes Rack 2.x compatibility for the few users of Raindrops::Middleware <https://bogomips.org/raindrops/Raindrops/Middleware.html>. Thanks to Dmytro Shteflyuk for this release. No need to upgrade unless you use Raindrops::Middleware with Rack 2.x. There's also a few minor, inconsequential cleanups. Dmytro Shteflyuk (1): Properly override respond_to? in Raindrops::Middleware::Proxy Eric Wong (2): Ruby thread compatibility updates tcp_info: remove unnecessary extconf.h include
2017-07-25Properly override respond_to? in Raindrops::Middleware::Proxy
Correct method definition according to Ruby documentation (https://ruby-doc.org/core-2.4.1/Object.html#method-i-respond_to-3F) is: respond_to?(string, include_all=false) → true or false Rack started using second argument starting from version 2: https://github.com/rack/rack/blob/master/lib/rack/body_proxy.rb#L14 If raindrops is used in Rack 2+ applications, an exception is raised: ArgumentError: wrong number of arguments (2 for 1) <ROOT>/gems/raindrops-0.18.0/lib/raindrops/middleware/proxy.rb:30:in `respond_to?' <ROOT>/gems/rack-2.0.3/lib/rack/body_proxy.rb:14:in `respond_to?'
2017-04-26tcp_info: remove unnecessary extconf.h include
ruby.h already pulls in extconf.h and has done so since Ruby 1.8.
2017-04-26Ruby thread compatibility updates
Drop vestigial Ruby 1.8 bits, and start using rb_thread_call_without_gvl on modern Rubies.
2017-03-23raindrops 0.18.0 v0.18.0
The most notable feature of this release is the addition of FreeBSD and OpenBSD TCP_INFO support. This includes the Raindrops::TCP for portably mapping TCP state names to platform-dependent numeric values: https://bogomips.org/raindrops/Raindrops.html#TCP Thanks to Jeremy Evans and Simon Eskildsen on the unicorn-public@bogomips.org mailing list for inspiring these changes to raindrops. There's also a few internal cleanups, and documentation improvements, including some fixes to the largely-forgotten Raindrops::Aggreage::PMQ class: https://bogomips.org/raindrops/Raindrops/Aggregate/PMQ.html 20 changes since 0.17.0: test_inet_diag_socket: fix Fixnum deprecation warning TODO: add item for IPv6 breakage ext: fix documentation for C ext-defined classes TCP_Info: custom documentation for #get! TypedData C-API conversion test_watcher: disable test correctly when aggregate is missing tcp_info: support this struct under FreeBSD define Raindrops::TCP hash for TCP states linux_inet_diag: reduce stack usage and simplify avoid reading errno repeatedly aggregate/pmq: avoid false sharing of lock buffers aggregate/pmq: remove io-extra requirement aggregate/pmq: avoid File#stat allocation Merge remote-tracking branch 'origin/freebsd' Merge remote-tracking branch 'origin/aggregate-pmq' doc: remove private email support address doc: update location of TCP_INFO-related stuff build: avoid olddoc for building the RubyGem doc: document Raindrops::TCP hash aggregate/pmq: update version numbers for Ruby and Linux
2017-03-23aggregate/pmq: update version numbers for Ruby and Linux
Linux is in the 4.x and Ruby is into the 2.x range, by now, so try to get with the times.
2017-03-23doc: document Raindrops::TCP hash
This is part of the stable API, so we shall document it for others to use.
2017-03-23build: avoid olddoc for building the RubyGem
This should make it easier for distros to pick this without picking up olddoc, too. We will still use olddoc for generating the website since it reduces bandwidth costs, and for generating NEWS for our own builds.
2017-03-23doc: update location of TCP_INFO-related stuff
2017-03-20doc: remove private email support address
Email was never private, and won't further burden myself or any future maintainers with trying to maintain someone elses' privacy. Offering private support is also unfair to readers on public lists who may get a watered down or improperly translated summary (if at all). Instead, encourage the use of anonymity tools and scrubbing of sensitive information when the sender deems necessary.
2017-03-18Merge remote-tracking branch 'origin/aggregate-pmq'
* origin/aggregate-pmq: aggregate/pmq: avoid File#stat allocation aggregate/pmq: remove io-extra requirement aggregate/pmq: avoid false sharing of lock buffers
2017-03-18Merge remote-tracking branch 'origin/freebsd'
* origin/freebsd: define Raindrops::TCP hash for TCP states tcp_info: support this struct under FreeBSD
2017-03-18aggregate/pmq: avoid File#stat allocation aggregate-pmq
File#size is available in modern Rubies so the extra syscall is avoided.
2017-03-18aggregate/pmq: remove io-extra requirement
IO.copy_stream is standard in 1.9+ and can use pread when given an offset. We do not need to use pwrite with fcntl locking, actually.
2017-03-18aggregate/pmq: avoid false sharing of lock buffers
And rely on frozen string optimizations in Ruby while we're at it.
2017-03-18avoid reading errno repeatedly
errno is in the thread-specific section and it is slightly cheaper to read it once rather than twice. Recent versions of mainline Ruby itself follows the same pattern.
2017-03-17linux_inet_diag: reduce stack usage and simplify
getnameinfo is overkill for NI_NUMERICHOST + NI_NUMERICSERV usage, and has a more complex and error-prone API than using inet_ntop and snprintf.