david - an HTTP load testing tool aimed at fork/threading servers david (as in david vs. Goliath) is a slow HTTP client intended for server administrators test behavior when many slow clients are connected to it. It is very small, but can open many active connections which can cause the server to deny connections from other clients. Highly recommended reading: http://kegel.com/c10k.html ======= The Problem ======= Misconfigured forking/threading servers can come under a lot of load when lots of traffic from slow clients connect to it and processes. Per-IP connection limits can prevent david from doing harm to other clients attempting to access a server, but it cannot prevent real-world scenarios where many users on slow Internet connections are all attempting to to access an HTTP site. Avoiding HTTP servers that use the per-client thread/process model is a good start. However if that's not possible (heavy reliance on mod_perl, mod_php, etc), then using a reverse proxy (nginx[1] is a good candidate) to buffer and spoon-feed slow clients works well, too. Servers using one client per thread/fork require an entry in the process table (or thread table), and additional stack memory usage. Even with shared memory and copy-on-write fork, this memory usage is much bigger than an implementation that avoids these things entirely and relies on I/O readiness notification for each. The addition of kernel thread or processes for each connection also places additional burden on the kernel scheduler and giving other tasks less time to run. [1] nginx just seems to drop connections that are going too slowly (1 byte/second). I doubt many users actually use connections that slow. ======= Inspiration ======= Many application servers I've seen often use at least 20-30 megabytes of memory (however not taking shared memory after fork into account). With Apache, when many clients connect, more processes are spawned (up to a limit defined in a configuration file). These server processes exist to execute application code quickly, however while they are often underutilized because they are waiting on I/O from slow clients. So while waiting on I/O from slow clients, the application does little else other than using a sizable chunk of memory on the system. ======= Usage ======= david sends HTTP requests from the first filename passed to it on the command-line so it can easily be customized by the user. Included is are barebones HTTP 1.0 and HTTP 1.1 request I use for testing. Configuration is done mostly via editing pre-processor #defines at the top of the source file. The HTTP_HOST environment variable must be defined for david to connect to a server. HTTP_PORT is optional and defaults to 80. Command-line: HTTP_HOST=www.example.com ./david http_1.1.txt ======= Implementation ======= david is a simple select()-based client. However, it uses child processes to get around file descriptor limits on select() and also per-process limits. I'm choosing select() for portability and simplicity over more exotic, less-portable I/O readiness schemes. david itself uses very little in the way of system resources other than file descriptors. Since it is used to simulate slow clients, it reads and writes to the socket very slowly and is not performance intensive. This has only compiled this on a Linux 2.6-based system, but it is believed to be portable to all systems supporting mmap, and non-blocking TCP sockets. mmap() is used so that a user could write arbitrarily large requests without causing david to use large amounts of resident memory. ======= Disclaimer ======= Like many tools, david can be misused with malicious intent. The author does not condone its usage on any machine without the express permission of its administrator(s). ======= License ======= This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA