Web Server Comparisons

"...factoring out I/O, the primary determinant to server performance is the concurrency strategy."

-- JAWS: Understanding High Performance Web Systems

Some comparisons between various web servers. We look at features, sizes, and performance. The headers in the table are links to explanatory notes below. Bright green entries indicate something unusually good, and red entries are something bad, sometimes with a link to a note explaining why.

Don't get too excited over the performance figures. Most of these servers have enough oomph to keep a T1 saturated, running on a lowly 100MHz Pentium or maybe even a 486. Very very few web applications need more power than that. So, the fact that Apache is not that fast shouldn't be of concern to most people.

Last updated 12jul98.

software				features					size				RPS		max users	name
name	version	released	active devel	model	auto conf	basic auth	chroot	throttling	tar bytes	source files	source lines	exe	small files	CGI	large files
thttpd	2.03	11jul98	yes	select	yes	yes	yes	yes	406,016	20	7,229	49,584	720	100	1000+	thttpd
Apache	1.3.0	05jun98	yes	pre-fork	yes	yes	no	no	4,925,440	149	73,381	397,152	250	90	150	Apache
mathopd	1.2b9	23may98	yes	select	yes	no	yes	no	112,640	8	3,658	49,088	770	75	500	mathopd
Roxen	1.2.29	10jun98	yes	threads	yes	yes	yes	no	14,274,560	1,480	247,789	502,284	170	11	50	Roxen
Boa	0.92	23dec96	yes	select	no	no	no	no	225,280	21	4,103	75,788	475	115	50	Boa
Jigsaw	2.0beta1	08apr98	yes	Java threads	n/a	yes	yes	no	11,341,824	594	95,744	1,841,601	45	n/a	25	Jigsaw
Acme.Serve	-	18may98	yes	Java threads	n/a	no	no	yes	302,592	27	4,718	79,943	45	n/a	25	Acme.Serve
CERN	3.0A	15jul96	no	fork	yes	yes	no	no	2,703,360	153	56,028	561,696	115	65	300	CERN
NCSA	1.5.2a-export	12oct96	no	pre-fork	no	yes	no	no	931,840	77	23,726	196,040	260	70	350	NCSA
Netscape FastTrack	3.01-export-us	02oct97	no	threads	n/a	yes	?	no	62,443,520	n/a	n/a	1,896,016	-	-	-	Netscape FastTrack
Netscape Enterprise	3.5.1-export-us	02feb98	yes	threads	n/a	yes	?	no	130,426,880	n/a	n/a	1,977,568	-	-	-	Netscape Enterprise
Zeus	3.1.4	10jul98	yes	select	n/a	yes	no	yes	3,747,840	n/a	n/a	1,367,976	1050	70	500	Zeus

Notes:

• Software - which web server.
  • Name - the name of the server, and a link to its home page.
  • Version - the version tested here.
  • Released - when it was released.
  • Active devel - whether it is currently under development.
• Features - some key server features.
  • Model - what kind of server it is. The models are:
    • fork - start a new process for each request.
    • pre-fork - pre-start a pool of processes which each handle multiple requests.
    • threads - use threads instead of processes.
    • Java threads - this version of the Java runtime uses "Green threads" instead of native threads, so it acts more like the select-based servers.
    • select - use non-blocking I/O and the select() system call to handle multiple requests in a single process, single thread.
  • Auto-conf - whether there's a script to automatically configure the build process for your OS.
    • mathopd did not compile out-of-box under Solaris 2.6, I had to make a minor edit.
    • Roxen did not compile under Solaris 2.6, I had to build under Solaris 2.5.1.
  • Basic auth - whether the server supports Basic Authentication, for password-protected web pages.
  • Chroot - whether the server lets you use the chroot() system call to enhance security.
  • Throttling - the ability to set bandwidth limits on certain pages, so they don't use more than their fair share of the server's resources.
• Size - various sizes of parts of the server
  • Tar bytes - the uncompressed source tarchive size.
  • Source files - how many source and header files.
  • Source lines - how many lines in the source and header files.
  • Exe - the executable size. For the compiled program this is size of the main executable file, stripped. For the Java servers it's the total size of the .class files or .zip files. For Roxen it's the size of the Pike interpreter.
• The benchmark test system is a 297MHz Ultra Sparc with 256MB RAM / 512MB swap running Solaris 2.6, otherwise totally quiescent. RLIMIT_NOFILE is 256 soft / 1024 hard, and v.v_maxup is 3941.
• The test loads were generated on a separate machine connected via 100MB/s ethernet. The test client is http_load, doing multiple requests in parallel, with checksumming enabled.
• All the test runs were done at least twice on different days, and all the shapes and inflection points etc. proved to be completely repeatable.
• All programs are in their out-of-box configuration, except as needed for the tests (e.g. enabling CGI). And except for mathopd, which doesn't have an out-of-box configuration - I used this one.
• RPS - maximum requests per second. This is determined by running the test load at various parallel-request rates and finding the maximum response rate. See the graph below for the full data.
  • Small files - the small-file test load consists of 1000 files, each 1KB long, requested randomly.
    • Acme.Serve exitted at 60 parallel requests, due to running out of file descriptors. Apparently the Java runtime uses three file descriptors per thread, or something like that. But still, the program shouldn't exit!
    • Jigsaw mysteriously refused to serve 40 of the 1000 files in the test load. Same 40 consistently. I took those 40 out of the test and the other 960 files worked fine. This sort of thing doesn't really build confidence in a server.
  • CGI - the CGI test load consists of a trivial "hello world" C program.
    • Roxen started returning bad checksums at only 3 parallel CGI requests.
    • Some of the servers do not implement the parts of the CGI spec that require header-parsing - thttpd, mathopd, Boa, perhaps others. This could be seen as giving them an unfair advantage at performance comparisons.
• Max users - maximum number of simultaneous users that can be handled. This is determined by running the test load at various parallel-request rates and seeing when it starts screwing up. Typical screwups are running out of memory or processes, so that requests start timing out or getting refused.
  • Large files - the large-file test load consists of 100 files, each 1MB long, requested randomly. Also, each connection is throttled to simulate a 33.6Kbps modem. Note that 1000 33.6Kbps connections is 3/4 of a T3.
• Some servers that I wanted to benchmark but didn't get to:
  • phttpd I couldn't get to run.
  • Netscape's license agreement prohibits publishing benchmark results (section 6 viii).

Here's a graph of the full data:

Comments:

• That nice diagonal line is very interesting. A bunch of very different servers follow it exactly for the first part of their curve. Its slope is 5 hits/second per user, indicating that each hit takes a minimum of 1/5th second to handle. But other servers don't follow it, showing that it's not an inherent limit in the test setup. thttpd-2.00 was following the line, while mathopd, a very similar server, does not, so I made some changes to thttpd to try and get its latency to be more like mathopd's. Turns out the change that made the difference was sending the response headers and the first load of data as a single packet, instead of as two separate packets. Apparently this avoids triggering TCP's "delayed ACK", a 1/5th second wait to see if more packets are coming in. thttpd-2.01 has the single-packet change.
• The sharp drop-off at 125 users for a bunch of different servers is mysterious, especially considering it applies to servers of three different models - pre-fork, threads, and select - and that other servers are immune.