We promised some numbers, but I've run into a snag that's slowed down my testing. Originally when we first started toying with the idea of pitting Mono + ASP.Net against PHP, I wrote a quick PHP script that would just make a cheap fopen call to whatever URL you wanted to test, however many times you wanted to hit it. This isn't really load-testing, it's more like a basic timer, and not a very accurate one at that. However, it gave me some quick raw numbers to work with.
Now, the cleverer ones amongst you will know that Apache Benchmark, or ab, will do pretty much the exact same thing, plus much more. This is where I was planning to get my first batch of results from. But something has happened that is leaving me puzzled as to how I'm going to make these tests fair.
I ran a few benchmarks against a basic PHP page, and it was pretty consistent, within the realms of what you'd expect for a page just coughing up the same content over and over. When I ran the same benchmark against a similar .aspx page however, things were not consistent!
Without going into a long post about what all results were, here's a brief summary. I originally noticed the problem when I ran ab -n 1000 -c 10 [url] from a remote machine that is on the other side of the country. ab was given a PHP URL and an ASP.Net URL for the test server here on my home network, and dutifully returned results for each. The PHP page was giving me results similar to the following (results in seconds for total test time, rounded for brevity):
75, 75, 74
Looks good. Kind of what you want to see in a benchmark. Predictable load times under a given circumstance means your application code is working predictably as well. I ran three tests and averaged the results. So what did I get from Mono? I restarted apache, hit a page to get rid of the initial .Net load time, then ran three tests. Check this out:
77, 77, 100
I thought that was kind of weird. Maybe the server was under some sort of load I didn't know about, so I thought I'd chuck the third benchmark as anomalous and run some more:
95, 107, 92, 106
Now I had a case of the WTFs. Thinking something was wrong, I rebooted apache (and hit a single .aspx page for the initial .Net load). Then I ran some more tests:
restart, 78, 79, 92
There it goes again! I switched to testing on the server machine itself under my desk to get rid of any network latency that might be occurring cross-continent. I ran some PHP benchmarks to see what I would get.
51, 51, 66, 52, 51, 51, 51
Looks good, save the spike in the middle. Back to Mono:
restart 56, 58, 61, 62, 68, 62, 63, restart, 57, 61, 61, 63, 67
It quickly became evident that the Mono environment is accruing some sort of overhead the more I pound on it. And it's not getting away from this overhead either. I left it alone for awhile to gather its thoughts (and maybe its garbage) while I took a break, but when I tried again it still showed much higher benchmark times than it would right after a server restart. The conclusion I came to over another hour of testing is that Mono would never serve pages as fast as the first benchmark on subsequent benchmarks.
This has delayed my actual collection of benchmark data between PHP and Mono ASP.Net, as I don't feel that it's a fair comparison between the two if Mono is accruing performance penalties the more I use it.
I'm going to have to research this a bit more. Maybe it's my configuration, maybe it's Apache, maybe it's mod_mono, maybe it's something wrong with the .aspx test page, maybe it's a spooky goblin. Maybe it's Mono itself. I don't have enough information at this point to be sure. I'll have to try to narrow this down, or continue testing knowing this limitation.
Blog Catalog
[...] I'm rebuilding the Mono test box from scratch. I'm not expecting this to resolve the previous issues I experienced while benchmarking Mono, but it is allowing me to tie up some loose ends. More on [...]
Asp.Mono in Linux? Not Yet……
During development of CppCMS I have always wanted to do some benchmarks against one of the most popular web development technologies: Asp.Net. I had found a blogging system: BlogEngine.Net that in its latest version had full support of mono under Linux…
[...] Benchmarking Mono ASP.Net vs. PHP – a slight problem | PHP vs .Net Benchmarking Mono ASP.Net vs. PHP – a slight problem [...]