Now, the cleverer ones amongst you will know that Apache Benchmark, or ab, will do pretty much the exact same thing, plus much more. This is where I was planning to get my first batch of results from. But something has happened that is leaving me puzzled as to how I'm going to make these tests fair.

I ran a few benchmarks against a basic PHP page, and it was pretty consistent, within the realms of what you'd expect for a page just coughing up the same content over and over. When I ran the same benchmark against a similar .aspx page however, things were not consistent!

Mono benchmarks -- an anomaly

Without going into a long post about what all results were, here's a brief summary. I originally noticed the problem when I ran ab -n 1000 -c 10 [url] from a remote machine that is on the other side of the country. ab was given a PHP URL and an ASP.Net URL for the test server here on my home network, and dutifully returned results for each. The PHP page was giving me results similar to the following (results in seconds for total test time, rounded for brevity):

75, 75, 74

Looks good. Kind of what you want to see in a benchmark. Predictable load times under a given circumstance means your application code is working predictably as well. I ran three tests and averaged the results. So what did I get from Mono? I restarted apache, hit a page to get rid of the initial .Net load time, then ran three tests. Check this out:

77, 77, 100

I thought that was kind of weird. Maybe the server was under some sort of load I didn't know about, so I thought I'd chuck the third benchmark as anomalous and run some more:

95, 107, 92, 106

Now I had a case of the WTFs. Thinking something was wrong, I rebooted apache (and hit a single .aspx page for the initial .Net load). Then I ran some more tests:

restart, 78, 79, 92

There it goes again! I switched to testing on the server machine itself under my desk to get rid of any network latency that might be occurring cross-continent. I ran some PHP benchmarks to see what I would get.

51, 51, 66, 52, 51, 51, 51

Looks good, save the spike in the middle. Back to Mono:

restart 56, 58, 61, 62, 68, 62, 63, restart, 57, 61, 61, 63, 67

WTF

It quickly became evident that the Mono environment is accruing some sort of overhead the more I pound on it. And it's not getting away from this overhead either. I left it alone for awhile to gather its thoughts (and maybe its garbage) while I took a break, but when I tried again it still showed much higher benchmark times than it would right after a server restart. The conclusion I came to over another hour of testing is that Mono would never serve pages as fast as the first benchmark on subsequent benchmarks.

This has delayed my actual collection of benchmark data between PHP and Mono ASP.Net, as I don't feel that it's a fair comparison between the two if Mono is accruing performance penalties the more I use it.

I'm going to have to research this a bit more. Maybe it's my configuration, maybe it's Apache, maybe it's mod_mono, maybe it's something wrong with the .aspx test page, maybe it's a spooky goblin. Maybe it's Mono itself. I don't have enough information at this point to be sure. I'll have to try to narrow this down, or continue testing knowing this limitation.

However, the growing pains have been a bit tougher than I expected. Not that I thought it would be as simple as compile-upload-browse right off the bat, but it did take a few hours to iron everything out, even with some relatively basic pages. For one, the generation of strongly typed datasets using XSD.exe under mono apparently just doesn't work. It took a while to realize that no amount of coercing the utility was going to give the same output as XSD.exe under Windows, so this forces an extra step if you're developing cross-platform. You have to manually generate your datasets and transfer them over as a .dll instead of relying on auto-compilation of your App_Code XSD file.

On top of that, I actually discovered my first mono bug. Say you've got a couple of ObjectDataSources on the page. Under Windows.Net, it's cool to have a DataList that uses one of them, and inside that DataList you nest a second DataList that references the second ObjectDataSource. Doesn't work in mono. I was able to work around it by using nested FormViews instead, but it puzzled me for a few minutes. Looking around at the mono svn changelogs, it looks like the FormView and DataList components handle the detection of the naming container differently. If I get a chance I'll put together a simple test project and submit it to the mono project. I actually have no idea which is the correct behaviour -- maybe it's a bug in Windows?

Regardless, I have a lot of confidence in mono. Partly because of projects like Plastic SCM, partly because I've heard other good stories about it, and partly because I just really want it to work for the sake of object oriented web programming. We'll see how the rest of the test site goes!

And coming up this week: our first PHP vs ASP.Net head to head! We test web server response times and some performance statistics…..

Just in the nick of time too. Or is it? Doing a google search for "PHP vs .Net" brings up -- as result #1 -- an article from SitePoint discussing the merits of the technologies. From… 2002.  The only blog that makes the Top 10 results bravely states that no one has any idea what they're talking about, followed by half a paragraph of useless blather and a recommendation for J2EE… from a post in 2003.

Is our debate irrelevant? Has it been beaten to death? I'd like to think that the answer to that question is irrelevant.  It's still pretty frustrating trying to find a useful comparison with hard numbers. Oh, .Net has extreme cache control you say… but PHP on MySQL can be optimized for extreme database reads? So which one is better again? If it's been beaten to death, they hid the body pretty well. The point is, we STILL see people confused about these technologies, their differences, and needing a 2 paragraph summary to put in their technical feasibility report that's due on their bosses desk in 15 minutes.

PHP and ASP.Net are both great languages. Any developer worth their salt knows you choose the right tool for the job.  But in our experience, most development shops are exclusively married to one technology for their web facing projects, and there's not a lot of professional development going on for people to learn both toolboxes.  So long story short, the context of this blog is to compare and contrast PHP and .Net on projects that we're working on.

So with grand ceremony, we will start off the journey by documenting the project that birthed the idea of this blog. I'm beginning to rebuild a website that was started with PHP and then abandoned for a while. I'm building it in ASP.Net and trying to run it on an old honky clunker with CentOS4.4 and Mono 1.2.6. Incidentally, Mono is the technology that will make most of our experiments possible, and will probably have a strong focus here. We've got some great articles coming, including some sciencey type benchmarking tests, but in the meantime I'm still finalizing the blog layout and getting cool people to link to it. If you're cool, which seems likely, feel free!

Since all the heavy lifting of starting the blog is done, we'll be letting the dust settle for the next week or so, inviting the robots take a sniff around, and heading off to Maui for a week.