I thought it would be fun to compare the response times of open source web frameworks. These come from ApacheBench 2.0.40-dev running on Linux (CentOS 5) inside VMWare. My computer is a dual Xeon E5345 @2.33GHz. The guest OS has 4GB of RAM and 1 virtual CPU. The purpose of this test is to determine the relative speed differences between frameworks for common tasks. Each test consisted of 10,000 requests. Concurrency was 1.
To provide a baseline for comparison, I built apache 2.2.8 and tested the index page.
Apache static file: 2991.65 Requests/sec with a time per request of 0.334ms.
Now on to the frameworks. First up is Ruby on Rails. The setup was about as basic as you can get. A simple view with no layout is rendered. The controller contains no code. Production environment was used. For the first test I ran against a single port in the mongrel cluster.
Rails on Mongrel: 485.85 requests/sec with a time per request of 2.058ms.
There’s a problem with this test. This is not a realistic way to run a rails site. You need to put a reverse proxy out front.
Rails on Mongrel behind Apache: 299.55 requests/sec with a time per request of 3.338ms.
Next up is PHP. There are a lot of different frameworks written in PHP. First I tested a simple PHP script that just prints hello world. I used the recommended php.ini.
PHP on Apache: 2518.25 requests/sec with a time per request of 0.397.
That’s not too far away from the static results. But to compare this to a framework is unfair. A full MVC framework does more by default. Common tasks such as mapping the URLs handlers, dealing with session cookies, and parsing templates add some overhead. It’s better to compare individual PHP frameworks.
The Symfony Project:
I used the sf_sandbox.tgz sample. I stripped out the graphics and CSS until I had a simple, un-styled, “Hello World”
Symfony: 32.21 requests/sec with a time per request of 31.048ms.
That’s so slow. I wonder how other PHP based frameworks do.
CakePHP: 34.90 requests/sec with a time per request of 28.657ms.
Also crappy performance. I’ll have to check with these projects to make sure there are no serious problems with my config.
Next is the Perl based Catalyst framework. I’m just testing the default index page.
Catalyst built-in server: 138.75 requests/sec with a time per request of 7.207ms.
Normally Catalyst is run in production under Apache/mod_perl.
Catalyst on Apache: 140.67 requests/sec with a time per request of 7.109ms.
I tried editing the Root controller to return “Hello World” rather than the larger welcome message. Results improved by about 6%.
I also want to try TurboGears but I’m having trouble getting it to run.
Conclusion: These preliminary results are not what I expected. Ruby has a reputation of being slow yet it outperformed the competition. The next step is to do a better job of setting up the tests to make sure each framework is producing exactly the same content and is performing the same amount of session and caching work.
Posted by seancomeau 