Python|Framework Performance
Framework Performance
We recently decided to switch our projects (what were they to start with? Perl / CGI?) to a MVC framework and our choice of the framework came down to three final contenders:
- Symfony on PHP http://www.symfony-project.com/
- Rails on Ruby http://rubyonrails.org/
- Django on Python http://www.djangoproject.com/
To better understand the pros and cons of each, we wrote a very simple demo application in all three. The app we chose was the one built in the Ruby on Rails tutorial Rolling with Ruby on Rails http://www.onlamp.com/pub/a/onlamp/2005/01/20/rails.html.
We really wanted to test the framework, not static file serving or database performance. So we didn't come up with complex queries or serve any static CSS or image files. All requests/hits/transactions are rendered pages. Also, all of the tests are GET operations (database SELEC Ts) for concurrent testing fairness and simplicity.
Each of the tested pages was configured to have a unique number displayed on it based on microtime (to verify the pages weren't being cached). Also, the same pages across all three apps were edited to have nearly the same number of bytes served.
DISCLAIMER: There are infinitely more test scenarios we could have performed. But we wanted something done in a finite time. We feel these results, though not the whole picture, offer hints about relative performance between the three frameworks/platforms.
Test Hardware:
- Dell 1U server (unknown model)
- Dual Xeon 3.2Ghz (6400 bogomips)
- 2 Gigs of RAM
- 2 240Gig (?) SATA drives
Test OS/Software:
- OS: SuSE 10.1
- kernel: 2.6.16.13-4-smp
- database: Mysql 5.0.18
- stress-test software: Siege 2.64 http://www.joedog.org/JoeDog/Siege
- Used stock RPM's for all software unless noted below
Symfony setup:
- Apache 2.2.0 (mod_php)
- PHP 5.1.2
- APC 3.0.10 (PHP accelerator, compiled from source)
- Symfony 0.7.1587 (not an RPM )
Django setup:
- Apache 2.2.0 (mod_python)
- Python 2.4.2
- Django 0.95-3336 (not an RPM)
Rails setup:
- Apache 2.2.0 (mod_proxy_balancer)
- Ruby 1.8.4
- Rails 1.1.4 (not an RPM)
- Mongrel and lighttpd web server (not an RPM)
Apache Prefork tuning:
StartServers 5
MinSpareServers 5
MaxSpareServers 10
ServerLimit 150
MaxClients 150
MaxRequestsPerChild 10000Results
For each of the platforms, we experimented a bit with some options to try to get the best performance by running 3-minute tests. Then we ran an extended 30-minute test on each platform. (Before the actual tests, we pre-ran siege for a minute or two to prime whatever cache's, etc.) Here are some of the results:
Symfony – 50 concurrent users
With PHP+APC: (30 min)
Transactions: 85862 hits
Availability: 100.00 %
Elapsed time: 1801.15 secs
Data transferred: 114.41 MB
Response time: 1.04 secs
Transaction rate: 47.67 trans/sec
Throughput: 0.06 MB/sec
Concurrency: 49.63
Successful transactions: 85862
Failed transactions: 0
Longest transaction: 4.15
Shortest transaction: 0.02Mysql: 7% of CPU
Siege: 5% of CPU
With APC disabled: (3 min)
Transactions: 2256 hits
Availability: 99.34 %
Elapsed time: 181.76 secs
Data transferred: 3.15 MB
Response time: 3.64 secs
Transaction rate: 12.41 trans/sec
Throughput: 0.02 MB/sec
Concurrency: 45.19
Successful transactions: 2256
Failed transactions: 15
Longest transaction: 30.57
Shortest transaction: 0.14NOTE: NOT having a PHP accelerator yields about 12 trans/sec! Horrible! Rumor has it that APC will be included by default in PHP6. In the above tests, we also had syck (for YAML parsing). However it didn't significantly change the results on a 3 minute test (less than 0.5 trans/sec).
Rails – 50 concurrent users
With 10 mongrels: (30 min)
Transactions: 158747 hits
Availability: 100.00 %
Elapsed time: 1800.82 secs
Data transferred: 211.38 MB
Response time: 0.57 secs
Transaction rate: 88.15 trans/sec
Throughput: 0.12 MB/sec
Concurrency: 49.89
Successful transactions: 158747
Failed transactions: 0
Longest transaction: 11.11
Shortest transaction: 0.00Mysql: 10% of CPU
Siege: 10% of CPU
NOTE: In the test above Rails was configured with 10 stand-alone mongrel processes (production mode) and web requests were redirected through Apache using mod_proxy_balancer. In subsequent 3 minute tests, having 20 or 30 mongrels didn't affect the transaction rate more than 0.5 trans/sec.
With lighttpd+fastcgi: (3 min)
Transactions: 15415 hits
Availability: 100.00 %
Elapsed time: 180.66 secs
Data transferred: 20.53 MB
Response time: 0.58 secs
Transaction rate: 85.33 trans/sec
Throughput: 0.11 MB/sec
Concurrency: 49.89
Successful transactions: 15415
Failed transactions: 0
Longest transaction: 1.25
Shortest transaction: 0.04NOTE: The above test was run with the default lighttpd.conf, but on port 80 and in production mode. The number of trans/sec was about the same, but the longest transaction time was much smaller!
Django, 50 concurrent users
With mod_python: (30 min)
Transactions: 224588 hits
Availability: 100.00 %
Elapsed time: 1800.52 secs
Data transferred: 298.94 MB
Response time: 0.40 secs
Transaction rate: 124.74 trans/sec
Throughput: 0.17 MB/sec
Concurrency: 49.92
Successful transactions: 224576
Failed transactions: 0
Longest transaction: 9.31
Shortest transaction: 0.00Mysql: 17% of CPU
Siege: 11% of CPU
NOTE: The above test was run with PythonAutoReload? Off and PythonDebug? Off (production mode). These are by far the best performance numbers of the three frameworks!
High-load Results
I re-ran 3-minute tests on all three platforms with 150 concurrent users to check high-load results.
Symfony – PHP+APC, 150 concurrent users
results
Transactions: 14363 hits
Availability: 98.68 %
Elapsed time: 183.12 secs
Data transferred: 3.72 MB
Response time: 1.34 secs
Transaction rate: 78.43 trans/sec
Throughput: 0.02 MB/sec
Concurrency: 105.06
Successful transactions: 122
Failed transactions: 192
Longest transaction: 30.67
Shortest transaction: 0.02NOTE: You can see there were very few successful transactions. Most were "500 – Internal Server Errors" and 192 failed to connect altogether. It looks like the 500 errors were caused by something with Mysql, because the dev interface returned: "connect failed Native Error: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock"
Rails – 10 mongrels, 150 concurrent users
results
Transactions: 14718 hits
Availability: 100.00 %
Elapsed time: 180.61 secs
Data transferred: 20.22 MB
Response time: 1.79 secs
Transaction rate: 81.49 trans/sec
Throughput: 0.11 MB/sec
Concurrency: 146.12
Successful transactions: 14718
Failed transactions: 0
Longest transaction: 18.39
Shortest transaction: 0.00NOTE: "Transaction rate" down, and "Longest trans" time is up (from 50 concurrent users), but no failed transactions.
NOTE: Attempting this test with lighttpd+fastcgi failed. The web server returned "500 – Internal Server Error"'s and absurd siege result numbers.
Django, 150 concurrent users
results
Transactions: 19903 hits
Availability: 100.00 %
Elapsed time: 180.78 secs
Data transferred: 26.73 MB
Response time: 1.35 secs
Transaction rate: 110.10 trans/sec
Throughput: 0.15 MB/sec
Concurrency: 148.32
Successful transactions: 19903
Failed transactions: 0
Longest transaction: 20.03
Shortest transaction: 0.01NOTE: "Transaction rate" down, and "Longest trans" time is up (from 50 concurrent users), but no failed transactions. Still the top performer.
Summary
Rails performed much better than Symfony. And Django performed much better than Rails.
Comment
When connecting rails to Oracle the performance dropped to the extent it made any production use of the product useless.
Question
"when connecting rails to Oracle the performance dropped to the extent it made any production use of the product useless". Any idea what went wrong there?
Comment
I am very new to Rails and am still learning, but I am experienced in Oracle. I would bet performance degrades dramatically when Rails connects to Oracle as Rails does not use Bind Variables or cache prepared statements. Not using bind variables in Oracle is the single most common mistake. When running the load test connected to Oracle, does Oracle consume a lot of the CPU?
Its unfortunate, as until Rails handles Oracle correctly, its not really fit to be used on it, and I was really hoping to use it there!
Comment
re: Oracle – I speculate that the time necessary to build a connection to Oracle for every request is what kills the performance.
Statistics Question
Is it possible to repeat the tests with httperf instead of siege?
Request: Zed, I think you are quite possibly the best person to do this test. I have read your blog and your concerns re: the common mistakes people make when doing perf testing.
Can you PLEASE PLEASE PLEASE do a proper comparison ???
Zed: Look, go read about "power" in statistics. Specifically just read power.t.test from R's help pages. There's a dead simple calculation you can do to find out how many samples and sample runs you have to do to make this accurate. Running it for 30 minutes don't mean jack, you need runs of samples.
Source Question
Is it possible to get the source for each example?
Helma
Have you consider http://helma.org ?
features inclues MVC use JavaScript instead of Ruby, compiled code to Java bycode for fast runtime performance codeless mapping of application objects to database tables (Just like active record)
- caches query in memory
Well, this comparison proves that RoR is not the most performant framework. But I bet most RoR users already knew that because they have different priorities (e.g. agility). For me, it is enough to know that (performance-wise) RoR is not complete crap. I bet a C solution could pretty much outperform all three solutions. But only masochists would prefer such a solution. There is much more than pure performance to consider.
Much more interesting would be a comparison how long it took to write the app and how much effort is needed should new requirements arise. but I admit that it is almost impossible to have meaningful measurements for this.
Comment
Just because C will always be faster than Rails, it does not mean Rails' performance cannot be improved. It can, and perhaps this data will help. How long it took to write the demo is irrelevant in this context.
Question
I am afraid to say but i don't think the benchmark was done properly, i can't tell about Ruby on rails since i never tested it, but i can about Django and Symfony, and, i do not believe the testers did the deployment procedures in Symfony as described here: http://www.symfony-project.com/book/trunk/deployment
This will increase performance alot, i can't believe symfony got such a bad performance (after working with both symfony and django), it really looks like the tests were done using '_dev.php' wich is the development 'view' with extended logging and debugging wich decreases alot the framework performance. So question is, what was the complete Symfony setup used to do this benchmarks including the URL's in symfony used to do so.
Comment
Well, after reading better some lines of the above article regarding the benchmark, i've just reached 1 conclusion: The testers have no knowledge whatsoever to benchmark anything PHP related, i've just read this small note to reach this conclusion,
"NOTE: You can see there were very few successful transactions. Most were "500 – Internal Server Errors" and 192 failed to connect altogether. It looks like the 500 errors were caused by something with Mysql, because the dev interface returned: "connect failed Native Error: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock" "
FIY, that is caused by using persistent connections to MySQL databases, meaning, mysql will open inumerous childs, when it reaches it's childs limit doesn't allow any more connections untill the previous childs are closed. Clearly states the lack of knowledge of both PHP AND MySQL. I am still astonished with this sentence 'It looks like the 500 errors were caused by something with MySQL' and you guys quickly throw the fault to Symfony/PHP who can take this tests seriously when this kind of comments blatantly show the complete lacking of knowledge of what they're testing.
Ridiculous.
Comment
Aside from earlier comments, what exceptionally whacked about this comparison is how running Rails / mongrel behind a reverse proxy still gets it's butt kicked by mod_python – that shouldn't happen (at least in theory). This comparison at least needed to be done with mod_ruby or stick the others behind a reverse proxy.
Would expect mod_python (1st) / mod_ruby to win any flat race over mod_php – mod_php's interface to Apache sacrifices some performance here in return for "refresh" of state on every new request – so long as you don't do anything like attempt persistant db connections, without knowing what you're doing…
Oh – and for more fun, trying adding this to python http://psyco.sourceforge.net ;)
Comment
The benchmark is completely bullshit. For example django performs much better with python-fastcgi and Apache2. Or running django over the cherrypy WSGI Server is faster than mod_python too. Also enabling psyco gives you a huge performance improvement. And with Python2.5 another Performance Improvement.
Also the symfony benchmark seem strange. In my test installation symfony outperforms Rails easily. No idea what's wrong on the setup here but something seriously is.
I don't want to say that rails is bad, if you choose rails you know that the performance isn't the best but development is fun.
Comment by Cald
The person who mentioned Helma asked for this, and I thought somebody else might also be interested.
From the article RailsvVsDjango:
"In this paper, we compare the two frameworks from the point of view of a developer attempting to choose one of the two frameworks for a new project."
http://docs.google.com/View?docid=dcn8282p_1hg4sr9
Comment
There are a lot of complaints / rants about the efficacy of this test. For those people… if you can do better, then "just do it." It does not take genius, initiative, or knowledge to find fault. It does take at least one of the above to design and run a benchmark test.
It may well be that the test is flawed. But… we would be better served by better tests than flaming someone else's initiative and labor.
I say kudos to the people who designed and ran this test. You gave us something to think about. To the flamers and detractors… put up or shut up.
Comment by KurtStephens
Not sure if this belongs here…
This RubyInline hack took 15% off our app execution time.
Date uses Rational heavily, which calls Integer#gcd for every new Rational.
http://kurtstephens.com/node/34
Comment by Elmer Thomas
You should be able to improve Symfony performance with this:
http://blog.thembid.com/index.php/2007/04/05/build-scalable-web-20-sites-with-ubuntu-symfony-and-lighttpd/
Comment by Massimo
my tests show that a loop in Ruby is about 15 times slower than the same loop in Python. That was enough for me to drop Rails. I'd love you to benchmark Gluon (http://mdp.cti.depaul.edu) which should be even faster than Django because it allows you to byte-code the web app, including the templates, so there is no parsing or text manipulation in production mode. No other framework allows it.
No comments:
Post a Comment