Sunday, March 30. 2008
Benchmark: PHP vs. Python vs. Perl vs. Ruby
Today, I was looking at a couple of PHP benchmarks (the one here for example), and they compared OOP benchmarks to Procedural benchmarks for the most part. I decided to write my own set of benchmarks using four of the most well-known web development languages that were ready-at-hand: PHP, Python, Perl, and Ruby. My benchmark also tests OOP vs. Procedural on a few terms: Execution speed and Raw lines of code written. I compare a visual estimate of memory usage between all four languages as well.
Now, one cannot draw very many conclusions from my benchmarks here, because the programs used don't really do anything useful, nor do they simulate "real-world" conditions, or practical use-cases. Remember, we learn multiple programming languages, not because of fads, but because it expands our own knowledge, and because some languages are better than others for some things; and it's helpful to be able to choose the right tool for the job.
Keep in mind the golden rule of benchmarking: All benchmarks are flawed benchmarks.
Now, one cannot draw very many conclusions from my benchmarks here, because the programs used don't really do anything useful, nor do they simulate "real-world" conditions, or practical use-cases. Remember, we learn multiple programming languages, not because of fads, but because it expands our own knowledge, and because some languages are better than others for some things; and it's helpful to be able to choose the right tool for the job.
Keep in mind the golden rule of benchmarking: All benchmarks are flawed benchmarks.
Conditions
CPU: 64-bit AMD Sempron 3000+ @ 1 GHz
OS: Gentoo Linux (x86-64) w/kernel: 2.6.22-gentoo-r6
RAM: 1 GB total
Scripting Language Versions:
PHP: 5.2.5-pl0-gentoo w/suhosin
Perl: 5.8.8
Python: 2.5.1
Ruby: 1.8.6 (2007-12-03 patchlevel 113)
All tests were run consecutively, and thus had approximately the same amount of memory and CPU time available to them.
Method
A bash script was used to automate testing and record results. A perl script was used to average the run times and give a result.
Test Programs
Two test programs were measured:
Hello - Prints "Hello World" to stdout.
Inc - Increments a variable, starting at 0.
Each program does one-million iterations, and is run ten consecutive times. It's also worthwhile to note that for these type of benchmarks use of OOP or a function would be impractical.
The test programs, shell script, perl script, and an Open Office spreadsheet can be downloaded here.
Results
Hello benchmark: (Times are in seconds)
Inc benchmark: (Times are in seconds)
Total Lines of code (including comments, etc.):
The memory usage is a percentage, obtained by watching top, and averaging the results across runs of the programs.
Conclusions
As I stated earlier, these benchmarks are not the kind you'd use OOP for (or even a function for code reuse.) However, these tests reveal that the slowest-executing programming method is: poorly-used OOP, which unfortunately is common-place nowadays. Excessive function calls also tend to add a good amount of overhead, and make for a good example of poor procedural programming. Performance really depends more on how the code is written as opposed to the paradigm in which it is written.
I'll admit, I despise 'traditional' OOP. It's messy, over-complicated, and completely unnecessary (I could rant all day.) I am, and always will be, a purely procedural programmer, and a pedantic bastard. One can have 'object-oriented' code, outside the scope of traditional OO, and traditional OO languages; in a more procedural fashion. That said, I have not biased these tests in any way.
The Neither column is the one that should be used to compare the four languages really, and going based upon that, one can conclude that writing these tests in any of the four doesn't require much more or less code than any of the others. From an execution-time perspective, Perl is the fastest overall. Perl also used the least memory overall.
Because of the number of iterations done within each test program (1 million), for simple scripts you're not going to see much of a difference in execution time between the four. However, for intensive tasks, you may see a very large difference, with Perl most-likely being faster than the other three; though as I previously stated, it all depends on how you write the code.
Eventually, I will write a few good benchmarks to show the substantial differences between OOP and procedural programming.
CPU: 64-bit AMD Sempron 3000+ @ 1 GHz
OS: Gentoo Linux (x86-64) w/kernel: 2.6.22-gentoo-r6
RAM: 1 GB total
Scripting Language Versions:
PHP: 5.2.5-pl0-gentoo w/suhosin
Perl: 5.8.8
Python: 2.5.1
Ruby: 1.8.6 (2007-12-03 patchlevel 113)
All tests were run consecutively, and thus had approximately the same amount of memory and CPU time available to them.
Method
A bash script was used to automate testing and record results. A perl script was used to average the run times and give a result.
Test Programs
Two test programs were measured:
Hello - Prints "Hello World" to stdout.
Inc - Increments a variable, starting at 0.
Each program does one-million iterations, and is run ten consecutive times. It's also worthwhile to note that for these type of benchmarks use of OOP or a function would be impractical.
The test programs, shell script, perl script, and an Open Office spreadsheet can be downloaded here.
Results
Hello benchmark: (Times are in seconds)
OO Func. Neither
PHP: 4.1248 3.7656 0.9309
Ruby: 2.8940 2.7772 2.1919
Python: 1.7893 1.7498 1.3138
Perl: 1.3188 1.1824 0.6986
Inc benchmark: (Times are in seconds)
OO Func. Neither
PHP: 3.7079 4.3501 0.6164
Ruby: 1.5032 1.3473 0.8810
Python: 1.0387 0.8543 0.5792
Perl: 0.4870 0.9355 0.4877
Total Lines of code (including comments, etc.):
OO Func. Neither
PHP: 32 24 14
Ruby: 41 26 18
Python: 33 23 16
Perl: 61 28 19
The memory usage is a percentage, obtained by watching top, and averaging the results across runs of the programs.
Conclusions
As I stated earlier, these benchmarks are not the kind you'd use OOP for (or even a function for code reuse.) However, these tests reveal that the slowest-executing programming method is: poorly-used OOP, which unfortunately is common-place nowadays. Excessive function calls also tend to add a good amount of overhead, and make for a good example of poor procedural programming. Performance really depends more on how the code is written as opposed to the paradigm in which it is written.
I'll admit, I despise 'traditional' OOP. It's messy, over-complicated, and completely unnecessary (I could rant all day.) I am, and always will be, a purely procedural programmer, and a pedantic bastard. One can have 'object-oriented' code, outside the scope of traditional OO, and traditional OO languages; in a more procedural fashion. That said, I have not biased these tests in any way.
The Neither column is the one that should be used to compare the four languages really, and going based upon that, one can conclude that writing these tests in any of the four doesn't require much more or less code than any of the others. From an execution-time perspective, Perl is the fastest overall. Perl also used the least memory overall.
Because of the number of iterations done within each test program (1 million), for simple scripts you're not going to see much of a difference in execution time between the four. However, for intensive tasks, you may see a very large difference, with Perl most-likely being faster than the other three; though as I previously stated, it all depends on how you write the code.
Eventually, I will write a few good benchmarks to show the substantial differences between OOP and procedural programming.
Comments
Display comments as
(Linear | Threaded)
Не могу поверить что ПХП настолько сильно отстает от того же перла.
using the php executable directly? you can use php with a lightweight webserver and reduce 500% the memory needed...
Thanks for your comment. However, my tests were with the CLI version of PHP and not with the Apache2 module, or any other web server. Thus, choice of web server would not have been a factor in my results.
There is a reason why PHP co-developer Zend developed a cache for PHP, being followed by others, like eAccelerator, MMcache. Many hosters also offer these cache already installed in the webserver.
Alas, it would make (more) sense to benchmark the languages based on any of these tools.
Alas, it would make (more) sense to benchmark the languages based on any of these tools.
Thank you for your comment.
These benchmarks were all performed with the CLI versions of the language interpreters. Web servers and bytecode caches weren't part of the equation.
These benchmarks were all performed with the CLI versions of the language interpreters. Web servers and bytecode caches weren't part of the equation.
This benchmark is useless. You can't do any comparison of a stupid hello world app!
You're basically measuring the loading of different interpreters, if anything with such a short script.
You'll need bigger test scripts than these if you plan to do any real benchmarks, this is not it.
You're basically measuring the loading of different interpreters, if anything with such a short script.
You'll need bigger test scripts than these if you plan to do any real benchmarks, this is not it.
ed,
Thank you for the comment. Please re-read the second paragraph of this entry.
Tim
Thank you for the comment. Please re-read the second paragraph of this entry.
Tim
LOL
U guys needed more than 30 lines for a hello world in PHP?
U guys needed more than 30 lines for a hello world in PHP?
This is a nice comparison which should be considered before selecting any web technology in web projects..
and thanks for investing your valuable time for this creative stff..
and thanks for investing your valuable time for this creative stff..
I found the bar graph a great tool to explain to my client the trade-offs of choosing various technologies to build their initial web infrastructure. My client could care less if it's a framework or a language - they want to know what they'll be stuck with when our team is gone. Yes there's opinions and caveats in the details but from a high level perspective it's great stuff.
Hi,
I was just looking for such benchmark to my thesis. I've downloaded your benchmark and run it myself and I have to disagree with your results.
First of all when I looked at inc-oo.perl it's identical to inc-p.perl. Maybe it's just a mistake in download version, because oo version was also included in Inc.pm. Anyway to get my results I've changed it to:
#!/usr/bin/perl -w
use Inc;
use strict;
my $inc;
$inc = Inc->new();
$inc->do_it();
Another problem was with PHP.
PHP by default loads many extensions and that's why it uses a lot of memory and it's run time was so long. When you turn them off results are severely different. As I recall other languages doesn't load by default extensions such MySQL support so chances should be equal. My results in incremental test were:
PHP:
oo 0.9726
f 1.1368
p 0.3347
Perl:
oo 1.6991
f 0.9703
p 0.4741
So in my opinion PHP looks quite promising. Especially when you take a look at PHP-CLI version.
I suggest you revise this benchmark or maybe I'm completely wrong.
Looking forward for your opinion.
I was just looking for such benchmark to my thesis. I've downloaded your benchmark and run it myself and I have to disagree with your results.
First of all when I looked at inc-oo.perl it's identical to inc-p.perl. Maybe it's just a mistake in download version, because oo version was also included in Inc.pm. Anyway to get my results I've changed it to:
#!/usr/bin/perl -w
use Inc;
use strict;
my $inc;
$inc = Inc->new();
$inc->do_it();
Another problem was with PHP.
PHP by default loads many extensions and that's why it uses a lot of memory and it's run time was so long. When you turn them off results are severely different. As I recall other languages doesn't load by default extensions such MySQL support so chances should be equal. My results in incremental test were:
PHP:
oo 0.9726
f 1.1368
p 0.3347
Perl:
oo 1.6991
f 0.9703
p 0.4741
So in my opinion PHP looks quite promising. Especially when you take a look at PHP-CLI version.
I suggest you revise this benchmark or maybe I'm completely wrong.
Looking forward for your opinion.
Thanks for the comment. I'm glad you found my benchmark useful.
You're correct. inc-oo.pl and inc-p.pl were the same in the archive. Thanks for pointing that out. I've corrected the file.
PHP does load a lot of extensions. However, I wanted to test with a typical configuration, so I used the configuration that comes out of the box. The key was that the configurations of the other languages were just as typical.
The variance between the numbers (per language) that you posted seem to be about what I'd expect.
The main point of the benchmark is not to compare PHP to Perl per se. The point is to show how different programming styles affect the length of source code, and execution speed in each of the respective languages. Comparing the languages to each other simply gives a brief look at the differences between each language for each programming style.
You're correct. inc-oo.pl and inc-p.pl were the same in the archive. Thanks for pointing that out.
I've corrected the file. PHP does load a lot of extensions. However, I wanted to test with a typical configuration, so I used the configuration that comes out of the box. The key was that the configurations of the other languages were just as typical.
The variance between the numbers (per language) that you posted seem to be about what I'd expect.
The main point of the benchmark is not to compare PHP to Perl per se. The point is to show how different programming styles affect the length of source code, and execution speed in each of the respective languages. Comparing the languages to each other simply gives a brief look at the differences between each language for each programming style.
In my work I needed strict performance comparison so my priorities were different
Anyway this extensions problem didn't give me a rest so I tried testing different configurations.
In the end I've found something quite comforting. It's not the default extensions that caused this lose of performance. but xDebug. It's used only in production time and is never installed on final servers - it's only for debugging and lower efficiency in this case is very understandable.
My results on my hardware:
PHP
oo: 0,9741; func: 1,1092; neither: 0,3359
PHP + default extensions:
oo: 1,0146; func: 1,1553; neither: 0,3695
PHP + default extensions and xDebug
oo: 5,6429; func: 5,8177; neither 0,7888
Try turning this extensions off
Anyway this extensions problem didn't give me a rest so I tried testing different configurations.
In the end I've found something quite comforting. It's not the default extensions that caused this lose of performance. but xDebug. It's used only in production time and is never installed on final servers - it's only for debugging and lower efficiency in this case is very understandable.
My results on my hardware:
PHP
oo: 0,9741; func: 1,1092; neither: 0,3359
PHP + default extensions:
oo: 1,0146; func: 1,1553; neither: 0,3695
PHP + default extensions and xDebug
oo: 5,6429; func: 5,8177; neither 0,7888
Try turning this extensions off
To all of you who are looking at these benchmarks as a way to pick the language for your web framework: beware that this is testing the *command line* execution time for these languages. Of the four of these, PHP was designed specifically to serve web pages, and its CLI support is rather pitiful and (obviously per the graphs) quite inefficient.
This means that you're comparing apples to beer if you expect CLI benchmarks to tell you how well a language will perform when serving up web pages (especially when you take things like optcode caching into consideration).
This means that you're comparing apples to beer if you expect CLI benchmarks to tell you how well a language will perform when serving up web pages (especially when you take things like optcode caching into consideration).
PHP, Ruby, Perl, Python - can anyone tell me a difference?
All are interpreted languages, All have extension, All have everything web developer need, All good for web development.
Measuring HelloWorl wouldn't give you any answer. "Default configurations" for every language differs by every distro.
I DOESN'T matter what language to use but DOES MATER how you're using it.
Plain C can be times slower than PHP, Java or .NET just because of dumb memory allocation logic made by dumb deve... sorry coder...
So. Treat these languages as fruits or vegetables. You can make any kind of salad from them an it will taste perfect.
More fruits you've tasted - more experience in fruits you have and you'll be able to compile exotic fruit salads on any "party" (project).
Just use your brain. All the time. And don't stop asking questions and seek answers!
All are interpreted languages, All have extension, All have everything web developer need, All good for web development.
Measuring HelloWorl wouldn't give you any answer. "Default configurations" for every language differs by every distro.
I DOESN'T matter what language to use but DOES MATER how you're using it.
Plain C can be times slower than PHP, Java or .NET just because of dumb memory allocation logic made by dumb deve... sorry coder...
So. Treat these languages as fruits or vegetables. You can make any kind of salad from them an it will taste perfect.
More fruits you've tasted - more experience in fruits you have and you'll be able to compile exotic fruit salads on any "party" (project).
Just use your brain. All the time. And don't stop asking questions and seek answers!
You're right on the money that it's poorly-wielded OO code that causes the misery. There are so few situations that call for object orientation in the first place (if you know how to write procedurally or functionally) that the inevitability of misuse looms as large as all the Java hype.
"It's like C++, but without pointers!"
Sigh.
Fun benchmark. I have no idea why people get so upset about comparing apples to beer. One gets the benefits of consuming both.
"It's like C++, but without pointers!"
Sigh.
Fun benchmark. I have no idea why people get so upset about comparing apples to beer. One gets the benefits of consuming both.
What I think would be interesting is to generate 100,000 random numbers or more and then use those numbers for each language and each paradigm possible to perform a quicksort. I tried Googling around to see if anyone has done that and posted nice looking graphs for it, but I didn't find any. (I would like to use it for a language presentation that I have tomorrow; not enough time to do it myself )
)
can you run again your benchmarks agains newest versions of the interpreters? python3, ruby1.9, php 5.3, etc.
i definitely agree that the best is to perform a test with a more common settings. no use in benchmarking php by removing all the loaded libraries/modules.
it would also be more interesting to see an up to date benchmark after the corrections and maybe a bit more intensive code.
it would also be more interesting to see an up to date benchmark after the corrections and maybe a bit more intensive code.
PHP is really no match to Python. I have been PHP developer for 3 years and switched to python because i feel Python is more like a hybrid language. Because of python flexibility features i can easily scale.
Calendar
|
July '10 |
| Mon | Tue | Wed | Thu | Fri | Sat | Sun |
| 1 | 2 | 3 | 4 | |||
| 5 | 6 | 7 | 8 | 9 | 10 | 11 |
| 12 | 13 | 14 | 15 | 16 | 17 | 18 |
| 19 | 20 | 21 | 22 | 23 | 24 | 25 |
| 26 | 27 | 28 | 29 | 30 | 31 |
Contact Me
| E-Mail: | tim\x40hentenaar.(com|net|org|info|biz|us) |
| AIM: | Matriark TerVel |
| ICQ: | 109613501 |
| Yahoo: | tim_hentenaar |
| Skype: | tim.hentenaar |
Comparativa del rendimiento de cada lenguaje cuando es utilizado en distintos paradigmas: POO, estructurada y en bruto. Proporciona las fuentes para poder comprobarlo en nuestro equipo.
Tracked: May 07, 22:21
This post was mentioned on Twitter by pradeepkrsharma: @kinshuksunil it's #Perl not PERL... but sir it's a very old technology and much stronger than PHP.. check this out http://bit.ly/458Phx ...
Tracked: Nov 12, 16:03