bzr, git, and hg performance on the Linux tree
OK, so I just did a historical comparison of git and bzr performance using the Linux source tree. One of the comments I got was “what about Mercurial?” Fair enough. I’ve really never done much with Mercurial because Ubuntu primarily uses bzr and git is what most of the other people I know using a DVCS use. However, there are a lot of projects using Mercurial, Mozilla being probably the most notable one. So, here’s a comparison of bzr and hg. You may want to read my previous post for details on the steps I’m doing.
Repo Initialization:
git bzr hg
0m0.086s 0m0.334s 0m0.137s
1 : 3.88 : 1.59
Add 2.6.0 Linux tree:
git bzr hg
0m14.269s 0m4.852s 0m2.526s
5.65 : 1.92 : 1
Commit 2.6.0 Linux tree:
git bzr hg
0m10.263s 0m43.968s 0m30.890s
1 : 4.28 : 3.01
Diff after copying in 2.6.25.2 Linux tree:
git bzr hg
0m24.425s 0m51.158s 0m37.846s
1 : 2.09 : 1.55
Committing large changes:
git bzr hg
0m28.468s 1m8.627s 0m47.948s
1 : 2.41 : 1.68
Diff after no changes:
git bzr hg
0m0.343s 0m47.448s 0m1.340s
1 : 138 : 3.91
Getting repo status after no changes:
git bzr hg
0m1.230s 0m4.027s 0m1.077s
1.14 : 3.74 : 1
Committing a trivial change:
git bzr hg
0m0.397s 0m9.010s 0m1.913s
1 : 22.7 : 4.82
Repository size (just VCS control directory):
git (gc) bzr (pack) hg
92 MB 112 MB 179 MB
So, Mercurial performs quite well. It generally sits somewhere between git and bzr. Hg runs somewhere around 2.75 times slower than git in the tested operations. Bzr runs around 5 times slower with the notable exception that bzr diff when there are no changes is 138 times slower than git and 35 times slower than Hg.
Thanks for the benchmarks.
Just curious… did you prime (or flush) the disk cache before running tests to make sure that all tests started with the same setup?
Also, were time measurements ran on a packed/gc’ed repository?
Lastly, can you quote the versions you used?
Cheers,
-Bart
bartman
2008/05/09 at 5:40 pm
No, I didn’t flush the disk cache. I’m sure there’s lots more that can be done to get really good benchmarks, but this isn’t my area of expertise as I’m just a curious VCS user
I did the packing and gc’ing after all the timing. I figure it’s highly unlikely that people are going to repack/gc after each operation.
The git/bzr/hg versions are the ones currently found in Ubuntu 8.04:
git (1.5.4.3), hg (0.9.5), and bzr (1.3.1)
laserjock
2008/05/09 at 5:59 pm
Do you remember what order git/bzr/hg were tested in, and was the ordering the same for the different tests?
Since the cache wasn’t flushed, the first test will be at a disadvantage compared to subsequent tests which might have lots of filesystem data already in memory. Thanks for posting these!
dap
2008/05/09 at 6:39 pm
Thanks for also benchmarking Mercurial! The results are about what I expected.
Robin
2008/05/10 at 7:59 am
Good work. Thanks for the comparison. Git is the best as I expected.
Bill Gates
2008/05/10 at 3:14 pm
Good benchmark, and , as you can see, mercurial performs quite well.
Using the benchmark, and doing some additional comparison about the features of those VCS, we could conclude that:
* Hg performs better than bzr in the tests
* Git doesn’t have a clean plugin support
* bzr and hg has have a nice plugin support.
So, with mercurial, you have an excellent performance and a good plugin support!
gerardo
2008/05/11 at 7:14 pm
Now, how many people work with the Linux kernel tree…
… not a lot, so it’s the bells and whistles that matter!
Vadim P.
2008/05/13 at 7:22 pm
These type of benchmarks don’t really say anything. They show how the tools behave in importing two specific versions of a single project, which is not a typical workflow for developers.
It would be more interesting to see benchmarks on repositories with thousands of commits. How does repository size increase for each commit? Do the tools get any slower when you have a lot of commits? How about diffing versus the last revision, or 100 revisions ago? Those are operations that can be slow in some tools, and would be worthwile to test
Pieter
2008/05/20 at 3:10 pm
I started a blog describing my adventures into benchmarking. It might be interesting to follow if you’re looking into this type of benchmarks.
Pieter
2008/05/27 at 1:18 pm
Somehow i missed the point. Probably lost in translation
Anyway … nice blog to visit.
cheers, Dazed.
Dazed
2008/06/18 at 9:58 pm
[...] did another comparison with bzr, git and hg, which showed that hg is actually fairly close to git in terms of [...]
In Traction » Blog Archive » Bzr vs git, the sequel
2008/06/26 at 1:18 pm
It is not mentioned how time was measured, is it user+sys result, or wallclock (real) time; see for example historical comparison of old versions of git and Bazaar-NG (or is it now GNU Bazaar?), where for Bazaar (bzr) real time and user+sys time differed *wildly*.
Also the details of hardware, operating system and filesystem used are not given.
BTW. I have added benchmark results from those two articles on Git Wiki: http://git.or.cz/gitwiki/GitBenchmarks#head-ac6a277715384041898e90485848b14b48f6c142
Jakub Narebski
2008/06/29 at 4:49 am
Do you think you could add monotone to this? Necessary commands follow.
Repo Initialization:
mtn db init -d ../linux.mtn
mtn setup . -d ../linux.mtn -b org.kernel.linux.2.6.0
Add 2.6.0 Linux tree:
mtn add . -R –no-respect-ignore
Commit 2.6.0 Linux tree:
mtn commit -m “Commit 2.6.0 Linux tree”
Diff after copying in 2.6.25.2 Linux tree:
mtn diff
Committing large changes:
(add new files, drop missing ones, this will treat any renames as drop+adds)
mtn add –unknown –no-respect-ignore
mtn drop –missing
mtn commit -m “Committing large changes”
Diff after no changes:
mtn diff
Getting repo status after no changes:
mtn status
Committing a trivial change:
mtn commit -m “trivial change”
Repository size (just VCS control directory):
du -sh ../linux.mtn
Matt
2008/11/21 at 2:58 pm
This monotone thing looks very interesting. I like that it’s not dependent on anything like ssh and that the sharing protocol is transportable over any network layer. This was really well thought out. Hope it catches on.
mrcaron
2009/11/19 at 8:43 am