OK, so I just did a historical comparison of git and bzr performance using the Linux source tree. One of the comments I got was “what about Mercurial?” Fair enough. I’ve really never done much with Mercurial because Ubuntu primarily uses bzr and git is what most of the other people I know using a DVCS use. However, there are a lot of projects using Mercurial, Mozilla being probably the most notable one. So, here’s a comparison of bzr and hg. You may want to read my previous post for details on the steps I’m doing.
Repo Initialization:
git bzr hg
0m0.086s 0m0.334s 0m0.137s
1 : 3.88 : 1.59
Add 2.6.0 Linux tree:
git bzr hg
0m14.269s 0m4.852s 0m2.526s
5.65 : 1.92 : 1
Commit 2.6.0 Linux tree:
git bzr hg
0m10.263s 0m43.968s 0m30.890s
1 : 4.28 : 3.01
Diff after copying in 2.6.25.2 Linux tree:
git bzr hg
0m24.425s 0m51.158s 0m37.846s
1 : 2.09 : 1.55
Committing large changes:
git bzr hg
0m28.468s 1m8.627s 0m47.948s
1 : 2.41 : 1.68
Diff after no changes:
git bzr hg
0m0.343s 0m47.448s 0m1.340s
1 : 138 : 3.91
Getting repo status after no changes:
git bzr hg
0m1.230s 0m4.027s 0m1.077s
1.14 : 3.74 : 1
Committing a trivial change:
git bzr hg
0m0.397s 0m9.010s 0m1.913s
1 : 22.7 : 4.82
Repository size (just VCS control directory):
git (gc) bzr (pack) hg
92 MB 112 MB 179 MB
So, Mercurial performs quite well. It generally sits somewhere between git and bzr. Hg runs somewhere around 2.75 times slower than git in the tested operations. Bzr runs around 5 times slower with the notable exception that bzr diff when there are no changes is 138 times slower than git and 35 times slower than Hg.
LaserJock 
Thanks for the benchmarks.
Just curious… did you prime (or flush) the disk cache before running tests to make sure that all tests started with the same setup?
Also, were time measurements ran on a packed/gc’ed repository?
Lastly, can you quote the versions you used?
Cheers,
-Bart
No, I didn’t flush the disk cache. I’m sure there’s lots more that can be done to get really good benchmarks, but this isn’t my area of expertise as I’m just a curious VCS user 🙂
I did the packing and gc’ing after all the timing. I figure it’s highly unlikely that people are going to repack/gc after each operation.
The git/bzr/hg versions are the ones currently found in Ubuntu 8.04:
git (1.5.4.3), hg (0.9.5), and bzr (1.3.1)
Do you remember what order git/bzr/hg were tested in, and was the ordering the same for the different tests?
Since the cache wasn’t flushed, the first test will be at a disadvantage compared to subsequent tests which might have lots of filesystem data already in memory. Thanks for posting these!
Thanks for also benchmarking Mercurial! The results are about what I expected.
Good work. Thanks for the comparison. Git is the best as I expected.
Good benchmark, and , as you can see, mercurial performs quite well.
Using the benchmark, and doing some additional comparison about the features of those VCS, we could conclude that:
* Hg performs better than bzr in the tests
* Git doesn’t have a clean plugin support
* bzr and hg has have a nice plugin support.
So, with mercurial, you have an excellent performance and a good plugin support! 😀
Now, how many people work with the Linux kernel tree…
… not a lot, so it’s the bells and whistles that matter! 🙂
These type of benchmarks don’t really say anything. They show how the tools behave in importing two specific versions of a single project, which is not a typical workflow for developers.
It would be more interesting to see benchmarks on repositories with thousands of commits. How does repository size increase for each commit? Do the tools get any slower when you have a lot of commits? How about diffing versus the last revision, or 100 revisions ago? Those are operations that can be slow in some tools, and would be worthwile to test
I started a blog describing my adventures into benchmarking. It might be interesting to follow if you’re looking into this type of benchmarks.
Somehow i missed the point. Probably lost in translation 🙂 Anyway … nice blog to visit.
cheers, Dazed.
Pingback: In Traction » Blog Archive » Bzr vs git, the sequel
It is not mentioned how time was measured, is it user+sys result, or wallclock (real) time; see for example historical comparison of old versions of git and Bazaar-NG (or is it now GNU Bazaar?), where for Bazaar (bzr) real time and user+sys time differed *wildly*.
Also the details of hardware, operating system and filesystem used are not given.
BTW. I have added benchmark results from those two articles on Git Wiki: http://git.or.cz/gitwiki/GitBenchmarks#head-ac6a277715384041898e90485848b14b48f6c142
Do you think you could add monotone to this? Necessary commands follow.
Repo Initialization:
mtn db init -d ../linux.mtn
mtn setup . -d ../linux.mtn -b org.kernel.linux.2.6.0
Add 2.6.0 Linux tree:
mtn add . -R –no-respect-ignore
Commit 2.6.0 Linux tree:
mtn commit -m “Commit 2.6.0 Linux tree”
Diff after copying in 2.6.25.2 Linux tree:
mtn diff
Committing large changes:
(add new files, drop missing ones, this will treat any renames as drop+adds)
mtn add –unknown –no-respect-ignore
mtn drop –missing
mtn commit -m “Committing large changes”
Diff after no changes:
mtn diff
Getting repo status after no changes:
mtn status
Committing a trivial change:
mtn commit -m “trivial change”
Repository size (just VCS control directory):
du -sh ../linux.mtn
This monotone thing looks very interesting. I like that it’s not dependent on anything like ssh and that the sharing protocol is transportable over any network layer. This was really well thought out. Hope it catches on.
Several notes:
You are using quite old Versions:
———————————
git:
You use 1.5.4.3 released 23-Feb-2008
1.7.5.4 is out
hg:
You use 0.9.5 released 07-Sep-2007
1.8.4 is available
bzr
You use 1.3.1 released 28-Apr-2008
2.3.3 is available
Using 3-4 old version for 6 year old project is not very serious.
How do you do make your git commit
————————————
git have specific mechanism were you prepare your commit before actually registering it. what command are you exactlyy using ?
can you provide the source ?
————————————
This will allow people to try it themself.
Mercurial and bazaar use python
————————————
They have an incompressible penalty at startup. simple comparison for fast operation should think about it.
DVCS doesn’t do the same thing during each operation
——————————————————-
It’s interesting to see how implementation impact various operation with the same name.
Git needs 14.2s to add the linux tree then 10.2s to commit it. it’s a total of 24.5s. This does not include optimizing repo space (repack)
Mercurial needs 2.5s add the linux tree then 30.9s to commit it. it’s a total of 32.4s This include optimizing repo space.
It look like git add prepare a significant part of the next commit during the add operation where mercurial do less during “add” but more during commit. Moreover git commit doesn’t include repack.
This highlight the difficulties of benchmarking small command independently.
Repository size:
——————-
You can use shrink-revlog.py to optimise the size of a mercurial repository.
Disk Cache
——————-
you can’t just ignore it. otherwise some tool bench will suffer from loading the cache for the next tool bench