bzr, git, and hg performance on the Linux tree
May 9, 2008 — JordanOK, so I just did a historical comparison of git and bzr performance using the Linux source tree. One of the comments I got was “what about Mercurial?” Fair enough. I’ve really never done much with Mercurial because Ubuntu primarily uses bzr and git is what most of the other people I know using a DVCS use. However, there are a lot of projects using Mercurial, Mozilla being probably the most notable one. So, here’s a comparison of bzr and hg. You may want to read my previous post for details on the steps I’m doing.
Repo Initialization:
git bzr hg
0m0.086s 0m0.334s 0m0.137s
1 : 3.88 : 1.59
Add 2.6.0 Linux tree:
git bzr hg
0m14.269s 0m4.852s 0m2.526s
5.65 : 1.92 : 1
Commit 2.6.0 Linux tree:
git bzr hg
0m10.263s 0m43.968s 0m30.890s
1 : 4.28 : 3.01
Diff after copying in 2.6.25.2 Linux tree:
git bzr hg
0m24.425s 0m51.158s 0m37.846s
1 : 2.09 : 1.55
Committing large changes:
git bzr hg
0m28.468s 1m8.627s 0m47.948s
1 : 2.41 : 1.68
Diff after no changes:
git bzr hg
0m0.343s 0m47.448s 0m1.340s
1 : 138 : 3.91
Getting repo status after no changes:
git bzr hg
0m1.230s 0m4.027s 0m1.077s
1.14 : 3.74 : 1
Committing a trivial change:
git bzr hg
0m0.397s 0m9.010s 0m1.913s
1 : 22.7 : 4.82
Repository size (just VCS control directory):
git (gc) bzr (pack) hg
92 MB 112 MB 179 MB
So, Mercurial performs quite well. It generally sits somewhere between git and bzr. Hg runs somewhere around 2.75 times slower than git in the tested operations. Bzr runs around 5 times slower with the notable exception that bzr diff when there are no changes is 138 times slower than git and 35 times slower than Hg.
May 9, 2008 at 5:40 pm
Thanks for the benchmarks.
Just curious… did you prime (or flush) the disk cache before running tests to make sure that all tests started with the same setup?
Also, were time measurements ran on a packed/gc’ed repository?
Lastly, can you quote the versions you used?
Cheers,
-Bart
May 9, 2008 at 5:59 pm
No, I didn’t flush the disk cache. I’m sure there’s lots more that can be done to get really good benchmarks, but this isn’t my area of expertise as I’m just a curious VCS user
I did the packing and gc’ing after all the timing. I figure it’s highly unlikely that people are going to repack/gc after each operation.
The git/bzr/hg versions are the ones currently found in Ubuntu 8.04:
git (1.5.4.3), hg (0.9.5), and bzr (1.3.1)
May 9, 2008 at 6:39 pm
Do you remember what order git/bzr/hg were tested in, and was the ordering the same for the different tests?
Since the cache wasn’t flushed, the first test will be at a disadvantage compared to subsequent tests which might have lots of filesystem data already in memory. Thanks for posting these!
May 10, 2008 at 7:59 am
Thanks for also benchmarking Mercurial! The results are about what I expected.
May 10, 2008 at 3:14 pm
Good work. Thanks for the comparison. Git is the best as I expected.
May 11, 2008 at 7:14 pm
Good benchmark, and , as you can see, mercurial performs quite well.
Using the benchmark, and doing some additional comparison about the features of those VCS, we could conclude that:
* Hg performs better than bzr in the tests
* Git doesn’t have a clean plugin support
* bzr and hg has have a nice plugin support.
So, with mercurial, you have an excellent performance and a good plugin support!
May 13, 2008 at 7:22 pm
Now, how many people work with the Linux kernel tree…
… not a lot, so it’s the bells and whistles that matter!
May 20, 2008 at 3:10 pm
These type of benchmarks don’t really say anything. They show how the tools behave in importing two specific versions of a single project, which is not a typical workflow for developers.
It would be more interesting to see benchmarks on repositories with thousands of commits. How does repository size increase for each commit? Do the tools get any slower when you have a lot of commits? How about diffing versus the last revision, or 100 revisions ago? Those are operations that can be slow in some tools, and would be worthwile to test
May 27, 2008 at 1:18 pm
I started a blog describing my adventures into benchmarking. It might be interesting to follow if you’re looking into this type of benchmarks.
June 18, 2008 at 9:58 pm
Somehow i missed the point. Probably lost in translation
Anyway … nice blog to visit.
cheers, Dazed.
June 26, 2008 at 1:18 pm
[...] did another comparison with bzr, git and hg, which showed that hg is actually fairly close to git in terms of [...]
June 29, 2008 at 4:49 am
It is not mentioned how time was measured, is it user+sys result, or wallclock (real) time; see for example historical comparison of old versions of git and Bazaar-NG (or is it now GNU Bazaar?), where for Bazaar (bzr) real time and user+sys time differed *wildly*.
Also the details of hardware, operating system and filesystem used are not given.
BTW. I have added benchmark results from those two articles on Git Wiki: http://git.or.cz/gitwiki/GitBenchmarks#head-ac6a277715384041898e90485848b14b48f6c142