bzr, git, and hg performance on the Linux tree

OK, so I just did a historical comparison of git and bzr performance using the Linux source tree. One of the comments I got was “what about Mercurial?” Fair enough. I’ve really never done much with Mercurial because Ubuntu primarily uses bzr and git is what most of the other people I know using a DVCS use. However, there are a lot of projects using Mercurial, Mozilla being probably the most notable one. So, here’s a comparison of bzr and hg. You may want to read my previous post for details on the steps I’m doing.

Repo Initialization:
git                bzr                hg
0m0.086s     0m0.334s     0m0.137s
1        :         3.88      :      1.59

Add 2.6.0 Linux tree:
git                bzr                hg
0m14.269s   0m4.852s      0m2.526s
5.65      :      1.92      :       1

Commit 2.6.0 Linux tree:
git                bzr                 hg
0m10.263s   0m43.968s    0m30.890s
1         :        4.28       :       3.01

Diff after copying in 2.6.25.2 Linux tree:
git                bzr                hg
0m24.425s   0m51.158s    0m37.846s
1        :         2.09      :      1.55

Committing large changes:
git                bzr                hg
0m28.468s   1m8.627s     0m47.948s
1        :         2.41      :        1.68

Diff after no changes:
git                bzr                hg
0m0.343s     0m47.448s    0m1.340s
1         :        138       :       3.91

Getting repo status after no changes:
git                bzr                hg
0m1.230s     0m4.027s     0m1.077s
1.14       :      3.74      :     1

Committing a trivial change:
git                bzr                hg
0m0.397s    0m9.010s      0m1.913s
1        :        22.7       :       4.82

Repository size (just VCS control directory):
git (gc)        bzr (pack)      hg
92 MB         112 MB          179 MB

So, Mercurial performs quite well. It generally sits somewhere between git and bzr. Hg runs somewhere around 2.75 times slower than git in the tested operations. Bzr runs around 5 times slower with the notable exception that bzr diff when there are no changes is 138 times slower than git and 35 times slower than Hg.

About these ads

15 thoughts on “bzr, git, and hg performance on the Linux tree

  1. Thanks for the benchmarks.

    Just curious… did you prime (or flush) the disk cache before running tests to make sure that all tests started with the same setup?

    Also, were time measurements ran on a packed/gc’ed repository?

    Lastly, can you quote the versions you used?

    Cheers,
    -Bart

  2. No, I didn’t flush the disk cache. I’m sure there’s lots more that can be done to get really good benchmarks, but this isn’t my area of expertise as I’m just a curious VCS user :-)

    I did the packing and gc’ing after all the timing. I figure it’s highly unlikely that people are going to repack/gc after each operation.

    The git/bzr/hg versions are the ones currently found in Ubuntu 8.04:
    git (1.5.4.3), hg (0.9.5), and bzr (1.3.1)

  3. Do you remember what order git/bzr/hg were tested in, and was the ordering the same for the different tests?

    Since the cache wasn’t flushed, the first test will be at a disadvantage compared to subsequent tests which might have lots of filesystem data already in memory. Thanks for posting these!

  4. Good benchmark, and , as you can see, mercurial performs quite well.
    Using the benchmark, and doing some additional comparison about the features of those VCS, we could conclude that:

    * Hg performs better than bzr in the tests
    * Git doesn’t have a clean plugin support
    * bzr and hg has have a nice plugin support.

    So, with mercurial, you have an excellent performance and a good plugin support! :D

  5. Now, how many people work with the Linux kernel tree…

    … not a lot, so it’s the bells and whistles that matter! :)

  6. These type of benchmarks don’t really say anything. They show how the tools behave in importing two specific versions of a single project, which is not a typical workflow for developers.

    It would be more interesting to see benchmarks on repositories with thousands of commits. How does repository size increase for each commit? Do the tools get any slower when you have a lot of commits? How about diffing versus the last revision, or 100 revisions ago? Those are operations that can be slow in some tools, and would be worthwile to test

  7. I started a blog describing my adventures into benchmarking. It might be interesting to follow if you’re looking into this type of benchmarks.

  8. Pingback: In Traction » Blog Archive » Bzr vs git, the sequel

  9. It is not mentioned how time was measured, is it user+sys result, or wallclock (real) time; see for example historical comparison of old versions of git and Bazaar-NG (or is it now GNU Bazaar?), where for Bazaar (bzr) real time and user+sys time differed *wildly*.

    Also the details of hardware, operating system and filesystem used are not given.

    BTW. I have added benchmark results from those two articles on Git Wiki: http://git.or.cz/gitwiki/GitBenchmarks#head-ac6a277715384041898e90485848b14b48f6c142

  10. Do you think you could add monotone to this? Necessary commands follow.

    Repo Initialization:
    mtn db init -d ../linux.mtn
    mtn setup . -d ../linux.mtn -b org.kernel.linux.2.6.0

    Add 2.6.0 Linux tree:
    mtn add . -R –no-respect-ignore

    Commit 2.6.0 Linux tree:
    mtn commit -m “Commit 2.6.0 Linux tree”

    Diff after copying in 2.6.25.2 Linux tree:
    mtn diff

    Committing large changes:
    (add new files, drop missing ones, this will treat any renames as drop+adds)
    mtn add –unknown –no-respect-ignore
    mtn drop –missing

    mtn commit -m “Committing large changes”

    Diff after no changes:
    mtn diff

    Getting repo status after no changes:
    mtn status

    Committing a trivial change:
    mtn commit -m “trivial change”

    Repository size (just VCS control directory):
    du -sh ../linux.mtn

    • This monotone thing looks very interesting. I like that it’s not dependent on anything like ssh and that the sharing protocol is transportable over any network layer. This was really well thought out. Hope it catches on.

  11. Several notes:

    You are using quite old Versions:
    ———————————

    git:
    You use 1.5.4.3 released 23-Feb-2008
    1.7.5.4 is out

    hg:
    You use 0.9.5 released 07-Sep-2007
    1.8.4 is available
    bzr
    You use 1.3.1 released 28-Apr-2008
    2.3.3 is available

    Using 3-4 old version for 6 year old project is not very serious.

    How do you do make your git commit
    ————————————

    git have specific mechanism were you prepare your commit before actually registering it. what command are you exactlyy using ?

    can you provide the source ?
    ————————————

    This will allow people to try it themself.

    Mercurial and bazaar use python
    ————————————
    They have an incompressible penalty at startup. simple comparison for fast operation should think about it.

    DVCS doesn’t do the same thing during each operation
    ——————————————————-

    It’s interesting to see how implementation impact various operation with the same name.

    Git needs 14.2s to add the linux tree then 10.2s to commit it. it’s a total of 24.5s. This does not include optimizing repo space (repack)

    Mercurial needs 2.5s add the linux tree then 30.9s to commit it. it’s a total of 32.4s This include optimizing repo space.

    It look like git add prepare a significant part of the next commit during the add operation where mercurial do less during “add” but more during commit. Moreover git commit doesn’t include repack.

    This highlight the difficulties of benchmarking small command independently.

    Repository size:
    ——————-

    You can use shrink-revlog.py to optimise the size of a mercurial repository.

    Disk Cache
    ——————-

    you can’t just ignore it. otherwise some tool bench will suffer from loading the cache for the next tool bench

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s