How do I …

use svn-buildpackage with the following repository (Debichem if you want to know):

/experimental/foo/debian/
…
/unstable/foo/debian/
/unstable/bar/debian/
…
/tags/foo/0.1-1
…

This sort of layout doesn’t seem to be standard and *-buildpackage are confusing me. Are *-buildpackage tools really worth it? Is it easier to just export the debian/ directory into an unpacked upstream tarball?

Also, as sort of a dare, does anybody know if hg,git,bzr-buildpackage will work off of such a SVN repo?

going against the grain

Well, of course I have to go against the grain and say -1 to Robert’s cry for eliminating self-produced bug mail. I like getting the bug mail from Launchpad when I file/comment on/work on a bug. It lets me keep track of what I’ve been doing.

On a somewhat related note, after talking with other developers I appear to be in the minority who primarily use Launchpad’s web interface when doing bug work. How about you? Do you primarily use the web or email interface? or perhaps a python-launchpad-bugs script like bughelper, etc.?

bzr, git, and hg performance on the Linux tree

OK, so I just did a historical comparison of git and bzr performance using the Linux source tree. One of the comments I got was “what about Mercurial?” Fair enough. I’ve really never done much with Mercurial because Ubuntu primarily uses bzr and git is what most of the other people I know using a DVCS use. However, there are a lot of projects using Mercurial, Mozilla being probably the most notable one. So, here’s a comparison of bzr and hg. You may want to read my previous post for details on the steps I’m doing.

Repo Initialization:
git                bzr                hg
0m0.086s     0m0.334s     0m0.137s
1        :         3.88      :      1.59

Add 2.6.0 Linux tree:
git                bzr                hg
0m14.269s   0m4.852s      0m2.526s
5.65      :      1.92      :       1

Commit 2.6.0 Linux tree:
git                bzr                 hg
0m10.263s   0m43.968s    0m30.890s
1         :        4.28       :       3.01

Diff after copying in 2.6.25.2 Linux tree:
git                bzr                hg
0m24.425s   0m51.158s    0m37.846s
1        :         2.09      :      1.55

Committing large changes:
git                bzr                hg
0m28.468s   1m8.627s     0m47.948s
1        :         2.41      :        1.68

Diff after no changes:
git                bzr                hg
0m0.343s     0m47.448s    0m1.340s
1         :        138       :       3.91

Getting repo status after no changes:
git                bzr                hg
0m1.230s     0m4.027s     0m1.077s
1.14       :      3.74      :     1

Committing a trivial change:
git                bzr                hg
0m0.397s    0m9.010s      0m1.913s
1        :        22.7       :       4.82

Repository size (just VCS control directory):
git (gc)        bzr (pack)      hg
92 MB         112 MB          179 MB

So, Mercurial performs quite well. It generally sits somewhere between git and bzr. Hg runs somewhere around 2.75 times slower than git in the tested operations. Bzr runs around 5 times slower with the notable exception that bzr diff when there are no changes is 138 times slower than git and 35 times slower than Hg.

git/bzr historical performance comparison

OK, I know git vs. bzr has been beat to death and that bzr speed seems to be often cited as its “Achilles’ heel“, but I was in #bzr the other day and somebody (a git fan I take it) said something to the effect of “well, bzr couldn’t be used to work with the linux kernel tree, that’s what git was made for”. Now, I have no experience of working on the linux tree, but it got me to thinking about if anybody had done any benchmarking of that kind of operation.

After some googling I found an old blog post from 2006 by Jo Vermeulen where he did some basic timing of common tasks such as adding files, doing diffs, commits, and finding repo status on the Linux 2.6 kernel tree using both git and bzr. Since both git and bzr have come a long ways since 2006  I thought I’d replicate Jo’s comparison (with git 0.99.9c and bzr 0.7pre) using current (by Ubuntu 8.04 standards anyway) versions of git (1.5.4.3)  and bzr (1.3.1). So, here’s the results:

First we unpack a Linux 2.6.0 tarball into linux-bzr and linux.git directories, then initialize the repos:

Initialization:
git (old)       bzr (old)       git (new)       bzr (new)
0m0.161s    0m1.593s     0m0.086s     0m0.334s

Nothing exciting so far. Now we tell the VCSs to track the files via bzr/git add :

Adding files:
git (old)       bzr (old)       git (new)       bzr (new)
0m42.121s  0m31.870s   0m14.269s    0m4.852s

In this case bzr not only wins in terms of absolute speed, but also in proportional gains with time. The git:bzr ratio in 2006 was 1.32:1 and now it’s 2.93:1 . Jo didn’t mention in his comparison how long it took him to then commit the initial 2.6.0 tree we added but for me it was 0m10.263s for git and 0m43.968s for bzr, a pretty clear win for git.

Next we’ll untar the latest 2.6.x kernel into our repos. Jo used linux-2.6.15.4 and I used linux-2.6.25.2. Perhaps I should have used the same version he did but considering we’re using entirely different hardware I don’t think our results are directly comparable anyway. OK, so now we want to see how long it takes to diff the changes:

Diffing changes:
git (old)       bzr (old)       git (new)       bzr (new)
2m26.982s  1m13.869s   0m24.425s    0m51.158s

This is one of the more fascinating results in my little experiment. The 2006 results gave a git:bzr ratio of 1.99 whereas my new results give a ratio of 0.48 . Apparently git has done a lot of work on speeding up diffing.

Next we commit our new 2.6.x changes:

Committing large changes:
git (old)       bzr (old)       git (new)       bzr (new)
0m54.964s  2m4.757s     0m28.468s    1m8.627s

so an old ratio of 0.44 and a new ratio of 0.41: not a lot going on there.

A really interesting test that Jo did was to do a bzr/git diff right after committing. Ideally this would take no time at all as we haven’t done anything since the commit, however:

Diffing no changes:
git (old)       bzr (old)       git (new)       bzr (new)
0m0.057s    3m51.918s   0m0.343s      0m47.448s

Back when Jo did his experiment the git:bzr ratio was 0.00025! Ouch. My results gave a ratio of 0.0072. In this case bzr has been gaining a lot of ground but it’s still rather remarkable how long it takes to diff when there are no changes.

The other things we would often do is a bzr/git status to see what’s going on:

Getting repo status:
git (old)       bzr (old)       git (new)       bzr (new)
0m0.442s    0m19.711s   0m1.230s      0m4.027s

The original git:bzr ratio was 0.022 and for the new one 0.305 so bzr has gained by an order of magnitude but still lags a bit.

Lastly, we look at what happens if you make a minor change (let’s just add our name to MAINTAINERS for fun) and then commit:

Small commit:
git (old)       bzr (old)       git (new)       bzr (new)
0m7.364s    2m6.685s     0m0.397s     0m9.010s

The times I got for both git and bzr are significantly faster than what Jo got in 2006. His git:bzr ratio was 0.058 and mine is 0.044, so some marginal gain by bzr here.

A last interesting note of comparison is the storage size that the VCS takes up. After all the operations above my .bzr directory is 112MB (or 23% of the total size of the repo+working tree) and the .git directory is 162MB (or 30% of the total size) so it seems that bzr has a bit better storage compression.

OK, so now the question is, what does it all mean? Well, I’m not entirely sure to be honest. When it comes to my original question of “Would bzr be usable working on the Linux tree” I would think, at least when it comes to common local operations, that the answer would  definitely be yes. It’s not the fastest thing around but it’ll get the job done.

I use both git and bzr on a regular basis and both are exciting and have their own strengths and weaknesses. Git is no doubt very fast, though I think other DVCSs are starting to catch up. Bzr is very user friendly and has great plugins. It’s really a cool time for code sharing, in my opinion. Rock on!

PackageKit Critique

If you haven’t heard, PackageKit is an exciting and upcoming project who’s goal is to create a user friendly package handling abstraction layer that is independent of distro and package format. Basically, the Grand Unified Package GUI for those more physics-minded. Sounds good like a good idea.

There are a few problems I have with the way PackageKit is heading, though, and hopefully this is a constructive critique that may help people think about these issues. I’m not a package management expert and I’ve only used PackageKit a bit on the soon-to-be-released Fedora 9 so please shout out if I get my facts wrong.

1. The PackageKit FAQ and people promoting PackageKit seem to often say something along the lines of “we’re not trying to replace <existing package management tool>“. I can’t help but think that, if it’s not designed to replace anything, is it really going to be all the unifying? If we’re gonna come up with a good distro-agnostic package management GUI it’d be nice if it actually replaced all the existing tools. Put another way, shouldn’t “unifying” package management imply replacing the disjointed bits that currently exist?
2. Another thing I’m having an issue with is the proposition that PackageKit is distro-neutral and is not trying to change existing “low-level” tools (apt, yum, etc.). However, discussion around the apt backend, Hughsie’s Law: “Authentication or license prompts can only be done before the transaction has started, and messages or notices about the transaction can only be shown after the transaction has completed.” and a similar FAQ statement “Upgrading, installing or removing packages has to be 100% silent.” indicate to me that perhaps it isn’t as neutral as it claims to be. Basically, PackageKit is neutral as long as your view of package management is the same as PackageKit author’s.
3. There are several places in the PackageKit FAQ and Debian wiki discussion where the PackageKit authors seem to indicate that simply erroring out of the transaction is a suitable, even if non-ideal, solution to problems that PackageKit can’t seem to cope with. I can’t think of anything more user-unfriendly than to have a package management tool bail because it can’t handle certain classes of packages (for instance packages that have a EULA, need input from users, or ask about config file changes). Users would assume, I would think, that the package is somehow broken and would either give up or report a bug to the distro.

I guess I would summarize by saying that it seems to me that PackageKit is painting itself into a corner where it is only useful to subsets of users and packages, while trying to still maintain that it is a unifying and user friendly tool. I see a lot of potential in PackageKit and appreciate all the time put in by Richard Hughes, but I’m not sure at this point how it’s going to reach it’s goals. Perhaps a more knowledgeable person can help enlighten me.

Quick look at Fedora

Lately I’ve been messing around with Fedora. I’ve been using and developing Ubuntu for over 2 years and I decided to take a little break and try something different. Right before the Gutsy release I had a brief look at openSUSE 10.3 betas. I think it’s very valuable for Ubuntu contributors to see what else is going on in the Linux/FLOSS landscape so that we can “think outside the box” and maybe better server our users.

So I’ve been running a dual-boot of Ubuntu and Fedora for a while. Fedora 8 was really pretty nice. Red Hat 7.2 was the first Linux distro I’d used and I tried a few of the early Fedora Core releases before landing in Ubuntu. I always found it a professional looking Gnome-based distro, though it often felt sluggish and experienced the dreaded RPM/yum hell. So, I was very pleasantly surprised to find in Fedora 8 a quick and responsive OS. Once the initial shock of using RPMs instead of debs wore off I found Fedora’s package management quite impressive.

One of my main complaints with openSUSE 10.3 was painfully slow and difficult to work with. In contrast, yum (and the yumex GUI) are very user friendly and responsive. Additionally, Fedora provides repo packages for both smart and apt that allow you to seamlessly use smart/apt/synaptic with Fedora repositories. Cool! Additionally Fedora 9 (which is currently a pre-release Preview) comes with PackageKit, which honestly seems like a less useful gnome-app-install. To be fair though, I haven’t played with it much so I might have missed some things. I can’t figure out how to install more than 1 package at a time which is a real deal-breaker to me. I’d love to be wrong about that though.

The Fedora artwork and polish are very impressive. I’ve always been impressed with Fedora’s artwork in general. It seems very consistent and professional. Ubuntu 8.04 has made some good steps in this direction. The Heron wallpaper is beautiful, in my opinion, and Ubuntu’s artwork seems to be getting a bit more consistent.

As far as hardware support goes both Fedora and Ubuntu work beautifully on my HP Pavilion dv6000. The only issue I’ve had is with my university’s wifi. The wifi network they’re currently phasing out uses a MS pptp VPN. Ubuntu has a nice network-manager-pptp package that works wonderfully. however that doesn’t seemed to have managed to end up in Fedora’s packages. I also got the new uni WPA wifi network to work fine in Ubuntu but it seems I’ve hit some sort of kernel/driver bug in iwl3945 in Fedora. I’ve gotta give huge props to Dan Williams who is the Fedora Network Manager guru. He spent several hours with me debugging my wifi problem. The Fedora/Red Hat developers I’ve interacted with have been very professional and knowledgeable so kudos to them. Hopefully we’ll figure out my iwl3945 issue soonish.

my very own LoCo and other news

I just found out I have my own Ubuntu LoCo, how sweet is that? For people who might not get the joke, I happen to fortuitously share the same name as a certain Middle Eastern country. I wonder if they’d make me an honorary member … Go Jordanians!

So I’ve been running Fedora in addition to Ubuntu on a daily basis for about a month now and today I decided since I’m already living “on the edge” running Hardy I should upgrade my Fedora 8 install to Fedora 9 “Sulphur”. After downloading 1.8 GB of new packages the upgrade went fairly well with the exception of selinux. The silly thing decided it didn’t like GDM and so I couldn’t even log in. I don’t know if this is a common issue (selinux freaking out) for Fedora release upgrades but it put a rather sour note on an otherwise smooth upgrade. At the moment I’m using the Fedora 8 kernel and all is good. Tomorrow I’ll try to get everything fixed up properly (any suggestions welcome). In the next few days I hope to work up a little Fedora 9/Ubuntu 8.04 comparison, from more of a developer perspective. As long as we keep things civil and positive I feel like distros can often learn quite a bit from each other.

On the Ubuntu front, I do believe I’m done for Hardy. Too much dissertation left, too little time. I finished off the edubuntu-docs translations and sponsored Matthew East’s final ubuntu-docs and gnome-user-docs packages. On my personal “Gosh I hope we get this fixed” list are bugs 215729 (Seahorse fails to import keys) and 185854 (Setting static IP in Network Settings doesn’t produce correct data). hint hint

me too, me too

Continuing on …

mantha@quark:$ history|awk ‘{a[$2]++ } END{for(i in a){print a[i] ” ” i}}’|sort -rn|head
259 ls
237 cd
53 exit
50 less
36 sudo
31 bzr
27 ssh
21 rm
21 nano
19 ping

So in the end it appears that I’m mostly wandering around my computer looking at stuff I also have a nasty habit of cd’ing to the directory I want to look at and then ls’ing in there, rather than just doing ls on the path. That “ping” is on the list reflects my uni’s crappy wifi/VPN situation the last two weeks. I can connect, but nobody’s home most of the time.

Long time .. no blog

It’s been forever since I blogged last. I’ve been deep into paper writing and trying to finish up my PhD. I have had a bit of time to do a little work (see the sweet 5-a-day sig below) and hopefully after Sunday I’ll be approved as a Debian Maintainer. I put in my application on Monday and my sponsor Michael Banck (azeem) was gracious enough to advocate for me. In Debian I’m currently the maintainer of plotdrop (gtk GUI for gnuplot) and just finished off the ITP for avogadro (really cool new chemical visualization app) with Michael’s help.

I’ve been long interested in the differences and similarities between Debian and Ubuntu. Many Ubuntu developers are current/former Debian developers and users but I think at this point probably a majority have “grown up” in Ubuntu like I have. Because Ubuntu is so heavily dependent on Debian, it behooves us developers to understand Debian processes, policies, and yes, even politics. I thought I’d just randomly share some thoughts on Debian from my Ubuntu-biased perspective:

• It’s all about who you know - you can have either an entirely discouraging or very uplifting experience with Debian depending on the people who you hang out with. Even though all projects are like this, I’ve found this to be most true in Debian. I’ve seen some horrible behavior by some Debian Developers but I’ve also made some really good friends and gotten a lot of quality help from others. I’ve had much much more success finding smaller special-interest groups to talk to (debichem, debian-science, and debian-mentors being good ones for me) than just reading debian-devel .
• When actively maintained, Debian packages are really really well taken care of. In general, Debian maintainers really know their software and are pretty quick to take care of things. On the flip side, I think it’s easier for Debian packages to “fall through the cracks” if a maintainer has no time. In Ubuntu we share the maintenance burden a lot which is I think is a boon for archive QA. But, I feel like a lot of times Ubuntu maintainers just don’t have an opportunity to really get to know the software they touch. We can be quite a bit quicker with things like library transitions and archive-wide changes, but also make silly mistakes because we don’t know the software well enough.
• The new Debian Maintainer thing is awesome. Allowing competent people to upload specific packages without having to go through the full process of becoming a Debian Developer will generate a significant amount of healthy activity for Debian, IMO. In Ubuntu, we do a lot of work to try to “create” new developers (MOTU Mentoring, MOTU School, Developer Week, the Packaging Guide, and some killer documentation for starters) but because of our small size (109 total developers including Canonical employees), team maintenance strategy, and fast release cycle an upstream developer or contributor wanting to focus on a single package or set of packages can easily get left out or turned off. I know Mark would like something similar to Debian Maintainers for Ubuntu and we pretty much just need the Launchpad bits to do it. I think it could be a positive move for Ubuntu.
• The entire FLOSS and Linux ecosphere owes a lot to Debian, it’s consistency, stability, philosophy, and even innovation. The Debian Free Software Guidelines were the basis of OSI’s Open Source Definition. I think it has spawned more major derivatives than any other distro. Debian also continues to be relevant and popular after almost 15 years in existence. It is really the only serious distro I know that’s not sponsored by a company. As I look around at how much Canonical, Red Hat, and Novell are putting into their respective projects it’s amazing to me how much Debian can get done.

Michael, Daniel Leidert and the rest of the Debichem team have really shown me a lot of the strength of Debian and how the two distros can work really well together and enhance each other. I think there are a lot more positive experiences like that than negative ones, which is just cool.

Anyway, I’m not sure if anybody cares about my babblings, but I just wanted to say, thanks Debian!

–
My 5 today: #117911 (qcad), #199368 (python-scipy), #196317 (gcompris), #103631 (auctex), #198439 (openbabel)
Do 5 a day - every day! https://wiki.ubuntu.com/5-A-Day

3rd Semiannual Golden Pony Awards

So, given that Gutsy was released a few months ago I figured it was again time for the Golden Ponies. This is a way for me to celebrate another release and reflect humorously (and sometimes with a healthy dose of satire) on the people and events involved with Ubuntu development.

Disclaimer: These awards are just a fun way for me to celebrate a new release. Obviously I can’t keep track of contributions of everyone in Ubuntu, so if you think you or someone you know should have gotten a Golden Pony and didn’t, I’m sorry. On the bright side, there’s always Hardy Heron.

So without further ado the Golden Ponies go to:

Free Software Advocacy - Scott Kitterman (ScottK) for his performance in Live Free or Die Hard

Best Mobster - Richard Johnson (nixternal) for his portrayal as a ChiTown thug in American *buntu Gangster

Advances in Multimedia Center on Linux - Mario Limonciello (superm1) and team for debunking the Windows Media Center monopoly in MythTV Busters

Distro Release Management - Jonathan Riddell (Riddell), Martin Pitt (pitti), Sarah Hobbs (Hobbsee), and Steve Langasek (slangasek) as the superhero action stars in Fantastic Four: Rise of the Gutsy Gibbon.

“Stickin’ in There” Award - Barry deFreese (bddebian) for his role as a MOTU oldtimer in No Distro for Old Men

Ubuntu Action Star - Luke Yelavich (TheMuso) for super kungfu, butt-kickin’ development action in TheMuso Ultimatum. Seriously, check out his team memberships.

Best in Comedy - Jorge Castro (jcastro/whiprush) and Daniel Holbach (dholbach) as a community wrangling duo in Hot Fuzz

Best Sea Captains - Brandon Holtsclaw (imbrandon) and William Grant (Fujitsu) for steering the ship in Pirates of the UbuntuWire

DE Animated Series - the entire Kubuntu team in KDE the Third … ummm … Fourth?

Best Community - the whole group of Ubuntu contributors (of course) who gave us the world-wide dance hit … License to Party: heck yeah we did it and we’ll do it again in six months baby!

To all you Golden Pony winners, here’s your award:

And Steven Kowalik … this is for you:

See ya’ll next time …