After reading Mackenzie’s post about 5-a-day on the Ubuntu UK podcast I have to say that:
Dave: “What I’m getting at is, the Ubuntu community is vast, why in the last 7 days is there only 35 people working on it?”
is exactly what I don’t like about 5-a-day.
An issue with any metric is that you need to make very sure that you’re actually measuring/reporting what people think you are. 5-a-day stats are exactly that, stats on the 5-a-day participants, not Ubuntu as a whole. A problem with these types of community metrics is that they’re most often based on voluntary participation and the stats actually become more about who participated in the statistical method than the actual “thing” you were trying to measure.
Now, I think a problem with 5-a-day is not that it’s a bad initiative. I think it’s done a lot of great things around the community and I want to see it continue. However, I do feel like there are some issues with it that I think people need to think about some:
- 5-a-day promotes quantity rather than quality. The problem is clearly that quality is very difficult to measure, but the fact remainst that you can create a lot of bug “churn” and not get much of any actual useful work done. We can create a highly participatory community that contributes pretty much nothing to its users, other distros, or the FLOSS landscape as a whole.
- 5-a-day is heavily promoted as the way to track bug triaging work in Ubuntu. While it certainly measures some, I believe 5-a-day only measure a minority of the actual work being done in Ubuntu
- The 5-a-day stats page gives no indication of what it’s actually measuring and could be easily misconstrued, IMO
- 5-a-day not only doesn’t track all contributors, it doesn’t necessarily track all contributions by a by a particular contributor. It relies on a person actively putting bug numbers in their 5-a-day applet/CLI tool.
Lastly, I think a lot of this comes down to 5-a-day being designed to be a fun “scoreboard” for the community to have a little competition. The problem is when people see it rather as serious business, and especially in these days where Ubuntu is being challenged about its contributions I think we need to be perhaps more careful with how these types of metrics are seen by “outsiders”.
LaserJock 
James:
Well, wouldn’t say I’m “pissed off” exactly, but I do think 5-a-day is only capturing a small amount of the total work so it’s value as a metric is quite low. However, as a fun community thing and to promote triaging in general, it’s pretty successful. I don’t want to kill it, I just think we should take a critical look at it and see what could/should be changed.
Daniel:
Launchpad karma at least is automatic and captures every participant that works in the bug tracker. Your doing metrics within the system itself.
I also don’t think it’s demonstrating that bug triaging is not daunting. If I were an inexperienced person looking at the 5-a-day stats page I’d conclude that a few “rock stars” (~10) do almost all the work and hence it’s probably not for me.
Daniel:
Yes, I am focusing on the scoreboard. 5-a-day is a stat tool, nothing more, as far as I can see. It’s not a communication tool, it’s not a collaboration tool, it’s not an educational tool, it just reports the bugs you’ve told it you worked on. How does 5-a-day help anybody ease into triaging? If we want to do MOTU School type days for triaging, fine. I think we should focus on educating and leading people towards making significant contributions to Ubuntu, and I don’t particularly see how 5-a-day is doing that. It’s a great little internal thing for us to have fun with, but I don’t see much use for it outside of that.
Daniel, point taken. The fun aspect is a good thing, for sure, but I’d rather people have fun while doing something “useful”. From what I’ve seen, most of these “get people involved in bug triaging” have pretty much been a failure when you consider the amount of effort put into them.
5-a-day would be incredibly more useful if you could click on a person in the stats page and get a list of the bugs they worked on. As it is, it’s just a scoreboard.
Anytime you introduce metrics you’re opening a can of worms because there are always going to be people who take them with more seriousness than is warranted by the metric.
Unless you’re pissed off because some people are getting notice for their contribution and others are not — which is fixable via published metrics and fanfare for those un-noticed contributions, not by killing of the 5-a-day.
I’d go further by saying that 5-A-Day and Launchpad karma are alike in terms of “metrics”: each is useful if its respective participants deem so. Overburdened core developers and maintainers alike tend to not use 5-A-Day, because they simply have few cycles remaining to play the metrics game. Moreover, metrics can be misconstrued, as you duly point out; however, the real value in 5-A-Day is demonstrating that the vast majority of bug triaging rout is not daunting.
The real kicker for 5-A-Day becoming a beast of its own is when its functionality is incorporated into Leonov, and a possible extension is that you get many eyes on the more important focus of a distribution, which is streamlining testing procedures to give the distribution’s users a real joy to use.
Granted, if we don’t have a good testing framework for packages incorporated into the development, then we’re not going far. I’m not just referring to puiparts. We need unit tests and more for legitimate bugs.
Jordan, I think you’re focusing a bit unfairly – or perhaps fairly, I don’t know, as it’s too “early” to tell – on the 5-A-Day scoreboard. I don’t feel that the aim of 5-A-Day is to promote metrics but to ease people into bug triaging. In isolation, sure, if a person peeks at the scoreboard, s/he may be turned off from bug triaging. However, if the “rock stars” are actively mentoring newcomers, and if these leaders have strong ties to Ubuntu development, then we’re in a better position as users of the distribution to “win”. Unfortunately, what often is lost in the noise is that 5-A-Day is useful as a tool, but its scoreboard does not gauge the health of the distribution. And that nuance is subtle, so thanks for raising the points that you have. The 5-A-Day introduction isn’t about winning the scoreboard; it’s about teaching, learning, and transferring knowledge. In essence, it’s akin to MOTU mentoring just from the bug triaging angle.
But to really tackle the issue at hand- how do we improve the distribution- we need not just metrics but a testing framework, which means that for every legitimate bug we close, we can demonstrate that the triager(s) learned something(s).
Jordan, the aggregation of small things, even internal things, can be useful. Look for a screencast on bug triaging soon.
The fun aspect can never be emphasised enough.
Software development is critically process. If we say, “here’s a bug”, “here’s how I fixed it”, “here’s a test to demonstrate the efficacy of the fix”, “here’s my list of bugs”, new K/Ed/X/Ubuntu members/contributing developers/etc. have something straightforward to give to their approving councils.
It’s fun (for those of us who dig stats) and it encourages daily bug activity. That’s all.
And I haven’t looked at the stats recently, but I don’t recall there being many “rock stars”. By and large simply doing 5 per day every day would catapult anybody into the weekly and monthly top 5.
Vadim,
5-a-day is not a method for triaging, it’s a method of collecting stats on triaging activity. People will continue to triage whether 5-a-day exists or not. Most of the bugs on 5-a-day are done by people who were triaging before 5-a-day existed. Besides, I’m not saying that 5-a-day should be gotten rid of at all, I’m trying to point out some things that I think could/should be addressed.
I didn’t see you list a better method for trianging 40+% of ubuntu bugs.
just complain only. how useful!
I think 5-a-day is misconstrued by some people.
5-a-day is not a metrics that measures anything important. 5-a-day is a motivational tool.
The idea is to work steadily on at least 5 bugs every day. It is not called As-many-as-you-can-a-day. Therefore, the big numbers come in the teams, when more people participate. Hence, the idea is again motivation.
What motivation do I talk about?
I am talking about the motivation to work on 5 bugs every day… ok.. I have not been very good in that due to moving household… however, I think every day, did I do my 5 bugs already, like I ask myself, did I east my 5 portions of fruit.
And I am trying to motivate as many people (especially people that are not participating and contributing yet) to join my team. Therefore 5-a-day also introduce a nice scalability. It is not about doing 100-a-day, it is about getting 100 friends to join.
Given that, the number do not mean anything, they are just the cake on top. A little game among ladies and gentlemen, like a nice round of golf in the evening. It is not about winning, it is not about stats, it is about participating and having some social interaction and community in doing this.
Therefore, I hope as many people as possible will continue to join. I would like to see not 35, but 100 or 1000 people every day. It is not one ant that builds the mount, but the whole colony!
This are my 2¢.
Wolfger’s right about if you actually do 5 each day you’ll jump up pretty high. That’d be 35/wk or 150/mo, and at that point, based on today’s stats, you’d be number 7 in both lists.
Yes, there needs to be more automation between LP and 5-a-day. It’d be nice if it auto-reported.
Jordan: I was trying, in my post, to address the belief that triaging is very hard to do. That’s why I linked to so many of the wiki pages and why I listed so many tasks that need to be done on bugs, like cleaning up bug titles. OK, reading a backtrace is hard, but that’s not the only thing there is to do. The bug reports in Portuguese, Russian, etc. are a bit of a problem for all of the monoglot triagers, so translators are welcome. I also wanted to publicize the Hug Day process and point out that triaging can be social (ex: Bug Jams). You’ll often find Daniel and I denting or going on IRC and naming a location and inviting others to join us for triaging. He did that one day, naming a coffee shop a few blocks from my apartment, so I met up with him. Two more kids from our LoCo team showed up as well.
The competition aspect does help though I think. Sometimes I’ll look at the board and realize “oh the person above me has only done two more bugs than I have. I’ll just find 3 more bugs to work on before I go to bed so I can jump them.” Well, that’s 3 bugs I wouldn’t have cleaned up, requested more info on, assigned a package to, or even tried to fix.
By the way, on that 35 people thing, there are currently 152 active members of the 5-A-Day team which says “If you join this team, you agree to do 5 bugs a day.”
I have to agree that 5-a-day promotes quantity over quality. It’s why I don’t participate and why I said I prefer a quality over quantity approach. Closing bugs and reducing bug count metrics is not a priority to me. Fixing them is. Unfortunately, the culture around 5-a-day is to increase the number of round trips and invalidate reports.
It’s much easier to ask a guy for information and mark a bug incomplete, knowing that 4 out of 5 times nothing will ever come back. But at some point this work needs to set the stage for better software and fewer bugs. How does a bug marked incomplete help?
We are closing this blog entry because it lacks the information we need to investigate the problem, as described in the previous comments. Please blog again if you can give us the missing information, and don’t hesitate to submit blog entries in the future. Thanks again!
@jldugger
You don’t know which of the 5 bugs you set incomplete will get a response with very good information. Not asking for more information deprives you of the opportunity to get it.
Furthermore, there is no automatic requirement to set a bug that does not get a response to invalid. The missing information can still be researched, the bug tried to be reproduced etc.
5-a-day has IMHO nothing to do with how bugs a triaged or closed or fixed. LP Karma can have the same effect. 5-a-day is as I said earlier merely a motivational tool to work every day a bit on bug triage and animate others to do the same. If you do not need this motivation or don’t want to participate, that is your prerogative. However, it is also not the right conclusion that 5-a-day leads necessarily to lower quality of work.