Some part of me will always be a QA guy, so it is nice to note that today is the tenth anniversary of my first formal bug filing (and first formal participation in Mozilla, I believe): mozilla bugzilla bug 8749, nested <DL> tags don’t display properly. Happy bugday to me, happy bugday to me… :)
QA
22
Apr 09
a rumbling about X QA
As I rebooted this morning as a result of RH bug 4733471 two serious questions popped into my head:
- do any of the major core X contributors2 employ a full-time X QA person? As far as I know the answer is ‘no’ but I’d love to be wrong.
- would a full-time X QA person funded fractionally by the major X contributors, reporting to the development managers for each of those contributors, but formally employed by freedesktop.org, make even more sense?
My sense is that this kind of position that may be hard for any one contributor to justify but that it is the kind of thing that is probably necessary for a complex piece of software to succeed, so a position with costs shared across the various contributors might make sense.
(This is only partially inspired by Owen’s recent call on behalf of Friends of GNOME and the sysadmin team, but I’ve always thought a full-time GNOME QA manager would make sense- it really is vastly more efficient for everyone involved if much of this sort of stuff is done upstream. And it just struck me today that probably the same is true for X.)
12
Feb 08
Bzzzzt.
giant by brom. License: 
The six month release cycle is not an all-controlling god, and bugs in one known, specific subsystem are not undebuggable without wide release (which was KDE’s most valid excuse). If it isn’t ready for wide use, it isn’t and shouldn’t be a GNOME .0. It isn’t ‘too late’ to decide that; you’re the QA team, dammit- it is your job to say no right up to the very last minute, and demand extra time to protect the users.
Distros follow our schedule because we promise high-quality .0s, so they should be thrilled we’ve admitted this won’t be high-quality, and they should either happily take a pass, or if they have a serious problem with it, they should provide resources to make it high-quality on time. (If they have a problem with it, their users should probably question doubt their commitment to shipping quality software.)
(cc’d to bugsquad shortly for real discussion.)
[on second thought: closing comments here because discussion on a topic of this sort should be on d-d-l or bugsquad, not on a blog.]
3
Jul 07
Infotopia, information-gathering, and software QA
A couple of weeks ago I finished reading Cass Sunstein’s Infotopia. While certainly not a perfect book by any stretch, it gives a stimulating overview of a central problem for any society- how it collects and filters information so that it can make decisions. Being a good U of C guy, he starts with Hayek’s notion that the price mechanism is an elaborate mechanism for ‘sharing and synchronizing local and personal knowledge‘ (to quote Wikipedia), and then goes on to discuss other mechanisms for getting information out of the heads which contain it- wikis, open source, democracy, polling, deliberation, prediction markets, etc. An interesting read to frame a lot of discussions around.
One of those discussions came up today. Quite simply, the big problem in QA is getting information about the state of the software out of the software and into the hands of developers as efficiently as possible.
This has three aspects: creating the information, getting it in the hands of the QA teams, and then filtering it into a form that is useful for developers to work on. Traditional QA has a very hard time getting the information- there are a lot of lines of code to be exercised, and very few people exercising the code (relatively speaking.) It is like squeezing water out of a stone, so they have to do a lot of things (like extensive automated testing) to get that information. The output is a relatively small amount of very regularized data, which is easy to present (though hard to weight efficiently and accurately.)
In contrast, open source QA has a whole ocean of information from the legions of volunteers willing to run pre-release code; the trick is to tap into that water without drowning in it. It isn’t regularized, but given a large enough body of users over time, you can be fairly certain that the bug reports will represent an accurate cross section of your problems, and the interaction with real users (instead of interaction with automated test tools or third-hand via the sales/customer relationship) can give you a fairly good idea of what bugs are actually important to real people.
If you’ve got one person to work on QA, I’d say you always want to swim in the ocean instead of doing any amount of automated squeezing information from the stone. This is not to say automated testing doesn’t have its place- in particular, good unit testing captures information at a very high-efficiency junction (when the original author is writing code) and then gives it back in a very compressed, efficient form that the developer should know immediately how to prioritize and deal with. Similarly, automated tests that attempt to capture regressions once a bug is fixed are also fairly efficient- they capture information which real humans in the field have identified as an important problem, and they again report simple, clear, efficient information- this bug # and commit # which were fixed are now not fixed. But generic ‘well, we’re going to write tests now because that is how we did it when we had no users willing to help us test’ testing is a very inefficient use of manpower- it is trying to dig a deep well to get information when you live next to a deep, clear, safe mountain lake.
So there you have it- proprietary QA is trying to squeeze information-water from a stone; open source QA is trying to learn how to swim in a sea of information. I know which problem I’d rather have.
18
Mar 07
productive testing tips
I was cited in an article on testing tips this past week. Here is some cut and paste from the email I sent to the author (Joe Barr) when he asked me for tips; my email goes into more detail than he was able to put in his article so I thought it might be worth posting.
- use a bugtracking system of some sort, and use as much metadata (including good titles) in that bugtracking system as you can. At first it is time-consuming, but over the long run, it’ll save you time by helping you avoid testing things or filing bugs twice. Relatedly, if we’re talking about things that can be done as a group, and not just as an individual, have a bugmaster- someone charged with organizing, sorting, and generally knowing what is going on with the bug tracker. That one person will help every other person who uses the bug tracker (both testers and developers) be more productive, which is invaluable.
- Where possible, use the very latest code. Don’t be afraid to rebuild things from CVS every night or every morning. The moreup-to-date your code is, the quicker you’ll catch things (again, helping everyone’s productivity) and the less time you’ll spend going back and forth with ‘is this in the latest version?’ Again, when speaking of a team, if someone can be charged with making this process as easy as possible, that one person has a productivity multiplier for the entire team- makes everyone more productive.
- If at all possible, write automated unit tests. The best way to be productive is to have the computer do the work for you. Again, this has up-front costs, but over the long run is a *huge* win. (If we’re talking about GUI software, try to focus on non-GUI code first when writing tests- GUI tests tend to be fragile, and you’re best off writing tests that will fail when things go badly wrong, not when a pixel shifted here or there.)
- dogfood, dogfood, dogfood. If at all possible, use the code you’re testing under real-life conditions to do everyday work. Real-life testing is always going to be more efficient than fake ‘do this, then do this, then do this’ checklist testing- that doesn’t catch edge cases, and it feels like work. If you’re using code for real work, you catch the edge cases that real users find, and you do it in the courseof doing something else- it doesn’t feel like work then. (Relatedly: if you dogfood and find a bug, make sure to do (1) file it immediately, and (3) write an automated test to duplicate the bug ASAP.)
- use automated crash reporting tools: every major OS now has ways to catch stack traces of catches and send them back to a bug database. Use those, and ship those- help the users help you.
- if you have an active volunteer community who are using nightly builds to do daily work, don’t spend your time testing things that they will inevitably test. For example, I’ve seen test plans that say things like ‘check to make sure it launches’. If it doesn’t run from the command line, a good community will let you know about it ASAP. If it crashes when you open the print dialog with 10,000 printers on your local network, well, most communities don’t run into that sort of thing- they have one printer. So spend your precious test cycles testing *that* kind of scenario, instead of testing basic stuff our community will catch like ‘does it start? does the file open dialog work?’
6
Mar 07
misc. post-weekend bits
- Diebold is considering selling their voting machine unit. RH, this is your chance.
- Actually used a Wii for the first time this weekend. What fun. This is real thinking outside the box. (I’ve never owned a game machine, and never previously been particularly tempted. But I am now, esp. since I didn’t buy myself anything for my birthday last week. :)
- Saw the first iPhone ad (unfree media, but at least unfree with an official player on Linux) on TV yesterday. Hope that Nokia will take the hint from Apple and put a cell chip in the N800′s successor- this is a market that Nokia should have owned a year ago, had they been willing to think a bit more outside their own boxes. I’m still glad they are doing all they are doing, and it is a unique and interesting little box, but it is frustrating to see that all the pieces are in place- except one- for the iPhone to have been a year earlier, and Free.
- Hopefully I’ll score an openmoko at the end of the month so I can replace the one proprietary component in my personal digital stack, even if it does look to be clearly suboptimal to an N800 with a cell chip. Openmoko folks, if you see this, you really, really need some free QA experts testing and helping you refine this thing :)
- Three bits of awesome news for GNOME distribution and testing- rpath-based images, a buildbot with integrated testing, and wider distribution of unstable builds.
- I miss Miami.
- I’m glad I’m not the only one who thinks that wordpress upgrades mar an otherwise excellent experience. Comments indicate fixes may be in the pipeline, yay.
- Interesting post on property rights and how they explode over at the Volokh Conspiracy. Look forward to the rest of the posts in the series, though I wish the citations provided links, or at least enough information to be googleable. (I’d link to SSRN, but SSRN is considered harmful.)
- Speaking of citations, am about to launch into a pile of bluebooking. Expect fun, law-school/data-organization related ranting to follow!
24
Aug 06
More on QA, Ubuntu, trust, etc.
Heard from Ubuntu this morning (in comments and via email, neither official) so I figured I owed an update, having slammed them fairly thoroughly here :) So some notes from email and comments:
- I didn’t see an official Ubuntu announce about this because I didn’t look in the most obvious place of all- ubuntu.com. Looking there points you to this message, with more details in a subsequent linked page. Good on Ubuntu for discussing the issue in the most highly visible place they can, and promising (albeit all the way at the bottom of the second page) that they are investigating the problem. Given that one of the most valuable things any distro has (especially Ubuntu) is the trust of its users, I would probably have given the ‘we are researching the problem’ statement much more prominence, but it is there, at least.
- To be very clear: I don’t expect Ubuntu to have researched the cause of the procedural problem and fixed it in 48 hours. That would be nice but unreasonable. I just expect them to very publicly say what they are doing about the problem, in terms of research, etc.
- To also be clear: I’m surprised I’ve seen nothing on planet ubuntu (not planet gnome), because I assume that at least some developers blog about what they are thinking about/working on, and if no developer blogs about this Very Big Fuckup, then… that ain’t good :)
- The negatives: apparently the problem was there for 17 hours. Not a good sign, but again, that is partially because I have high standards for Ubuntu.
- Apparently the reason I didn’t know about dapper-proposed is that it isn’t fully deployed yet. That is mixed news, I guess- good that there is a reason I didn’t know about it; bad that something like dapper-proposed was not fully tested and in place before the LTS release. (Note here that again I’m holding Ubuntu to a very high standard; as far as I know no other distro has such a queue for their long-term distros yet either. Of course, every distro should. If I’m wrong, and other distros do have it, I’d love to know- please let me know in comments.)
- James: I’ve not considered an LWN article on distro QA because for quite some time (really since around when I left Novell) I’ve been pondering writing the definitive serious white paper on the subject. As dobey is about to find out, writing anything of that length is hard :) We’ll see if these blog posts coalesce my thinking enough to get something LWN-length out, though.
- error27, others who have discussed enterprise distros: Enterprise distros have substantial resources directed at identifying stable upstream versions, and stabilizing them even more. So of course we should expect that at this point enterprise distros are very stable; more so than their more bleeding-edge community counterparts. However, traditional enterprise distros can only be resourced from within the company that produces them, and their users are explicitly paying not to worry about it- the payment is mostly in lieu of other forms of contribution. In contrast, a community distro like Fedora or Ubuntu should be virtually unlimited in terms of the amount of testing, feedback, triage, etc., that it receives from community members. Given that, if coordination and communication problems are solved, community distros should be of at least equal quality to enterprise distros. (It should be of no surprise, given that coordination/communication problems are perhaps the biggest stumbling block to this, that I think everyone needs a bugmaster.)
- Go read the comments in last night’s post for more comments on the Edgy/Unstable differences. They are all dead on; no need for me to repeat them, except to say that obviously there are a lot of various layers to the disparity. Still, the basic question stands: how do you get more people onto unstable, and get them contributing?
I swear I’ll write something about law school soon :)
23
Aug 06
Notes about distros, QA, etc.
Yesterday I flamed Ubuntu, I think with cause, for breaking X. Followups:
Points out of the comments in that thread yesterday:
- Fedora now believes that they are going to be able to support (apparently already have supported) distro->distro upgrades, like Debian has done for years and Ubuntu has done since day 1. This is very big for Fedora. Along with a growing selection of packages in Extras, two of Debian/Ubuntu’s biggest selling points to the technical community are under siege. Yay for competition :)
- One of the Fedora dudes clarified my understanding of the FC5/X7.1 situation- it was not nearly as broken as I’d thought. That said, part of what Fedora should be trying to do is build a culture around QA and quality- which means clear messaging about these sorts of things. So I’m glad that they were doing the right things for roughly the right reasons, but they need to get better about communicating those so that their culture grows up with them.
- rpath points out that the obvious solution to problems like the one I had yesterday is to be able to rollback packages, and that conary (rpath’s system management tool) can do that. I continue to think that rpath is doing really interesting stuff; this would be one good demo of why. (Yes, I know red carpet and other tools have done rollback for a while, but conary’s implementation, from what I can grok of it, is nice and well-integrated.)
- Lucas did a really interesting (and totally unscientific) ‘survey’ of Ubuntu and Debian users by way of CTCP in IRC, and discovered that something like 4% of Ubuntu users were using edgy, while 76% of Debian users were using unstable. My hunch is that this says more about Ubuntu’s incredible success in getting newbies into #ubuntu than it does anything else, but the core question (‘what percentage of our users are using and testing our development branch? what steps are we taking to raise that percentage?’) is a really interesting one which every free software project should ask itself. (NB that GNOME is failing here, and has been since Ximian stopped funding packaging of unstable builds years ago. Ubuntu’s unstable builds have been a huge pickup in that respect.)
- Shockingly, no real response from Ubuntu that I can see anywhere (planet, bugsquad list, forums), other than the fast fix. Remember that much of this is about expectations- I expect a lot from Ubuntu, so when they fuck up, (1) I get very very pissed, because I trusted them and (2) I expect openness about why it happened this time and how they are going to prevent it from happening next time, so that I can again trust them.
Some things I personally should have explained better:
- I am not switching distros. All things considered, at this time Ubuntu still offers the best mix of maintainability and support for my needs, especially now that I know not to actually trust their support packages. Silly and naive of me to have done so earlier, though, and led to (frankly) great anger.
- QA for this sort of thing (big, big bug in package everyone uses) is not really very hard to do. Ubuntu has been leading the way for quite some time in open source distro QA implementation, pushing packages early and quickly in their unstable branch so that their stable release is both well tested and fairly up-to-date- exactly what every other distro should be doing, and some do to various degrees. But Ubuntu have not quite pushed hard on the last mile for stable- getting users to test proposed updates before they ship. I only discovered yesterday that they have a ‘proposed updates’ channel for apt. Quite simply, if I (who am completely obsessive about open source QA) don’t know about your proposed updates channel, you haven’t pimped it enough. Every distro should have a proposed updates channel, like Ubuntu does, and pimp it heavily to their skilled users, which AFAICT Ubuntu does not. Skilled users who are not running unstable should, in response, consider it nearly a moral obligation to use the proposed updates channel on any non-mission-critical boxes. That combo, used effectively, should have caught this before it went out. If Ubuntu is post-morteming this (which they should be) that would be the big question to ask- why did the community not catch this for us?
- It is worth remembering that every significant community linux distro has a community of thousands who will gladly test anything you throw at them, so distros must actively encourage and take advantage of that. Any distro which doesn’t (and many don’t) is throwing away free time and free money. (Relatedly, I firmly believe that as a result of the opportunity for free QA, most open distros should in practice be more stable than their ‘enterprise’ alternatives, which have smaller user bases who would rather pay for someone else to do the work. That in practice enterprise distros tend to be more stable points to inefficiencies in how open distro QA is done, IMHO, not just the obvious points about business models.)
22
Aug 06
!@#@!#@!- still learning what ‘long term support’ means
Things that are not good:
- put all your class notes in something X-based
- see an X update from Ubuntu before you go to class
- decide not to install the X update, because, hey, you wouldn’t want a broken X right before class
- read some email, have breakfast
- remember you’re running not just any distro, but hey, the ‘Long Term Support’ distro- the one that presumably has, you know, a QA process. And no one would put out a package that breaks X in their Long Term Support, enterprise-ready distro, right?
- install the upgrade
- turn off the computer
- go to class
- turn on the computer
- discover that you have no X, and class started two minutes ago.
Furious would not begin to describe how I felt. The internal dialog in my head was ‘!@!@#@!#. What actually well-supported distro can I switch to?’ because lets be clear- I’m running a stable distro for the first time in ages specifically to avoid shit like this. If the ‘stable’ distro still breaks my fucking X, it isn’t stable. Period. End of discussion. So I need another distro.
To ubuntu’s credit, there was an update in apt within a few minutes of when I got to class, so I was able to fix it by apt-get’ing again. But if your QA process for the Long Term Support distro let through an X update that broke X, well, your QA process still needs some work. (I understand that given the vast diversity of hardware X runs on, it isn’t possible to do perfect QA, but if it breaks a lot of machines, which it did, something went deeply wrong in your process.)
Side note: Abi’s XML is pretty noisy when you’re using outline mode. Turns out emacs + abi xml was not quite the savior I would have hoped it would be in my initial, paniced moments.