Intro ===== A couple years ago a friend in a summer job decided that his summer firm needed a bugmaster, the catch being that he didn't quite know what one was. So he asked me if I'd written anything on the subject. I hadn't, and that ball was dropped. A couple months ago someone at Novell asked 'well, how do you deal with all these incoming bugs? Doesn't it drive your engineers insane?' The answer, again, was 'bugmaster.' But again, it was a little hard to describe what a bugmaster is, or why exactly it makes sense for large projects to have them. So... hopefully this little drop will help explain why a bugmaster makes sense for many projects, and what a good (and great) bugmaster looks like. [This is a preliminary version of the text, call it 0.1. AKA 'not really ready for prime time.' But I'd love it if people familiar with my work for gnome and/or who are just good editors read it and commented... I hope eventually to push it to the bugzilla community for more discussion, but I'd like to not completely embarass myself first :) Also, I might split it up into several docs, 2.2K words is too much as it is, and it doesn't even feel done yet.] what is a bugmaster, anyway? ============================ Bugmastering in a nutshell: The bugmaster voraciously reads as many bugs as possible, and attempts to help the organization deal with those bugs by constantly improving the quality of the bug information hackers/developers, leaders/managers, and QA/volunteers have. Most frequently, improving bug information means triage- the process of assigning bugs severity and priority, to help developers allocate their bug-fixing time, and cleaning the system of bugs which are in some way not useful- duplicate bugs, incomplete bugs, and bugs which have been fixed but not marked as such in a bug tracker. But it also can take a number of other forms. Among these would be mediating between developers and bug reporters in disagreements; working with duplicates; disambiguating bugs filed by non-skilled volunteers; helping other volunteers work with bugs and the bug system; and helping hackers (or managers) better understand the state of their own project. All of these roles are important for a large software project to be really effective, and they are all too easy for projects (both open and proprietary) to overlook or do poorly, despite best intents. why everyone needs a bugmaster ============================== While open and proprietary projects have a lot of differences, they share enough similarities that some of the motivations for having a bugmaster are shared between them. These include: * By taking the rough edges off of bug tracking and doing lots of dirty work, bugmasters let hackers do what they want to do most- focus on creating great software. For both proprietary and open projects, anything that isn't coding or designing is overhead that detracts from either hacking more or having a life. A great bugmaster delivers an organized, consistent, friendly bug tracking system to hackers, and lets hackers hack, and that is guaranteed to help morale. All this without the detrimental effects of shielding developers from feedback, which is often the failure mode developers and managers fall back into when a bug system is difficult or unpleasant to use. * In my experience, it is rare in any type of large project that people have a true sense for the quality of the whole product (outside of bug counts); a more holistic approach to QA done by someone who has dealt with massive numbers of bugs on the lowest level can help planning and release strategy quite a bit, by providing better and more qualitative information than the graphs and counts that are the raw tools of most QA management. Saying 'bug counts are doing down by XYZ a week' is great, but supplementing it with the understanding of someone who has read hundreds or thousands of bugs can help catch problems early. * Good bugmastering understands a project's priorities and works constantly with hackers/developers to achieve those, reinforcing project direction in a constant, developer-friendly, low-level manner during their participation in every bug. In open projects, even in the case of projects that have an iron-fisted BDFL [http://en.wikipedia.org/wiki/Benevolent_Dictator_for_Life], that BDFL is unlikely to be able to oversee all aspects of the project. Good bugmasters help open project leaders (BFDL or otherwise) move the project in the direction they want it to go, by prioritizing bugs in line with the BFDL's vision and helping clarify that vision when hackers or bug reporters aren't clear on it. Similarly, in closed projects, while product and project managers define a project's direction, they can't read every bug and are often poor at communicating this vision to hackers. A good bugmaster bridges that gap, using the bug process as an opportunity to move the project in the specified direction, one bug at a time. This bridging of developers and goals also works in reverse- a good bugmaster is likely to be among the first to see that a particular portion of a project's goals are unworkable, nonsensical, or just unlikely to be finished in a desired time period, and can help communicate that to leaders who can figure out where to go from there. why open source projects (in particular) need a bugmaster ========================================================= Of course, bug tracking when you have paid QA in a proprietary situation and bug tracking when you have volunteers in an open project are two different beasts. Open source projects need bugmasters most obviously and most badly; besides the reasons listed above, you have: * Massive numbers of external testers can quickly make a bugzilla useless. Massive open beta is one of free software's biggest strengths, but it is useless for large projects without organization- hackers just can't keep up. Too many projects have thrown up their hands and said 'I can't keep up without more hackers' when the real solution is not to throw more hackers at bug system maintenance, but to throw an expert at bug system maintenance and let the same number of hackers hack. This is less of a problem in proprietary projects, where the ratio of QA to developers is likely to be small, instead of the (effectively) hundreds to one in successful free software projects. * A good bugmaster can help create effective testing and QA communities- strong, directed leadership is always helpful in creating communities, and a good bugmaster can help create this. In the best cases, open source communities do happen almost by magic, but having someone who feels responsible for helping create such a community can help a lot, especially in QA, which is not typically a 'glamour' job in open source and hence isn't as likely to create community as hacking or art. why proprietary projects (in particular) need a bugmaster ========================================================= Bugmastering is not just for open source projects, I believe, though I have less personal experience with this. Traditional software QA has some failings that can be remedied (all 'frequent', none 'always') by applying another layer of review and communication on top of traditional QA<->engineer communication. * Proprietary QA frequently has blinders on. In many cases, QA is assigned to a small number of subsystems and works tightly on small, specific parts of functionality with specific developers. This division of labor is of course necessary in a large organization, but it isn't sufficient for a healthy QA organization- it makes patterns harder to spot, makes it more difficult to have a qualitative big-picture view of product quality, and it encourages cozy relations between hackers and QA which can fail to push the bug gathering process hard enough. [FIXME: these last two points are phrased really weakly, and there are better reasons I'm totally drawing a blank on tonight... need to ponder more on motivation and structure here.] * Traditional software QA tends to focus on the automatable, since they have the time and tools to do this, tend to dislike manual testing. This is again understandable and reasonable, and in most cases quite useful. But obviously it has shortcomings, and someone who deals with a large number of bugs by reading and dealing with them can often see patterns in them that are missed by people who concentrate on specific functional areas and/or specific testing tools. (Expert QA teams can also work around this problem by doing escape analysis after a product is released to see what their automated testing did not catch, but this does not directly help brand new products or brand new features.) * Similarly, since traditional QA tends to be firewalled from customer relationships, and because PM (in my very limited experience) can fail to communicate goals very well with QA, it is easy to focus on tests that directly reference specific features, without stressing cross-component integration that isn't well specified. a good bugmaster ================ * is an enabler- always subverts his own agenda to the project's agenda, and knows his biggest role is to make the jobs of others easier by providing them with the best information possible. * is a pattern-finder- can quickly draw good conclusions from scattered data points. * promotes consistency, defining standards and terms that help everyone find what they need * stands up to developers- NOTABUG sometimes means 'I don't want to fix it', and a good bugmaster knows the difference and acts on it. * understands his co-workers- willing to listen very hard, particularly to hackers/developers. They do know what they're doing most of the time. * advocates for users- users are all too easy to forget about in the heat of the development moment. stand up for them. * is not afraid of volunteers/QA- you will be told 'no one will use this product if this bug is in a release', and you have to be willing to tell them 'you're on crack' if that's the case. * understands the business/game plan (what's our target user/market? when a bug filer says 'no one will ever deploy this if it still has this bug', which they will say, are they right? do the hackers care?) * is willing to get hands very dirty- will need to do unpleasant, repetitive gruntwork * has a fairly good technical background- doesn't have to be perfect, but must be able to quickly analyze and understand the general nature of a bug report. a great bugmaster ================= * is a compulsive organizer * has a very big-picture view- instinctively knows you we have too many bugs and where new bugs fall in the great scheme of things. (Conversely, should not be too obsessed with * is good at automating the gruntwork * strong leadership qualities- motivator, listener, etc. where do bugmasters fail? ========================= * problem spaces requiring /deep/ expertise which the bugmaster does not have. 100% accuracy is not typically a massive problem for bugmastering- bug tracking is, by nature, an iterative process involving dialogue and discourse, and so mistakes can be fixed by the hackers, QA volunteers, and leadership. But being able to at least get in the right ballpark is crucial- since a major part of the job is understanding and clarifying (in the form of triage, for example) getting it roughly right before handing it off to experts is important. This means that fields where 'roughly right' is very difficult to understand are going to be problematic for bugmasters. The canonical example of this in the free software world is libraries and APIs- since getting the impact of an API bug right is usually hard for someone without deep technical expertise in the API, initial triage of API bugs by a bugmaster is not likely to save developers much time, or to give leaders an accurate understanding of the state of the API. Any other project where very deep knowledge is necessary to understand a bug (say, UI design of very specialized interfaces, or accessibility work for more unusual disabilities) is likely to suffer from similar problems unless the bugmaster also has such deep knowledge. * if they disagree with developers or management on priorities. This is not a huge problem for gnome- stable, polished products are agreed to be a high priority by everyone. But we've seen problems where a bug volunteer has their own agenda, or entire projects where quality just isn't a priority. If that is the case, a bugmaster is unlikely to succeed. They can help push an agenda (indeed, that is one of the strengths I identified above) but the seeds for success of that agenda must have been sown by others for it to be successful- it can't start with the bugmaster. what isn't a bugmaster? ======================= * traditional QA- they are not responsible for creating bugs. Sure, they are concerned about the quality of the product and so will use it as much as possible and file bugs whenever appropriate, but their primary task is to understand and clarify incoming bugs from other sources (QA or volunteer users) and not to provide them themselves. * management- a great bugmaster has many of the qualities of a manager, and is probably typically a good candidate to become management in the future, but a bugmaster who is also a manager can easily become a nag instead of an assistant, and a neutral bugmaster who is seen to represent hackers, users and management equally is likely to be more respected by all parties, making them more useful.