Since publishing my stream of consciousness about software quality on Friday (What is a software quality?) I’ve continued to think about the subject. In part putting the thoughts down started to structure my own thinking, in part a few comments made on Twitter caused me to think more, and, simply having written something caused me to reflect.

There are four main points I’d add to Friday’s entry.

First, what I wanted, what I needed for Xanpan, was (is) a good enough working definition of software quality, I don’t need a perfect definition. But even this turns out to be surprisingly difficult. And as Steve Smith pointed out this is not confined to software quality, defining “quality” at all tends to lead you in the direction of Zen and the Art of Motorcycle Maintenance.

Next, what I really need are two things: A definition of software quality and a definition of a software defect. I think I’m starting to approach workable definitions of both. But, definitions (by definition almost) should be free of assumptions about how you achieve – or don’t achieve – that aim. In other words, the definitions should not imply or assume an Agile, Waterfall or any other approach.

Third, one of the things I was concerned about was gold plating or over engineering. Really this is part of the “how do you achieve your aim.” A definition of software quality – and low defects – need not rule out (or embrace) any particular approach. I need to separate this out. Similarly the knowledge issue, while important, needs to be put on the side here.

Finally, I’m coming to the conclusion that we need two different terms for software defects. The clue was in that Tom Gilb quote really.

• An Absolute Defect is, as Gilb says, free from personal opinion or taste. It is objective. For example, function called “Add Two Numbers” always returns 5 – whether you give it 2 and 2, 2 and 3, or 100 and 200.
• A Common Defect (and I’m prepared to find a better name and I don’t imply any link to Deming) is anything which someone at some time decided to report – or classify – as a defect.

Absolute defects are a perfect subset of common defects, i.e. all absolute defects are also common defects but only some common defects are absolute defects.

I think its important to make this distinction because so much time and energy is spent by people arguing over what is and what is not a “defect”. If we define separate terms then we can at least reason about this situation and the fact that individual, teams and organizations often have incentives for classifying something as a defect whether it is or is not.

Jon Jagger pointed out in his comment on the previous blog that “Jerry Weinberg’s definition of quality is “value to someone” or “value to some persons”.” A common defect most definitely fits that description. Someone, somewhere, considers it valuable to call this thing a defect.

But I’m not sure I go along with that statement as a definition of quality as a whole. It is too open and not specific enough for me – or perhaps for the uses I want to put it to!

Some observations about defects to end:

• All defects cost time and probably money: even logging a report of a potential defect costs time (and thus money) and action taken as a result (investigation and potential fixing) costs time. Defects themselves – even when not reported – may well costs money, e.g. overpayment or lost customers. (This also implies an element of the unexpected to defects.)
• Absolute defects cost time and money; is the same true for common defects? Well maybe my opinion of something does result in a different financial outcome than someone else’s but does that make it a defect? Even then it can be surprisingly hard for some organizations to tell what makes, and what looses, money. I’d like this to be the dividing line but I don’t think it is.
• Common defects need to be administered, managed and possibly fixed just the same as absolute defects. In my opinion, if someone reports a defect it should be considered against other defects – common or not – and action (fixing) taken or not. The debate between whether a particular defect is an absolute defect or a common defect isn’t very useful, it is the software equivalent of “my Dad is bigger than your Dad.”
• If there exists some pre-code definition (specification, requirement, formula, etc.) of what the code should do it might be easier to classify something as an absolute defect but this isn’t enough reason to create that definition.
• High quality software has few defects of either sort and is perceived as having few defects.
• Software which has a lot of reported defects doesn’t quality as high quality.

If any of you have heard me speak in a training session or conference you’ll know I am found of quoting Philip Crosby: “Quality is free!”. Crosby was talking from a background in missile production but the message was picked up by the car industry and silicon chip industry (“The Anderson Bombshell” in 1980 explained how Japanese RAM manufacturers were cheaper than Americans and better quality). I am quite fond of applying the argument to software.

I like to cite Capers Jones: “projects with low defect potentials and high defect remove efficiency also have the shortest schedules, lowest costs, and best customer satisfaction levels.” (Capers Jones, Applied Software Measurement, 2008)

He’s not alone in this, Tom Gilb says: “The reduction in defects … saves ‘rework’, which otherwise is about half of all effort in software projects.” (Tom Gilb, Competitive Engineering, 2005).

Jon Jagger pulled me up on some sloppy discussion of software quality, defects and rework in Xanpan. So I’m trying to come up with a (concise) definition of what I mean by software quality and it turns out that this more difficult than you might think.

To my disappointment Jones doesn’t give a definition.

Gilb offers “Defect: a failure to observe a formal, written, required rule. It is not a personal opinion or personal taste. It is a failure to observe a group norm, or required best practice.”

That sounds good until you take the statement to pieces:

• What if rules are informal? Well I suppose we can allow informal, tacit, rules, because they are “group norms”.
• “written” assumes there is something written which isn’t an assumption I can accept. Jones points out that documentation is the second most expensive activity after fixing defects, so I’d hate to eliminate defects at the expense of increase writing costs.
• “personal opinion or taste” seems fair enough but putting this into practice can be incredibly difficult. I know plenty of times when I would call a defect a personal taste but the person raising the issue wouldn’t
• “group norm” is particularly difficult when you are developing products which will change group norms
• And “best practice” …. who says it is best practice? who says it can’t be bettered?

I like Gilb’s definition but I don’t think is enough. Crucially even in saying “not a personal opinion” it does nothing to avoid the “one man’s bug is another man’s feature” problem.

What can we say about software quality and defects?

• Software quality is inversely proportion to the number of defects in the system: high quality implies few defects and vice versa
• Defects have undesirable consequences
• Defects incur costs, in all likelihood financial costs but there are others, time in particular. Even if defects are not fixed they will incur costs, e.g. over payments from a financial system or people ringing the helpline to report a spelling mistake
• Removing defects requires rework and rework costs time and money

This is probably the start of a longer list, what I am describing are the attributes – or qualities – I attribute to “high quality software”.

The list is also self fulfilling: everything I have said so far implies that low quality, lots of defects, will increase costs, so the quotes from Crosby, Jones and Gilb all become self fulfilling. Perhaps this isn’t a problem, perhaps the quality attributes we want from our software is that costs are kept down.

But there is another quality I would like from high quality software which is insidious. High quality software should be changeable – actually all software is changeable (its soft!) but some code is easier to change than other code.

High quality software as easy to change

Lets leave to one side a definition of easy, I agree it should be quantified but not right now.

What do I mean when I say “change” ? I think the spirit is captured by an old John Vlissidees quote I’ve long been fond of:

“A hallmark – if not the hallmark – of good object oriented design is that you can modify and extend a system by adding code rather than hacking it…. In short, change is additive, not invasive. Additive change is potentially easier, more localized, less error-prone, and ultimately more maintainable than invasive change.” John Vlissides, The C++ Report, February 1998

I’m prepared to generalise this to all software, not just OO software. I might even go as far as focusing on the “rather than hacking it” – although one then needs to define “hacking”. Good software needs to allow for change rather than having change forced into it.

Actually this quote also provides the attributes we need to define “Easy to change”

• Change is localized
• Change is less error-prone – perhaps better stated as “change does not inject defect” (Somewhere in Jones writing he suggests 7% of defect fixes inject new defects so high quality software would have a bad fix injection rate less than 7%)
• Change is more maintainable, i.e. changing software does not detract from the changeability of the software

If have these attributes (qualities if you prefer) then software quality is high and as a result change is cheap(er). The relationship between quality and costs appears again.

But the way I describe the quality-cost link is the reverse of the way many people perceive it: the stereotypical Project Manager views quality as an attribute that can be reduced in order to accelerate development and reduce costs. I have to say I have difficulty in actually understanding this point of view but perhaps its because of the way I am defining quality.

Apart from that there is another danger of approaching: Over engineering.

Given all we have said so far you could make an argument for spending a lot of time designing your software to exhibit all these attributes. You could seek to build, design, software which would not require the design itself to be revisited. That after all is rework isn’t it?

Now I’ve long believed there is rework and there is rework:

• Rework to fix bugs, defects, is bad and wasteful because you shouldn’t have put the bug in there in the first place.
• Rework to change software for new requirements, even if that means reworking (refactoring) the design is good, or at least acceptable, because you couldn’t know about this up front therefore any effort to cater for this requirement might be misplaced and could actually end up complicating the design. In other words it is self defeating.

As I see it there is a question of knowledge here: you need to engineer within your knowledge, if you know, or could easily find out, some piece of information which would cause you to work differently then you should. But if there is information you don’t know, and would be time consuming/costly, or even impossible, to find out then it is acceptable to defer knowing and accept rework will be required later.

So the question starts to become one of knowledge acquisition. One way of acquiring more knowledge is through feedback, when feedback is rapid, timely and cheap to get we can rapidly expand the knowledge we are working with.

High quality software should be as free as possible from deficiencies given the current knowledge of what is required, but open to change when new knowledge becomes available which necessitates a change.

It’s tempting to write this as:

Quality(T) = Changeability(T) / Known defects(T)

Where T represents some point in time, as time progresses onwards changeability may well decrease which known defects may go up or down.

Notice I’ve said: “Known defects” not “Known changes”. For a piece of living, successful, software there will be a list of changes people would like made to the software. The existence of this list actually demonstrates another attribute of quality software: people use it and value changes. (Low quality software on the other hand may be so buggy that people avoid using it and thus don’t request changes.)

Excluding non-defect changes like this does leave open the problem of whether a defect report (a bug) is actually a defect report or a request for change. In some organisations such debates are heated but usually they are pointless. Sometimes they are really a Cap Ex v. Op Ex discussion, sometimes they are a “Who will do it?” or a “When will it be done?” discussion, sometimes they are a “Who will pay?” discussion. All these, and more, problems get in the way of this measurement.

While I would like to throw the door open and say: “Its all work to be done, one backlog” to do so would be to blow the equation and argument out of the water because, as I just said, high quality software may well have a longer list of change requests than low quality software.

So now we have to consider the argument about internal v. external quality…. but this blog entry is all ready too long.

Does any of this make sense?

Does any of this help?

Am I any closer to defining software quality?

Perhaps but I don’t think I’ve answered my own question yet!

Writing this entry has helped me. I think I’ve found a possible definition of quality, although I still need a definition of defect. I think we need to consider the attributes of both quality and defects. I think there is a temporal issue here related to knowledge (but I don’t know how to model or define that.) I’m even more confused then ever about the relationship between cost and quality because it appear circular.

Anyone got any better ideas? – or just comments?

Some Agile evangelists are very keen on the idea of “Commitment.” i.e. the development team “commit” to doing an amount of work within a time-box (normally an iteration.) The team do this work come hell or high-water. They do what it takes. Once they’ve said they’ll do it they do it.

I believe the idea of Commitment was baked into Scrum – see the Scrum Primer by Larman, Benefield and Deemer for example. And I’ve heard Jeff Sutherland proclaim the power of Commitment in person. But it seem Scrum is less keen on it these days. The October 2011 “Scrum Guide” by Schwaber and Sutherland does not contain the words Commit or Commitment so I guess “commitment” is no longer part of Scrum. Who knew?

I’ve long harboured my doubts about Commitment (e.g. from last year see “Unspoken Cultural differences in Agile & Scrum” and “My warped, crazy, wrong version of Agile”, and from 2010 “Two ways to fill an iteration”) but now I’m going to go further and say the Commitment protocol for filling an iteration is actively damaging for software development teams in anything other than the very short run. This reassessment has been triggered by a) watching the #NoEstimates discussion on Twitter and b) visiting clients were teams attempt to follow the Commitment protocol.

1) Commitment can lead to gaming by developers who have an incentive to under commit (my argument in “Two ways to fill an iteration”).

2) Commitment can lead to gaming by the need/business side who have an incentive to make the team over commit

3) Commitment combined with velocity measurement and task estimation leads to confused and opaque ways of scheduling work into a sprint, Since (#1 and #2) developers over estimate stories and tasks and the business representatives apply pressure to reduce estimates. This prevents points and velocity of floating free instead they become a battle ground. (Points are a fiat currency, if you don’t allow it to float someone, somewhere, has to provide currency support; overtime from developers, or , more likely, inaccurate measurement.)

4) At one client commitment led to busy developers for the first half of the sprint (with testers under worked) and then as the sprint came to a close very busy testers with developers taking things a little easier. Except developers where also delivering bugs to Testers, the nice pile of bugs kept developers busy but meant that a sprint wasn’t really closed because each sprint contained bugs. On paper, on the day the sprint closed the sprint was done, but it soon required rework. There was also a suspicion that as the end of sprint approached Testers lowered their acceptance threshold and both Developers and Testers asked fewer probing, potentially disruptive, questions later in the sprint.

5) Developers under pressure – even self imposed – to deliver may choose to cut corners, i.e. let bugs through. Bugs are expensive and disruptive.

6) Developers asked to Commit ask for more and more detail before the sprint starts. A “cover your ass” attitude takes hold and stores start to resemble functional requirements of old with all the old problems that brought.

7) Developers become defensive pointing to User Stories and Acceptance Criteria and saying “Your bug isn’t included in the criteria so I didn’t do it” (the other end of a “cover your ass” attitude.)

8) Developers who have not totally mastered Test Driven Development will be tempted – even incentivised – to skip this practice in order to go faster. They may even go faster in the short run – building up “technical debt” – but in the long run will go far far slower.

9) Business/Customers conversely have no motivation to support development adoption of TDD or to invest in automated acceptance test (ATDD, BDD, SbE etc) of their own because, after all, the developers are committed.

Maybe I should say that I currently believe Estimates can work, I have sympathy with the #NoEstimates argument but I have clients where Estimates do work, one manager claims “to be able to bring a project in to the day” using estimates. So I have trouble reconciling #NoEstimates with experience.

Part of the #NoEstimates argument is that “estimates” are too easily mistaken for “commitments” and when they do so teams cannot be expected to honour them but some people do. Obviously if you remove commitment then the transmission mechanism is removed and estimates might still be useful.

While I’ve been suspecting much of the above its taken me a while to come to these conclusions. In part this is because I don’t see that many teams that actually do Commitment. Most of the teams I see are in the UK and I’ve always thought Commitment was a very American idea – it always creates images of American Football teams in my mind – “win one for the Gipper”.

Actually most teams I see are teams I have taught so they don’t do it. (they do some variation on Xanpan if I’m being honest). While I talk about Commitment I teach the Velocity protocol and it is estimation and velocity that is baked into Xanpan. (I hope to be able to push out my notes on Xanpan very soon so join the list.)

2. Validated backlog: items from the opportunity backlog which have been examined, researched and discussed enough to be considered valid candidates for future development.

• Opportunity backlog contains big items with little breakdown, Epics. These may be happen sometime in the longer term, over future quarters and years. They may appear on a roadmap or they may be more speculative.
• Validated backlog items should be at the story size – small enough to be deliverable soon but demonstrating real business value. These items may be developed sometime in the next quarter so they appear on the quarterly plan.
• Iteration backlog items are here and now, they are task sized and are in the current iteration.

There is no point in doing more work on any item until it moves to the next backlog, into the next planning horizon. At each stage some items will disappear, upon closer examination they will not be judged worthwhile.

• Items in the Iteration backlog may well have detailed effort estimates arrived at from work breakdown if needed
• Items in the Validated backlog probably have ballpark estimates and, very importantly, value estimates (how much is this item worth?)
• Items in the Opportunity backlog might have effort scores taken from historical averages (why spend time estimating something that might not happen?) and can only move to the Validated backlog when a value estimate has been made and the item has been judged as worth doing relative to everything else

Three backlogs. What do you think? Good idea or silly?

I was at the Nordic Developer Conference (NDC) in Oslo last week (well, me and about 1700 other people!). I took a risk, I spoke to the title “Do things right and then do the right thing” (now on SlideShare). I was deliberately challenging accepted management practice.

In many ways this was an experiment itself, the presentation is based on a feeling I’ve had for a few years now – dating back to the Alignment Trap – but never really sat down to argue in full. In part I was looking for people in Oslo to shoot down my argument – or support it! I’m glad to say that so far most of the comments I’ve had have been broadly supportive.

Basically my argument is this:

• It is incredibly difficult to know “the right thing”
• Therefore it is better to try something, get feedback (data etc.) and adjust ones aim; and repeat
• However, in order to do this one needs an effective delivery machine
• In other words: you need to be able to iterate.

I used the analogy of using of trying to hit a target with a gun. First you shoot, then you use what you learned to adjust your aim and shoot again. Again you use the feedback to refine your aim.

You might choose to use a machine gun, rapid fire, rapid feedback. Cover the target in approximate shots.

Alternatively you might use a sniper rifle. One perfectly aimed shot and bang! Hit first time.

While I’m sure many organizations would like to think they are using a snipers rifle (one bullet is cheaper than many) I’d suggest that many are actually using something with significantly poorer performance. An older weapon, one without sharp shooter sights, on which requires manual reloading, one which is prone to breaking.

As a result they try to make every shot count but they just aren’t very good at it.

While one might like to think that organizations make a rational choice between these different approaches I think it is more a question of history – or path dependency to use the economic term. You use the weapon you have used before. You use the weapon which your tools provide for.

I’m not saying this argument is universally true but I think it increasingly looks like the logical conclusion of Agile, Scrum and Lean Start-Up. I am also saying you need to try many times.

In technology our tools have changed: when teams worked with Cobol on OS/360 making every shot from your 1880 vintage gun count was important. But for teams working with newer technologies – especially the likes of Ruby, PHP and such on the web – then a trial and error approach might well be the best way.

Possibly the right answer is nothing to do with your organisation but rather a question of your competitors. You want to choose a weapon which will allow you to out compete the competitor, perhaps through asymmetric competition.

Of course the key to doing this is to work fast, and fail fast, and fail cheap. If you take a long time to fail, or if it is expensive then this technique isn’t going to work.

After the presentation Henrik Ebbeskog sent me this blog post via twitter which is beautiful example of exactly what I’m talking about: You Are Solving The Wrong Problem.

Then I went to the PAM Summit conference in Krakow were James Archer included a wonderful quote in his presentation:

“Faced with the choice between changing one’s mind and proving that there is no need to do so, almost everyone gets busy on the proof.” economist JK Galbarith.

This adds another dimension to my hypothesis: that when we invest a lot of time, energy and/or money in detraining what the “right thing” to do it it becomes more difficult to change our mind.

For example, suppose we invest a lot in building a new feature on our website, and suppose that in the week after launch the new feature performs poorly, or at least less well than other recent new features. Those who have invested a lot in arguing for the feature, those who feel closely associated with the feature may be more prone to say “Give it another week, lets get more data” while those who are more distant might be prepared to review what is happening sooner.

In order to accept “failure” we cannot invest too much of ourselves in any feature, shot or attempt.

I’m still not completely convinced by my own argument. The management doctrine “Do the right thing, then do it right” is so strong in me – and so logical. Although I can build reasonable argument for “Do it right, then do the right thing” that I’m still uncertain that I believe it.

What do you think?

Truly, I’d love to have more comments on this.