I think I get quite a few comments on this blog and I publish most of them. Almost all of the ones I reject are spam/graffiti type, people looking to put a link to their site on my blog. Sometime I reply to a comment but not always, it depends on whats on my mind and how much time I have.

Last week a comment arrived on my Failures in Agile process entry from April 2007. Its worth pointing out that the comment has very little to do with what I wrote in the entry. Except for the fact that both my entry and the poster are concerned with “Agile failure” the content of the post seems to have little connection with what I wrote.

What’s interesting is that this post exists. Its a reminder that many people out there still believe a lot of myths about Agile development. The poster is right about some stuff but then just lists a bunch of myths with no supporting evidence or examples.

So lets take the post and dissect the myths:

• “Considering the reasons for failure you named someone may conclude there is no difference between Agile project failure and RUP project failure. In other words the software development approach obviously does not make difference.“

True: indeed I concluded this myself when I wrote: ”Failure of an Agile project looks a lot like the failure of a Waterfall or RUP project.“

The RUP folks increasingly claim that ”RUP is Agile”. There is long post on that subject but not today. If the RUP folks are right then you would expect RUP to fail like an Agile method because it is an Agile method.

There is a stream of thought that say that method doesn’t make much difference. Check out “Amethodical systems development: the deferred meaning of systems development methods” by Truex, Baskerville, and Travis if you want the full details.

It is widely felt that people, not process or method, are the key ingredient for successful software development. So how do you get good people? I think you need to grow them, develop your own developers. And I think the way you do this is through a learning culture. And I argue that Agile development has its roots in Organizational Learning so is better suited than RUP, SSADM, Waterfall, etc. to do this.

For my full argument: Buy the book, Changing Software Development.

• “The main bottleneck I came across is the failure to define good system requirements. In other words: garbage in – garbage out, principle applies here.”

True: Not sure it is always the main bottleneck but I think can be and it is always significant. This is one of the reasons I’ve devoted a lot of blog entried (and the odd conference presentation) to Product Management.

Although most Agile methods say little about requirements I think they do help. By improving deliver they free up management time and energy to think about that they actually want.

• “Agile wins more attention of management becacuse they discovered the way to shift the responsibility and blame for project failure to programmers…”

Not in my experience. I think managers have always blamed developers and developers have always blamed management.

But blame is not a very useful concept.

• “Downsides of agile:”

To the myths…

• “agile may only be used on small projects”
• “maximum 5 developers on project“

False: this myth started because the early writers on Agile hadn’t applied it to large projects when the wrote the first books.

I’m working with a large .com at the moment to introduce Agile to a large (50+) development team. Jutta Eckstein wrote about large Agile projects four years ago.

Software development always goes better with small teams. Since when did RUP, Waterfall, etc. etc. guarantee success? At the very least Agile is no worse than the next best methodology.

One of my personal mantras is: ”Inside every big project is a small project struggling to get out“. To find this project you need to a) be able to deliver, b) actually deliver business value.

• ”all developers must be higly skilled (no learning moments are allowed)”

First part: Highly skilled developers help any project go better. Agile is no different. (With the possible excpetion of Mongolian Hordes I’m not aware of any development technique that claims to do better with less skilled developers.)

Second part: False, Agile is all about learning, there are lots of learning moments. Again, buy the book, Changing Software Development – I’m not going to repeat myself.

• “developers must be in the same physical room with the project owner.“

False: there is a lot of experience with distributed development teams using Agile.
And once again, any development method goes better when teams are in the same room.

Frankly, distributing your development teams is always going to hinder your efforts. Putting them in one room is the most effective way. However sometimes you just can’t do it. And sometimes companies are prepared to take the productivity hit because the cost savings compensate for it.

This came up in the work I did on Conway’s Law.

• ”project owner must be available 24/7 since good programmers also work extra hours”

False: There is no law that says “good programmers work long hours” so the assumption is wrong. In fact, Agile emphases sustainability, you don’t get that by working people long hours.

Lets be clear: Agile does not mean working long hours. If you are working long hours you have a problem and you should fix that.

There is a problem with Karoshi (death by over work) in some cultures but this can’t be blamed on Agile or Lean. Yes it exists, yes it occurs at Toyota in Japan. No I have not heard of any stories of it occurring in Agile Software Development teams or at Lean enterprises like Dell or Amazon. Or even at Toyota outside Japan.

The original XP project, C3, did cause the near nervous breakdown of the first “Customer” – Business Analyst or Product Manager really which was a failure in C3. However, this does some way to supporting the view (held by myself and the poster) than knowing what you are doing (requirements) is important and thus you need more effort here.

To work someone to death or a breakdown is a problem that needs to be fixed.

• “agile does not support use of virutual teams or outsourcing constructions“

False: Where does this idea come from? I’d love to know so I can understand what to disprove!

OK, lets try…

”virtual teams“ – see above.
”outsourcing“ – ThoughtWorks has built a pretty big business as an Agile outsourcer.

• ”agile provides poor quality of software since documentation usually does not exist. If it does – it is written in “reverse engineering” style.”

False.

I have never seen any evidence that documentation leads to high quality software. Year ago I worked on rail privatisation, we had lots and lots of documentation, and then some more. The software was rubbish.

Agile provides high quality software because it simply wouldn’t work without quality. Look at TDD, pair programming, customer involvement, etc. etc. Its all about eliminating re-work, i.e. improving initial quality.

The tests produced by Agile developer are documentation, executable documentation at that but documentation all the same.

And reverse engineering documentation is actually the only sort worth having. Since design and requirements are emergent anything written in advance will change.

• “selling “agile project” as “fixed price & fixed scope” is mission imposible since scope of the project is not defined.“

False: it just means you need a different type of contract.

If Anonymous would like to reply to me please do, but please, next time you make some of these wild claims please explain your thinking or cite your evidence.

An interesting analysis in McKinsey quarterly about the increasing costs of off-shore manufacturing – Time to rethink offshoring. As is so often the case this piece is US centric, and it is about manufacturing not services, still, some of the data and analysis are worth looking at:

• Wage rises in China mean manufacturing workers now earn $4,140 a year up from $1,704 in 2003
• Weaker dollar also erodes the advantage (and since the pound is low at the moment the same is true in the UK)
• Higher oil prices mean getting raw materials to China, and finished products from there is also more expensive

They also hint at the delays in supply when you ship from China – because you have to order so far in advance and get stuff shipped.

In short, manufacturing high-tech products in China isn’t clearly cheaper any more.

And to proove the point, yesterday’s FT reports: “The latest cheap manufacturing site for European companies is not in Asia or eastern Europe but the US”. (Its a little more complicated than that but have a read for yourself.)

So what of the software industry? Lets apply this thinking, and extend it to India and elsewhere.

• Wages have been rising here too, I don’t know how much by but is manufacturing staff have more than doubled their pay IT workers can’t be far behind
• Weaker dollar has erodes the advantage (ditto the pound sterling)
• Higher oil prices mean flying staff between your offshore development centre and your actual location is more expensive

And of course you have the supply chain delays too. McKinsey doesn’t go into this angle greatly but there is a good analysis in Womack and Jones Lean Solutions: did you know stores have to order their Nike’s a year in advance? Bottom line: you can have lean manufacturing on the other side of the world but you face challenges if you want a lean company spread over that distance.

Regular readers will know I’m all in favour of developing software in India, Malaysia or wherever. However I think the benefits are exaggerated. Or rather, the downsides aren’t appreciated. Somewhere there is a balance.

People I know who know about graphics and data visualisation tell me Edward Tufte is The Authority. I also hear he is a great presenter and his tutorials worth attending – if you into graphics and visualisation that is. So when I see his name on something I pay attention.

As regular readers know I have a thing about Conway’s Law. So when Tufte’s name came up in connection with Conway’s Law on a recent search I paid attention. Tufte’s point is actually about the inadequacies of PowerPoint.

I agree with him. I really dislike PowerPoint and wish I could give it – and similar tools – up. I’ve blogged about this before. I should report I have failed to give it up. I have two up coming presentations – BCS Project Management and Oredev – and I’ve prepared PowerPoint decks for both.

Why?
• Because they are a crutch to lean on, parts of me is scared without it
• Because the event organizers expect it
• Because I want the attendees to have something to take home
• Because it helps to plug my book (nice big picture!)

The comments and links on Tuft’s piece are even more interesting. Seems NASA are concerned about engineering by PowerPoint and the US military also have PowerPoint trouble.

All the more reason to redouble my efforts and kick the habit.

Once in a while I get asked my opinion on Open Source Software projects. Personally I think Open Source software (OSS) is great: Apache, GNU C++, Linux to name a few. But as a project management or product development technique I can’t say I recommend it. There are two reasons I give for this.

First there is no single OSS development model: Apache has a stack of IBM cash, GNU C++ is largely volunteers with chip makers (and others) contributing expertise and resources to help their own products, Mozilla was seeded by Netscape and has done a good job of marketing itself, etc. etc. Some OSS projects have paid development teams, some have single individuals and so on. It is not enough to say “We will do it open source” you need to be more specific.

Second: most OSS projects fail. “Go look at the number of web browsers on Source Forge” I say – a quick search this morning shows 19,362. There are lots and lots of OSS project that go no where. Commercial projects (famously) fail too but the important point is: OSS is no silver bullet.

So with this in mind I was interested to find and read a paper by Kieran Healy and Alex Schussman from the University of Arizona entitled: The Ecology of Open-Source Software Development. Although from 2003 I think the analysis and arguments probably still stand. (This paper appears in several places on the web so I guess it is already well know, maybe I’m late to the party.)

The authors did some analysis on projects in Source Forge and came up with some interesting results. Basically it turns out that when most people talk about the Open Source model, and successful examples, they are talking about the top 1% or less of all Open Source projects. The true OSS model has a long tail.

Taken as a whole OSS projects follow the Power Law – its that law again! The authors say:

“The median number of developers is one. The 95th percentile is 5. Relative to the whole field, only a tiny number of projects have more than a handful of developers. The median number of cvs commits is zero. At the 75th percentile it is 1 and at the 90th percentile it is less than 100. This indicates that there is little or no programming activity taking place on more than half of the projects. Examining the number of message authors across all forums shows that only projects at the 90th percentile and above have more than two contributing message writers.”

And later:

“In summary, we found power-law type distributions for all activity measures in our dataset. In each case, a tiny number of projects dominate an activity-type when measured by volume.”

The projects which are actively developed (in this study these include Crystal Space 3D, Open CASCASE, gkernal) are not the most downloaded (in this study CDex, VirtuakDubm ZSNES). In fact the authors say:

“Measures of user interest in a project — such as site views and downloads — are not closely related to measures of developer activity on a project.”

So what are we to make of these findings?

Well I think it bears out my initial comments. OSS is no silver bullet.

The papers authors propose three hypothesis for further consideration – and given that the paper was five years ago they may well have developed these further.

First they suggest that the involvement of professional developers is key. Despite the gifted ametuer image of some projects it helps to have professionals along. Second the role of project leader is critical – nothing new here perhaps but more evidence that OSS alone is not enough. (Actually, I was reminded of Fred Brook’s Chief Programmer model while I was reading this.)

And third, most interesting to me: the authors suggest that successful OSS projects with have hierarchical structure between leader and development team. Many OSS proponents (e.g. Eric Raymond) suggest a more chaotic structure in OSS projects which may not be the case (remember Healy and Schussman are suggesting a theory here without proof).

Personally all three suggestions ring true for me. There are wider lessons here too for the Agile and business communitiies.

There are those in the Agile community who see it as a route to emulate the success of OSS. However this study shows that the OSS success if not good enough for commercial projects.

Second some elements of leadership and hierarchy are necessary to have a successful project. In other words, relying exclusively on self-organising teams and emergent behaviour may not be enough. Some element of control is justified.

In the business community there has been a lot of talk in the last few years of applying Open Innovation – that is applying Open Source ideas to business problems and products. Proctor and Gamble have one example, Walkers Crisps in the UK is trying an Open Source type idea at the moment. If the OSS experience is anything to go by open innovation may come up with some good ideas but it is not enough alone.

From time to time the dark side of Lean surfaces. On the whole a lot of this dark side is bunk (rubbish), Lean gets the blame for something that isn’t really anything to do with Lean. Though Lean is not without its problems most of the well publicised cases (like karoshi, death by overwork) seems to have more to do with (Japanese) society than Lean itself.

But it hadn’t occurred to me until today that there were also financial problems with Lean, or to be specific, the transition to Lean. The summer issue of the MIT Sloan Management Review has a piece about financial problems (How to Manage Through Worse Before Better) which can afflict companies transitioning to Lean.

Although this piece is about manufacturing companies not software companies I think its worth discussing the reasons and how they might also effect software companies:

• Lean companies hold little or no inventory, so a company transitioning to Lean will reduce the amount of completed work held in stock (not to forget raw materials and work in progress). But this stock is also shown as an asset on the balance sheet. So reducing inventory reduces the value of the company.

• It is not only the transition company which hold stocks, so too do customers. As the company becomes more Lean the lead time for orders will reduce so customer no longer need to order so far in advance and hold buffer stocks. They will adjust their orders, rather than ordering 6 weeks in advance they may order 1 week in advance. Therefore sales will appear to reduce as customer change their schedule.

• Although the Lean will make the company more productive in the short run it is difficult for the company to realise this benefit. It is not possible to reduce the work force during the transition because managers and workers need to co-operate so managers can’t fire workers (well you could, but imagine what it would do to the rest of your change programme).

• Neither can the extra productive capacity be used for new manufacturing because the company is still in transition and it takes time to introduce new products to manufacturing and optimise the system along Lean lines.

So what about software companies? Actually I think they do, perhaps to a lesser degree:

• Software companies don’t hold stock, but work in progress could appear on the balance sheet, it all depends how the company accounts for R&D so this could be an issue.

• Customers might delay orders once they know they can, this depends on your sales model.

• Software companies face the same problem in laying off surplus workers, if anything the problem is worse. Even though most software companies I’ve ever seen don’t have enough people to do the work they are asked to it these benefits won’t show on the balance sheet or cash flow statement.

• Finding work for newly available resources is also going to be a problem. I’ve blogged before about the importance of Product Managers and the idea that The Bottleneck has moved – if you have developers who are more productive you need more Product Managers – so costs will go up!

Actually things might be worse in software development because of the quality issue. A Swiss friend of mine claims that his software company was forced out of business when they adopted TDD and improved quality. It seems many of their customers were buying new versions of the software, and paying support fees, to get bug fixes. Improve the quality, reduce the bugs and why do they need to spend more money?

My reply is always: its not much of a business model if you rely on your customers paying for fixes. But the point here is that during the transition phase you loose the revenue before the benefits kick in.

I’ve not read the second half of the Sloan piece yet so I don’t know how to resolve these issues but I think this analysis is helpful in understanding why companies find it hard to transition.

I was expecting slightly more response to my “Why managers don’t get Agile” post last week because this is an issue that comes up again and again and again. Hans Wegener (of metadata fame) did respond and for some unknown reason his comment ended up on the wrong blog post (May’s post on making strategy) – his comment is here.

I certainly didn’t intend to come across as anti-control. True, I do believe that less control and more self-organization can improve results but I know that some control is necessary. The point i was trying to make is: Agile methods do not discuss the role of the manager much, and they appear to loosen control on the development effort. Therefore it is hardly surprising that managers don’t “get” Agile.

Actually I believe that Agile development improves control in an organization for several reasons.

First the illusion of control is exposed. Estimates, GANTT charts and “management by exception” never really controlled development work, they just presented an illusion of control. So much upfront planning was just planning theatre, it never really changed much, development work carried on much the same, work took as long as it took whether the estimate was 1 day or 1 month. Agile strips away many of these illusions – that’s why it is painful at first.

Secondly Agile improves control because work can be stopped at any time and still produce benefits. Stop a traditional project half way through and you have a lot of code which implements some features but is so full of bugs that it is usually unusable. Stop an Agile project half way through and you may have fewer features but the features you have are usuable, there is no need for a test-fix-test cycle.

The removal of the test-fix-test cycle also improves management control. Traditionally this cycle takes up an unpredictable amount of time at the end of the project. Because it is unpredictable control is lost.

Third – and final for the moment – because Agile teams are focused on delivering what the business wants they don’t develop anything else and do deliver what was asked for. Again more effective control.

However this third point does put a responsibility on the customer/business side of things to be able to articulate what it actually wants and to take a part in creation and verification. Again this might look like a loss of control but it isn’t.

For example, a traditionally request might be “Add a widget to the website” and at some unknown date in the future the widget would be added – the colour, size, shape and functionality of the widget might not be what was expected but was there. Such a request given to an Agile team is likely to result in a whole bunch of questions: “What shape do you want it? Colour? When by?” etc. etc. This might look like a loss of control but in fact the team is giving more control to the requester.

One of the things I always tell clients is that Agile exposes problems, and in this case a problem with the management of IT work has been exposed.

Hans ends his post with the comment: “I can’t prove this, but I have a theory: developers don’t “get” management” and I have to agree with him. Superficially management is like coding, its about organizing stuff, connecting things and putting things in order. However it is very different: the fundamental building block of management is not lines of code but people. Mangers work in ambiguous environments with a lack of information and few ways of testing a hypothesis.

I’m also convinced that many (even most) of the problems software development faces are the result of poor management not technology. Developing software may look easy but it is hard, not only do you need good coders and testers but you need good managers. Being a good coder does not make you a good manager and being a good manager does not make you a good development manager.

I hear it time and time again: “they don’t understand Agile” – “they don’t get it” – “they won’t let us do Agile” – “they” means managers. I get frustrated when I hear this because a) I’m now on management side of software development, b) I get Agile, c) To me “Agile management” is just the application of good management.

The problem was repeated on ACCU General last week when someone said:

“I often find there is little to no problem convincing the programmers. The problem is convincing people “higher” up the corporate ladder. The ones who control the purse strings”

Recognise the problem?

The poster suggested a reason:

“but typically they have only a shallow understanding of the nature of software development.”

I think the poster is right, typically this is the case but it is not the whole story, there are other factors.

Last week I was putting together my ideas for Øredev 2008 were I’m giving a talk entitled “Hitch Hikers Guide to Management.” In the session I want to talk about the role of managers in Agile development. It was then that it all came into focus…

Managers don’t get Agile because Agile appears anti-management. Not only is there no clear role for managers in Agile but many of the Agile methods set out to remove management.

Lets look at the evidence, and for once Scrum is the best example.

• Scrum does not define a “Project Manager” role – it does define a “Scrum Master” role (increasingly organisations are interpreting “Scrum Master” as a “Project Manager” which kind of defeats the objective)
• Scrum does not define a “Business Analyst” or “Product Manager” role, instead it defines a “Product Owner” role (and does a bad job in my view)
• Scrum strongly advocates self organising teams, i.e. no manager or leader

Just about all Agile methods are developer centric, they were created and pioneered by developers. XP does away with just about all roles except customer and developer. So at first sight Agile methods are a threat to managers.

Somewhere in the back of my mind I remembered Peter Block’s comment in Flawless Consulting:

“Maintaining control is the center of the value system of most organizations. There is a belief in control that goes beyond effectiveness or good organizational performance. Many managers believe in maintaining control even if keeping control results in poorer performance. There is case after case demonstrating that more participative forms of management are more productive, yet the practice of participative management is not too common. … Control is the coin of the realm in organizations.”

Agile, as often presented, threatens control in an organization, it appears to say: “remove the managers and give control to the code monkeys software engineers. (i.e. the guys who don’t wash, the same guys managers don’t want speaking to the customers).”

Contrast this with something like PRINCE 2. As regular readers will know I qualified as a PRINCE 2 Practitioner earlier this year. I learned very little about project management on the five day course, and much of what I was taught flew in the face of my own experience. But what PRINCE 2 does present is a control mechanism, or rather lots of control mechanisms.

And there are many management roles: Project Manager, Project Executive, Chief Supplier, Chief Customers, etc. etc. PRINCE 2 will ensure the ritual and appearance of control even if a project is widely out of control. In other words: PRINCE 2 is not a threat to anyone in management.

In contrast Agile, and the organizational learning approach I advocate, do represent a threat to management. They don’t remove management from the picture but they do change the role. And so far Agile methods haven’t done a good job of explaining where management fits in.

So, is anyone still surprised that management don’t get Agile?

I have been convinced for a while now that the main problem in software development is not technology but poor management. Its why when I moved from development to management I went off and got a management degree. I am also convinced that Agile and modern, management practice are totally aligned – go read Changing Software Development if you don’t believe me.

Anyone for Agile Management ?

The Financial Times has reported more about fallout from the Moody’s credit rating bug in the last couple of days: Moody’s to investigate staff over rating bug (page 1) and Moody’s to check on accuracy.

I blogged about this a few weeks ago and had a very interesting comment pointing out that despite the bug Standard & Poor’s gave the same instruments a similar rating, that is worrying.

According to the FT:
“We have no reason to believe that our record [with computer bugs] is unusual,” Mr Cantor [of Moddy’s] said, noting that bugs often appear in software development throughout the computing world.

Indeed, that a bug existed is not unusual, that it took over a year to fix is not unusual. That is may have lost companies lots and lots of money probably isn’t even unusual.

I don’t know what’s worse, that fact that this situation is regard as “usual” in the industry or that Moody’s are using everyone else’s failings to excuse their own.

We – well some of us at least – know the solutions, we can fix this, but if Moody’s and the rest of the industry accept this as “normal” what’s the point?

Here’s a suggestion: Agility is a measure of the time it takes between an organization deciding something and action being taken.

Ideally you would also measure the time it takes from an event to the organization recognising the event, realising a decision was needed, making it and then acting. But some events aren’t events, they are gradual build ups, at some point they pass the “tipping point” and action is required. How do you know when you should recognise the event?

Moving on slightly.

Some people have suggested that organizations should be modular. (John Seely Browns argument in The Only Sustainable Edge touched on this.) If a unit isn’t working swap it out and up in a better one, or a more high performing one. Business design is taking on aspects of software design. The modular corporation can swap its call centre in Aberdeen for one in Mumbai.

People who advocate this also suggest that the corporation becomes more more Agile as a result. If you need more call centre operators just power up another centre somewhere in the world. If you need more sales people just hire a sales force module. But I’m not sure.

I can see how this would allow you to flex capacity, and for a stable organization it may allow them to respond more quickly. However, in a rapidly evolving environment, one were you don’t know what’s happening its going to take time to identify the module you want, sign the contracts and switch them on. And that’s assuming you can buy in the business module you want and you have compatible interfaces. If you can’t you might have to build an adaptor.

In an innovative environment that is changing fast, or just one that is poorly defined, bringing in a third party is going to complicate things. Someone else to talk to, another organization, what are their standards? Do they understand the domain? Are the property rights clear? And so on.

In some environments I think Agility is enhanced by having everything done in house. Learning and change should be a lot faster when everyone works for the same corporation, have the same objectives, and don’t have to keep secrets.

When you add organizational boundaries it slows down recognition of events, decisions and action.

Couple more thoughts I picked up last week…

I finally found out the name of something I’ve know about for a while: Confirmation bias.

This is the brain’s tendency to seek out information that supports are existing beliefs, and to put more faith in evidence that supports out current beliefs; rather than take a purely balanced view. Confirmation bias means that we human’s aren’t a rational and neutral as we like to think we are.

Several of the presentations a Tom’s seminar last week demonstrated techniques that can be used to expose our assumptions. Assumptions are a bad thing, they blind us to things we need to know. However they are useful, they serve as short cuts which enable us to process information faster.

I had a lecturer as an undergraduate who told us to “Always document your assumptions” which I tried to do. Then he would knock marks off for not documenting assumptions. Trouble is, I didn’t know I was making them. Assumptions can be difficult things to recognise.

Especially when you take in confirmation bias.