My blog from last week touched on this but I think its worth spelling this out. There are two problems with Agile, and two answers:
Question 1: Why do Agile techniques work? – forget the technical explanations of why TDD is a good thing, why RUFD is preferable to BUFD or how value stream mapping helps. Think for a moment: there are a lot (an awful lot) of software development groups which are not effective.
Problem 1: How can it be that the same set of (Agile) practices can help all these groups?
Answer 1: Because these groups all have similar problems, probably because too many people were taught (and believed) the same set of practices that have created the problems.
Question 2: How can Agile work without business alignment? The basic Agile techniques pay little attention to what the business wants. The XP onsite customer is really lightweight, XP assumed the customer knows what they want but often they don’t. Scrum is better but the product owner is still left to their own devices, Scrum says little about how the product owner knows what’s what.
Problem 2: How can Agile be effective without business alignment?
Answer 2: It doesn’t have to be. The MIT Sloan Review piece shows that just getting to be effective is such a massive improvement over being ineffective that it is a success. You don’t have to be aligned with the business to show improvement.
In the long run, for organizations which have effective Agile teams these problems will show up. And in time, as more companies achieve Agility, the competitive advantage of Agile Software Development will diminish.
The advantage will move to the companies that address these second order problems. Such companies need to: learn how to improve Agile practices beyond what we currently know, and, get IT aligned with the business.
In an Agile world this means pushing Agile further. Tom would call this Evolutionary, I call it learning, it amounts to the same thing in the end.