The internet, we were told, would make location irrelevant. But it hasn’t. Some examples.

• My brother lives in the UAE so we talk by phone. We would use Skype but the local telephone company blocks Skype and various other services.

• I have several kitchen containers from a New Zealand company. I bought them in the UK and would happily buy more but they are no longer stocked in any shop I know – and believe me I’ve tried. I could order them online from the US but the American shops won’t ship outside of the USA and Canada.

• When I’m searching the web, particularly if I’m looking to buy something, and I find the kind of site I’m looking for I want to know where are they based? London? Cardiff? Chicago? Mumbai? I have nothing against India companies, or American companies (provided they will ship abroad) but when I’m faced with a site I’ve never heard of I want some kind of reassurance. Their location is part of that. OK, there are fraudsters in London as there are in Chicago, I’m unlikely to chase them myself, and they could be lying about where they are, but it all adds to my understanding.

In short: the internet makes many things possible over long distanced but location still makes a difference.

The same is true in software development teams.

You can have people located in different offices, different parts of the world, and you can use telephones, wiki’s, Skype, video-conference kit, etc. etc. to work but it isn’t the same as having everyone in the same place.

When you locate people elsewhere you incur extra costs. Obviously the cost of telephone calls or bandwidth but these are a small part. As you use more technology (conference calls, video conferencing, shared whiteboards, etc. etc.) you incur more and more direct costs. Even wiki’s take time to installl, and time to use.

You also incur costs because conversation often need to be arranged. So I might IM someone and say ‘Can you talk?’ and 20 minutes later they get back and say ‘In 5’. The need synchronise before talking slows down the process and there is a cost of asking.

Then there is the cost of missed conversations. We make small talk, trivia, at the water cooler, or walking to lunch, or because someone is there and we want to moan. When someone is remote these conversations are lost. We incur cost because we don’t get the benefits of these costs.

Then there is the cost of setting up meetings to compensate. A few years ago Lise Hvatum and myself ran a focus group at EuroPLoP entitled ‘What do we think of Conways Law now’. One of the recommendation we came up with was: when you remove an informal communication channel you need to replace it with a formal one.

So when we have lots of remote workers you need more formal, arranged in advance, meetings. This ads to the costs again.

And when you do talk the level of information transfer is less. Even with video conferencing your primary communication mode is voice. Communication is centred on explicit words. Face to face much communication is implicit, its in the tone of the voice, the facial expressions, the body language and how other react. You loose all this when you are remote.

Lets face it, software developers don’t have a great reputation for communicating well anyway so why put obstacles in the way?

Every time we put an obstacle in the way we reduce information transfer. This goes for cubical walls and offices too, open plan offices are my preference. Every barrier, cubical partition, different floors, different buildings, different continents, reduces information flow. The bigger the barrier the bigger the loss.

And for a development team – focus on that word: team. Who ever heard of a football team which played on different pitches?

Team work is about just that, team work. You can’t do it when you are alone. Every barrier makes team work more difficult.

It is an illusion to think that you can develop software when your team is spread around a single building let alone the world.

This is especially true for people who are in leadership positions in an organization. You can’t have a CEO, a team leader or an architect who is removed from those he leads.

Sure there are exceptions. A few companies are completely distributed, Sleepycat software – before Oracle bought them – springs to mind, as do Open Source projects. On the whole these work because they are born on the internet.

The other reason remote working teams work is because they aren’t so much a team but a bunch of individuals, each working on their part of the system and seldom interacting with each other. I think of these as Swordfish teams – a bunch of developers flying in formation. As a result (Conway’s Law again) you are likely to end up with a very diverse system architecture.

Don’t get me wrong, I’m a big fan of working form home once in a while. The odd day spent working at home can be really productive – precisely because I have few conversations and can focus on one or two things. Working from home is great for reflecting and preparing oneself.

And it is acceptable for large teams to have the odd person working remotely. But the smaller the team is greater the need for them to be in one place.

Maybe once upon a time developers could work alone, remotely. They could put on headphones and switch off to the rest of the world. But today developing software is a team sport and needs to be played like one.

I say that as one who has never liked team sports such as football, rugby, cricket, etc. I’ve always preferred solo sports like skiing, running and swimming. I have to adopt and change too.

Software development is easy. Don’t listen to what people say. I was 12 when I started programming and I picked it up no problem. Businesses are full of people who programme without even knowing it: Excel is programming by another name, and how many people write macros?

Yes, programming is really easy. You can buy ‘Programming for Dummies in …’ – well just about any language you want. People can, and do, create small world changing programs without ever really being taught how to program.

That’s why software development and new businesses go hand in hand. Its one of the reasons why so much innovation today appears as software.

But….

Software development is also one of the hardest things. In fact, developing good software is so complex it might be the most complex thing man has ever attempted.

Moon landing? – it used software, compared to the complexity of the systems in the international space station I’m sure Apollo looks simple.

Nuclear power? – compared to the maths used by the finance industry today it looks simple. And that is all in software.

The thing is, writing a small piece of software can be (but not always) very easy. Problems set in when you want to grow the software, want more people to use it, want to sell it, want to package it, want to re-use the code or ideas, take bugs out of the system, add features, make it run 24×7.

It is very very easy to write simple software and have it do something useful. It is an order of magnitude harder to get it to commercial quality and keep doing it. And, to make it harder, there is no single accepted method for doing this and getting it right every time.

It sometimes amazes me that we get any of this right. More amazing is that software which is fundamentally flawed works, or at least seems to work. Stuff which by any engineering criteria is broken is used by companies every day, they even base their business on it. (There is another blog entry in this.)

Developing good software, delivering on time, producing with sufficient quality, etc. etc. – all the things you expect from ‘professional systems’ is a matter of getting your Ducks in a Row. What I mean is…

Creating good software, meeting expectations, delivering on time and the rest is not just a matter of getting one thing right. It is a matter of getting many things “right” – or at least workable. You can get some of these wrong and still delivery something – maybe a little late, or lacking features but you will do something.

Each time you do something badly, each time one of your ducks is out of position, you won’t break the whole thing. This isn’t a chain, instead of breaking you will reduce your productivity, or move away from the target, or reduce quality. With each duck that moves out of position it gets worse but it still works, somehow.

Put it another way. If creating ‘the best software ever’ means scoring 100%. Each time you move a duck out of position you remove 10%. So:
• don’t co-locate your developers, loose 10%
• start work without talking about design, loose another 10%
• work to a fixed specification or work to no specification loose 10% as well.

Down to 70%.

• Fire developers half way through the project, loose 10% for lost productivity, loose 10% for moral.
• Projects looking late, add more developers, loose 10% for violating Brooks Law.

40% doesn’t look good does it?

But there aren’t many case where you need to score 100%. 90% will do, even 80%, 70% often, some customers will be happy with 40% – or less.

The people who get upset most by this?

Not customers, they get annoyed but quite often there is nothing they can do about it.

It is the developers themselves. They see each loss, they know where all those 10% are being lost. And these guys are problem solvers, they want to fix these problems.

Now go back to my falling off a log theory of business.

I said: you do something right. Not 100%, maybe you just need 10%. But that one thing is about doing something useful, something other people want. Its about benefit and effectiveness.

Viability of your software product/company =
benefit of your software
x
effectiveness of your organization

If your software doesn’t bring many benefits then you need an effective organization to stay in business. You have to be better than the rest.

If your software brings lots of benefits then it doesn’t matter how ineffective your organization – shoot the ducks! – you will stay in business.

If your software has few benefits and you can’t deliver, then Game Over.

Which leaves… lots of benefits and a highly effective organization. Then you have a world beater.

Unfortunately few companies fall into the last category which leave most people working in one of the first three.

When I was at business school my classmates and I read lots of good stuff about how to run a good company. Books like ‘In Search of Excellence’ and ‘Built to Last’. But I remember some of my classmates asking: ‘Why aren’t the companies I’ve worked for like these?’ To most, if not all, my classmates these books described something that was not the case.

That was five years ago. Since then I have met many people, few if any of them have worked for ‘Excellent’ companies.

Indeed most of the people I meet (particularly in software development) tell me crazy things that have happened at their companies. Decisions to close down offices just before a contract is won. Objective setting sessions were the objectives set are meaningless. Management which does the opposite of what they know is best.

One company I worked with in the last five years came closer to ‘excellence’ than any other company I know. Still, it got into trouble and has had three rounds of downsizing in the last two years. From what I hear things are pretty crazy now. Apparently in the last round they paid off four engineers, soon discovered they needed them, tried to hire them back, HR got involved and slowed the whole thing down during which time three of them got new jobs.

Another company I worked with was perpetually shooting themselves in the foot. So much so that they even let go the one person go who could make sense of the development team. Just this week I learned the same company which is letting one of their key managers go.

I end up hoping the end comes soon but these companies survive. The books say they should go out of business but they don’t.

So it is that I find myself asking ‘where are the excellent companies?’ And I feel pain when someone else tells me another story of the stupid things their employer is doing.

I’ve come up with a theory which explains it. I apply it only to the software industry because this is what I know but I suspect it might be more universal. It goes like this…

Like having a baby there is no law against starting a company. Actually there are a few laws, and in some countries it can be hard, but in most Western countries, provided you do not have a criminal record you have a right to start a company. In the UK it can take less than a day to form a company.

Most of these companies will go out of business in under a year. If I remember the statistics less than 10% survive year 1.

This 10% have done something right. Maybe they hired the right people, maybe there were in the right place at the right time. Maybe their competition was more of a mess than they were. Perhaps they really did have ‘first mover advantage.’ In other words, many of these companies survive purely on luck. They got something right. That doesn’t guarantee they got anything else right.

Some of these companies will go out of business in the years that follow but the longer a company remains in business the greater its chances of surviving even longer.

Over time these companies will gain some customers. Now customers can be surprisingly resilient, not because they are ‘loyal’ but because switching costs are great and because they have to decide to go elsewhere. Remember, these companies may be dysfunctional themselves so changing isn’t easy.

When you have a few customers, particularly big ones, you can make the money go a long way. Software companies which pull in their wings and go into maintenance mode can shed most of their staff and still stay in business for years. In fact, I would argue that almost any company you think is about to fail can last another year.

Notice there is nothing here which says the company will be ‘excellent’, or even that it will do what the management text books say. All I’m saying is that crazy mixed up companies can survive for a long time when they shouldn’t. Many of these companies have little right to exist by business school standards but they do.

As a consequence many, if not most, companies are, to some degree dysfunctional. Many people work for these companies. Many people are abused by these companies. And these companies are enough to put you off companies altogether.

I call this my ‘Falling off a Log Theory of Companies’: Starting a company is as easy as falling off a log. Most companies which come into being will fail quickly, the majority of those fail will be to some degree dysfunctional but will survive. Consequently most of the people I know will work for dysfunctional companies.

I think this theory applies especially to software companies for two reasons. Firstly there are still many many opportunities in software to be found and exploited, i.e. you can still get lucky. Secondly, the IT industry, and those who fund it, are massively optimistic. We accept failure too easily.

A few months ago I explained this theory to someone not in software. He said he’d seen the theory in action too. He also suggested the book these companies need is not ‘In Search of Excellence’ but ‘In Search of Mediocracy.’

There are many, many books about software development. The technical ones alone (e.g. Java, C++, Apache, .NET, etc. etc.) take up meters and meters of shelf space. The ones about project management take up a bit less space but they are still plentiful, and the ones about people in the process take up even less space but are longer lasting.

It is the later category that interests me most theses days. Some books seem perennially relevant and may never go out of print, books like:

• Mythical Man Month by Fred Brooks
• Psychology of Computer Programming by Gerry Weinberg
• Peopleware by Tom DeMarco and Timothy Lister

But often you don’t need a full book to make a point. On these occasions an essay or journal article will do. Often they make the point better simply because they are shorter. Unfortunately these tend to be overlooked too often – books get all the publicity. I would like to highlight half a dozen essays which I think are perennially relevant and often overlooked. In some cases this is because the articles can be hard to get hold of or, more likely, people don’t know they exist.

1. How do committee invent? by Melvin Conway – the origin Conway’s Law. Although written in 1968 this piece is truer than ever. Arguments made by Conway explain much of what actually happens in software development. I’ve examined and looked at this subject myself elsewhere.

Conway suggests that organization that create systems (not specifically software systems) will create copies of their organizations. So, if you have one developer writing the entire system it will be all entangled, few interfaces and difficult for anyone else to work with. And if you have half a dozen developers who don’t communicate you will get six different coding styles, and six versions of most common functions.

What Conway didn’t foresee is that many of system now create the reverse force. Organizations are limited by the systems they use, thus their structures become copies of the systems in use.

2. Worse is Better by Richard ‘Dick’ Gabriel – this started as a keynote talk, became an essay and then a series of letters in which Dick argued anonymously with himself.

If you are one of these developers who think the best code will always win the day you need to read this. Dick suggests that sometimes the solution which is technically worse is economically superior. In a similar vein see Brian Foote’s Big Ball of Mud pattern.

3. No Silver Bullets from the author of Mythical Man Month, Fred Brooks. This piece came out in the mid 1980’s. The Wikipedia summary is here, otherwise you will need to by the Anniversary edition of Mythical Man Month to read it.

Brooks argues that software development is hard, and there is no ‘silver bullet’ which will make it easier. No language, no operating system, no methodology or any other tool will fix it. I have been told that Brooks has since said the if there is a Silver Bullet it is Agile Software Development but I can’t find this quoted anywhere so I don’t completely believe it.

4. A Rational Development Process and How to Fake It by Dave Parnas. Originally published by the IEEE this also appear in Software Fundamentals: The Collected Papers of David L. Parnas. You might also find some copies stashed on the Internet. Dave Parnas deserves to be more widely read by software engineers today so it you haven’t read much of his work buy the collected papers, well worth reading.

In this paper Parnas argues that while we may want to use a rational development process, and while it makes sense to do so it is impossible. This is because software development requires intuition and inspiration – or as I would put it learning. And since you can’t schedule these things or guarantee they will happen the chances of following a rational process are negligible.

However, in order to make our work accessible to those who come after us we need to fake the development process so it looks like we followed a rational process.

I like the logic, I agree with much of it but I have one disagreement. Each generation of software engineers comes along and believes the previous generation were rational and get confused by what they find. They slowly learn that they cannot be rational – causing a lot of angst. They also wonder how the previous generation made such a mess. So frustration is built up.

5. Cathedral and the Bazaar – by Eric Raymond: an explanation of Open Source software. And an explanation of why software created in corporations is frequently a mess. I don’t think Open Source is the answer to all our problems, but I think it is an answer to some and is here to stay.

6. Enrollment Management by Peter Conklin: until recently this was one you really have to hunt down, but it is worth it. Originally published in the Digital Technical Journal (Digital as in DEC or Digital Equipment) in 1992 it was republished by the IEEE in July 1996. HP now have back issues of the Digital Journal online so you can read it here.

This piece tells the story of the Alpha AXP programme in the late 1980’s and early 1999’s. In theory, on paper, on GANTT charts the project couldn’t be done. The author tells how enrollment management was used to engage those working on the project. Although written in 1992 this paper is about Agile development. And it is about Agile development in the large – over 2,000 engineers worked on the project.

I wrote about this article before, a few years go. One of the small stories in this paper always brings a smile to my face. The OpenVMS team committed to a delivery schedule and said ‘We don’t know how to achieve this, but we commit to finding a way.’

So there you have it. Six essays I recommend you read. And if you only have time for one? Well, chances are you know about Open Source and the Silver Bullet so I recommend you read Enrolment Management or A Rational Process, and if you push me more I’ll say: Enrolment Management.

Tom Gilb sent me a link to this presentation from Ryan Shriver about the use of Scrum with Tom’s EVO techniques. Very interesting, and it returns me to the subject of IT/business alignment – which in part is also a discussion of where Agile software development is wrong.

My blog from last week touched on this but I think its worth spelling this out. There are two problems with Agile, and two answers:

Question 1: Why do Agile techniques work? – forget the technical explanations of why TDD is a good thing, why RUFD is preferable to BUFD or how value stream mapping helps. Think for a moment: there are a lot (an awful lot) of software development groups which are not effective.

Problem 1: How can it be that the same set of (Agile) practices can help all these groups?

Answer 1: Because these groups all have similar problems, probably because too many people were taught (and believed) the same set of practices that have created the problems.

Question 2: How can Agile work without business alignment? The basic Agile techniques pay little attention to what the business wants. The XP onsite customer is really lightweight, XP assumed the customer knows what they want but often they don’t. Scrum is better but the product owner is still left to their own devices, Scrum says little about how the product owner knows what’s what.

Problem 2: How can Agile be effective without business alignment?

Answer 2: It doesn’t have to be. The MIT Sloan Review piece shows that just getting to be effective is such a massive improvement over being ineffective that it is a success. You don’t have to be aligned with the business to show improvement.

In the long run, for organizations which have effective Agile teams these problems will show up. And in time, as more companies achieve Agility, the competitive advantage of Agile Software Development will diminish.

The advantage will move to the companies that address these second order problems. Such companies need to: learn how to improve Agile practices beyond what we currently know, and, get IT aligned with the business.

In an Agile world this means pushing Agile further. Tom would call this Evolutionary, I call it learning, it amounts to the same thing in the end.

The Financial Times carried a piece on Wednesday entitled ‘Agility: Flexibility takes over from planning’. It starts off about agile business but actually spends most of piece talking about agile software development. Both the presence of the piece, and the evidence in it suggests agile is getting more prevalent.Nice definition of company agility from Professor Donald Sull of London Business School:

“he defines [agility] as a company’s ability consistently to identify and seize opportunities more quickly and effectively than rivals.”

What is interesting about this definition is that it says nothing about what you do, or how you do it. Instead it measures agility by comparison with your competitors.

Slightly disappointing that the examples the article cites are the same examples that come up again and again, Zara, BT and perhaps slightly new, Borland.

If you have the time, this piece from the same issue is also worth a look: ‘Why do so many technology projects end in failure?’ This picks up on the EDS / BSkyB court case I commented on a couple of weeks ago.

Its a shame the editors didn’t link the two pieces. In the long term Agile is about seizing opportunities before your competitors, in the short term it is about delivering IT projects on time, on budget and without court cases.

I have spent the last two days at the XP Day 2008 conference here in London. A very good conference and highly recommended. Unusually for me I wasn’t speaking at this one which made for a nice change. No worries about presentations, rooms and being in the right place at the right time.I picked up a whole bunch of ideas, and had another bunch of thoughts. Before I forget them I want to quickly record them – some of these ideas are probably worth an entire blog on their own.

Multi-media keynotes: Both keynote speakers incorporated multi-media elements in their presentation. Jeff Patton included music and Yvonne Rogers video. I think this is a sign of two things: multi-media is becoming the norm; second, people are increasingly bored with the PowerPoint (’death by’) and presenter talking at you (chalk-and-talk) type presentations.

Media companies are more receptive to Agile: A lot of the people attending XP Day had connections with media companies, the BBC and BSkyB were both represented, as were set-top box manufactures, advertising companies and others in the media. Of course there were old industry (pharma, banks, etc.) there too but they have always around.

This is a trend I’ve observed elsewhere. It seems to me that media companies are more receptive to Agile software development than others. There are perhaps two forces at work here. Firstly media companies are fairly new to software as a key part of their business. Yes they always had IT systems but not as part of the product. Agile is the current ‘best practice’ when the BBC, Sky, etc. started to development it was already the norm.

Perhaps more importantly, the second force: media companies expect just in time delivery. You can’t delay the 6 O’clock news because it isn’t ready, you know well in advance you have 30 minutes to fill, and you have to prioritise. The nature of media companies is closer to Agile development so they are more receptive.

Process as a substitute for team work? Of cause Agile software development is all about development process and practice. But scratch deeper and everyone agrees development depends on good people and team working. In fact team working has got more important as software has got bigger and more complex.

It seems to me that a lot of what we call process is actually trying to substitute for team work. We spend time talking about the process rather than the team, we devise process rather than build teams. Somehow we expect the process to substitute for a good team.

Another Agile failure mode: I forget who mentioned this but someone pointed out that Agile teams often fall apart when someone leaves, particularly if that individual is forces to leave by redundancy. This kind of fits with my experience. When an Agile team works well the team bond. The departure of one person is likely to trigger others to leave of their own accord.

Agile Business: This is a topic I’ve been thinking about a lot, so has Chris Matts and others. David Stoughton and Nick Coutts did a great session on the future of business. Not necessarily Agile but a lot of the trends they see in society and the economy suggest the successful business of tomorrow look a lot like Agile organizations.

One point that stuck in my mind above all else was the suggestion that we have moved from a supply economy to a demand economy. Rather than successful companies being those who could produce things the cheapest, and persuade the most people to buy them – a push system – the successful companies of the future will operate a pull system. They will produce what the customer wants, when the customer wants it. Rather than being cost driven they will be value driven.

For my money this was the best session at the conference.

Future of Agile: There are many many software development organizations that are a mess. A little bit of Agile can fix make a great improvement. Agile software development thinking has a long way to run and an awful lot of implementation to deliver. However, Agile has flaws.

One of these flaws is that if you examine in too closely it collapses. The Agile manifesto is actually pretty meaningless, nobody could really disagree with it. If you try and write down a high level value statement of what Agile delivers it sounds pretty much like all the other methodologies and tools that have proved to be snake-oil in the past. Even some of the Agile practices don’t stand up to examination, e.g. pair programming – most developer hate it! Retrospective? – most companies don’t do them.

Agile is still state-of-the-art, the best we have.

It is also a marketing term; a term that collects together a bunch of ideas and allows them to be labelled and grouped together. As a marketing term it is open to abuse.

Increasingly people are increasingly asking: what next?

The people asking this question most often are those with the most experience so some of this questioning is simply boredom, they are looking for the next thing. Some of it is very real and necessary.

(There is some excitement about a new Agile methodology called Kanban, I don’t know much about it and I don’t think this is the next big thing. Something here, but I haven’t had a chance to read it myself.)

So what next?

To me the next big thing will represent a break with Agile. The Agile community came from the Patterns community, Agile is not Patterns. The Patterns community came from the OO community. Patters are not OO. So we shouldn’t expect the next thing to be Agile+.

The next big thing will be People and Team. Software development always comes back to people. Each successive move (OO –> Patterns –> Agile) takes us closer to people. As I said above, sometimes we use process as a substitute, sooner or later we need to address the real issue.

The trick to improving your software development efforts is quite simple: improve your people. Expose them to new ideas, get them learning, help them put those new ideas and learning into action.