estimation – Why do developers habitually underestimate work?

No seriously, answer that and we might be able to get around to estimating changes to it.

What most people will tell you is that its the code and digital artefacts used by that code. Fair enough.

But… It needs something to run on. Say an OS, or some other platform.

But… The platform needs more libraries, platforms, frameworks, and a expertly crafted grain of sand.

But… The expertly crafted grain of sand needs many more grains of sand interlinked by wires.

But… The wires need electrical current, so we need a power planet and a power distribution system.

… I could keep on going down, where does the software end exactly?

How about up. That software talks to other things. Like Humans, and robots.

… Those humans need training, and those robots need software to actually be able to use this software.

.. And those humans need organising into businesses, and the like to really guide that software into being useful.

… and we can keep going up. Again where does it end?

My point here is that each of these levels is an abstraction of sorts, and like all abstractions details leak.

Even if we take the original definition that software is just the code and digital artefacts (as good a definition as any), it has to deal with those leaky abstractions going up, and the leaky abstractions going down.

What leaks in is messy reality:

  • Bugs, Flaws, Glitches, and power outages from below
  • Common Sense Stupidity, Accidents, and Environmental Systems with positive and negative feedback loops running in size and power from tiny, through to large, quickly through to slowly.

So realising that software is one aspect of a System it makes sense to look over at what sorts of systems we have: S, P, and E

S Systems are our nice well-behaved problems like sorting. We can define the problem, and we have many ways of solving that problem that are well described and researched.

P Systems are ones where its clear what we want to solve, we have good problem definitions, we might even know an algorithm that will eventually solve the problem provided: good inputs, enough time, or a machine actually capable of running it. Weather prediction fits here:

  • We don’t have perfect scanners that can detect the state of every atom on earth at once, so we have poor/approximate input.
  • If physics is correct then we have an algorithm that will solve it, its just not runnable on any machine that exists.
  • The algorithms that are runnable are approximations and can’t solve the problem, but can give reasonable guesses.

But we have a very simple way to check if the answer was right, we observe the real weather.

E Systems are the worst. Solving the problem, changes the problem. Even partially solving the problem, changes the problem. Its easier to describe this kind of system using an example: Air traffic control.

There are five planes wanting to land. The software tracks this and assigns them runways. queueing two of them up because there are three runways. Except now a plane wants to take off. So it directs it to the third runway. Except on landing the plane crashes blocking runway three. So the plan has to be rerouted back to runway 2 except one of the landing planes is currently using the taxi way. And fire crews need to get to runway 3.So an operator needs to stop the next plane from landing on runway 2 to clear it, to clear the third runway. And we still have to operate landings and takeoffs without runway 3 for a while. Maybe some planes need to be redirected to another airport, or need to be told to not take off from those other airports because they won’t be able to land.

See how quickly the problem went from assigning planes to runways, to stopping planes at other airports from taking off.

Now try estimating for that.

The core problem is this. If something already existed that solved the problem at hand, we would just copy it and be done. We do this everyday with applications like Word. The estimation would be just for the time to locate the solution and copy some files – something language, library, framework, and application authors have been working on for over half a century.

But copying only works for S and P type systems that have been explored and are no longer novel. You can’t copy E systems because that means copying the world around them, so every E system is by definition novel.

Novel means we haven’t explored it – Which means learning.

Now estimate how much learning you have to do to understand X.

You can’t. The best we can do is find some examples that look similar and approximate.

A P System algorithm. Which means we don’t have a usable exact algorithm, we only know how to verify the answer.

To boot we developers are also human, and humans are generally speaking optimists. You can go and review the psychology research to verify that.