Software Psychology

How Do You Estimate for That?! by Bruce Nielson

Trying to estimate software projects is difficult to say the least. But sometimes, it’s just impossible.

I just finished resolving a problem on my project that I spent the last two days working on. The task was to issue a Purchase Order (PO) to a vendor using the SAP interface available to me. 

It all started when the Purchasing group called to tell me they couldn’t release the PO because they got an error in SAP that says it would put the project over budget. They tried everything and couldn’t get it to work. They had no idea how to fix it, so they threw it back to me to resolve. I then started calling people all over the place. I talked to accounting, to finance, to IT, and I even called the help desk which resulted in a conversation with someone in some other country that I couldn’t understand.

Finally, IT sent me over to taxes, who told me that the reason it wasn’t working is because when they make a PO, they automatically add an amount to it for tax even if the purchase is services and not taxable. So any project amount you enter must have taxes added to it even if there are no taxes to be charged.

Armed with this knowledge, I then had the project amount raised to account for these ghost taxes and the problem disappeared.

Do you really believe it’s possible to estimate something like this in a project?

Does Robert Glass’ Formula for Estimation Success Actually Work? by Bruce Nielson

In a previous post I mentioned Robert Glass’ “fact” that estimates are made at the beginning of the project before the problem is even defined, thus the estimate is invalid from the get go.

While I don’t disagree with Glass, I do believe he is under estimating (pun intended!) why we human’s prefer estimates – even known bad ones – over no estimate at all. I referred to this phenomenon as “water breathing” because if you are about to drown anyhow, you might as well see if you can breath under water.

Up front estimates are much the same – if you don’t make them at all, you’ve failed. If you make them and they are bad, you might (and often will) have time to correct them later. So that’s the real reason we make bad up front software estimates regularly. It’s a good reason once properly understood.

I know this probably sounds defeatist and I don’t mean it to be. In later posts I’ll discuss alternatives to making bad upfront estimates, even when you have “no choice.” But before I explore alternatives, I think we could benefit from looking at all of Glass’ “facts” about estimation. (See Facts and Fallacies of Software Engineering, p. 27 – 42)

Fact 8: One of the most common causes of runaway projects is poor estimation.

Fact 9: Most software estimates are performed at beginning of the life cycle. This makes sense until we realize that estimates are obtained before the requirements are defined and thus before the problem is understood. Estimation, therefore, usually occurs at the wrong time.

Fact 10: Most software estimates are made either by upper management or by marketing, not by the people who will build the software or their managers. Estimation is, therefore, done by the wrong people.

Fact 11: Software estimates are rarely adjusted as the project proceeds. Thus those estimates done at the wrong time by the wrong people are usually not corrected.

Fact 12: Since estimates are so faulty, there is little reason to be concerned when software projects do not meet estimates targets. But everyone is concerned anyway.

Fact 13: There is a disconnect between management and their programmers. In one research study of a project that failed to meet its estimates and was seen by its management as a failure, the technical participants saw it as the most successful project they had ever worked on.

Look over that list carefully. I can see everyone nodding their head approvingly. “Yup, it’s all so true! Those stupid pointy haired managers are the real cause of the bad estimates!”

While these facts all seem generally correct to me, I feel like something is missing. Glass seems to be advocating that if we do the inverse, we’ll be fine. Likewise, as you shake your head knowingly and wag your finger at management, you are advocating the same. But is this really true?

Let’s try taking the inverse of all his facts and see if we really believe the software estimation problem is solved. Can you honestly say you’ll agree with the following statement as being factually correct?

You’re software estimates will all be wildly successful if you just:

• Make good estimates instead of poor ones (inverse of fact 8) by…
• Making the estimates after the problem is defined – so after requirements gather is completed (inverse of fact 9)
• Have the developers make the estimate, not marketing or management (inverse of fact 10)
• And adjust your estimates when there are clear changes to the scope of the project (inverse of fact 11)

Since all of the above are true for your project, we can then conclude that it’s perfectly acceptable for management to get very concerned if you miss your estimates, since they “did it all the right way” (inverse of fact 12) and so we know there will never be another disconnected between management and programmers again. (inverse of fact 13).

Do you believe it?

I don’t.

Personally, I have never let management or marketing make estimates for the programming team (except to add fudge factor on top), I’ve always performed the estimate after requirements gathering when we’d supposedly “defined the problem”, and I always adjust my estimate for clear changes of scope. But even when I “do it all right” my experience is that programmers still can’t generally keep to their estimates.

So my conclusion is that while Glass has hit on a series of true problems about software estimation – and I’d agree if you don’t at least do the above, you have no hope at all in making your estimates – I feel he is failing to address the true underlying problems with software estimation.

So then what are the real reasons we all seem to suck at software estimation? To find that answer, we need to learn a lot more about how software and human psychology collide before we have any real hope of finding a solution to the problem.

Sure It’s the People – But Which People? by Bruce Nielson

So if we admit that software is really about human intelligence, not tools, then we know that human factors matter the most.

Tom Demarco and Lister, in their famous (infamous?) book Peopleware, were the first to make popular the idea that it was people that mattered the most.

But I can't shake the sneaking suspicion that they missed something. You see, I've taken top notch programmers and had them succeed on one project and failure miserably on another.

I remember one client that, whenever he called, I'd cringe. I knew – knew – that no matter who I assigned to the project, it was already a failure from the moment he called. But when his counterpart at the same company called, I knew – knew -- the project was going to succeed no matter what.

Somewhere along the line the thought occurred to me: could it be that the customer was the primary success or failure factor on my projects?

But how could that be? I knew tools weren’t the main factor, but how could programmers not be? How could it be the customer -- who never wrote a line of code!?

Thus began my study of the broader software project success factors.

The Chaos Report

Perhaps you are familiar with the original Chaos study from the Standish group and it's project failure and project impair factors. This was a study that started out trying to determine what affect tools has on the success or failure of a project. As they started interviewing people, people would volunteer things like “well, the tools don't really matter, but let me tell you what does.”

Eventually, due to the overwhelming evidence, they had to rethink the study based on the documented factors that correlated with success.

First, they found that a lot of software projects –nearly a third! – fail entirely. As in, you spend a lot of money and get nothing back for it. (See this link for more information.) This is sort of scary and I hope word doesn't get out about this little fact about software production.

Think about that for a moment. If one third of all software projects fail (or did back when the study was done, anyhow) then that means that every software project you do already costs you one third more than you thought, even if you come in on time and on budget. Okay, I admit that's bad math. In reality, it's the big projects that fail, so it's probably closer to half or more if you're a big company with big projects and a lot less if you're a smaller company with smaller projects. But you get the point.

The Chaos report further showed that over half of projects will cost 189% or more of their original estimate. Imagine if that happened to your house when you built it.

“Hey, buddy, sorry about that. I told your house would be $200,000 – greater starter home by the way – but instead I'm going to have to charge you $378,000 for it. But don't worry, you can mortgage it and pay it back over 30 years. That's there what we in this industry call capitalization!”

What Were the Real Success / Failure Factors?

More interesting was the fact that the success and failure factors that we normally think about, such as having a good project team, didn't turn out to be that important to the project after all. Here is the list they found in the study:

Project Success Factors % of Responses

1. User Involvement 15.9%
2. Executive Management Support 13.9%
3. Clear Statement of Requirements 13.0%
4. Proper Planning 9.6%
5. Realistic Expectations 8.2%
6. Smaller Project Milestones 7.7%
7. Competent Staff 7.2%
8. Ownership 5.3%
9. Clear Vision & Objectives 2.9%
10. Hard-Working, Focused Staff 2.4%

Other 13.9%

Apparently a competent staff only mattered 7.2% of the time. You get another 2.4% if they are hard-working and focused. That's a whopping 9.6% of the time that the development team ended up causing the success of the project.

What really amazes me is which members of the extended project team really mattered: End Users (15.9%) and Executive Mismanagement (13.9%).

But it gets better. It turns out that the process you used (Run for the hills! He's talking about process!) mattered more than the development team!

If your “process” included a “clear statement of requirements,” you score an extra 13.0%. If it included that little thing called “planning,” you get another 9.6% (that's equivalent to the entire development staff under best circumstances!). If you have “smaller project milestones,” score another 7.7%.

So apparently “process” does matter and maybe even more than the development team. Counter intuitive? You betcha ya!

It’s Not the Development Team That Matters!?

So, as it turns out, people are what matter the most, but it's not primarily the “developer-people” that cause projects to succeed or fail.

How can this be? Since when does a project get determined by the people not doing the work on the project? Welcome to the topsy turvy roller coaster ride that is software projects. Buckle up and grab the barf bag. You're going to need it.

What are Water Breathers? by Bruce Nielson

In Robert Glass' excellent book, Fact and Fallacies of Software Engineering, one of his “facts” (I.e. Qualified opinions) if that “Most software estimates are preformed at the beginning of the life cycle... before the requirements are defined and thus before the problem is understood.”

He goes on to say that, unlike many of his “facts” in the book, there seems to be no controversy on this one; there seems to be general intellectual agreement that we lock in on estimates too soon. This being the case, he says, “Someone should be crying out to change things. But no one is.”

He then adds:

At the very least, you need to know what problem you are to solve. But the first phase of the life cycle, the very beginning of the project, is about requirements determination. ... Put more succinctly, requirements determination is about figuring out what problem is to be solved. How can you possibly estimate solution time and cost if you don't yet know what problem you are going to be solving?

Why No Outcry?

While I agree the basic sentiment behind what Glass is saying, namely that we do lock in on estimates far too soon, I can't help but feel that Glass has missed several important points. In future posts, I'll contrast the approach Glass is advocating to my own approach to software estimation and demonstrate that Glass is actually misunderstanding certain critical points about software development and estimation. He’s also confusing the subjective nature of a so-called “problem space.” 

But at the moment, I want to look at a different aspect of this paradox that Glass is uncovering: if everyone agrees software estimates are locked in too soon, why isn't there a general out cry for change?

Glass holds this lack of outcry up as something strange to behold. Why would we all, time and again, slit our own throats like this? Surely we're smart enough to change blatantly stupid behavior like this!

Human Psychology and Software

In my opinion, Glass has fallen into the trap of overlooking the overriding importance of human psychology as it pertains to software solutions. The problem of “premature software estimations” can't be fixed by simply pointing out that the estimates will be bad at that point and failing to address the real problem.

Glass has failed to realize that upfront software estimates are what I call a “water breather.”

Imagine that you angered the mafia. The Godfather has you caught, puts you in cement boots and drops you into the Brooklyn river. There you are, holding your breath at the bottom of the river. You are starting to pass out. This is it, you're going to die.

Why not go ahead and see if maybe you can breathe water?

Sure, normally trying to breath water is a bad idea and I don't recommend it. But if you are in the situation above, you really might as well. You're dead anyway and there has got to be at least some very (very very) slight chance that you happen to have a genetic adaption that you didn't know about until this point whereby you can breathe underwater. Since the risk of death by drowning yourself has been eliminated, it really is time to try to breathe water. Go for it.

Of course you're probably not genetically adapted for water breathing and you're probably going to die. But you were definitely going to die if you didn't try, so what the heck.

As silly as this example seems, its actually quite logical. And it explains quite accurately the real reason we lock in on premature software estimates over and over again despite overwhelming evidence that they don't work.

Business Plans Are Part of Reality

What I am trying to say, is that Glass is ignoring all together the very real need – overriding need – to make business plans. Once we factor in this overriding need, upfront software estimates make perfect sense. Sure, they don't statically work. But if you're only other alternative is to never do the project in the first place – and I contend that is almost always the case – of course you're going to make an upfront estimate and go for it. Why? Because it's the single most sensible thing for you to do under that circumstance.

And actually, this is better than water breathing. In water breathing, testing your genetic adaption to breath water only buys you another faction of a second at the most. But when it comes to upfront software estimates, making a bad estimate buys you months or even years to come up with good reasons why the estimate was “right, except things changed” and convince everyone (often including yourself) that you need a “minor” change of plans or maybe a “follow on project.”

In short, “premature software estimates” are here to stay because they make rational sense compared to the next best alternative. People innately “get it” and do what is in their and often their company’s best interest: they go ahead and make the premature estimate and roll the dice and then start working on their best excuses. I just can't argue with this logic and I certainly wouldn't want to see it changed or “fixed” until we really do have a better alternative available. 

Tools or People? by Bruce Nielson

What matters more to the outcome of a software project? Good tools or good people? Okay, that one is a no brainer, right? We all know people matter more than tools, right?

But try this thought experiment. Take an average group of programmers today and set them up in a contest with the brainiest programmers of 1000 years ago. Who wins?

No, no, no, that's not right because computers didn't exist 1000 years ago. But isn't a computer just a tool? So apparently tools matter in some cases.

So let’s make this more fair by flashing forward to, say, the 1950s and retry the thought experiment. Again, the average programmers in 2009 will win hands down on a productivity basis. They can make more complex programs using Java or .Net then the 1950s programmers could ever dream of. So again, tools do appear to matter more than we given them credit for.

The problem is that the “tools vs. people” dichotomy we talk about doesn’t really exist. Apparently, what we mean when we say that “people are more important than tools” is that given roughly modern tools, one new snazzy tool isn't going to make much of a difference and it's certainly not going to cause your productivity to go through the roof. However, over longer periods of time, changes to paradigms, as supported by tools, can and will significantly increase a programmer’s productivity.

Tools allow a person to be more productive by changing the level of abstraction the programmer is forced to think at. 1950's programmers had to us machine language or punch cards or, gulp, both. No matter how smart they were, they were limited in their productivity by the tools. Today's programmers use modern compilers that allow code in something similar to English, virtual machines, automatic garbage collection, built in error handling, build tools, automated unit tests, etc. Throw in there a nice GUI-based drag and drop interface and now you're talking real productivity gains. So tools matter and they matter a lot.

With each new tool paradigm, we get closer and closer to the way a real human thinks. When we are dealing with punch cards and machine language the amount of translation between what the programmer is able to think and the way to implement it is huge. With modern tools, the gap has narrowed and thus productivity has climbed to match.

Now if you followed my argument above, we have uncovered that real source of productivity in software – a programmer’s ability to think! The basic unit of productivity in software has always been a “thought unit” (whatever that is.) No matter how good your tools get, the best programmers will always be the clear thinkers that require less thinking to accomplish their goals.

Tools are used to make thinking easier and more human-like and nothing more. So it turns out tools are really just a way to remove obstacles that naturally exists between the programmer's ability to think and the computer's need for formalized instructions.

Software Psychology Blog

Nate has generously allowed me to add a new blog to his site. My topics are going to be a bit different, however. I am not interested in code so much as the human factors in software development. I will do a series of posts about such topics.

Hope to talk to you all soon.

Bruce Nielson