Software Psychology : Design, Software Develoment Psychology

Software is Not Coding, It’s A Cooperative Game of Communication by Bruce Nielson

BruceNielson — Mon, 20 Sep 2010 06:04:25 GMT

In my post on “code is really design” post, I mentioned that I would further address the paradox that software is created in “thoughts units” but the most important people to the success or failure of a project are the sponsor and customer. In this post, I’ll explore that paradox further by discussing Alistair Cockburn’s idea of Software as a cooperative game.

In Agile Software Development, Cockburn says:

Software development is therefore a cooperative game of invention and communication. There is nothing in the game but people’s ideas and the communication of those ideas to their colleagues and to the computer.

In Crystal Clear: A Human-Powered Methodology for Small Teams, he writes:

[Successful teams] view developing software as an evolving conversation – an economic-cooperative game, actually. In any one move they go forward and create reminders for each other.

But in what sense is software a game and how does this help us understand software development better?

I’ve never been fully comfortable with Cockburn’s calling it a “game.” I’m not sure images of Monopoly and Halo (which is what comes to my mind, anyhow) is what he intended.

It’s helpful to refer to other non-software examples that Cockburn uses. One of my favorite of his examples is making laws in a legislature.

Can You Estimate How Long it Will Take to Pass a Law?

Imagine if the republicans asked the democrats, “How long will it take you guys to pass that new health care law of yours?”

Now in theory, the democrats should be able to estimate this. It’s just a bunch of words on a pieces of paper, right? Someone has to sit down to write it, they have to talk to experts, they have to run a few numbers and come up with a budget, then they have to pass it through a committee, then bring it to the floor, and finally get a vote. So the question the republicans are asking seems reasonable on the surface.

But in reality, part of what the democrats have to estimate is how long it will take them to convince enough republicans to pass a version of the bill. If this step were unnecessary, making an estimate would probably be easy. But if the democrats need to actually convince some key republicans to vote for the bill to get it passed, then part of their estimate will have to be an estimate for how long it takes to keep revising the bill until the republicans accept it – if ever.

Once we realize that is the case, the republican’s question seems less legitimate. The Republicans have a significant say in determining when the bill is “done” and the democrats don’t actually control that part of the process. The democrats cannot, even in principle, make an estimate for how long it will take to pass the law.

I see software much like this. We have customers and sponsors on the one hand and the development team (programmers, testers, designers, project managers) on the other hand. But instead of capturing ideas into specific laws to be interpreted by other humans, they are cooperating together to capture business rules into an automated programs that will be interpreted by computers.

There is nothing else going on when we make software. There is no analogy here to building a house or a wall, or anything else physical. When my customers ask me “how long will it take to program the fitz-bart module?” I am literally incapable of answering them because part of my estimate would have to be how long they will choose to iterate before they accept the software as matching their needs. There is literally no limit to how many times they might choose to keep tweaking the requirements and I’ve been in cases where they never become satisfied with it.

If I didn’t have to worry about customers having to accept the software, estimates would be a lot easier. But with customer acceptance required, it’s impossible to estimate anything beyond the first iteration of the creation of the module.

An Alternative – Cooperation

Now consider an alternative example. Lets say that both the republicans and democrats both are anxious to pass a health care bill and they both have more or less synchronous goals. Now is it possible to estimate how long it will take to pass a health care law?

Absolutely!

Why? Because now both sides are working together to meet a target date. This means both sides are motivated to make the date and, more to the point, both will be willing to accept some compromises to meet that date.

Software works the same way. If the customers/sponsors and the development team both want to make a certain release date or a specific budget target, there is almost always a way to come up with a workable (though usually manual, and thus less than ideal) solution to any software problem such that you can meet those goals. It just requires both sides to agree that they will make the target date or the target budget and to be willing to negotiate on scope until they match those goals.

The Law of Lossy Requirements by Bruce Nielson

BruceNielson — Thu, 21 Jan 2010 19:15:00 GMT

Lossy Compression

In computer science, compression is an indispensible tool. Anyone familiar with .zip files knows what I mean. Interestingly, there are two kids of compression, lossless and lossy. Lossless compression is like .zip compression, you put a file of, say, 100kb in and the end result is a file of, say, 50k. But when you reverse the process, you get back your original 100kb file.

Lossless compression relies on the fact that in real life information contains patterns. Just imagine taking this post and finding the most common used words in it. Then replace the most common word with a the tag “<1>” and the next most common with “<2>”, etc. The end result would generally be a smaller file, yet by replacing “<x>” with the original words you could recover the original file in its entirety.

I used to wonder what possible use lossy compression could ever be. Why in the world would I ever want to save off a compressed version of my files if I couldn’t get them back to their original state ever again?

Of course for files, lossy compression is useless but for computer graphics it’s quite useful. Because the human eye is not capable of picking up all the details in an image it’s possible to reduce the details in the image without the human eye being able to tell the difference.

What’s even more interesting is that you can continue to increase that lossy compression on an image until the human eye can tell the difference, yet the brain will still be able to tell what the image is. It just looks a bit more blurry.

In fact, you can continue to compress and compress using lossy compression to any size. At some point you end up with a single solid color, of course, which isn’t useful, but even if you take a very large image – let’s say 1024 x 1024 pixels – and shrink it via lossy compression down to the equivalent of a 16 x 16 image, the human brain fills in the details and still makes sense of the image.

Requirements Are Lossy Compression

If Code is Design, one of the most under appreciated points of software development is that requirements are equivalent to lossy compression of graphics, only in reverse. (See also Tell Me What To Do, Not How to Do It for an example of this.)

What we call “requirements documents” (or even “design documents”) are really a blurry picture of what is to be built. In other words, requirements and design documents by definition do not contain all the details.

Why do we do this? There are actually many reasons, but here are two key reasons:

1. Until the software is actually built, it doesn’t exist. So there isn’t really a choice but to imagine it with broad brush strokes and then refine it into reality.

2. As was discussed in a previous post, human beings more easily understand abstractions rather than precision. If we didn’t have abstract and blurry views of the design, no one but the programmer would ever understand what is to be done. Actually, even the programmer would only understand it in chunks and not holistically.

The Law of Lossy Requirements

If you followed my argument so far, then The Law of Lossy Requirements will now make sense to you. This law states:

All requirements documents are a lossy compression of the business rules. If you made it non-lossy, the cheapest way to document it would be in code.

Therefore there are two corollaries to this law:

Corollary 1: There will never be a point where you know all details of what software is to be created until the software is completed.

Corollary 2: There is no such thing as documenting everything in the software because to do so would require synchronizing essentially a second copy of the code.

Code is Design by Bruce Nielson

BruceNielson — Sat, 02 Jan 2010 09:00:00 GMT

The Story So Far

Let’s return to the primary concern with Software. We’ve talked about how most software projects either fail altogether or run significantly over budget. Along with that thought, we considered the statistics that show that the development team itself plays little role in the success and failure of the project in comparison to users and sponsors. However, contrasting that we’ve looked at how software development is really developed in “thought units” more so then actual code.

These two facts seem on the surface to be at odds with each other. If software is really developed in “thought units” then why do users and sponsors have more control over the success and failure of the project then the developers?

Coding As Design

I will now explore this question further in a series of posts. But first, we need to look at the concepts of “software design” vs. “software coding”

Rumor has it that software is an engineering discipline and because of that, our industry adopted much of the vocabulary and process used by real engineers. This led to the creation of the both famous and infamous “waterfall methodology” where we pass through water tight stages that start with Requirement Gathering, move on to a documented Design, and then finally Construct what was designed, finishing off with Testing and then Release.

It’s all a bunch of posh, of course.

Actually, despite it’s recent (and partially deserved) bad reputation, the waterfall methodology has a lot to recommend it. To this day, I’ve never come across any methodology, Agile or otherwise, that doesn’t owe a great debt to the Waterfall methodology. So let’s not throw out the baby with the bath water.

However, what we software developers do is not really engineering in the true sense and probably never will be. This is why a strict waterfall approach, which serves so well in other engineering fields, has failed us miserably.

Jack W. Reeves classic article, “What is Software Design?” addresses this directly. (See also this link, for further information.) What Reeves correctly points out is that what we call “Software Design” is not a complete design at all, but is rather what he calls a “high level structural design” yet we treat it as if it’s some sort of complete detailed design (even often calling it “The Detailed Design”).

Now maybe you’re now thinking, “So it’s a high level design instead of a detailed design. So what, who cares?”

Yet, as I’ll show in future posts, this seemingly small distinction might just be the entire difference between understanding the real problems of software development and turning a blind eye to them.

But let’s let Reeves speak for himself. He says:

For almost 10 years I have felt that the software industry collectively misses a subtle point about the difference between developing a software design and what a software design really is. … This lesson is that programming is not about building software; programming is about designing software.

…the software industry has created some false parallels with hardware engineering while missing some perfectly valid parallels. In essence, I concluded that we are not software engineers because we do not realize what a software design really is

A program listing is a document that represents a software design. Compilers and linkers actually build [manufacture or construct] software designs.

…it is cheaper and simpler to just build the design and test it than to do anything else. We do not care how many builds [constructions or manufactures] we do—they cost next to nothing in terms of time.

[The] designs will be coded in some programming language and then validated and refined via the build/test cycle.

Is Programming a Form of Engineering at All?

Reeves questions if Programmers should be called Engineers if they don’t even realize that programming isn’t “manufacturing” or “construction” at all, but is really “detailed design.” But Reeves never quite questions if programming might not be Engineering at all.

But this leaves one uncomfortable. If our industry is full of programmers that do detail design and think they are doing “manufacturing” (i.e. “construction or building of a design”) how could they possibly misunderstand their own engineering discipline so severely?

Agile Maven, Alistair Cockburn, suggests how by questioning the very notion of “software engineering”:

Software development is not “naturally” a branch of engineering. It was proposed as being a branch of engineering in 1968 as a deliberate provocation intended to stir people to new action [Naur-Randell]. As a provocation, it succeeded. As a means for providing sound advice to busy practitioners, however, it cannot be considered a success. After 35 years of using this model, our field lacks a notable project success rate [Standish], we do not find a correlation between project success and use of tidy “engineering” development processes [Cockburn 2003a], and we do not find practitioners able to derive practical advice to pressing problems on live projects. (See link)

Elsewhere, Cockburn points out that the whole notion of “software engineering” was actually just an attempt to provoke less than obvious parallels between programming and true engineering fields. “The term ‘software engineering’ was coined in 1968, in the NATO Conference on Software Engineering,” Says Cockburn, “It is fascinating to read the verbatim reports of that conference and compare what those attendees were saying with what we have created since then.”

He then quotes from that conferences notes on the background of the conference:

The Study Group concentrated on possible actions which would merit an international, rather than a national effort. In particular it focused its attentions on the problems of software. In late 1967 the Study Group recommended the holding of a working conference on Software Engineering. The phrase ‘software engineering’ was deliberately chosen as being provocative, in implying the need for software manufacture to be based on the types of theoretical foundations and practical disciplines, that are traditional in the established branches of engineering.

It may well be time for us to abandon the very notion of “Software Engineering” and to base our development techniques off of some more natural match. Reeves agrees:

In many ways, software bears more resemblance to complex social or organic systems than to hardware.

Ramifications of Software As Design

Rethink, for a moment, how accepting “Coding” as really being “Design” changes the way we think of software development. The ramifications are substantial.

First of all, it means software development, being a design activity, will always change after the requirements gathering and high level design are “completed.” We would no more expect software to merely follow the “original design” (which we now know is only a high level design without all the needed details) then we’d expect Intel to design their next generation process only at a high level and then lock it in permanently. Any attempt to do so would be met with the same sort of frustration that plagues the software industry.

Another ramification is that it often makes sense to “get to coding” (i.e. detailed design) as quickly as reasonable. Contrary to popular belief, not all rushes to code are “hacking.”

Indeed, any methodology that encourages “design” (i.e. high level design) to be simultaneous with “coding” (i.e. detailed design) will be superior to one that doesn’t. This explains the popularity of methodologies such as “spiral,” “rapid prototyping,” and now “agile.”

Furthermore, testing software is actually a design activity as well. It’s how we validate the design details. Thus any methodology (a la Agile) that encourages testing during “coding” will be more successful than one that doesn’t.

Conclusion: It’s All Design

Wait! Did you read that right? Is it really true that design, construction, and testing phases are all really just design?

As Reeves so aptly put it:

The overwhelming problem with software development is that everything is part of the design process. Coding is design, testing and debugging are part of design, and what we typically call software design is still part of design. Software may be cheap to build, but it is incredibly expensive to design.