Nate's Stuff

While I am a big fan of unit testing I often try to point out that Code Coverage tells us little more than “are there unit tests” not “are we unit testing”. The former indicates that the code is indeed being executed. The latter indicates that the code is being exercised.

Scott Hanselman describes this in a podcast available here:
http://www.hanselman.com/blog/HanselminutesPodcast103QuetzalBradleyOnTestingAfterUnitTestsAndTheMythOfCodeCoverage.aspx

I personally think testing really needs to be evolved to the level our architecture and code. Testing frameworks are clumsy and it is very difficult to all of our code. Most people are quick to point out that unit testing UI is difficult and usually yields the least amount of benefit. Depending on the architecture and implementation of that specific application, that may well be true.

Two other major hurtles that are well known. First, when we unit test we want as “authentic” of a test as we can get. This includes the data. How much time and effort do we spend getting authentic data to our unit tests or providing an abstracted data layer for our code to operate on? This is a big problem not just because of the hours it takes to provide such data abstraction implementations but because the time it takes to run Unit Tests is greatly increased. I argue this because I believe that when unit testing takes 30 min’s to complete, developers are not going to run them before checking in, and that I believe is an issue.

The other major unit testing hurdle is the concept of the “Oracle” as described by Scott. This term simply refers to the ability of a piece of code to assert that a test has indeed run in such a way that an end user is going to get what they expect. Some of these sceneries can be very complex and it is impossible to test for all permutations of a given rule.

What do I think is needed in Unit Testing?

1. The disparity between code and “UI” needs to be removed! It needs to be no harder to test a piece of UI then a API Class Library.
2. Unit testing “language” needs to be more declarative and functional. I need to be able to test a wide variety of test scenarios quickly and easily. Functional languages are good at this. I do not suggest that we adopt Python for unit testing but maybe a testing language developed for this purpose.
3. There needs to be a healthy attitude toward testing in the industry. I watch every day at any place I go where people “fix” the code in production. If they had better testing before the code went to production they could have easily caught these errors! A production line is down for an entire day, but if you ask -- no one can afford
4. There needs to be more automation in Unit Testing. For example, can’t the computer automatically generate tests for the edge cases? Coulden’t it generate input that would “execute” every branch of code possible? It may be of marginal usefulness but it would be free.
5. Unit testing needs to run faster! I have long mentioned the need to use multiple threads to execute tests – I even modified a release of nUnit to run on the ThreadPoool, but these need to be adopted and standardized
6. There needs to be a better way to test with real data.

Of course, the best sets of testing will not eliminate all bugs.  In fact, you must adopt your own testing philosophy when in charge of a development effort.  The question you have to ask yourself is two-fold. First, How important is it that my code has very few bugs.  Would a bug in your software, if unchecked, cost your company a lot of money?  Could a potential bug cost the company thousands, millions, billions, or even someone's life?  If so, then your testing should reflect that.  It's good risk management to spend 2x or more the cost of software development to avoid this type of costly loss.  On the other hand, if software has little consequence other than user annoyance when a bug is present then perhaps less of the development dollars need go toward testing.  In some sceneries it is completely fathomable that none of the software budget is spent on automated testing. 

The other question you will need to answer is will adding automated testing to my project yield a return on investment?  Before you make a snap judgement on this second point I may add that systems that have good testing in place (automated testing in particular) have a much cheaper lifecycle than applications that do not have this type of safe-guard.  Automated testing is a lot like life insurance in this way.  Having critical pieces of your code unit tested will save your bacon almost guaranteed!