Talk About Quality

When software development has a Lives of the Greats book, a textbook that reads like a prime-time mystery series, and even a children’s picture book, we have finally come of age. Here are the books, and my reviews. Happy reading, and onward to the next age!

• Coders at Work, by Peter Seibel (review by talkaboutquality)
• If I Only Changed the Software, Why is the Phone on Fire? by Lisa K. Simone (review by talkaboutquality)
• I am a Bug, by Robert Sabourin (review by talkaboutquality)

Automated testing is essential for test-driven development, and for regression testing. But remember:

1. If you have a system that runs automated tests, look at the results. Leaving it on “FAIL” won’t help.
2. Supplement with adversarial testing, to catch what only the human eye sees

Still not sure?

Okay, what test was ignored or missed before releasing this context-sensitive advertising software?

(all relevant credit to The New York Times)

Google’s “Mr. Automated Testing”, Miško Hevery, talks about Software Testing Categorization. From the title, it doesn’t sound like much, but it prompted a lot of good discussion. Some excerpts on key points, edited for length and clarity:

Know what you’re talking about

You hear people talking about small/medium/large/unit/integration/functional/scenario tests but do most of us really know what is meant by that? — Miško Hevery

Need for speed

Let’s start with unit test. The best definition I can find is that it is a test which runs super-fast (under 1 ms) and when it fails you don’t need debugger to figure out what is wrong. — Miško Hevery

Make slow tests faster by running them in the background on spare machine cycles. Works for static analysis tools too. For example, Riverblade’s Visual Lint add-on for Visual Studio and Gimpel Software’s PC-lint. — Talk About Quality

I heard a senior test manager present at a conference who espoused the sophistication of his automated test regime (on a mainframe no less). Countless tests ran every evening, unattended. Lots of clapping in the audience. In the break I met a guy who worked for the presenter. He asked, “do you want to know why the tests run so fast? We took out all the comparison checks”. — Paul Gerrard

You’re right to have a healthy concern. Whether automated tests run in a millisecond or in several hours, they always have to be re-reviewed to see if they test something useful. Passing tests are especially dangerous. Everyone thinks green is great so they don’t look at the tests to discover that features have moved on and the tests don’t test anything. Automated tests are like automatic shift (does anyone still drive stick like I do?): very nice and we take it for granted, but when you press on the gas you still have to look where you’re going so you don’t cause an accident. — Talk About Quality

Test Planning applies to automated testing too

At my previous employer, the standard unit tests took upwards of an hour (for fewer than 2000 tests). This discouraged developers from running them, which resulted in broken builds for everyone. When we worked on fixing this, mock objects provided most of the salvation. However, a good portion of the tests were really integration tests and by correctly identifying them as such, we were able to remove them from the standard suite, and instead run them nightly. What Misko is missing from his post is how test classifications go hand in hand with scheduling tests. — TheseThingsNeedFixed

Thanks to Bulkan Evcimen for tweeting about an article I doubt I would have found otherwise: “A Genetic Programming Approach to Automated Software Repair” (pdf link here), by  Stephanie Forrest et al. They describe there how they fed a genetic programming algorithm the following inputs:

1. Source code with a known defect
2. A negative test case
3. Several positive test cases

The authors show how software with the genetic algorithm is capable of repairing the defect such that the software under test no longer fails, while it still passes the positive test cases. They complete the activity with additional software that reviews the change and reduces its scope to the minimum.

Is this the end of the maintenance programmer? If so, this time there would be a happy ending: John Henry lives, and goes on to be a railroad engineer.

Consider a function in software code, and how to write it properly so that it meets its requirements and doesn’t fail. The standard way of looking at it is that a function has an API — a signature — of input and output variables and types. The code for the function has a pretty standard form. There are checks for invalid input, and a function body which implements a mini-flowchart of conditional checks, and maybe a state machine if needed. All this to take every given input vector to the correct output.

Unit tests, whether they’re written before coding (test-driven development), or after, apply test input vectors to the function and check if the expected output is produced. The same story applies to a module: a collection of functions or classes that implements a feature. There are still valid inputs to check if they give correct output, and invalid inputs that must be rejected. Best-practice test design methods, such as boundary-value checking and equivalence classes, help the developer choose the best test vectors from the sea of possibilities. The ones that will produce good tests that cause failures early, while the code is still in the hands of the developer.

So why do we still have so many “surprise” failures? Unit-tested code that nevertheless comes back from system testing with failures? And the Steps to Reproduce — so simple — like, “I left the system running overnight and when I came back in the morning it had crashed.”

The answer may be in the flow of data. Functions and modules in embedded software don’t just deal with single inputs one by one. Generally they are expected to process streams of data in real time. For example, a modern television set-top box, with a built-in disk, has to process multiplexed video data and metadata, the same from the hard disk, as well as a much slower stream of user input via the TV remote control. If the code slips up, or leaks memory, sooner or later, you get a wrong, stuck, or crash situation that QC proudly reports.

Where have we seen this before? How about those robot car competitions that university engineering departments often hold? Contests which have expanded outward to worldwide challenges to design, for example, an auto-piloted car. (See “The DARPA Urban Challenge“). These contests demand software that will keep a driverless car on track for long periods of time, avoiding stationary and moving obstacles (other cars).

Could it help, then, to see a lowly function in embedded software not as a flowchart of if-then-else statements, but as a little car — or perhaps delivery truck– that must be kept on track, avoiding obstacles in the data stream while delivering packages to the right place? How would that view affect code layout, code review, and unit test design? See you at the track!

This week, I moved from a general process improvement group to a software development department, to specifically focus on code quality. The most mature tool and process I left behind was a defect tracking system. Of course, in our development department, we use that defect tracking system. In case anyone asks me what it’s for, here’s my answer.

Prevent Errors, Fix Defects, Describe Failures

There are many definitions of those terms, but since when we fix problematic code we close “defects” in the “defect tracking system”, I started with that word in the middle. Leads to these definitions:

Error: The mistake someone made. (The root cause is the earliest error in the chain.)

Defect: The wrong code (missing, extra, or incorrect).

Failure: The unacceptable behavior of the code when running.

Detect each as early as possible

Each can be detected with appropriate tools. Detect earlier to minimize cost and annoyance.

Meanwhile, do it right

How?

With tools, learning, and discussion for writing good code.

When I park my (manual-shift) car on a slope, pointing downward, I set the parking brake, and I leave it in reverse as an extra safety measure. Of course that means that when I’m ready to start out again, I have to remember to take it out of gear (as well as releasing the brake), otherwise the car just jumps backwards and stalls.

Today I was reading Joel Spolsky’s “tips for reading other people’s source code”. In spite of the title (Reading Code is Like Reading the Talmud), you don’t have to have studied Talmud to get his point.

You also don’t have to conclude that his point is universally applicable.

Yes, when you have to work with undocumented, unclear code from an unavailable 3rd party, you might have to reverse engineer it to figure out how it does what it does. Or even just what it does.

But when you’re writing the code, especially as part of a team, there’s no reason to write it so that someone else will need to reverse engineer it. Design well, and write clearly, so that you and others will be able to understand it easily and modify it quickly.

Then everyone can move forward without delay.