Talk About Quality

When software development has a Lives of the Greats book, a textbook that reads like a prime-time mystery series, and even a children’s picture book, we have finally come of age. Here are the books, and my reviews. Happy reading, and onward to the next age!

• Coders at Work, by Peter Seibel (review by talkaboutquality)
• If I Only Changed the Software, Why is the Phone on Fire? by Lisa K. Simone (review by talkaboutquality)
• I am a Bug, by Robert Sabourin (review by talkaboutquality)

What kind of tool is required for code review? What are we trying to accomplish? How much of the code must be reviewed at a time, and why? These are some of the questions that we get caught up in, that prevent us from deploying effective code review in a development organization. One way to cut through the arguments and move forward is to realize that there are different types of code review, with different goals. Here are some parameters of two major, but different types:

Code Review type 1

Key question: Does this change accomplish what it was supposed to, while doing no damage?
Desired code author response to comments: Revise changes
Under review: changed files with diff
Tool: diff comparer
Required feature as a code review tool: right-click submit comment
Nice-to-have features (but could be in a separate app): right-click definitions

Code Review type 2

Key Question: Is this code designed well, so that it is easy to fix and extend?
Desired code author response to comments: Refactoring / renaming
Under review: any subset of entire codebase, starting from a given area of the code
Tool: source code browser
Required feature as a code review tool: right-click submit comment
Nice-to-have features (but could be in a separate app): code editing

As good as the misconceptions surrounding Toyota make it sound, the truth is even better.

– Stewart Anderson in Quality Digest’s Quality Insider 20-Oct-2009

Thanks to Stewart Anderson, both for bringing us the real story about how Toyota, famed for focus on quality, actually goes about doing it. Put aside for a moment what you know about ISO, QMS, metrics etc and read the carefully crafted wisdom of experience. How quality really works at Toyota. And yes, you really can apply these ideas elsewhere, even in software development.

All quotes are from the linked article by Stewart Anderson.

Toyota’s basic pattern for improving a process is based on a simple three-part model:

1. Understanding the current condition.
2. Developing and defining a target condition.
3. Understanding and tackling problems which need to be overcome to move from the current condition to the target condition.

This model has learning at its heart [to identify] actions to solve problems in the current condition.

The primary responsibility [of the team leader] is to monitor the process, ensure that standard work is being followed, and coach and mentor the work team in improving the process. Team leaders receive special training in process improvement and problem solving ….

To read the full article click here.

Related reading: Gemba Kaizen: A Commonsense, Low-Cost Approach to Management (my review here).

Full disclosure: My Toyota is over 15 years old and still running fine. My last car was also a Toyota.

Automated testing is essential for test-driven development, and for regression testing. But remember:

1. If you have a system that runs automated tests, look at the results. Leaving it on “FAIL” won’t help.
2. Supplement with adversarial testing, to catch what only the human eye sees

Still not sure?

Okay, what test was ignored or missed before releasing this context-sensitive advertising software?

(all relevant credit to The New York Times)

Jesse Gibbs at Atlassian sent me to the following post from Scott Bilas at the game software company Loose Cannon Studios. In Peer Code Reviews: Good Commenting Practices, Bilas says code reviewers should seek architectural issues, and adherence to good software development practices and coding standards. And that they should look for mentoring opportunities. At the same time, he lets them off from other responsibilities, saying “Reviewers aren’t expected to catch everything,” and, “Reviewers aren’t expected to catch deep or systemic design problems.”

It’s a pretty good tutorial on the current best practice of code review. So why does it feel like something big is missing?

Upon reflection, it turns out that, unlike, say, editing and commenting on a book, code review is not really a reviewer-author relationship at all. An editor may make a lot of changes, but those changes end with publication. Code, on the other hand, will be fixed, extended, and refactored by future developers. Each future, or “next” developer will need exactly two things from the code in order to do his or her job: readability and understandability. Code is readable if it has the right balance of abstraction vs. detail at each level of the code. Code is understandable if the reviewer can, without help from the author, see what the code does, and why. If code is clear, that is, readable and understandable, modifications will be easy and error-free.

So the question “what are the responsibilities of a code reviewer” turns out to be a trick question. Like this one, from driving class:

Question: In the following traffic diagram (imagine any diagram you want), who has the right-of-way?

Answer: Nobody has the right-of-way. Right-of-way can never be possessed, only given.

The code reviewer does not have any review responsibility. The code author has all the responsibility to write clear code, and provide it to the code reviewer. We should stop calling that person a “code reviewer”; instead, say “code user”.

Code review becomes a simple and easily explained two-step process. First the “code user” reads the code and notes down what s/he doesn’t understand how to use, and second, s/he meets with the code author to present what was not clear. In that manner, the “code user” successfully stands in for the “next developer”, and gives the code author an early chance to make things better. As a bonus, all the things a traditional code reviewer was supposed to check, or at least the important things, will be checked. Mentoring will happen. All this when we realize that code review is code use.

Someone comes to me with a problem that sounds like,

“I wanted to do X but it doesn’t work.”

When I hear that request, often they know a lot more about the technical details than I do. I need them to teach me in a way that lets me put things in order. My method starts from success … and progresses to failure. Strange? But it works.

The questioner is focused on the failure scenario. I crave success. There must be some around — just have to find it. I ask what similar scenario succeeds. What version of the file did work? What build compiled successfully? When did this test last pass? As we discuss the successful scenario, I learn why it worked. What requirements were met by that configuration. I repeat those requirements out loud for confirmation.

With a success in hand, well understood, we have a checklist to apply to the failed configuration. We can go over the failure and check whether all the things that made something similar work are present. Of course, one will be missing. We have found the problem. I may not know how to fix it. But usually the questioner does.

Google’s “Mr. Automated Testing”, Miško Hevery, talks about Software Testing Categorization. From the title, it doesn’t sound like much, but it prompted a lot of good discussion. Some excerpts on key points, edited for length and clarity:

Know what you’re talking about

You hear people talking about small/medium/large/unit/integration/functional/scenario tests but do most of us really know what is meant by that? — Miško Hevery

Need for speed

Let’s start with unit test. The best definition I can find is that it is a test which runs super-fast (under 1 ms) and when it fails you don’t need debugger to figure out what is wrong. — Miško Hevery

Make slow tests faster by running them in the background on spare machine cycles. Works for static analysis tools too. For example, Riverblade’s Visual Lint add-on for Visual Studio and Gimpel Software’s PC-lint. — Talk About Quality

I heard a senior test manager present at a conference who espoused the sophistication of his automated test regime (on a mainframe no less). Countless tests ran every evening, unattended. Lots of clapping in the audience. In the break I met a guy who worked for the presenter. He asked, “do you want to know why the tests run so fast? We took out all the comparison checks”. — Paul Gerrard

You’re right to have a healthy concern. Whether automated tests run in a millisecond or in several hours, they always have to be re-reviewed to see if they test something useful. Passing tests are especially dangerous. Everyone thinks green is great so they don’t look at the tests to discover that features have moved on and the tests don’t test anything. Automated tests are like automatic shift (does anyone still drive stick like I do?): very nice and we take it for granted, but when you press on the gas you still have to look where you’re going so you don’t cause an accident. — Talk About Quality

Test Planning applies to automated testing too

At my previous employer, the standard unit tests took upwards of an hour (for fewer than 2000 tests). This discouraged developers from running them, which resulted in broken builds for everyone. When we worked on fixing this, mock objects provided most of the salvation. However, a good portion of the tests were really integration tests and by correctly identifying them as such, we were able to remove them from the standard suite, and instead run them nightly. What Misko is missing from his post is how test classifications go hand in hand with scheduling tests. — TheseThingsNeedFixed

A new project is ramping up and needs more resources. Another project is winding down — maybe it can do with less.

Let’s move some people from one team to the other

What’s wrong with this common resource allocation decision? It’s not the decision. Moving resources to the project that needs them makes sense. What’s wrong is the words, which hide a false assumption.

Here, “team” means location. Move people from one place to another. Like conductors putting people on a train: the “teams” are two train stations. One called “Project A Station” and the other one, “Project B Station”.

But teams are not train stations. A team is not the room it sits in, or its box on the org chart. A team is the particular combination of people — their skills and personalities — and their shared experience. They know their project or product best right now, and equally important, they know each other and the special role each person fills. Reassigning people from one team to another is more than moving a person. It is actually dismantling the first team, and disturbing the second. The movement of people based on the train-station model of teams leads to a contradiction: both teams — origin and destination — cease to exist.

If we must model people and treat them as resources, a better model is a hand at cards. The managers are the players and partners; the game is getting more projects completed. There’s a fixed deck — the “human resources” of the company. Some cards may be similar in strength or level, but every card is unique.

The value of your hand is based entirely on the combination of cards you’re holding right now. If you or your partners need a stronger hand, everyone knows that some cards will have to be exchanged. Put down 2, pick up 2. I pick up what you put down. Resources. Cards. I know this doesn’t sound any better to workers than “counting heads”, but it is. Because every card player thinks twice before putting down a good card, and prefers to take over an entire good hand if they can.

It’s pretty well known that there are three ways to improve software quality: improve process, improve methods, and improve tools. But these are the generic concepts — applicable to any software development effort. Today’s topic is more specific and closer to home. Product quality.

Isn’t it obvious? The software product is the running software, and product quality is how well it meets its requirements. Simply put — does it do what people expect it to do, and not crash or lose data?

But organizations which release software focusing only on that one aspect of product quality are often unpleasantly surprised by inconsistency. They are doing all the right things — following a good process, using good methods and tools, and regularly reviewing all those to see what can be improved. Yet sometimes the product is better, sometimes worse. Why?

External product quality — in software, that’s how well the software serves its users when it’s running, is not enough. There’s also internal quality. The readability of the codebase. Code that’s well-written — easy to understand quickly — is easier to maintain. Fewer programmer errors, and fewer schedule slips. We can add to that clear and organized debug output, which avoids false positives, and gives an accurate record of when the running software seemed to be fine, but was really operating on the edge — just barely recovering from failures.

But there’s another part of the product that’s often overlooked. Overlooked when reorganizing teams, or just when parceling out the daily assignments from multiple projects. The developers. If you take a single iteration of a software product, with development at its best, you’ll apply best process, methods, and tools to put out the next version of the software, and maybe even have codebase improvement (cleanup, refactoring) on the list too. Get to release and you deliver good running software, more maintainable code and … the new state of the team that developed it.

The team members may have learned new methods or technologies. Or just how to avoid pitfalls from the past. Equally important, that particular team learned how to work together — not in general, but specifically in today’s process, methods & tools, and codebase. And with each other. This third part of the product cannot be ignored or thrown away, just as nobody would think to delete a code branch delivered to the customer, or switch out a working library in mid-project.

Looking further at this third component of the product — the team that delivered it — we see clearly that those quality improvement concepts of process, methods, and tools do not exist in a vacuum. They are not just abstract ideas. Rather, they exist in the team members who develop a product, and are unified by teamwork — which means this team learning to work together with these people on this product.

So when you think of product quality improvement, plan to improve all of these:

1. How the product works
2. How the codebase looks
3. How the team works

And at the end of an iteration, make sure to review and protect all three. That’s the guarantee that the next version of the product will be even better.

Thanks to Bulkan Evcimen for tweeting about an article I doubt I would have found otherwise: “A Genetic Programming Approach to Automated Software Repair” (pdf link here), by  Stephanie Forrest et al. They describe there how they fed a genetic programming algorithm the following inputs:

1. Source code with a known defect
2. A negative test case
3. Several positive test cases

The authors show how software with the genetic algorithm is capable of repairing the defect such that the software under test no longer fails, while it still passes the positive test cases. They complete the activity with additional software that reviews the change and reduces its scope to the minimum.

Is this the end of the maintenance programmer? If so, this time there would be a happy ending: John Henry lives, and goes on to be a railroad engineer.