Talk About Quality

As good as the misconceptions surrounding Toyota make it sound, the truth is even better.

– Stewart Anderson in Quality Digest’s Quality Insider 20-Oct-2009

Thanks to Stewart Anderson, both for bringing us the real story about how Toyota, famed for focus on quality, actually goes about doing it. Put aside for a moment what you know about ISO, QMS, metrics etc and read the carefully crafted wisdom of experience. How quality really works at Toyota. And yes, you really can apply these ideas elsewhere, even in software development.

All quotes are from the linked article by Stewart Anderson.

Toyota’s basic pattern for improving a process is based on a simple three-part model:

1. Understanding the current condition.
2. Developing and defining a target condition.
3. Understanding and tackling problems which need to be overcome to move from the current condition to the target condition.

This model has learning at its heart [to identify] actions to solve problems in the current condition.

The primary responsibility [of the team leader] is to monitor the process, ensure that standard work is being followed, and coach and mentor the work team in improving the process. Team leaders receive special training in process improvement and problem solving ….

To read the full article click here.

Related reading: Gemba Kaizen: A Commonsense, Low-Cost Approach to Management (my review here).

Full disclosure: My Toyota is over 15 years old and still running fine. My last car was also a Toyota.

Jesse Gibbs at Atlassian sent me to the following post from Scott Bilas at the game software company Loose Cannon Studios. In Peer Code Reviews: Good Commenting Practices, Bilas says code reviewers should seek architectural issues, and adherence to good software development practices and coding standards. And that they should look for mentoring opportunities. At the same time, he lets them off from other responsibilities, saying “Reviewers aren’t expected to catch everything,” and, “Reviewers aren’t expected to catch deep or systemic design problems.”

It’s a pretty good tutorial on the current best practice of code review. So why does it feel like something big is missing?

Upon reflection, it turns out that, unlike, say, editing and commenting on a book, code review is not really a reviewer-author relationship at all. An editor may make a lot of changes, but those changes end with publication. Code, on the other hand, will be fixed, extended, and refactored by future developers. Each future, or “next” developer will need exactly two things from the code in order to do his or her job: readability and understandability. Code is readable if it has the right balance of abstraction vs. detail at each level of the code. Code is understandable if the reviewer can, without help from the author, see what the code does, and why. If code is clear, that is, readable and understandable, modifications will be easy and error-free.

So the question “what are the responsibilities of a code reviewer” turns out to be a trick question. Like this one, from driving class:

Question: In the following traffic diagram (imagine any diagram you want), who has the right-of-way?

Answer: Nobody has the right-of-way. Right-of-way can never be possessed, only given.

The code reviewer does not have any review responsibility. The code author has all the responsibility to write clear code, and provide it to the code reviewer. We should stop calling that person a “code reviewer”; instead, say “code user”.

Code review becomes a simple and easily explained two-step process. First the “code user” reads the code and notes down what s/he doesn’t understand how to use, and second, s/he meets with the code author to present what was not clear. In that manner, the “code user” successfully stands in for the “next developer”, and gives the code author an early chance to make things better. As a bonus, all the things a traditional code reviewer was supposed to check, or at least the important things, will be checked. Mentoring will happen. All this when we realize that code review is code use.

That’s right, not “development” but “developer”.

The latest issue of The Embedded Muse newsletter—#179 (pdf)—has a reference to an interesting and very different kind of book about C. It’s about the business pressures and the thought processes of C software developers. And a lot more. It’s also long—1600 pages! So download a copy and skim for what interests you:

The New C Standard: An Economic and Cultural Commentary

Some lines that caught my eye, just in the first 100 pages, skimming with the “Page Down” key:

“Software developer interaction with source code occurs over a variety of timescales [for example, those] whose timescale are measured in seconds.”

“The act of reading and writing software has an immediate personal cost.”

“Source code faults are nearly always clichés; that is, developers tend to repeat the mistakes of others and their own previous mistakes.”

“A list of possible deviations should be an integral part of any coding guideline.”

“The following are different ways of reading source code, as it might be applied during code reviews …”

Other related references from the book’s author Derek M. Jones (http://www.knosof.co.uk/):

• Blog: http://shape-of-code.coding-guidelines.com/
• Article:  Coding Guidelines: Fact and Fiction
• Article: The 7±2 Urban Legend (small pdf)

And somehow related, from another author, Les Hatton (http://www.leshatton.org/):

• Article: Testing the value of checklists in code inspections

A co-worker forwarded me this article ”5 Reasons for Software Developers to Do Code Reviews (Even If You Think They’re a Waste of Time)” which certainly sounded promising. Even when I don’t think code reviews are a waste of time. But as I read through it, it became clear that more is less. The article says too much, and detracts from its own message.

1. Developers know their code will be evaluated, so they work harder. “The most useful thing about a code review is the fact that the coder knows that someone is going to review the code,” says Oliver Cole, president of OC Systems and also lead for the open-source Eclipse Test and Performance Tools Platform project.

Work hard because you enjoy it. And of course your code will be evaluated, but not primarily by code review. Rather, the main user of your code is the “next developer”—possibly someone on your team, or even you yourself a few months later. That’s where the evaluation happens.

 2. It improves a developer’s own programming skills.
In your heart, you might not care that much about the success of this particular software project. But most programmers want to improve their personal skills, and that means learning from other people.

If you don’t care about the success of the project, code review won’t help.

3. It’s an opportunity for mentoring, so the programmers you work with get smarter (and thus, more fun to hang around with).” [...] While the intention [to mentor individuals] is generally well meaning, it can often lead to individual discomfort and perceived or actual criticism. In these cases, the greatest opportunity for mentoring usually exists in the context of small collaborative teams working together to realize goals and not in a code review.”

Criticism is not bad, it is essential. It is not personal, but professional. And of course, the smaller the meetings (down to even just 2 people — reviewer and author), the better.

4. It creates consistency and a culture of quality across the project. [...] Developers are quick to complain about being judged on the wrong metrics, but, says Gary Heusner, client partner at custom software developer Geneca, “We have to change the rules to allow for quality and efficient development to be valued over making milestones that are really yardsticks of process more than milestone of value delivered.” Code reviews are a big part of that.

Code reviews are simply part of good software development.  When management, together with the team, track value delivered, that is a big part of creating a culture of quality. Only when the environment is right can code reviews have a chance of being effective.

5. It encourages team bonding. “People think code review is just about finding bugs, but it brings people together, says Smartbear’s Jason Cohen. Often, he says, it can deliver far more than expected.

“Success stories happen very often when performing code reviews,” says Dave Katauskas, senior architect at Geneca. “But the best success story is the pattern that develops once a team has gelled. The longer you’re into a project, the better quality code is created. This is all due to the code review process and governance that occurred up stream in the beginning of the project.”

I had to read this one a few times. Right answer for wrong reasons. I will not credit code review where credit is not due. Even the writer with the “success story” realizes the true origin of the success is the gelled team.

But still, I had to click on the Jason Cohen link to see why code review “brings people together”. Go ahead—click below on “Lightweight Code Review Episode 5: Team Building for the Cold, Dark, and Alone”. But first, get ready to read it right: code review doesn’t create good teams. Rather, good teams benefit from code review. OK, now click.

Lightweight Code Review Episode 5: Team Building for the Cold, Dark, and Alone

“Isn’t it nice to know that, when all else fails us, we have an innate decision-making tool to fall back on?”

Robert L. Glass, “Intuition’s Role in Decision Making” (IEEE Software, January/February 2008)

Yes, Glass admits, estimates that come unbidden from a manager’s subconscious seem the very opposite of quantitative or rational. But most decision-making methods have a common theme: using historical data to decide what to do next. Quantitative estimation puts everything out in the open. Rational support for a desired deadline is at least based on the facts. Intuition, at the other extreme, pulls that data, informally known as “experience”, from the subconscious. And since a good manager has experience, intuition works.

Sort of.

What’s hidden by Glass’ essay are the unstated quality standards that justify each method. Only a complete, quantitative estimate, matched by continuous measurement and adjustment, can promise high quality by the target date. Intuition, on the other hand, even coming from an experienced manager, makes quality likely, but hardly guaranteed. His successes are what keep him in his job, and his few failures are forgiven. Rightly so. He is employed to deliver good enough software, quickly, to meet modern society’s ever-growing appetite for computerized life.

So should we favor intuitive, seat-of-the-pants estimating, with its benefit of early delivery, but its cost of uncertain quality?

Digging deeper, we find that there are some very different reasons why we write software, which strongly influence how we plan our work.

1. Profit
2. Functionality
3. Beauty

Anyone who gets paid for writing commercial software must acknowledge that profit pays his or her salary. Profit leads eventually to pretty good quality, via competition. In the short run, though, we experience a lot of pressure, a lot of errors, and many customer-accepted (if sometimes societally unacceptable) failures. In that context, it seems, we should estimate quickly and intuitively, but admit that quality is expendable.

In a well-funded, non-profit organization (e.g. NASA, famed for error-free software — wholly separate from failure-free flights), software can be about complete functionality. Take as much time as needed to implement everything, perfectly. After all, when the spaceship flies past Jupiter, there’s no second chance.

Where, then, is guaranteed perfection, estimated correctly and delivered quickly? The only place we see that in life is artistic performance. A pianist plays, from memory, a piece that she only decided a few months ago to perform. An hour long, no mistakes. The event starts, and finishes, on time. The customer — the audience — rises to its feet in noisy appreciation. The enabler of perfection is seeking beauty, or doing a thing for its own sake.

Many software developers write software because it’s beautiful, fun, or spiritually rewarding. And that reason engenders the highest-quality work. A quality that delivers on time, with no costly rework. Functionality. Profit too.

Somehow, in the commercial context, that reason why we write software must be harnessed and encouraged by software managers. Otherwise, dismissing a developer’s data with an intuitive estimate is not a fall-back, but falling backwards.

This week, I moved from a general process improvement group to a software development department, to specifically focus on code quality. The most mature tool and process I left behind was a defect tracking system. Of course, in our development department, we use that defect tracking system. In case anyone asks me what it’s for, here’s my answer.

Prevent Errors, Fix Defects, Describe Failures

There are many definitions of those terms, but since when we fix problematic code we close “defects” in the “defect tracking system”, I started with that word in the middle. Leads to these definitions:

Error: The mistake someone made. (The root cause is the earliest error in the chain.)

Defect: The wrong code (missing, extra, or incorrect).

Failure: The unacceptable behavior of the code when running.

Detect each as early as possible

Each can be detected with appropriate tools. Detect earlier to minimize cost and annoyance.

Meanwhile, do it right

How?

With tools, learning, and discussion for writing good code.

Usually I just write my book reviews on Amazon, but this book is important enough that I wanted to post my review here too. Dreaming in Code by Scott Rosenberg.

We’re Not in Kansas Anymore

And we haven’t been for a long time. To hear Rosenberg tell it, we briefly entered modernity in the 1950’s with computer hardware, and then quickly regressed into an ironic Greek tragedy of software, where all of us spend our time playing either Sisyphus, fixing endless “bugs”, or Tantalus, finding on-time delivery methods just beyond our reach. Ironic because in this story, Sisyphus is often happy.

Dreaming in Code is not about coding. Nor is it fiction. It is disturbing, realistic fact. An outsider’s diary of a software project, blessed with a successful visionary leader—Mitch Kapor of Lotus 1-2-3 fame—and almost no resource constraints, which nevertheless exhibits all the “usual” problems.

Rosenberg competently brings in software history and references all in the field will recognize. More important, though, he shows software to be one of those great advances that is fraught with deadly imperfections. Like, say, hospitals, or automobile travel.

To software people. You can keep your eyes closed, come to work, and enjoy dreaming in code. Or you can wake up for a day and read this book.

(An extra note, since I’ve read some of the reviews, including those on the back cover. Some miss the point. This book is not about the Chandler project, nor should members of apparently successful software projects ignore it lightly. It is also not a disguised methodology book on how to develop software. It is about our expectations from software and from ourselves, to show us ourselves in a mirror and raise questions.)

To everyone else. I haven’t the slightest idea what non-software people will think reading this book. But Rosenberg has made it accessible, using few technical terms and explaining them simply. So you owe it to yourself to look here. And we need your insight— don’t leave the field just to us.

Postscript

I remember back in the 60’s and 70’s, a two-sentence conversation I often used to have with people outside of the computer (software) field, when space adventure movies (I’m thinking of Kubrick’s “2001“) were popular:

Them: Gosh, imagine what it would be like out on a spaceship, depending only on a computer for life support.

Me: You don’t have to imagine anymore. We are both on one right now.

I write so much about code review that I may forget to emphasize the obvious.

First, “code review” means different things to different people. I mean only that

Someone, who knows good code when s/he sees it, reads the code and gives the code author helpful comments.

Second, code review is not a goal — only a means to an end. The goal is better code now, and better developers as time goes on (which means better code in the future).

That said, I can get to today’s point, which is:

Code review is expensive, so use it wisely.

Do Not Use Code Review

Do not use code review to ensure that fixed text, such as copyright notices and authorship headers, are correct. For those, create code file templates and use them.

Do not use code review to find text layout problems. Instead, adjust your code editor so that it helps you lay the text out according to your style guide, and only lets you lay it out that way. In fact, skip the style guide — write a code editor configuration file, and perhaps some macros, instead.

Do not use code review to find where the language has been used incorrectly or unclearly. The person who wrote your compiler did that work already. Just turn the compiler warnings all the way up, and maintain zero compiler warnings. (And as Gerard Holzmann says in his rule #10, “The rule of zero warnings applies even when the compiler or the static analyzer gives an erroneous warning: If the compiler or analyzer gets confused, the code causing the confusion should be rewritten.”

Do not use code review to look for functional errors. Write unit tests instead. Not only is the CPU faster than a reviewer, and never gets bored, but the activity of writing unit tests often helps the code author see functional errors him/herself.

Do not use code review to find misuse of the language where the construct is error-prone. Consider the costs and benefits of moving to a language more suited to your environment, product, and developers, and if benefits outweigh costs, migrate to the better language.

Finally, do not use code review to find more instances of a problem if root cause analysis of past failures has already identified the problem as serious and systemic. Correct the problem first.

What’s Left for Code Review?

After all of the above is being addressed, then the valuable time of your code reviewers can be invested identifying readability, maintainability, and extensibility issues. For the most part, it takes a person to identify these issues. The reviewer is standing in for another developer—the one who will have to work on the code next.

These issues are the ones which affect how long it takes to fix, change, or extend the code, and how often fixes are required. The time saved by code that addresses these issues is what your time investment in code review is supposed to return, hopefully several times over.

When I park my (manual-shift) car on a slope, pointing downward, I set the parking brake, and I leave it in reverse as an extra safety measure. Of course that means that when I’m ready to start out again, I have to remember to take it out of gear (as well as releasing the brake), otherwise the car just jumps backwards and stalls.

Today I was reading Joel Spolsky’s “tips for reading other people’s source code”. In spite of the title (Reading Code is Like Reading the Talmud), you don’t have to have studied Talmud to get his point.

You also don’t have to conclude that his point is universally applicable.

Yes, when you have to work with undocumented, unclear code from an unavailable 3rd party, you might have to reverse engineer it to figure out how it does what it does. Or even just what it does.

But when you’re writing the code, especially as part of a team, there’s no reason to write it so that someone else will need to reverse engineer it. Design well, and write clearly, so that you and others will be able to understand it easily and modify it quickly.

Then everyone can move forward without delay.

I may be an exception, but if there’s one thing I enjoy, it’s buying the right tool for a specific job, when the job needs doing. It gives me an excuse to buy a new toy, as well as the pleasure of having a task go smoothly. Faithful readers may remember my first post here, Choose and Use the Right Tools, where I didn’t, and how that turned out.

Today I got a second chance, when my son decided to move his PC to the other side of his desk. With my wife’s permission, we set out to drill another hole for the computer cables. (My wife designed the desk, which is probably worth a good deal more than the PC.)

This time, when I got home from work, I was already presented with a new special-purpose hole saw. There was a little concern that it cost $20, but I reassured everyone that it was fine—this time we would use the right tool and get the job done nicely.

We got all prepared: cleared the work area, measured and marked the hole center on the bottom of the desk to ensure clearing the frame of the desk, put masking tape on top to prevent chipping, and drilled a starter hole all the way through. Came to putting the hole saw in the drill, though, and … it wouldn’t fit.

My son sped up—started thinking of all sorts of quick ways to attach it to the drill anyway.

I slowed down. I decided I would either become much more knowledgeable about hole saws in 15 minutes, or we would put the whole thing away ’til tomorrow and ask at the store. Well, I learned about what a hole saw and drill look like together, what mandrels are and what arbors look like, and finally settled on the understanding that my drill should definitely accept bits with up to a 1/2″ diameter shank. Pretty technical, huh!

All I can say is that this newfound knowledge gave me the confidence to open the drill chuck a bit larger, and then the hole saw (1 cm metric shank—do the math) did fit. We continued the job, including drilling the hole half of the way from the top, and the other half of the way from the bottom, so that the finishing cut would be in the middle of the desk thickness where it wouldn’t damage either surface. Out popped a plug of wood, just like in the picture.

What, you might wonder, does all this have to do with learning, and software development?

For years I’ve been trying to learn about regular expressions because they seemed so useful. But I never got very far until this week. I had to prepare a demonstration of integrating a source code editor (my current favorite—Source Insight) with some XML-modified output from PC-Lint. Well, it took me more than a hole saw’s 15 minutes, but a lot less than “years”. About an hour-and-a-half and I had—rather suddenly, it seemed—taken control of regular expressions for myself.

What made the difference? I had a specific task to do. I was no longer focused on the learning, but, as with the hole saw, on the necessity of the task at hand. A real task, whose result I had promised to someone else. With that promise, the learning just came by itself.