With compiler warnings, that means setting warnings-as-errors in your compiler so you cannot complete your build. Yes, you also need tests that can fail and a source control system that refuses promotion of versions that fail test. And code review to ensure that code changes to address compiler warnings only improve the code. But the key is warnings-as-errors.

While error-proofing is simpler with fast-food trays, the idea is the same. Read “Simple, Brilliant, Error Proofing at the Amazing In-N-Out Burger” (Mark Graban  |  04/11/2011  Quality Digest) and look for where else you can error-proof your process.

1. Notice that many tests don’t find defects
2. Decide we need a new definition for the word “test”
3. But the current definition is useful, so move it to the word “check”
4. Now the word “test” is open for redefinition
5. Define “test” as in Testing vs. Checking

Quick summary of new and useful definition:

Let  “test” mean an exercise of the software under test that helps a thinking tester answer the question, “Is there a problem here?”

The most common tool on a software development project is always the “bug tracker” or defect tracking system. It’s really a change request tracking system, with two kinds of requests for changes: enhancements (performance improvement or new feature requests) and defects (fix requests). Tracking requests for changes. That’s good.

Now look at the other major tool in every software development group — it’s usually called a “version control system”. But see above —  users don’t care about versions. And developers have to report to the defect tracking system what they’ve done, and that system is which is full of requests for changes, not requests for versions. So why would we want a version control system?

That’s why the light went on for me when I read Joel Spolsky’s Hg Init: a Mercurial tutorial. Not because Mercurial (or Git) makes branching cheap and merges easy. (It does.) But because, as Spolsky writes:

Subversion likes to think about revisions. A revision is what the entire file system looked like at some particular point in time. In Mercurial, you think about changesets. A changeset is a concise list of the changes between one revision and the next revision. (When Spolsky writes “revisions”, read “versions”.)

Now I understand why there’s so much trouble connecting our defect (change) tracking and version control workflows. Testing too — what we really want to know is, “Which change broke the build?” And bringing in code review, more problems, because people want to review whether each change accomplishes the goal of adding or fixing something.  Software development is all about changes: specifying them, designing them, coding them, testing them, and delivering them. Not versions. Changes. Sure, versions have a place in software delivery — they are labels for the software after a certain number of changes. (Mercurial supports them, as “tags”.) So if you want all the tools to work together and make your software development life easier, switch to a software repository that lets you control changes.

What should we call them? Maybe “software changeset control systems”. The two big ones these days are Mercurial, and Git. For Mercurial, see Spolsky’s tutorial. For Git, listen to Linus Torvalds on git.

Software is not construction, no matter how popular that statement may be. A program is not a building. Parts are not chosen first and then fitted into place. Design continues almost to the last minutes of coding. Further, almost nobody spends time creating new code, at least not for very long. Sure there are new products and new codebases. But even then, after a short period, most additional coding is extending or changing an existing codebase. So if coding is not construction, and mostly involves changing what’s already created, what is it?

Well, coding is writing. It’s using text to communicate a message. That may not sound like much, but understanding this helps a lot. Instead of struggling to make up new ways of thinking and learning, as if programming were a completely new human activity, we can look at how good writers write. People have been writing for many hundreds of years, and have become pretty good at it.

Learning to write

How do people get into writing? Do we start, as most do in programming, full grown, writing essays? No, not at all. We start as children, and learn first to talk, then to read, and finally to write. Yes, I know that school seems to teach writing, at least forming the letters, before reading. But it’s an illusion: listening to a bedtime story is essentially reading – paying attention to someone else’s writing, and understanding it – and comes before writing. Most people don’t try writing that communicates a message – expository prose – until secondary school or later.

Good writers have their audience in mind at all times. They edit their writing through several drafts, as well as getting others to review it, and give written comments. And they talk about their writing. Maybe not always directly, but when a political columnist goes to a dinner party, you can bet he’ll talk politics. When a novelist chats in the park, she’s talking about the same human struggles that will later appear in her books. Even for the solitary writer typing in a darkened room, the life that feeds the writing is one of talking with, listening to, and reading about people.

Software is Writing

Now back to software: software is not like writing, it is writing. In our industrial setting, it is too late or too radical to suggest re-teaching programming, by teaching first how to talk, then to read, then to write programs. But we can highlight where daily programming work already involves each of these three activities, and where it doesn’t, but should.

Talking includes formal design reviews and code reviews, but also whiteboard conversations and hallway arguments. How should I write this here? Why is that data structure that way? Why does the compiler give that warning? Don’t just think alone – talk it out with someone nearby. It is the same with tests. Why does that scenario fail intermittently? How should I write it differently? And even for defect entries -what does the submitter really mean? Phone him up. Why does this change fix the problem? Find the developer and ask.

Reading is for design documents, training materials and perhaps professional books. But mainly, it includes reading lots of existing code. That’s reading it until you’re really sure what it does, so you know how to change it, fix it, extend it. There is also code review — where the author asks you to read their code, not just to say that it looks fine, but to identify what’s wrong, or unclear.

Writing is the coding, of course. But who is the audience? There are two audiences. One, the compiler, which is your personal translator, from a non-native language (nobody’s mother spoke to them in C), to an almost unknown language – the machine language that the hardware actually understands. The compiler is a rigid, simple-minded, sometimes unpredictable audience, just a ghost of the person who wrote it. Here you have to be very precise, careful, and limit yourself to what the compiler understands and to what the hardware will accept. The other audience, is so very different – the next developer. This may be yourself a few weeks later, or someone else, but there will always be another developer, coming back months or years later, to fix or extend the code. She wants comments, explanations, meaningful names for things, and an easily readable structure. Because now you’re long gone, she can’t talk to you: she must read, so you must write.

Tools!

Well, with all this talk about programming being writing, let’s come back to earth. Software developers are technical people, and expect to use the best tools to do their jobs. There’s the whiteboard in the hallway — we need more of those, for talking. Defect tracking systems, where we can make sure we write clear resolution notes. And for automatically sharing and discussing code, web-hosted code tools for static analysis and code review are the way to go.

–
Originally published in company internal newsletter. Edited and released here with permission.

Another effective technique is to explain your code to someone else.  This will often cause you to explain the bug to yourself.  Sometimes it takes no more than a few sentences, followed by an embarrassed “Never mind, I see what’s wrong.  Sorry to bother you.”  This works remarkably well; you can even use non-programmers as listeners.  One university computer center kept a teddy bear near the help desk.  Students with mysterious bugs were required to explain them to the bear before they could speak to a human counselor.

Brian Kernighan and Rob Pike, in The Practice of Programming

Seems that there are rules that we learn, then love to break, and are always sorry afterwards. Here’s how.

My code doesn’t work

• I’ll just try this change to see if that fixes everything. And then this change, and this other change.
• I’ve been stuck for 10 minutes, but since nobody knows this code like I do, I won’t ask anyone for help.
• I know I don’t understand what every line of code does, but it’s OK — I’ll figure out the problem anyway.
• A complete build takes only a few minutes, so there’s no need to try a 3-line sample instead.
• I’ve read the documentation and I’m sure I’ve understood it, so I won’t ask anyone for a second opinion.
• I already asked someone 3 times for help and got it, so I can’t bother him or her a fourth time.
• It doesn’t compile (cleanly, or at all) just now, but soon it will again if I just keep at it

(The only way I got home today was that somehow I did realize it was, um, OK to call the tool support line.)

What are your favorite ”justifications” for why it’s better to stay stuck than to get help, or at least take a break and share your problem with someone else?

While I’m waiting, I’ll go talk to my teddy bear.

Here’s an image that can help: when code review participants enter the review, imagine everyone sitting on the same side of the table, with only the code on the other side. The code author is no longer the author or owner — just another reviewer. The two or more people in the room, or in the online review, form a team together to find as many things as possible in the code to improve. Now nobody needs to defend the code, because everyone is a reviewer. Still, though, if criticism is in the air, the real code author will feel it.

The desired result of a code review is a list of important things to improve. Comments which the code author can take back to the code and say, “I understand what I need to do to make this code better.” Take a concept from the topic of assertive communication: The “I” message. Best explained by an example:

Let’s say I’m a busy, pressured group leader who has to lead meetings, and someone is always late for the group meetings — much later than everyone else . I decide to speak to that person. What can I say to him that will cause him to feel what I feel, and based on that, to join me in solving the problem? Not to feel criticized, and to become defensive. An “I” message would be:

“I’m under a lot of pressure when leading our weekly status meetings, and when I see you come in 20 minutes late to the meeting, it makes me more tense, and makes it hard for me to keep the meeting on track.”

The idea is that, while I have mentioned the person’s lateness, I have described only what I see, and described a problem that I have. All the focus is on me and my problem, which the group member may realize is his problem too, since he may benefit from a calmer group leader and a more effective meeting. Hopefully, he is drawn into solving my, or our, problem.

Similarly with code review. Instead of a critical comment like, “this code is very confusing,” say, “when I read through this function — both the comments and the code — I get to the end and I still have no idea what it’s really supposed to do. If I had a change request on this code, I would not be able to confidently make the change.” Given that the code author probably does not have the same problem, this comment draws the author into saying, “I understand what the function is supposed to do, but apparently you don’t from reading the code. I’ll try to improve the comments or clarify the code so that you understand what I understand.”

Try this method in your code reviews to get everyone working together against the code, to improve the code.

When microprocessors first came out the buzz on the news was that soon they were going to be inside everything. I remember wondering what do we need them in everything for? My washer and my microwave oven, for example, worked just fine with a mechanical dial. But I was wrong and now they are in everything.

Much earlier on, complex systems were based on the Six Simple Machines. I can’t say which is simplest, but maybe it’s the lever. What’s important here is that the behavior of the lever is governed by physical properties of solids, which by now are pretty well understood. Push down on one end, and the other end goes up. Exceptions exist: solids can stretch under tension, deform under pressure, and if you push too hard, they can break. Apparently Toyota had problems with this too, in their gas pedal problem and fix.

Technical advances brought hydraulics — based on the properties of fluids and gases. I also remember replacing a brake line on an old car once, and getting to see how a leak in the line leads to spongy braking, and, if I hadn’t been refilling it weekly before the replacement, to brake failure.

In both of those technologies, the basic idea is that the system user (e.g. the driver) pushes on the control, force is transmitted immediately through a physical medium, and the desired effect occurs. Failure modes are well understood and (usually) are prevented.

Now replace the lever with software. No more physical properties: the behavior of the software transmission line is governed by the behavior of people — software developers writing lists of instructions which are translated several times before reaching the hardware. Now you push on the control, and a cacophony of instructions (okay, I’m being unfair — if the software is well-written, it’s a concert of instructions) rush down a wire to an actuating motor at the other end.

Let’s be clear about this: real-time software is a myth. The metal of the lever, or even the fluid of the hydraulic line, has been removed and replaced by your entire software development organization. Materials science out the door, organizational sociology in its place. If you want to know where that leaves us, read this short post about drive-by-wire, and the many comments there.

What’s the answer? In the code itself — perhaps it’s avoiding interrupts and multi-threading, and moving to synchronous programming where everything proceeds in lock-step. At least then a modern car would run like clockwork. Other than that, notice that many of the suggestions in the discussion talk about mechanical back-up systems, which are basically putting the lever back into the system.

But software development is not, as commonly believed, about technology. It is about writing, reading, and communication in ever-larger groups. Those who have said, “The pen is mightier than the sword” were thinking of politics. Now writing, in the form of software, interposes itself between every hand and hammer. We have broken the lever; do we know how to wield its replacement?

Try this: put aside the image of code review as a meeting, or even code review as an independent activity. Limit your description of code review to: someone else is reading your code, you’re discussing it together, and there’s effective writing as a result. If you see those three things happening, you’re seeing code review.

Debugging

The great hope was that code review would be solely a preventive activity: do code review before execution, and there won’t be anything to debug. That’s not going to happen. Too much code is delivered under pressure, without code review. But all is not lost. Just review the code when debugging. Look again at the three steps — how does code review, as defined above, change debugging? First, don’t debug alone. Whatever you may think of pair programming for or against, pair debugging is a must. Especially if you wrote the code, having someone else look for the problem, or better, restate the problem, is essential. Second — discussion. Debugging that rushes to a fix often produces the wrong fix. And finally, effective writing. Sure there’s the writing which is fixing the code. But what about adding comments? And perhaps more important and effective, especially if it’s a quick fix: submit a new defect or change request to your tracking system for what really needs changing to reduce the chance of this failure recurring.

Static Analysis

Tools! If I have tools that can find problems, I won’t have to review code. Yes, you will. And you should want to. Static Analysis tools are great for finding simple errors, and potential defects. For narrowing the field of focus. But then, to find out what’s really wrong … code review. Additional benefit of the tool — since it has no feelings, it can be two of you against the reviewer: read the code with a partner, discuss why the tool thinks there’s a problem and whether you agree, and again, effective writing. Not just the fix, but why that fix, and also perhaps an adjustment to the tool so it will do better next time. High-end static analysis systems let you document that right inside the tool.

Testing

Testing, and its partner, code coverage, are often seen as the opposite of code review. Whatever I couldn’t catch by code review (and static analysis) is left for exercising the code under controlled conditions, causing failures, and fixing them. Actually code review is the most important part of developer testing. Not only reviewing the code under test in order to design good white-box tests. But also reviewing the code based on the test results and coverage results. Why does the software intermittently fail this test? Why is it so difficult to cover all the branches in this function? Again, read together, discuss, write. Here the effective writing is likely to be refactoring the code. If someone wants code review “documentation” — give them a diff and a one-sentence explanation of what you did and why.

Code Review is All-Purpose

There are many more activities in the developer’s daily life. Detailed design, branching and merging, delivery — all can benefit from in-line code review with this three-step model of reading, discussion, and effective results-writing. And by acknowledging this model, we see that code review, or almost code review, is happening all the time. Just have to complete the steps to get code review that real people will do.

• Coders at Work, by Peter Seibel (review by talkaboutquality)
• If I Only Changed the Software, Why is the Phone on Fire? by Lisa K. Simone (review by talkaboutquality)
• I am a Bug, by Robert Sabourin (review by talkaboutquality)

Code Review type 1

Key question: Does this change accomplish what it was supposed to, while doing no damage?
Desired code author response to comments: Revise changes
Under review: changed files with diff
Tool: diff comparer
Required feature as a code review tool: right-click submit comment
Nice-to-have features (but could be in a separate app): right-click definitions

Code Review type 2

Key Question: Is this code designed well, so that it is easy to fix and extend?
Desired code author response to comments: Refactoring / renaming
Under review: any subset of entire codebase, starting from a given area of the code
Tool: source code browser
Required feature as a code review tool: right-click submit comment
Nice-to-have features (but could be in a separate app): code editing