Talk About Quality

Software is not construction

Software is not construction, no matter how popular that statement may be. A program is not a building. Parts are not chosen first and then fitted into place. Design continues almost to the last minutes of coding. Further, almost nobody spends time creating new code, at least not for very long. Sure there are new products and new codebases. But even then, after a short period, most additional coding is extending or changing an existing codebase. So if coding is not construction, and mostly involves changing what’s already created, what is it?

Well, coding is writing. It’s using text to communicate a message. That may not sound like much, but understanding this helps a lot. Instead of struggling to make up new ways of thinking and learning, as if programming were a completely new human activity, we can look at how good writers write. People have been writing for many hundreds of years, and have become pretty good at it.

Learning to write

How do people get into writing? Do we start, as most do in programming, full grown, writing essays? No, not at all. We start as children, and learn first to talk, then to read, and finally to write. Yes, I know that school seems to teach writing, at least forming the letters, before reading. But it’s an illusion: listening to a bedtime story is essentially reading – paying attention to someone else’s writing, and understanding it – and comes before writing. Most people don’t try writing that communicates a message – expository prose – until secondary school or later.

Good writers have their audience in mind at all times. They edit their writing through several drafts, as well as getting others to review it, and give written comments. And they talk about their writing. Maybe not always directly, but when a political columnist goes to a dinner party, you can bet he’ll talk politics. When a novelist chats in the park, she’s talking about the same human struggles that will later appear in her books. Even for the solitary writer typing in a darkened room, the life that feeds the writing is one of talking with, listening to, and reading about people.

Software is Writing

Now back to software: software is not like writing, it is writing. In our industrial setting, it is too late or too radical to suggest re-teaching programming, by teaching first how to talk, then to read, then to write programs. But we can highlight where daily programming work already involves each of these three activities, and where it doesn’t, but should.

Talking includes formal design reviews and code reviews, but also whiteboard conversations and hallway arguments. How should I write this here? Why is that data structure that way? Why does the compiler give that warning? Don’t just think alone – talk it out with someone nearby. It is the same with tests. Why does that scenario fail intermittently? How should I write it differently? And even for defect entries -what does the submitter really mean? Phone him up. Why does this change fix the problem? Find the developer and ask.

Reading is for design documents, training materials and perhaps professional books. But mainly, it includes reading lots of existing code. That’s reading it until you’re really sure what it does, so you know how to change it, fix it, extend it. There is also code review — where the author asks you to read their code, not just to say that it looks fine, but to identify what’s wrong, or unclear.

Writing is the coding, of course. But who is the audience? There are two audiences. One, the compiler, which is your personal translator, from a non-native language (nobody’s mother spoke to them in C), to an almost unknown language – the machine language that the hardware actually understands. The compiler is a rigid, simple-minded, sometimes unpredictable audience, just a ghost of the person who wrote it. Here you have to be very precise, careful, and limit yourself to what the compiler understands and to what the hardware will accept. The other audience, is so very different – the next developer. This may be yourself a few weeks later, or someone else, but there will always be another developer, coming back months or years later, to fix or extend the code. She wants comments, explanations, meaningful names for things, and an easily readable structure. Because now you’re long gone, she can’t talk to you: she must read, so you must write.

Tools!

Well, with all this talk about programming being writing, let’s come back to earth. Software developers are technical people, and expect to use the best tools to do their jobs. There’s the whiteboard in the hallway — we need more of those, for talking. Defect tracking systems, where we can make sure we write clear resolution notes. And for automatically sharing and discussing code, web-hosted code tools for static analysis and code review are the way to go.

–
Originally published in company internal newsletter. Edited and released here with permission.

After spending more time than I should have troubleshooting a coding (actually static code analysis) problem, I shared the story with a co-worker, who reminded me that Kernighan and Pike had been there before:

Another effective technique is to explain your code to someone else.  This will often cause you to explain the bug to yourself.  Sometimes it takes no more than a few sentences, followed by an embarrassed “Never mind, I see what’s wrong.  Sorry to bother you.”  This works remarkably well; you can even use non-programmers as listeners.  One university computer center kept a teddy bear near the help desk.  Students with mysterious bugs were required to explain them to the bear before they could speak to a human counselor.

Brian Kernighan and Rob Pike, in The Practice of Programming

Seems that there are rules that we learn, then love to break, and are always sorry afterwards. Here’s how.

My code doesn’t work

• I’ll just try this change to see if that fixes everything. And then this change, and this other change.
• I’ve been stuck for 10 minutes, but since nobody knows this code like I do, I won’t ask anyone for help.
• I know I don’t understand what every line of code does, but it’s OK — I’ll figure out the problem anyway.
• A complete build takes only a few minutes, so there’s no need to try a 3-line sample instead.
• I’ve read the documentation and I’m sure I’ve understood it, so I won’t ask anyone for a second opinion.
• I already asked someone 3 times for help and got it, so I can’t bother him or her a fourth time.
• It doesn’t compile (cleanly, or at all) just now, but soon it will again if I just keep at it

(The only way I got home today was that somehow I did realize it was, um, OK to call the tool support line.)

What are your favorite ”justifications” for why it’s better to stay stuck than to get help, or at least take a break and share your problem with someone else?

While I’m waiting, I’ll go talk to my teddy bear.

One of the big human challenges to code review is ego. For all the talk about “egoless programming”, ego still means “self” and when a person writes code, or owns it for maintenance, s/he identifies with the code. That kind of “ego” or “self” thinking is what lets the code author work through problems by saying, “let’s see, when I receive this signal, I enter this loop and I calculate that value.” So how can code author and code reviewer(s) go into a code review, whether it be a meeting or online, and yet avoid criticism and defensiveness?

Here’s an image that can help: when code review participants enter the review, imagine everyone sitting on the same side of the table, with only the code on the other side. The code author is no longer the author or owner — just another reviewer. The two or more people in the room, or in the online review, form a team together to find as many things as possible in the code to improve. Now nobody needs to defend the code, because everyone is a reviewer. Still, though, if criticism is in the air, the real code author will feel it.

The desired result of a code review is a list of important things to improve. Comments which the code author can take back to the code and say, “I understand what I need to do to make this code better.” Take a concept from the topic of assertive communication: The “I” message. Best explained by an example:

Let’s say I’m a busy, pressured group leader who has to lead meetings, and someone is always late for the group meetings — much later than everyone else . I decide to speak to that person. What can I say to him that will cause him to feel what I feel, and based on that, to join me in solving the problem? Not to feel criticized, and to become defensive. An “I” message would be:

“I’m under a lot of pressure when leading our weekly status meetings, and when I see you come in 20 minutes late to the meeting, it makes me more tense, and makes it hard for me to keep the meeting on track.”

The idea is that, while I have mentioned the person’s lateness, I have described only what I see, and described a problem that I have. All the focus is on me and my problem, which the group member may realize is his problem too, since he may benefit from a calmer group leader and a more effective meeting. Hopefully, he is drawn into solving my, or our, problem.

Similarly with code review. Instead of a critical comment like, “this code is very confusing,” say, “when I read through this function — both the comments and the code — I get to the end and I still have no idea what it’s really supposed to do. If I had a change request on this code, I would not be able to confidently make the change.” Given that the code author probably does not have the same problem, this comment draws the author into saying, “I understand what the function is supposed to do, but apparently you don’t from reading the code. I’ll try to improve the comments or clarify the code so that you understand what I understand.”

Try this method in your code reviews to get everyone working together against the code, to improve the code.

Sometimes I get the feeling that when people consider and talk about code review, they’re talking about different things but they don’t know it. I see code review everywhere during coding.

Try this: put aside the image of code review as a meeting, or even code review as an independent activity. Limit your description of code review to: someone else is reading your code, you’re discussing it together, and there’s effective writing as a result. If you see those three things happening, you’re seeing code review.

Debugging

The great hope was that code review would be solely a preventive activity: do code review before execution, and there won’t be anything to debug. That’s not going to happen. Too much code is delivered under pressure, without code review. But all is not lost. Just review the code when debugging. Look again at the three steps — how does code review, as defined above, change debugging? First, don’t debug alone. Whatever you may think of pair programming for or against, pair debugging is a must. Especially if you wrote the code, having someone else look for the problem, or better, restate the problem, is essential. Second — discussion. Debugging that rushes to a fix often produces the wrong fix. And finally, effective writing. Sure there’s the writing which is fixing the code. But what about adding comments? And perhaps more important and effective, especially if it’s a quick fix: submit a new defect or change request to your tracking system for what really needs changing to reduce the chance of this failure recurring.

Static Analysis

Tools! If I have tools that can find problems, I won’t have to review code. Yes, you will. And you should want to. Static Analysis tools are great for finding simple errors, and potential defects. For narrowing the field of focus. But then, to find out what’s really wrong … code review. Additional benefit of the tool — since it has no feelings, it can be two of you against the reviewer: read the code with a partner, discuss why the tool thinks there’s a problem and whether you agree, and again, effective writing. Not just the fix, but why that fix, and also perhaps an adjustment to the tool so it will do better next time. High-end static analysis systems let you document that right inside the tool.

Testing

Testing, and its partner, code coverage, are often seen as the opposite of code review. Whatever I couldn’t catch by code review (and static analysis) is left for exercising the code under controlled conditions, causing failures, and fixing them. Actually code review is the most important part of developer testing. Not only reviewing the code under test in order to design good white-box tests. But also reviewing the code based on the test results and coverage results. Why does the software intermittently fail this test? Why is it so difficult to cover all the branches in this function? Again, read together, discuss, write. Here the effective writing is likely to be refactoring the code. If someone wants code review “documentation” — give them a diff and a one-sentence explanation of what you did and why.

Code Review is All-Purpose

There are many more activities in the developer’s daily life. Detailed design, branching and merging, delivery — all can benefit from in-line code review with this three-step model of reading, discussion, and effective results-writing. And by acknowledging this model, we see that code review, or almost code review, is happening all the time. Just have to complete the steps to get code review that real people will do.

Jesse Gibbs at Atlassian sent me to the following post from Scott Bilas at the game software company Loose Cannon Studios. In Peer Code Reviews: Good Commenting Practices, Bilas says code reviewers should seek architectural issues, and adherence to good software development practices and coding standards. And that they should look for mentoring opportunities. At the same time, he lets them off from other responsibilities, saying “Reviewers aren’t expected to catch everything,” and, “Reviewers aren’t expected to catch deep or systemic design problems.”

It’s a pretty good tutorial on the current best practice of code review. So why does it feel like something big is missing?

Upon reflection, it turns out that, unlike, say, editing and commenting on a book, code review is not really a reviewer-author relationship at all. An editor may make a lot of changes, but those changes end with publication. Code, on the other hand, will be fixed, extended, and refactored by future developers. Each future, or “next” developer will need exactly two things from the code in order to do his or her job: readability and understandability. Code is readable if it has the right balance of abstraction vs. detail at each level of the code. Code is understandable if the reviewer can, without help from the author, see what the code does, and why. If code is clear, that is, readable and understandable, modifications will be easy and error-free.

So the question “what are the responsibilities of a code reviewer” turns out to be a trick question. Like this one, from driving class:

Question: In the following traffic diagram (imagine any diagram you want), who has the right-of-way?

Answer: Nobody has the right-of-way. Right-of-way can never be possessed, only given.

The code reviewer does not have any review responsibility. The code author has all the responsibility to write clear code, and provide it to the code reviewer. We should stop calling that person a “code reviewer”; instead, say “code user”.

Code review becomes a simple and easily explained two-step process. First the “code user” reads the code and notes down what s/he doesn’t understand how to use, and second, s/he meets with the code author to present what was not clear. In that manner, the “code user” successfully stands in for the “next developer”, and gives the code author an early chance to make things better. As a bonus, all the things a traditional code reviewer was supposed to check, or at least the important things, will be checked. Mentoring will happen. All this when we realize that code review is code use.

It’s Thom Holwerda — keeping things simple for us:

Simple Code Quality Metric

Here’s his site — with a search for “code quality” (warning: if you can’t ignore “Evony” web game ad graphics, stay away).

A medical radiation machine operator types the letter ‘x’, realizes it’s an error, backspaces, types ‘e’, and continues. Consequences of the error and related defects lead to the patient’s untimely death.

Ten years later, a Japanese stock broker mistakenly switches the share price and amount on the sell order of a new stock. He tries to cancel the order, but fails. His employers lose $225 million, and are involved in lawsuits for years.

A blogger clicks the “back arrow” by mistake, gets a warning message, concludes he meant to continue writing, and clicks “OK” to continue. He loses his work, and has to rewrite it.  Here, the consequences are annoying, if trivial by comparison.

What do all these cases have in common? Canceling a request. If it’s not hard enough to program computers to do what we want them to do, who would have thought that telling them not to do it would be hard too?

The cancellation scenario — or “use case” as it’s called in software design — is the silent partner of every positive request supported by a piece of software. It has to cover giving the user clear options, executing the cancellation, and rolling back any partial results. Things get more complicated if authorization is required, or if the transaction has already gone through (both of those requirements figured into the Japanese broker error story). Cancellation and rollback are also part of automatic requests that may occur if one software module (the “server”) cannot complete a request by another (the “client”) and has to make sure to put everything back the way it was, and send the proper response code.

So the next time you’re designing a piece of software, no matter how simple, think what it’s supposed to do, but also what it will do if the user or client calls out, “Cancel that!”

You can do anything in software. Hordes of eager computer science graduates, not to mention brilliant open source coders, keep joining the ranks of software development because it’s fast and fun.

The fact that software is instructions to a machine (the “hard” in “hardware”) seems to have been the only thing John Tukey had in mind when he coined the word “software” back in 1958. (Dr. Tukey, an accomplished statistician, was more focused on computerizing the Fast-Fourier Transform, and criticizing the Kinsey Report for its questionable sampling methods.)

While software attracts developers with its ease in creating things, it tempts us all with its other “softness”: amenability to change. Software can be endlessly fixed, extended, improved. And that advantage demands something of developers which was unexpected, and, well … hard.

Hardware must fit form to function so it’s easy to use. And come with good documentation for maintenance and repair. But that’s all on the outside. Software, always ready for change, has to be clear and readable on the inside too. In other words, software developers have to be good writers, because the next developer will have to read and quickly understand what’s going on in order to change it.  Written changes, again, have to leave the software in a clear and readable state.

In high school, I avoided English class like the plague (and got bad marks for using cliches in my papers), preferring to go into school on snow days to use the (one) computer. Good writing is not why people become programmers. But it’s exactly what we need. Clear written communication. Equal in impact to life-changing books (pen mightier than sword and all that), software crucially affects our lives — from cars, to food transport, to the electric grid.

That good writing is an unwelcome requirement of sofware is why developers code quickly and obscurely, hate documentation, and shun code review. And  why managers push for features, delivery, and fixes, while devaluing internal quality.

Is there hope? The only one I can think of must exploit other likes and dislikes: managers want software changes fast, while developers like writing new code more than fixing someone else’s (or their own) bugs. Good writing is the only way to make code support that scenario, and reap the advantages of software.

I know what I want: highlight-and-comment capability in a text editor. And since plain text really shouldn’t be marked up, I’d want it to save the comments in a separate text file, with line numbers of what was commented on, and the comments’ text. I’m not the first one looking for this capability: “climbanymountain” asked about it on Joel on Software back in 2006: “win32 text editor for code review?“

I started thinking about writing a plugin for some popular text editor, such as Notepad++ or Notepad2 (the latter is my favorite for simple editing). But before the “build or buy” decision, I thought I’d ask around the office.

A lot of ideas, but here was the simplest. Light and off-the-shelf. Use a diff tool. 

The following procedure works fine, and “supports” any version control system. In my example, I’m using open-source WinMerge as my diff tool, IBM Rational ClearCase as my version control system, and Microsoft Windows Explorer as my tree browser. Say the source file to review is called “MyFile.c”, and the reviewer’s name is Firstname Lastname.

The parts in square brackets are optional: omit them if you don’t want to clutter up your version control system with code review records, but just want to use them to prepare for sitting with code author.

1. Browse to the source code file I want to review; [check it out and] open it.
2. Immediately Save As MyFile_flastname_review.c in the same directory. Close the source code file.
3. Select the source code file and the review file (they are next to each other in alphabetical order), right-click and open WinMerge on them.
4. Go to the pane of the review file and type review comments anywhere I want. Save regularly with CTRL-S. (Optionally, for better syntax highlighting, can precede comments with comment character such as “//”. But don’t have to.)
5. Generate a diff (patch) file, call it the same name as the review file, but add .diff on the end. (In WinMerge, that’s on the menu with Tools > Generate a patch.)
6. [Check in all 3 files (adding 2 new ones to source control) -- check-in activity per defect tracking system, and comment with one-line summary of the review.]

From this point, code review comments are easily viewable side-by-side with the code by viewing in WinMerge or similar viewer.

Also, the diff file clearly identifies the line number commented on, and the comment text. I figure I could write an AWK script to turn that into CSV, for tabular display and tracking in Microsoft Excel, if I wanted to.

Having a standalone consumer application get stuck or crash, requiring reboot, is not the worst thing that can happen. (Worse is incorrect behavior that causes data loss or physical harm.) But requiring a reboot is the most annoying failure in non-safety-critical systems.

If there’s any good news, it’s that the list of fault modes is short:

• System resources exhausted
• Mistakenly idling
• Waiting for acknowledgement that never comes
• Deadlock

Did I miss any?

Only exception-safe code can avoid these undesired end states.

Design by Contract (DbC) is one way to exception safety.

Failure mode and effects analysis (FMEA) helps you plan a path to get there.