Talk About Quality

After spending more time than I should have troubleshooting a coding (actually static code analysis) problem, I shared the story with a co-worker, who reminded me that Kernighan and Pike had been there before:

Another effective technique is to explain your code to someone else.  This will often cause you to explain the bug to yourself.  Sometimes it takes no more than a few sentences, followed by an embarrassed “Never mind, I see what’s wrong.  Sorry to bother you.”  This works remarkably well; you can even use non-programmers as listeners.  One university computer center kept a teddy bear near the help desk.  Students with mysterious bugs were required to explain them to the bear before they could speak to a human counselor.

Brian Kernighan and Rob Pike, in The Practice of Programming

Seems that there are rules that we learn, then love to break, and are always sorry afterwards. Here’s how.

My code doesn’t work

• I’ll just try this change to see if that fixes everything. And then this change, and this other change.
• I’ve been stuck for 10 minutes, but since nobody knows this code like I do, I won’t ask anyone for help.
• I know I don’t understand what every line of code does, but it’s OK — I’ll figure out the problem anyway.
• A complete build takes only a few minutes, so there’s no need to try a 3-line sample instead.
• I’ve read the documentation and I’m sure I’ve understood it, so I won’t ask anyone for a second opinion.
• I already asked someone 3 times for help and got it, so I can’t bother him or her a fourth time.
• It doesn’t compile (cleanly, or at all) just now, but soon it will again if I just keep at it

(The only way I got home today was that somehow I did realize it was, um, OK to call the tool support line.)

What are your favorite ”justifications” for why it’s better to stay stuck than to get help, or at least take a break and share your problem with someone else?

While I’m waiting, I’ll go talk to my teddy bear.

One of the big human challenges to code review is ego. For all the talk about “egoless programming”, ego still means “self” and when a person writes code, or owns it for maintenance, s/he identifies with the code. That kind of “ego” or “self” thinking is what lets the code author work through problems by saying, “let’s see, when I receive this signal, I enter this loop and I calculate that value.” So how can code author and code reviewer(s) go into a code review, whether it be a meeting or online, and yet avoid criticism and defensiveness?

Here’s an image that can help: when code review participants enter the review, imagine everyone sitting on the same side of the table, with only the code on the other side. The code author is no longer the author or owner — just another reviewer. The two or more people in the room, or in the online review, form a team together to find as many things as possible in the code to improve. Now nobody needs to defend the code, because everyone is a reviewer. Still, though, if criticism is in the air, the real code author will feel it.

The desired result of a code review is a list of important things to improve. Comments which the code author can take back to the code and say, “I understand what I need to do to make this code better.” Take a concept from the topic of assertive communication: The “I” message. Best explained by an example:

Let’s say I’m a busy, pressured group leader who has to lead meetings, and someone is always late for the group meetings — much later than everyone else . I decide to speak to that person. What can I say to him that will cause him to feel what I feel, and based on that, to join me in solving the problem? Not to feel criticized, and to become defensive. An “I” message would be:

“I’m under a lot of pressure when leading our weekly status meetings, and when I see you come in 20 minutes late to the meeting, it makes me more tense, and makes it hard for me to keep the meeting on track.”

The idea is that, while I have mentioned the person’s lateness, I have described only what I see, and described a problem that I have. All the focus is on me and my problem, which the group member may realize is his problem too, since he may benefit from a calmer group leader and a more effective meeting. Hopefully, he is drawn into solving my, or our, problem.

Similarly with code review. Instead of a critical comment like, “this code is very confusing,” say, “when I read through this function — both the comments and the code — I get to the end and I still have no idea what it’s really supposed to do. If I had a change request on this code, I would not be able to confidently make the change.” Given that the code author probably does not have the same problem, this comment draws the author into saying, “I understand what the function is supposed to do, but apparently you don’t from reading the code. I’ll try to improve the comments or clarify the code so that you understand what I understand.”

Try this method in your code reviews to get everyone working together against the code, to improve the code.

Someone comes to me with a problem that sounds like,

“I wanted to do X but it doesn’t work.”

When I hear that request, often they know a lot more about the technical details than I do. I need them to teach me in a way that lets me put things in order. My method starts from success … and progresses to failure. Strange? But it works.

The questioner is focused on the failure scenario. I crave success. There must be some around — just have to find it. I ask what similar scenario succeeds. What version of the file did work? What build compiled successfully? When did this test last pass? As we discuss the successful scenario, I learn why it worked. What requirements were met by that configuration. I repeat those requirements out loud for confirmation.

With a success in hand, well understood, we have a checklist to apply to the failed configuration. We can go over the failure and check whether all the things that made something similar work are present. Of course, one will be missing. We have found the problem. I may not know how to fix it. But usually the questioner does.

A new project is ramping up and needs more resources. Another project is winding down — maybe it can do with less.

Let’s move some people from one team to the other

What’s wrong with this common resource allocation decision? It’s not the decision. Moving resources to the project that needs them makes sense. What’s wrong is the words, which hide a false assumption.

Here, “team” means location. Move people from one place to another. Like conductors putting people on a train: the “teams” are two train stations. One called “Project A Station” and the other one, “Project B Station”.

But teams are not train stations. A team is not the room it sits in, or its box on the org chart. A team is the particular combination of people — their skills and personalities — and their shared experience. They know their project or product best right now, and equally important, they know each other and the special role each person fills. Reassigning people from one team to another is more than moving a person. It is actually dismantling the first team, and disturbing the second. The movement of people based on the train-station model of teams leads to a contradiction: both teams — origin and destination — cease to exist.

If we must model people and treat them as resources, a better model is a hand at cards. The managers are the players and partners; the game is getting more projects completed. There’s a fixed deck — the “human resources” of the company. Some cards may be similar in strength or level, but every card is unique.

The value of your hand is based entirely on the combination of cards you’re holding right now. If you or your partners need a stronger hand, everyone knows that some cards will have to be exchanged. Put down 2, pick up 2. I pick up what you put down. Resources. Cards. I know this doesn’t sound any better to workers than “counting heads”, but it is. Because every card player thinks twice before putting down a good card, and prefers to take over an entire good hand if they can.

It’s pretty well known that there are three ways to improve software quality: improve process, improve methods, and improve tools. But these are the generic concepts — applicable to any software development effort. Today’s topic is more specific and closer to home. Product quality.

Isn’t it obvious? The software product is the running software, and product quality is how well it meets its requirements. Simply put — does it do what people expect it to do, and not crash or lose data?

But organizations which release software focusing only on that one aspect of product quality are often unpleasantly surprised by inconsistency. They are doing all the right things — following a good process, using good methods and tools, and regularly reviewing all those to see what can be improved. Yet sometimes the product is better, sometimes worse. Why?

External product quality — in software, that’s how well the software serves its users when it’s running, is not enough. There’s also internal quality. The readability of the codebase. Code that’s well-written — easy to understand quickly — is easier to maintain. Fewer programmer errors, and fewer schedule slips. We can add to that clear and organized debug output, which avoids false positives, and gives an accurate record of when the running software seemed to be fine, but was really operating on the edge — just barely recovering from failures.

But there’s another part of the product that’s often overlooked. Overlooked when reorganizing teams, or just when parceling out the daily assignments from multiple projects. The developers. If you take a single iteration of a software product, with development at its best, you’ll apply best process, methods, and tools to put out the next version of the software, and maybe even have codebase improvement (cleanup, refactoring) on the list too. Get to release and you deliver good running software, more maintainable code and … the new state of the team that developed it.

The team members may have learned new methods or technologies. Or just how to avoid pitfalls from the past. Equally important, that particular team learned how to work together — not in general, but specifically in today’s process, methods & tools, and codebase. And with each other. This third part of the product cannot be ignored or thrown away, just as nobody would think to delete a code branch delivered to the customer, or switch out a working library in mid-project.

Looking further at this third component of the product — the team that delivered it — we see clearly that those quality improvement concepts of process, methods, and tools do not exist in a vacuum. They are not just abstract ideas. Rather, they exist in the team members who develop a product, and are unified by teamwork — which means this team learning to work together with these people on this product.

So when you think of product quality improvement, plan to improve all of these:

1. How the product works
2. How the codebase looks
3. How the team works

And at the end of an iteration, make sure to review and protect all three. That’s the guarantee that the next version of the product will be even better.

A co-worker forwarded me this article ”5 Reasons for Software Developers to Do Code Reviews (Even If You Think They’re a Waste of Time)” which certainly sounded promising. Even when I don’t think code reviews are a waste of time. But as I read through it, it became clear that more is less. The article says too much, and detracts from its own message.

1. Developers know their code will be evaluated, so they work harder. “The most useful thing about a code review is the fact that the coder knows that someone is going to review the code,” says Oliver Cole, president of OC Systems and also lead for the open-source Eclipse Test and Performance Tools Platform project.

Work hard because you enjoy it. And of course your code will be evaluated, but not primarily by code review. Rather, the main user of your code is the “next developer”—possibly someone on your team, or even you yourself a few months later. That’s where the evaluation happens.

 2. It improves a developer’s own programming skills.
In your heart, you might not care that much about the success of this particular software project. But most programmers want to improve their personal skills, and that means learning from other people.

If you don’t care about the success of the project, code review won’t help.

3. It’s an opportunity for mentoring, so the programmers you work with get smarter (and thus, more fun to hang around with).” [...] While the intention [to mentor individuals] is generally well meaning, it can often lead to individual discomfort and perceived or actual criticism. In these cases, the greatest opportunity for mentoring usually exists in the context of small collaborative teams working together to realize goals and not in a code review.”

Criticism is not bad, it is essential. It is not personal, but professional. And of course, the smaller the meetings (down to even just 2 people — reviewer and author), the better.

4. It creates consistency and a culture of quality across the project. [...] Developers are quick to complain about being judged on the wrong metrics, but, says Gary Heusner, client partner at custom software developer Geneca, “We have to change the rules to allow for quality and efficient development to be valued over making milestones that are really yardsticks of process more than milestone of value delivered.” Code reviews are a big part of that.

Code reviews are simply part of good software development.  When management, together with the team, track value delivered, that is a big part of creating a culture of quality. Only when the environment is right can code reviews have a chance of being effective.

5. It encourages team bonding. “People think code review is just about finding bugs, but it brings people together, says Smartbear’s Jason Cohen. Often, he says, it can deliver far more than expected.

“Success stories happen very often when performing code reviews,” says Dave Katauskas, senior architect at Geneca. “But the best success story is the pattern that develops once a team has gelled. The longer you’re into a project, the better quality code is created. This is all due to the code review process and governance that occurred up stream in the beginning of the project.”

I had to read this one a few times. Right answer for wrong reasons. I will not credit code review where credit is not due. Even the writer with the “success story” realizes the true origin of the success is the gelled team.

But still, I had to click on the Jason Cohen link to see why code review “brings people together”. Go ahead—click below on “Lightweight Code Review Episode 5: Team Building for the Cold, Dark, and Alone”. But first, get ready to read it right: code review doesn’t create good teams. Rather, good teams benefit from code review. OK, now click.

Lightweight Code Review Episode 5: Team Building for the Cold, Dark, and Alone

This week, I moved from a general process improvement group to a software development department, to specifically focus on code quality. The most mature tool and process I left behind was a defect tracking system. Of course, in our development department, we use that defect tracking system. In case anyone asks me what it’s for, here’s my answer.

Prevent Errors, Fix Defects, Describe Failures

There are many definitions of those terms, but since when we fix problematic code we close “defects” in the “defect tracking system”, I started with that word in the middle. Leads to these definitions:

Error: The mistake someone made. (The root cause is the earliest error in the chain.)

Defect: The wrong code (missing, extra, or incorrect).

Failure: The unacceptable behavior of the code when running.

Detect each as early as possible

Each can be detected with appropriate tools. Detect earlier to minimize cost and annoyance.

Meanwhile, do it right

How?

With tools, learning, and discussion for writing good code.

I’ve often thought about the advantage that athletes and musicians have over software developers: time to practice—lots of it. Most time is spent in exercises and rehearsals, to produce the best peformance.

In software, everyone spends most of the time producing, with a bit of “time off” for learning new technologies. No wonder perfection seems so far off.

A friend pointed out that Dave Thomas has proposed CodeKata as a way of doing that practice. I read it, and while the ideas look good, I felt there was something missing. The “katas” (formal patterns in karate, and here, in design and coding) seemed somehow schoolwork-like and disconnected from real work. I could only imagine myself “practicing” on assignments where I needed the result, even if for something trivial like importing a bunch of e-mails into SharePoint (more about that another time).

I don’t have an answer today, but the question is much bigger than just finding time to practice in software development. So instead, have a look at these posts and presentations where people are discussing the issue, and see what you think.

Level 5 means never having to say you’re sorry (Jeff Atwood)

Big Macs vs. The Naked Chef (Joel Spolsky)

No Best Practices (James Bach)

Herding Racehorses and Racing Sheep (.ppt) (The Pragmatic Programmer)

Competence is a Habit (.ppt) (David Leach)

When I park my (manual-shift) car on a slope, pointing downward, I set the parking brake, and I leave it in reverse as an extra safety measure. Of course that means that when I’m ready to start out again, I have to remember to take it out of gear (as well as releasing the brake), otherwise the car just jumps backwards and stalls.

Today I was reading Joel Spolsky’s “tips for reading other people’s source code”. In spite of the title (Reading Code is Like Reading the Talmud), you don’t have to have studied Talmud to get his point.

You also don’t have to conclude that his point is universally applicable.

Yes, when you have to work with undocumented, unclear code from an unavailable 3rd party, you might have to reverse engineer it to figure out how it does what it does. Or even just what it does.

But when you’re writing the code, especially as part of a team, there’s no reason to write it so that someone else will need to reverse engineer it. Design well, and write clearly, so that you and others will be able to understand it easily and modify it quickly.

Then everyone can move forward without delay.

Want quality improvement, professionalism, and mentoring? Just deploy this short list of responsibilities for each and every person in your company:

• Do your assigned job correctly
• Regularly take a step back and find ways to improve how you do your job
• Demonstrate these improvements to those around you

What’s missing, though?

These guidelines don’t tell you where reviews fit in. Specifically, when somebody, who is doing his job correctly and trying to improve how he does it, comes and asks you for your review and comment on some work product.

We know that reviews can catch defects at 1/10th the cost for each stage earlier we do them. Going further, reviews can be a mentoring platform, and teach extremely effectively, because they are based on people’s current, real work.

Still, many of us react to such a review request, at best, by agreeing to do it, maybe even doing it on time, but all the while feeling, “I’m helping someone else, but I’m taking time away from my own work.”

So that’s the question: is there a fourth item on the list:

• Review others’ work when they ask your help

I think there can be, but only if management, in its leadership role, declares that reviewing others’ work is part of each person’s job. And in its management role, rewards time spent on reviewing equally with time spent on the assigned job.

Getting the benefits of any kind of review — code review, design review, or others — depends on it.