Read Before Running

Debugging: How to Start (commonly accepted method)

Search for “debugging” on the internet and most entries will tell you the first step is to reproduce the problem (properly called “the software failure”) and the second step is to create the simplest configuration that also reproduces the problem.

I agree that it is essential to reproduce the problem, because you need to know you’re working on the right problem.

Fun to Run, or “the thrill of the chase”

But programmers, myself included, like to run things—to let the computer do the work—so there’s a great temptation to reproduce the problem, change something, run the software, add some “debugging statements”, and run it again. And there goes an hour of valuable time.

Does readability fit in here somewhere?

If the code is written clearly, and by that I mean that it’s readable, with meaningful variable names and the proper levels of abstraction, then the second step might be to actually read through the code.

A short debugging story

Today, I showed myself both the wrong way, and the right way, to respond to a “bug report”.

Where I work, I’ve turned out to be the owner of a small set of text-parsing scripts, written in awk. We use them to pull out compiler warning messages from the long build logs, so that developers can see them and address them. These parsing scripts are short—half a page of code including comments—and are tested only with the few build logs that I’ve had time to try. Given limited time (it is not my main job to write parsing scripts!), I have to hope that all the build logs for a given compiler are pretty similar, so whatever pattern I’ve identified will be followed in other build logs. It isn’t always so.

A developer noticed (first by eye, confirmed with a simple “grep”) that there were more compiler warnings in his build log than in the parsed csv (comma-separated-value) output file. Not good. I set about debugging!

It took about 5 minutes to reproduce the problem. Really just gathering the sample input and bad output files, and setting up a copy of the relevant scripts. One run (takes about 2 seconds) showed that, indeed, some warnings were missing.

My next step, by the standard debugging technique, was to start printing all the partial results, run the script over and over again, and look to see where it was failing. Then some more time to adjust the failing script and retest.

Haven’t I seen this somewhere before?

After the first 5 minutes (reproducing the problem), I already had a pretty good idea of which script was failing. A sed one-liner that adds newlines after each compilation command line, and before the first warning message, so that the awk script would be able to separate the first warning message as a record and select it. What was odd was that when I opened that script (just one executable line—the rest is comments including change history), the last modification comment was about how I had fixed exactly the same problem before. Briefly I wondered about that, but rushed onward to more than half an hour of running parts of the script pipeline to prove to myself exactly what was failing and why.

I could have saved myself the time.

Debugging: How to Start (the code-reading method)

After reproducing the initial problem (again, just 5 minutes), and arriving at the suspect script with the worrisome comment, I should have stopped running the code, and started reading it. After all, if my last modification was to fix this same problem, then apparently that modification was either wrong, or insufficient.

It turned out, and this was visible by comparing the one-line script and the new build log on which it failed, that it was insufficient. The new build log had some extra spaces at the end of the compilation command line, and the sed one-liner, designed to identify such lines, was unprepared for the extra spaces.

It did take me another 15 minutes to verify that most new build logs had a random number of extra spaces, and thus to pick the right two-character adjustment to the regular expression. Regression testing took another half hour.

But I could have saved myself almost an hour in the middle of “debugging-by-running” if I had applied a few minutes of “debugging by reading the code”.

Even better would have been reading it out loud, or reading and explaining it to someone else.

Why We Write Software

“Isn’t it nice to know that, when all else fails us, we have an innate decision-making tool to fall back on?”

Robert L. Glass, “Intuition’s Role in Decision Making” (IEEE Software, January/February 200

Yes, Glass admits, estimates that come unbidden from a manager’s subconscious seem the very opposite of quantitative or rational. But most decision-making methods have a common theme: using historical data to decide what to do next. Quantitative estimation puts everything out in the open. Rational support for a desired deadline is at least based on the facts. Intuition, at the other extreme, pulls that data, informally known as “experience”, from the subconscious. And since a good manager has experience, intuition works.

Sort of.

What’s hidden by Glass’ essay are the unstated quality standards that justify each method. Only a complete, quantitative estimate, matched by continuous measurement and adjustment, can promise high quality by the target date. Intuition, on the other hand, even coming from an experienced manager, makes quality likely, but hardly guaranteed. His successes are what keep him in his job, and his few failures are forgiven. Rightly so. He is employed to deliver good enough software, quickly, to meet modern society’s ever-growing appetite for computerized life.

So should we favor intuitive, seat-of-the-pants estimating, with its benefit of early delivery, but its cost of uncertain quality?

Digging deeper, we find that there are some very different reasons why we write software, which strongly influence how we plan our work.

1. Profit
2. Functionality
3. Beauty

Anyone who gets paid for writing commercial software must acknowledge that profit pays his or her salary. Profit leads eventually to pretty good quality, via competition. In the short run, though, we experience a lot of pressure, a lot of errors, and many customer-accepted (if sometimes societally unacceptable) failures. In that context, it seems, we should estimate quickly and intuitively, but admit that quality is expendable.

In a well-funded, non-profit organization (e.g. NASA, famed for error-free software — wholly separate from failure-free flights), software can be about complete functionality. Take as much time as needed to implement everything, perfectly. After all, when the spaceship flies past Jupiter, there’s no second chance.

Where, then, is guaranteed perfection, estimated correctly and delivered quickly? The only place we see that in life is artistic performance. A pianist plays, from memory, a piece that she only decided a few months ago to perform. An hour long, no mistakes. The event starts, and finishes, on time. The customer — the audience — rises to its feet in noisy appreciation. The enabler of perfection is seeking beauty, or doing a thing for its own sake.

Many software developers write software because it’s beautiful, fun, or spiritually rewarding. And that reason engenders the highest-quality work. A quality that delivers on time, with no costly rework. Functionality. Profit too.

Somehow, in the commercial context, that reason why we write software must be harnessed and encouraged by software managers. Otherwise, dismissing a developer’s data with an intuitive estimate is not a fall-back, but falling backwards.

Looking Back, Looking Forward

Some return for your U.S. taxpayer dollars:

CrossTalk, The Journal of Defense Software Engineering is an approved Department of Defense journal. CrossTalk’s mission is to encourage the engineering development of software in order to improve the reliability, sustainability, and responsiveness of our warfighting capability and to inform and educate readers on up-to-date policy decisions and new software engineering technologies.

Whatever you may think about “warfighting”, CrossTalk sometimes has some really good articles, and even some great ones. Here are two, from the December 2007 issue:

Very good, about software maintenance:

Geriatric Issues of Aging Software

Great, about moving beyond “escaped defects” to defect prevention: 

Advancing Defect Containment to Quantitative Defect Management

Both are thorough and comprehensive—worth reading!

Process + Craft = Quality

This week, I moved from a general process improvement group to a software development department, to specifically focus on code quality. The most mature tool and process I left behind was a defect tracking system. Of course, in our development department, we use that defect tracking system. In case anyone asks me what it’s for, here’s my answer.

Prevent Errors, Fix Defects, Describe Failures

There are many definitions of those terms, but since when we fix problematic code we close “defects” in the “defect tracking system”, I started with that word in the middle. Leads to these definitions:

Error: The mistake someone made. (The root cause is the earliest error in the chain.)

Defect: The wrong code (missing, extra, or incorrect).

Failure: The unacceptable behavior of the code when running.

Detect each as early as possible

Each can be detected with appropriate tools. Detect earlier to minimize cost and annoyance.

Meanwhile, do it right

How?

With tools, learning, and discussion for writing good code.

Two Parts Human, One Part Machine

With the enormous popularity of Google, who could resist an article in the New York Times (front page, four pages long!) about how Google Keeps Tweaking Its Search Engine? Maybe we’d learn some upcoming changes (hardly) or inner secrets (definitely not). But in among the intentionally vague references to formulas and advanced mathematics, the following caught my eye:

Mr. Singhal often doesn’t rush to fix everything he hears about, because each change can affect the rankings of many sites. “You can’t just react on the first complaint,” he says. “You let things simmer.”

So he monitors complaints on his white board, prioritizing them if they keep coming back.

Singhal is responsible for nothing less than Google’s search algorithm—surely a mixture of mathematics and magic.
The article also tells us that Google has a computerized problem-reporting system where

Any of Google’s 10,000 employees can use its “Buganizer” system to report a search problem, and about 100 times a day they do — listing Mr. Singhal as the person responsible to squash them.

So why is he prioritizing his work using… a whiteboard?

Change scenes now to Yahoo’s pragprog discussion list, where experienced developers discuss their craft. (You have to sign up to see the messages.) A few days ago someone asked for recommendations on the best tool for UML. The first answer from one of the members was:

“I’ve always found a pen and a large sheet of paper rather effective myself :)”

Software development is still largely a human activity. Planning is human. Design is human. Only some of coding gets down to the machine level.

Humans like and need simple, flexible tools that don’t hold much more than people can think about at once. With planning—in its essential activity of prioritizing—if you get more than 10-15 items, the bottom of the list never gets done. No point keeping them on the list at all. For that, even better than paper, the whiteboard comes with a great accessory: the eraser. Likewise with design: the important part is the discussion, the thinking out loud. Whiteboards go great in hallways for such work.

Yes, sooner or later there’s code, and compilers, and syntax, regression test suites and repeatable builds. For those machine activities, use computerized tools and scripted automation.

But for people, stick with the whiteboard.

Don’t Watch Your Fingers

When I talk about tools for coding, some people ask,

“How can we be free when the environment is computer-controlled?”

It’s all about the right metaphor. One of mine is woodworking.

“By definition, a jig guides your tool and a fixture guides your work.”

Thus begins Jim Stack in The Best Jigs and Fixtures for Your Workshop. And he claims the 30 best in his book. But here’s a similar example from another source, to give you the idea.

There’s no improving on the authors’ explanation:

“If you’ve ever built a picture frame you’ve experienced the frustration of trying to get all of the parts to fit perfectly. While trying to “dry-fit” and clamp the frame’s pieces together the parts often fit perfectly. But add glue, and a little clamp pressure, and the parts begin to slide apart. It can be a frustrating race against the clock to even things out before the glue begins to set.

(By the way, by Stack’s definition, this example is really a fixture, not a jig.)

If you’ve ever tried framing a picture, you may know what they’re talking about.

Equally well if you’ve ever raced to get your program to compile and run before you go home.

Yes, that picture frame looks pretty tightly constrained. But it’s not limiting my creativity as a carpenter.  It’s just holding the pieces in alignment so they don’t all fall apart. I don’t find myself thinking creatively when I have glue all over my fingers and wood stuck to the carpet.

Your tools include the IDE and the compiler. Your work is, clearly, the code.

What kinds of add-ons (”jigs”) guide your tools? What kind of fixtures can help guide your work?

Here’s the rest of that site: http://www.woodzone.com/tips.htm. Look through it and then make your own list for coding.

You don’t even have to watch your fingers.

Craftsmen (and women) in the Modern Factory

Used to be that products were made by craftsmen. Lots of art, skill, and tools in the hands of an experienced individual.

The industrial age brought us the factory, with an emphasis on automation, process, and soon, statistical control.

More recently,  high-quality producers worked to combine the two, with individual or small-group responsibility for an entire component as part of an industrial manufacturing system.

In software, the building construction analogy has fallen. The important, human part of programming is design. Construction, so often error-prone, need not be. Take an ingredient from each part of the history above: from craftsmanship, discipline. From manufacturing, computerized tools.

Here’s how I see a modern factory of software craftspeople working:

• The best tools for writing good code attached to each developer’s IDE
• The same tools centrally build-automated (e.g. with CruiseControl), with IDE-compatible output too
• Code review for readability, maintainability, and extensibility of code, and for continuous developer improvement
• Constant professional discussion amongst developers and their group leaders about coding and detailed design
• Detailed root cause analysis to prioritize improvement efforts

No Free Lunch

Recently I read an article by Norman Balabanian (”On the Presumed Neutrality of Technology”, IEEE Technology & Society magazine, reprinted Winter 2006, pp. 15-25; originally published in 1980), where he addresses, among other things, the popular claim that people choose new technologies out of free choice. That if we use a technology, whether it be a car, or a refrigerator, or permanent-press clothing, we freely choose the benefits, and have accepted the costs. In several examples, Balabanian shows that we do not. We live modern life as part of an interlocking, consumption/profit-driven system which requires us to accept most of the new technologies, or starve.

I decided to do a small experiment in the household, to see up close what Balabanian was talking about.

The Experiment

I set out to collect and put aside all the food packaging that our family opened and threw away as part of our meals at home for one week. Not paper plates or napkins, or non-food trash such as newspapers. Just the containers that our food comes in. My kids wondered at first, but after a day or two I convinced them that even after school people still learn, and this was my self-assigned science project this week. As material rapidly piled up, I cut off “data” collection at 4 days.

What did I find?

I collected a half-full garbage bag of paper and cardboard, similar of plastic bags and bottles, and a few metal cans, for a total of 1 kg of waste material. By weight, the packaging was a bit less than 4% of the food net weight. Not much at all, but I was not concentrating on waste vs. recycling.

Rather, I was looking at the question: have we freely chosen to buy our food packaged?

Looking at the list below, I have to say “no”. Very few of these foods are available for purchase either at our corner grocery, or our large supermarket, without the packaging (i.e. take in your own container).

Reminds me of a cynical saying we used to have in high school: “You have a choice … and it has been made for you.”

The Challenge to Innovators

Don’t tell me how to recycle these materials. (We do already—as much as our city provides for.) That’s more forced choice: take the food in packages, and then recycle the packages.

Instead, suggest ways that we or the food distribution system could change so that we could eat our (reasonably) healthy diet without all the packaging in the first place.

Appendix 1: Four Days of Food Packaging

Food Amount (g) Type Wrapper Type Comments

Cheetos™ 55 Prepared Plastic Silverized
Pudding 330 Prepared Plastic
Fruit 1000 Prepared Plastic Styrofoam
Flour 2000 Prepared Paper
Cola 3000 Prepared Plastic
Milk 3000 Raw Plastic
Cereal 1200 Prepared Paper Waxed cardboard
Cereal 0 Prepared Plastic
Cottage Ch. 250 Prepared Plastic
Sugar 1000 Raw Paper
Potato Chips 50 Prepared Plastic Silverized
Eggs 1400 Raw Paper Cardboard
Soda Water 1500 Prepared Plastic
Smoked Salmon 100 Prepared Plastic
Granola 500 Prepared Plastic
Brown Sugar1000 Raw Paper
Yogurt 450 Prepared Plastic
Fruit Juice3000 Prepared Plastic
Margarine 2000 Raw Paper Waxed paper
Tuna Fish 320 Prepared Metal
Pineapple 825 Prepared Metal
Crackers 325 Prepared Plastic Silverized
Egglplant Spread 1000 Prepared Plastic
Mineral water 3000 Raw Plastic
Total Food Weight 27305

We’re Not in Kansas Anymore

Usually I just write my book reviews on Amazon, but this book is important enough that I wanted to post my review here too. Dreaming in Code by Scott Rosenberg.

We’re Not in Kansas Anymore

And we haven’t been for a long time. To hear Rosenberg tell it, we briefly entered modernity in the 1950’s with computer hardware, and then quickly regressed into an ironic Greek tragedy of software, where all of us spend our time playing either Sisyphus, fixing endless “bugs”, or Tantalus, finding on-time delivery methods just beyond our reach. Ironic because in this story, Sisyphus is often happy.

Dreaming in Code is not about coding. Nor is it fiction. It is disturbing, realistic fact. An outsider’s diary of a software project, blessed with a successful visionary leader—Mitch Kapor of Lotus 1-2-3 fame—and almost no resource constraints, which nevertheless exhibits all the “usual” problems.

Rosenberg competently brings in software history and references all in the field will recognize. More important, though, he shows software to be one of those great advances that is fraught with deadly imperfections. Like, say, hospitals, or automobile travel.

To software people. You can keep your eyes closed, come to work, and enjoy dreaming in code. Or you can wake up for a day and read this book.

(An extra note, since I’ve read some of the reviews, including those on the back cover. Some miss the point. This book is not about the Chandler project, nor should members of apparently successful software projects ignore it lightly. It is also not a disguised methodology book on how to develop software. It is about our expectations from software and from ourselves, to show us ourselves in a mirror and raise questions.)

To everyone else. I haven’t the slightest idea what non-software people will think reading this book. But Rosenberg has made it accessible, using few technical terms and explaining them simply. So you owe it to yourself to look here. And we need your insight— don’t leave the field just to us.

Postscript

I remember back in the 60’s and 70’s, a two-sentence conversation I often used to have with people outside of the computer (software) field, when space adventure movies (I’m thinking of Kubrick’s “2001“) were popular:

Them: Gosh, imagine what it would be like out on a spaceship, depending only on a computer for life support.

Me: You don’t have to imagine anymore. We are both on one right now.

The Practicing Developer

I’ve often thought about the advantage that athletes and musicians have over software developers: time to practice—lots of it. Most time is spent in exercises and rehearsals, to produce the best peformance.

In software, everyone spends most of the time producing, with a bit of “time off” for learning new technologies. No wonder perfection seems so far off.

A friend pointed out that Dave Thomas has proposed CodeKata as a way of doing that practice. I read it, and while the ideas look good, I felt there was something missing. The “katas” (formal patterns in karate, and here, in design and coding) seemed somehow schoolwork-like and disconnected from real work. I could only imagine myself “practicing” on assignments where I needed the result, even if for something trivial like importing a bunch of e-mails into SharePoint (more about that another time).

I don’t have an answer today, but the question is much bigger than just finding time to practice in software development. So instead, have a look at these posts and presentations where people are discussing the issue, and see what you think.

Level 5 means never having to say you’re sorry (Jeff Atwood)

Big Macs vs. The Naked Chef (Joel Spolsky)

No Best Practices (James Bach)

Herding Racehorses and Racing Sheep (.ppt) (The Pragmatic Programmer)

Competence is a Habit (.ppt) (David Leach)