Talk About Quality

We may never know all the details of the recent Toyota failures, because, unlike the Challenger and NASA, Toyota is a commercial entity rather than taxpayer-funded government. And it’s a bit early to be looking for definitive analysis. What we can do, as embedded software developers, is realize what we’re up against, and how it didn’t start yesterday.

When microprocessors first came out the buzz on the news was that soon they were going to be inside everything. I remember wondering what do we need them in everything for? My washer and my microwave oven, for example, worked just fine with a mechanical dial. But I was wrong and now they are in everything.

Much earlier on, complex systems were based on the Six Simple Machines. I can’t say which is simplest, but maybe it’s the lever. What’s important here is that the behavior of the lever is governed by physical properties of solids, which by now are pretty well understood. Push down on one end, and the other end goes up. Exceptions exist: solids can stretch under tension, deform under pressure, and if you push too hard, they can break. Apparently Toyota had problems with this too, in their gas pedal problem and fix.

Technical advances brought hydraulics — based on the properties of fluids and gases. I also remember replacing a brake line on an old car once, and getting to see how a leak in the line leads to spongy braking, and, if I hadn’t been refilling it weekly before the replacement, to brake failure.

In both of those technologies, the basic idea is that the system user (e.g. the driver) pushes on the control, force is transmitted immediately through a physical medium, and the desired effect occurs. Failure modes are well understood and (usually) are prevented.

Now replace the lever with software. No more physical properties: the behavior of the software transmission line is governed by the behavior of people — software developers writing lists of instructions which are translated several times before reaching the hardware. Now you push on the control, and a cacophony of instructions (okay, I’m being unfair — if the software is well-written, it’s a concert of instructions) rush down a wire to an actuating motor at the other end.

Let’s be clear about this: real-time software is a myth. The metal of the lever, or even the fluid of the hydraulic line, has been removed and replaced by your entire software development organization. Materials science out the door, organizational sociology in its place. If you want to know where that leaves us, read this short post about drive-by-wire, and the many comments there.

What’s the answer? In the code itself — perhaps it’s avoiding interrupts and multi-threading, and moving to synchronous programming where everything proceeds in lock-step. At least then a modern car would run like clockwork. Other than that, notice that many of the suggestions in the discussion talk about mechanical back-up systems, which are basically putting the lever back into the system.

But software development is not, as commonly believed, about technology. It is about writing, reading, and communication in ever-larger groups. Those who have said, “The pen is mightier than the sword” were thinking of politics. Now writing, in the form of software, interposes itself between every hand and hammer. We have broken the lever; do we know how to wield its replacement?

Recently I found an excuse to dust off some old lab books of introductory science experiments. They were the curriculum for a 9th grade Physical Science I class at my high school; I wrote them as a summer job during college. I didn’t invent the experiments. Rather, the head teacher gave me outlines, and my job was to try them out and make sure they would work, so that prospective students wouldn’t get frustrated and turned off by science.

The excuse was a friend, a scientist, who is meeting a public service requirement of her studies by volunteering to teach science to schoolchildren, and to their teachers. I offered my science course as possible material for her classwork. As we looked over the experiments, we doubted a bit how even 9th graders might learn from them, or be as excited about them as I was when I built and tested them myself. Sure, the projects were easy, and they had worked. But would the experience of building these projects — static electricity demonstrations, electromagnets, motors — be enough to convey the principles behind their operation? Would the kids see the point?

Yesterday I was in the kitchen preparing a simple lunch of tacos and beans. Washing and cutting lettuce, chopping an onion, peeling a cucumber, grating cheese, warming the beans with some tomato sauce. In the quiet afternoon, I couldn’t help noticing the crunch of the knife through the lettuce. Or wondering how best to preserve the other half of the cucumber, now leaking water from its open end. And why do we grate cheese? So much science, here in the kitchen!

Now with the internet, you can search “science in the kitchen” and get pages and pages of websites with all sorts of neat science projects in the kitchen. Let alone YouTube with some pretty exciting and dangerous experiments. (Be warned!) I have no doubt that school science textbooks have taken this to heart and now include experiments that relate to the real world. And yet, science enrollment declines.

Even with the best of intentions, popular science experiments in the kitchen won’t do it. Science is not in school, nor is it in the kitchen. Science is not an activity, but a way (just one way) of looking at the world and making sense of it. What excited me in science class was not the sitting and listening to the lecture. It was making the connections with daily life outside class, and enjoying the beauty of the natural world with new understanding.

Those understandings could come from things as simple as basic cell structure in biology. There’s lots of water in a living cell, so when you peel and cut a cucumber, it gets wet. Or as complex as thermodynamics. One college winter, I learned that there’s really never a flow of “cold”, but only heat transfer. For a good month after that, climbing the steps to class, I would grasp the banister outdoors, and instead of feeling cold, I felt the heat flowing out of my hand into the metal. These experiences are what brought me back to study even more.

Science teachers have to have and share that excitement. If they’re not science experts themselves, no matter. They are learning adults, who can build their own understanding and catch the excitement from science mentors. The key has to be in giving the right answer to the question so many students ask: “But what is this good for?” The wrong answers are the allegedly practical ones — advancing technology, getting a job, or passing the test. The right answer, the one that should guide science teaching, and bring the kids back to class, is “Because the world around you is beautiful, and science gives you eyes to see it.”

A medical radiation machine operator types the letter ‘x’, realizes it’s an error, backspaces, types ‘e’, and continues. Consequences of the error and related defects lead to the patient’s untimely death.

Ten years later, a Japanese stock broker mistakenly switches the share price and amount on the sell order of a new stock. He tries to cancel the order, but fails. His employers lose $225 million, and are involved in lawsuits for years.

A blogger clicks the “back arrow” by mistake, gets a warning message, concludes he meant to continue writing, and clicks “OK” to continue. He loses his work, and has to rewrite it.  Here, the consequences are annoying, if trivial by comparison.

What do all these cases have in common? Canceling a request. If it’s not hard enough to program computers to do what we want them to do, who would have thought that telling them not to do it would be hard too?

The cancellation scenario — or “use case” as it’s called in software design — is the silent partner of every positive request supported by a piece of software. It has to cover giving the user clear options, executing the cancellation, and rolling back any partial results. Things get more complicated if authorization is required, or if the transaction has already gone through (both of those requirements figured into the Japanese broker error story). Cancellation and rollback are also part of automatic requests that may occur if one software module (the “server”) cannot complete a request by another (the “client”) and has to make sure to put everything back the way it was, and send the proper response code.

So the next time you’re designing a piece of software, no matter how simple, think what it’s supposed to do, but also what it will do if the user or client calls out, “Cancel that!”

“Britain’s Ugly Duckling Breaks Out in Song” I slowly translated off the showbiz page of a foreign-language newspaper. I had to work at it to figure out that it said “Susan Boyle” and then look up her appearance on YouTube from a week ago. Anyone who wants to be moved by song, and doesn’t mind (or enjoys) the contrast with “beautiful people” celebrity judges putting feet in their mouths, should stop and listen.

I thought I would have nothing to add to the commentary on a musical appearance that saw over two million views (and that’s just on one upload, let alone the original TV broadcast). But after reading the 5-day-old Wikipedia entry, and some of the newspaper articles in the references, I wondered why nobody offered the obvious. Good singing comes from interpretive ability, soul, and poise. Anyone who has enjoyed opera would have no reason to be surprised–and every reason to be moved–by Boyle’s voice. Similarly by that of her “predecessor” Paul Potts. It is the furthest thing from coincidence that Potts sang opera, and Boyle sang from a musical, both genres that are formally performed live without a microphone.

So, while Tanya Gold and others may not be wrong in their social analyses, they are missing the point. Song, in contrast to the child of yesteryear, is to be heard and not seen.

I’m reading and excellent book Expert C Programming: Deep C Secrets, by Peter Van Der Linden. It’s the book all C programmers need, because it’s an explanation of why this ever-popular language works (or doesn’t work) the way it does. It also prompts me to review why “code quality” is necessary and what it is.

Code Quality

Ways of writing code that affect software maintenance time and correctness (the “people side”), and that affect computer execution performance and correctness (the “machine side”).

Naturally, it follows that good quality code is code which is written so that maintenance is easy and execution is fast, efficient, and correct.

Today, for a change, I’d like to talk about the “machine side” of things. Re-reading about the details of C, a language known for being high-level but “close to the machine”, made me want to review, from the bottom up, what a computer is, so that code, and code quality, can be placed in context.

I’m taking a big risk offering these definitions without looking them up (I may do that later), but here goes. I am trying to give only the essentials—the absolute minimum required to define the terms. Even though I am an electronics engineer, I have deliberately left out the word “electronic” as an unnecessary popularization of one application of electricity. At the same time, apologies in advance to physicists and chemists who will notice my skipping over their levels of mechanics and electricity. Keeping it simple here.

Machine

A thing which allows action at a distance. Generally has a defined input-output function: person does this to it, and it produces that response to the action.

Simple Machine

There’s a famous short list of them out there—here’s a fun example. To name just one, a lever: press this down over here, and over there, that goes up.

State Machine

A machine that has more than one state, or position of its parts, that it can be in. Specific actions take it from state to state. State machines can be mechanical. Even a see-saw is a state machine.

Clocked State Machine

A state machine that proceeds from state to state by having each state create the next action, which action is applied at the next independently-determined, regular time interval. Not suprisingly, the pendulum clock is the prototype, mechanical clocked state machine. Hence the name “clock” in computers (which we didn’t get to yet).

Electric State Machine

A state machine whose “position” is in fact the pattern of electrical charge. Even a lightbulb is an electric state machine. So is a bit of computer memory.

Computer

A clocked electrical state machine. As we will see later, this definition is enough to make it a generic machine—a machine that can do almost anything people want it to do.

Digital Computer

A computer where all the states are combinations of parts’ states which can only take N fixed integer values.

Binary Digital Computer

A computer where N is 2. Generally the two values are called 0 and 1. But of course the 0 and 1 don’t exist physically. They appear as two different charge patterns in the electrical parts of the computer.

Machine Language

A small set of binary numbers, with corresponding computer state-change responses. When a special part of the computer is forced to take on the state represented by one of these numbers (popularly called “loaded into memory”), at the next (one or a few) clock cycle(s), the computer will change to the corresponding new state. Also, a machine language is written by the computer parts manufacturer, and supplied with it.

Assembly Language

A small set of letter combinations which map 1:1 to the machine language. Exist only because most people remember letter combinations better than number combinations.

Computer Program

A list of combinations of language elements (“statements”) that, when loaded into memory along with a “start” instruction, cause a computer to proceed automatically from state to state.

High-level Language

A set of words, and rules for combining them, that, when used in a computer program, which is passed through another computer program (called a “compiler” if processed all at once, or an “interpreter” if processed one word at a time), produce a machine language computer program.

Software Design (activity)

Deciding how a computer program should be organized to best cause the running program to be compiled or interpreted so that the computer will do what was required. (See “Requirements”, immediately below.)

Requirements

A set of statements, in a human natural language, each one containing “shall” or “must”, which mostly describe how a computer should respond to actions applied to it.

Well, that was a lot, but much shorter than a college textbook!

Where Code Quality Fits In

Go back and read “high-level language”. That’s where code quality fits in, and why it’s a challenge. The code must both represent the design description, and meet the constraints of the particular high-level language. Further, that language may have been written for the convenience of the compiler writer. Finally, a large part of the machine language that makes up the running program does not come from the developer’s high-level language program, but from third-party programs written by multiple hardware manufacturers. It’s like copying a painting while looking at the painting in a mirror, and looking at the canvas in another mirror.

No wonder that under those constraints, writing code that is both clear to people, and correct for the compiler, is difficult. But with the twin tools of code review and static analysis, it is possible.

Recently I read an article by Norman Balabanian (“On the Presumed Neutrality of Technology”, IEEE Technology & Society magazine, reprinted Winter 2006, pp. 15-25; originally published in 1980), where he addresses, among other things, the popular claim that people choose new technologies out of free choice. That if we use a technology, whether it be a car, or a refrigerator, or permanent-press clothing, we freely choose the benefits, and have accepted the costs. In several examples, Balabanian shows that we do not. We live modern life as part of an interlocking, consumption/profit-driven system which requires us to accept most of the new technologies, or starve.

I decided to do a small experiment in the household, to see up close what Balabanian was talking about.

The Experiment

I set out to collect and put aside all the food packaging that our family opened and threw away as part of our meals at home for one week. Not paper plates or napkins, or non-food trash such as newspapers. Just the containers that our food comes in. My kids wondered at first, but after a day or two I convinced them that even after school people still learn, and this was my self-assigned science project this week. As material rapidly piled up, I cut off “data” collection at 4 days.

What did I find?

I collected a half-full garbage bag of paper and cardboard, similar of plastic bags and bottles, and a few metal cans, for a total of 1 kg of waste material. By weight, the packaging was a bit less than 4% of the food net weight. Not much at all, but I was not concentrating on waste vs. recycling.

Rather, I was looking at the question: have we freely chosen to buy our food packaged?

Looking at the list below, I have to say “no”. Very few of these foods are available for purchase either at our corner grocery, or our large supermarket, without the packaging (i.e. take in your own container).

Reminds me of a cynical saying we used to have in high school: “You have a choice … and it has been made for you.”

The Challenge to Innovators

Don’t tell me how to recycle these materials. (We do already—as much as our city provides for.) That’s more forced choice: take the food in packages, and then recycle the packages.

Instead, suggest ways that we or the food distribution system could change so that we could eat our (reasonably) healthy diet without all the packaging in the first place.

Appendix 1: Four Days of Food Packaging

Food Amount (g) Type Wrapper Type Comments

Cheetos™ 55 Prepared Plastic Silverized
Pudding 330 Prepared Plastic
Fruit 1000 Prepared Plastic Styrofoam
Flour 2000 Prepared Paper
Cola 3000 Prepared Plastic
Milk 3000 Raw Plastic
Cereal 1200 Prepared Paper Waxed cardboard
Cereal 0 Prepared Plastic
Cottage Ch. 250 Prepared Plastic
Sugar 1000 Raw Paper
Potato Chips 50 Prepared Plastic Silverized
Eggs 1400 Raw Paper Cardboard
Soda Water 1500 Prepared Plastic
Smoked Salmon 100 Prepared Plastic
Granola 500 Prepared Plastic
Brown Sugar1000 Raw Paper
Yogurt 450 Prepared Plastic
Fruit Juice3000 Prepared Plastic
Margarine 2000 Raw Paper Waxed paper
Tuna Fish 320 Prepared Metal
Pineapple 825 Prepared Metal
Crackers 325 Prepared Plastic Silverized
Egglplant Spread 1000 Prepared Plastic
Mineral water 3000 Raw Plastic
Total Food Weight 27305

Usually I just write my book reviews on Amazon, but this book is important enough that I wanted to post my review here too. Dreaming in Code by Scott Rosenberg.

We’re Not in Kansas Anymore

And we haven’t been for a long time. To hear Rosenberg tell it, we briefly entered modernity in the 1950′s with computer hardware, and then quickly regressed into an ironic Greek tragedy of software, where all of us spend our time playing either Sisyphus, fixing endless “bugs”, or Tantalus, finding on-time delivery methods just beyond our reach. Ironic because in this story, Sisyphus is often happy.

Dreaming in Code is not about coding. Nor is it fiction. It is disturbing, realistic fact. An outsider’s diary of a software project, blessed with a successful visionary leader—Mitch Kapor of Lotus 1-2-3 fame—and almost no resource constraints, which nevertheless exhibits all the “usual” problems.

Rosenberg competently brings in software history and references all in the field will recognize. More important, though, he shows software to be one of those great advances that is fraught with deadly imperfections. Like, say, hospitals, or automobile travel.

To software people. You can keep your eyes closed, come to work, and enjoy dreaming in code. Or you can wake up for a day and read this book.

(An extra note, since I’ve read some of the reviews, including those on the back cover. Some miss the point. This book is not about the Chandler project, nor should members of apparently successful software projects ignore it lightly. It is also not a disguised methodology book on how to develop software. It is about our expectations from software and from ourselves, to show us ourselves in a mirror and raise questions.)

To everyone else. I haven’t the slightest idea what non-software people will think reading this book. But Rosenberg has made it accessible, using few technical terms and explaining them simply. So you owe it to yourself to look here. And we need your insight— don’t leave the field just to us.

Postscript

I remember back in the 60′s and 70′s, a two-sentence conversation I often used to have with people outside of the computer (software) field, when space adventure movies (I’m thinking of Kubrick’s “2001“) were popular:

Them: Gosh, imagine what it would be like out on a spaceship, depending only on a computer for life support.

Me: You don’t have to imagine anymore. We are both on one right now.

Rarely does a topic stir up more controversy than code review. And global warming doesn’t either, though perhaps it should.

I was grateful for the opportunity to see Al Gore’s very slick hour-and-a-half documentary, An Inconvenient Truth, about global warming and carbon emissions. Whatever you may think of Gore, blatant promos for Apple laptops, or computer-generated tear-jerker sequences of drowning polar bears, it is worth seeing as a thought-provoking presentation.

Remember to think.

Some of the questions I was left with after recovering from the “mild thematic elements” (yes, that’s what gave it a PG rating) are:

• What does “carbon neutral” mean? And “carbon offset”?
• Are carbon subtraction programs working as planned?
• Is carbon emission reduction going to save us from a global warming disaster?

Nothing is as simple as it seems.

Carbon neutral refers to “calculating your total climate-damaging carbon emissions, reducing them where possible, and then balancing your remaining emissions, often by purchasing a carbon offset. (Related term: carbon negative.) “

(If anyone has a more formal source than a word popularity article, let me know.)

Hmm. What about “carbon negative”? It’s easy to get distracted on the web, but the only real answer is planting more trees. (All that stuff about buying carbon dioxide emission reductions from other organizations is nonsense — not that it’s not helpful, but it’s not taking carbon out of the atmosphere – just paying for someone else to put less in rather than reducing one’s own emissions.) A bit disturbing to find, then, that even Friends of the Earth has raised concerns about how carbon-reduction tree planting is being carried out. Humbling too—one more example of how we never quite know the effects of one man-made intervention carried out to mitigate another.

Finally, if one can entertain the thought that world climate change might be a bit more complicated than a 100-minute movie, there’s short, medium, and long reading to be found at JunkScience.com.

What seems to me, though, is that the reductions—that are currently fashionable to call carbon emission reductions—are probably good for “old-fashioned” (1960′s) reasons: reduce landfill, keep air clean for breathing, improve personal health through exercise, and increase spiritual health by slowing down.

Will all this get lost in the Carbon Rush?

I was reading Steve Yegge’s rant on Agile, praise of Google, and all the responses there. I won’t get into all that, but one topic that Steve mentioned, as did one of his commenters, Ryan Baker, was “work queues”.

Attraction: (Yegge, in part) “With nothing more than a work queue (a priority queue, of course), you immediately attain most of the supposedly magical benefits of Agile Methodologies.”

Concern: Before writing this post, I thought I should search the term. I found, for example, a utility from The Code Project. Why does that worry me? Because structures that are good for computers are not necessarily good for people. (Unfortunately people who work in computers know those structures best and are often tempted.)

Then I realized that I had had a work queue experience just this past week.

Last Thursday I had 6 Splint warnings in my code. My goal: to get rid of them one by one, and gain some hands-on experience with this open-source static analysis tool. My method: See it as a work queue of 6 items, and attack each one in turn. Well, I tried a half-hour on each two or three of them, and when I was done, I had … 8 Splint warnings.

Why?

From just a quick initial look, I had realized that the warnings were talking about two concepts that I happened not to understand in detail in C just then. (If you’re curious, they are internal/external scope of variables and definition vs. declaration, and how and when to free malloc’d memory — both in the context of structs).

I ignored that realization and started on my work queue anyway. I tried this, and I tried that, and of course failed. Now I will go ask a C programmer who knows. But I could have done that after the first 5 minutes (reading the warnings and understanding them).

What’s the problem?

Work-queue-based work has no direction—no guidance.

It’s good for washing dishes, but not for any activity that takes skill.

For that, get guidance — get a mentor.