Talk About Quality

Google’s “Mr. Automated Testing”, Miško Hevery, talks about Software Testing Categorization. From the title, it doesn’t sound like much, but it prompted a lot of good discussion. Some excerpts on key points, edited for length and clarity:

Know what you’re talking about

You hear people talking about small/medium/large/unit/integration/functional/scenario tests but do most of us really know what is meant by that? — Miško Hevery

Need for speed

Let’s start with unit test. The best definition I can find is that it is a test which runs super-fast (under 1 ms) and when it fails you don’t need debugger to figure out what is wrong. — Miško Hevery

Make slow tests faster by running them in the background on spare machine cycles. Works for static analysis tools too. For example, Riverblade’s Visual Lint add-on for Visual Studio and Gimpel Software’s PC-lint. — Talk About Quality

I heard a senior test manager present at a conference who espoused the sophistication of his automated test regime (on a mainframe no less). Countless tests ran every evening, unattended. Lots of clapping in the audience. In the break I met a guy who worked for the presenter. He asked, “do you want to know why the tests run so fast? We took out all the comparison checks”. — Paul Gerrard

You’re right to have a healthy concern. Whether automated tests run in a millisecond or in several hours, they always have to be re-reviewed to see if they test something useful. Passing tests are especially dangerous. Everyone thinks green is great so they don’t look at the tests to discover that features have moved on and the tests don’t test anything. Automated tests are like automatic shift (does anyone still drive stick like I do?): very nice and we take it for granted, but when you press on the gas you still have to look where you’re going so you don’t cause an accident. — Talk About Quality

Test Planning applies to automated testing too

At my previous employer, the standard unit tests took upwards of an hour (for fewer than 2000 tests). This discouraged developers from running them, which resulted in broken builds for everyone. When we worked on fixing this, mock objects provided most of the salvation. However, a good portion of the tests were really integration tests and by correctly identifying them as such, we were able to remove them from the standard suite, and instead run them nightly. What Misko is missing from his post is how test classifications go hand in hand with scheduling tests. — TheseThingsNeedFixed

It’s not hard to find a list of the better-known prerequisites for Continuous Integration. See, for example, Take Advantage of Continuous Integration, in a 2006 issue of Visual Studio magazine. They list:

• source code repository
• fully automated build process
• automated tests

But slap those together and turn it on, and you may still be left with a build dashboard full of “FAIL” messages, and nobody quite sure how to fix them and keep them fixed.

From my recent experience, there are at least two additional requirements. As usual, it’s the implicit requirements–the ones everyone assumes are alread met but really aren’t–that make all the difference. So, continuing the list,

• reliable builds, identical between developer’s workstation and the central build system

Sounds obvious, right? It’s really two requirements that go together, and neither is so obvious in practice.

Reliable: when a build doesn’t always work exactly the same way on a developer’s machine, s/he can re-run it, and may never find the time to troubleshoot why it doesn’t work 100% of the time. But when a central build system runs it, everyone expects a passing build to pass every time. Fail has to mean “something’s wrong with the code”, not, “oops, I guess that build didn’t work this time.”

Identical: the build results obtained by the developer and by the build system must match. Nothing saps faith in a build system like the “WOMB” (“works on my box”) response. (Thanks to Don Hass for that acronym, in his post Who cares if the build is broken?)

And speaking of which, Don was actually commenting on Derrick Bailey’s original notes, The Build Is Broken… Who Cares? Our last prerequisite for today:

• Somebody has to care that the build is broken

Derrick concludes that

“The entire team cares.” If we create an atmosphere where a broken build is a bad thing, then we have created a more productive team.

But to create that atmosphere in the first place, there has to be a first person who cares. Someone who looks at the entire build system display, often with multiple components and platforms, identifies who needs to do what to get each build passing, and follows up daily to get the display all green. It can be a thankless job, but it’s well worth it, to ensure that all the builds are reliable and passing, create that self-sustaining teamwork atmosphere, and make Continuous Integration really work.

Software is all design, with construction being automated or at least automatically tested. So said Jack W. Reeves in his article What is Software Design back in 1992. James Shore followed up and defined good design as one which:

“… minimizes the time required to create, modify, and maintain the software while achieving acceptable run-time performance.”

Always Ready to Run (and Install)

Continuous Integration with Test-Driven Development (and other automated tests on the build) ensures that functionality is always ready. Include automated build of installation script and the software is always ready to install and run.

The automated build system part of continuous integration knows how to return the essential “broke the build” or “build OK” message. The first follows the “fail early” principle. The second indicates perfect functional quality of the features that are in the build; all functional discussions can focus on which features are ready and whether they are as desired.

What about changes and new features?

What ensures, though, that the software is always ready to easily modify and extend?

Static analysis tools and complexity measures can help, but there’s a lot of important stuff that no machine can tell you.

For example, just one key attribute of easy-to-change code is:

Meaningful entity names

Write me a rule that finds the ones that aren’t.

A much easier way to find, and recommend improvements to, such people-oriented code attributes is code review.

On top of a continuous integration system with automated analysis and test, and a developer practice of TDD and constant refactoring (at the lowest level — not turning the entire design upside-down every week), all new code, and the edges of the existing code it interfaces with, should be reviewed.

Automated static analysis should find the language or construction-related problems, leaving human code reviewers to focus on things like readability, simplicity, and other design principles that only a person can find.

What kind of code review?

Pre-check-in review is an absolute minimum so that code review issues prevent check-in and thus prevent a build that runs, but whose code (design) isn’t ready to work with.

Pair programming is a nice idea but must need good pairing and people who are very comfortable with each other to make a difference.

Weekly desk review (the reviewer’s desk, by him or herself first) and discussion of all code written that week is best: it allows reviewer to see and advise on the bigger picture.

It’s the only way to keep the design which is in your code, as well as its functionality, Always Ready.