Testing vs. Checking ... my 2 cents.

I was pleased to see Michael Bolton's series on Testing vs. Checking. If you haven't been following, what I consider to be the central thread of the topic (and the unfortunately inevitable fallout that seems to happen in "testerland" almost any time someone says something that makes sense).

From Michael:

From James Bach:

From Scott Barber:

See, I hopped up on the "test vs. check" soapbox back in '05 when I got sick of being asked for performance requirements and test cases that start with "The system shall [x]" and "Verify that the system does [x]" (I provide much more detail about my distaste for Performance Test Cases in my Chapter of Beautiful Testing, due out in October). I used the terms "investigate vs. validate" in the hope of getting people to focus on what I was saying instead of arguing about whether or not validating (or checking) is "testing". It didn't really work. Instead of discussing/assessing/debating/enhancing the value and usefulness of the distinction, I found myself endlessly fielding question about whether I really *meant* to say "verifying" not "validating" because "At my company that's what we call it."

So I guess I'm jumping on the bandwagon, but it never hurts to have more supporting references.

As an aside, I cannot express how tired I am of hearing "At my company we call that...", "Well, that wouldn't work at my company", and "We don't do that at my company" used as a dismissal in response to almost any idea that requires thought, consideration, or (gasp) change. Ya'know what folks? The entire world is not like "your company." And if it were, then what, exactly, do you expect to find in training courses, magazines, blogs and conferences that will be of value to you? Seriously, if you think you know it all, stop being so greedy and start sharing it so the rest of us can catch up. Otherwise, what harm would it do you to actually take a moment to understand and consider what others have to say instead of being so rudely dismissive? Maybe what others are saying is appropriate for "your company" and maybe it's not, but how can you be sure when you're spending your energy dismissing the possibility rather than considering the possible applications, implications and derivations that *just might* be valuable or useful to "your company," or maybe even "your next company".

Honestly, to me it feels like testing, on the whole, is simply afraid to change. I don't get it. The rest of the software industry is changing constantly, but the overall state of testing seems to be bolted firmly in place. This isn't true for every individual, nor every company, so please, if this description doesn't fit you, don't take it personally. But if you don't believe this is the general state of affairs, go to testing conferences -- once every three years or so will do -- and tell me if you see any significant advancement (other than an amazing advancement in folks ability to repackage old ideas as new ones).

However, if you are contributing to testing's fear of change,to the fear of having to admit that someone else might have come up with something potentially or situationally valuable, or to the inertia that appears to be the root of why many testers are unwilling to consider ideas different from their own, then do the industry a favor and stay out of training classes, stop reading blogs, and above all please stop spending your time deliberately derailing discourse among those who actually want to collaborate thoughtfully in the hope of advancing the state of software testing beyond the '70's. When you're ready, come back with an open mind, your thinking cap on, your experiences at your side and you'll be welcomed with open arms -- at least by me.

This is just another case of the same 'ole debates resurfacing every few years. Maybe someday we'll advance beyond outdated ignorance and inertia, but in case that doesn't happen soon, I'm marking my calendar to republish my Investigate vs. Validate article in 2013 (with links to these discussions, and enhancements to encompass what I learn between now and then.) If the value, applications, and implications of thinking about the distinction doesn't catch on this time around, maybe it will then.

Scott Barber
President & Chief Technologist, PerfTestPlus, Inc.
Executive Director, Association for Software Testing
Co-Author, Performance Testing Guidance for Web Applications
"If you can see it in your mind...
     you will find it in your life."


I just stumbled across this blog entry by Aaron Evans. It is a wonderful example of the kind of response I'd expect from a tester. He demonstrates understanding of the concept, he makes distinctions that demonstrate his own thoughtfulness and experience, then goes on to disagree with the semantics of "test vs. check". This is the kind of "discussing/assessing/debating/enhancing the value and usefulness of the distinction" I mention above.

Aaron gets a "gold star" from Scott Barber, and he doesn't even agree with me.

See, it's really *not* that hard to both disagree *and* enhance the conversation. That's all I ask of folks. If you're going to enter a conversation, add something to it.

Believe it or not, many of the folks in this field who get labeled as being "bullies", "arrogant" or whatever (people like James Bach and myself, for example), we *like* when people disagree/debate with us and we respect the people who do it. Debate and disagreement helps us sharpen our ideas, helps us express them better, helps us evolve and enhance them, and sometimes even helps us to recognize that we're barking up the wrong tree with a particular idea.

I won't go so far as to speak for anyone else, but *I* have all the time and energy in the world for someone who disagrees with me and is willing to engage in a conversation intended to find the points of agreement/disagreement and explore whether one/both of us suffer from a logic error, failed to consider a context, and/or whether there may be some "grand unifying idea" behind the whole concept. What I don't have respect for is people who dismiss ideas out of hand, speak out against them, but are unwilling/unable to do so much as discuss what it is about the idea that doesn't work for them.

I see it like this. My 6-year old son saying seems to *always* want to be snacking on something, but he often over does it and spoils his meals, so I've told him to ask me before getting a snack. Sometimes he asks and I answer "Not right now, we'll be having dinner soon". If he responds by saying "Will it be much longer dad? I just want a little one to hold me over until dinner." I'm likely to reverse my decision, especially if I'm not yet actually in the process of making dinner. If, however, he said "Yeah, whatever dad." while walking away from me and grabbing a snack anyway, well, let's just say that would not result in such a positive outcome.

Anyway, nice work Aaron. For those of you who don't have Aaron's blog in your "read feed", I recommend it. http://fijiaaron.wordpress.com/


Scott Barber
CTO, PerfTestPlus
Executive Director, AST

I can really relate to your 6 year old.

Aaron Evans