Informed guess as a testing tool
Submitted by Ainars Galvans on Fri, 20/04/2007 - 12:53.
exploratory test patterns | general software testing | metaphors | perspectives
I want to talk about one of the tools that I use as a tester: guessing. Not a random guess! I’m talking about a guess based on experience and analysis. The fastest experience analysis happens in pre-conscious mind, more over it has access to cognitive experience of a lifetime.
A long time ago I compared test execution with a sword, but test documentation with a shield. My intend was to compare defense with credibility but offence with achieving results and that good results may constitute credibility in certain context. So the informed guess as a tool is a two handed sword: you can’t parry or dodge, or block – you could only attack, but your attach is incredibly powerful.
Sometimes called intuition or experience based testing
See this post for more experiences. But let me quote the following:
(...) I work in aviation, and yes, I agree, I wouldn't trust applications tested via exploratory methods. At the same time, (...)
We used to call it "intuitive testing" and add time in the test schedule specifically for such testing. I think an intelligent, thinking individual is not going to be limited by a set of test cases if they see something odd, different, or that sparks their curiosity. (...)
Linda shows perfectly well how what I’m going talk about works well even in industry such as aviation. In industry I work defects found in production are not such expensive as they are perhaps in aviation. As a result I do not limit my practices of intelligent (intuitive) way to perform test beyond scripted test cases. I use it a lot to save time by not performing what described in test cases.
Anatomy of what they call intuition or experience-based
In 19th century Freud described the phenomena that are still referenced in both cognitive science and psychology. I found a simple and short description:
"The sub-conscious contains all of the emotional and cognitive experience of a lifetime" "The pre-conscious mind is like a problem-oriented and independent file-clerk. It looks over the shoulder of the conscious mind: when a problem is being considered, it conducts a search into the sub-conscious mind for clues that it considers relevant. “
So sub-conscious contains my experience with this and other software applications, other defects and issues I've seen. That experience may help me to recognize the same patterns in what I observe now. But I need to give some information for the pre-conscious mind to give him search criteria. So I need to run a test and see the pattern in order to trigger pre-conscious mind perform the search.
Need a balance between “intuition” with logic
I agree with those who say testing goal is to provide information about quality. I also agree that the best way to do that is the cognitive process of concurrent product learning, test design and test execution.
More over to provide the information we need to process it logically, analyze it objectively. Divide that information so that reader could look at parts. We want to prove that our testing was sequential and achieve a rational coverage.
For a purpose I've used above all the terms associated with the left-brain in this study:
http://www.funderstanding.com/right_left_brain.cfm
The right brain associated with random, intuitive, Holistic, Synthesizing, etc... are however extremely useful during collection of the data. And the main idea is that in this process we use experience/memory of a lifetime – not only that acquired during product learning, test design and test execution.
Let me show this at 3 different experiences
1) Guessing reasons for not-repeatable bugs. I like this type of challenges . To guess the reasons I actually need to guess more tests that would localize the problem and at the end come up with test that repeat the problem in 100% of the time.
I know ideas sometimes pops-up in my mind from nowhere. Well I don’t say I’m so lucky to get the repeatable test idea, only idea for test that gives me more clue – more information to analyze.
2) Regression tests
If all the defect fixes and improvements done in a component are retested by me I could analyze all these changes to guess the risk of regression issues in certain functionality. The risks depend on: this particular product, this (those) developer(s), for the type of functionality, for programming language, architecture, design pattern, etc. There are too much of this information for conscious mind to process, but pre-conscious mind could come up with pretty good guess.
3) problem-solving skills or at least one ingredient of it is based on our ability to recall experiences.
Problem: confidence level unknown and ways to increase it.
In conclusion I would like to add some negative aspect. Pre-conscious mind’s “criteria for relevance do not always seem logical to the conscious mind, and therefore the ego learns to censor certain kinds of information from the sub-conscious, preventing them from rising higher into full consciousness” .
The issue is that pre-conscious mind actually never tells you what are the criteria of relevance and your conscious mind wouldn’t always understand it anyway. So you never know how confident you could be about the search results, especially when empty results are received: do you need to refine your search criteria or give up?
A long time ago I compared test execution with a sword, but test documentation with a shield. My intend was to compare defense with credibility but offence with achieving results and that good results may constitute credibility in certain context. So the informed guess as a tool is a two handed sword: you can’t parry or dodge, or block – you could only attack, but your attach is incredibly powerful.
Sometimes called intuition or experience based testing
See this post for more experiences. But let me quote the following:
(...) I work in aviation, and yes, I agree, I wouldn't trust applications tested via exploratory methods. At the same time, (...)
We used to call it "intuitive testing" and add time in the test schedule specifically for such testing. I think an intelligent, thinking individual is not going to be limited by a set of test cases if they see something odd, different, or that sparks their curiosity. (...)
Linda shows perfectly well how what I’m going talk about works well even in industry such as aviation. In industry I work defects found in production are not such expensive as they are perhaps in aviation. As a result I do not limit my practices of intelligent (intuitive) way to perform test beyond scripted test cases. I use it a lot to save time by not performing what described in test cases.
Anatomy of what they call intuition or experience-based
In 19th century Freud described the phenomena that are still referenced in both cognitive science and psychology. I found a simple and short description:
"The sub-conscious contains all of the emotional and cognitive experience of a lifetime" "The pre-conscious mind is like a problem-oriented and independent file-clerk. It looks over the shoulder of the conscious mind: when a problem is being considered, it conducts a search into the sub-conscious mind for clues that it considers relevant. “
So sub-conscious contains my experience with this and other software applications, other defects and issues I've seen. That experience may help me to recognize the same patterns in what I observe now. But I need to give some information for the pre-conscious mind to give him search criteria. So I need to run a test and see the pattern in order to trigger pre-conscious mind perform the search.
Need a balance between “intuition” with logic
I agree with those who say testing goal is to provide information about quality. I also agree that the best way to do that is the cognitive process of concurrent product learning, test design and test execution.
More over to provide the information we need to process it logically, analyze it objectively. Divide that information so that reader could look at parts. We want to prove that our testing was sequential and achieve a rational coverage.
For a purpose I've used above all the terms associated with the left-brain in this study:
http://www.funderstanding.com/right_left_brain.cfm
The right brain associated with random, intuitive, Holistic, Synthesizing, etc... are however extremely useful during collection of the data. And the main idea is that in this process we use experience/memory of a lifetime – not only that acquired during product learning, test design and test execution.
Let me show this at 3 different experiences
1) Guessing reasons for not-repeatable bugs. I like this type of challenges . To guess the reasons I actually need to guess more tests that would localize the problem and at the end come up with test that repeat the problem in 100% of the time.
I know ideas sometimes pops-up in my mind from nowhere. Well I don’t say I’m so lucky to get the repeatable test idea, only idea for test that gives me more clue – more information to analyze.
2) Regression tests
If all the defect fixes and improvements done in a component are retested by me I could analyze all these changes to guess the risk of regression issues in certain functionality. The risks depend on: this particular product, this (those) developer(s), for the type of functionality, for programming language, architecture, design pattern, etc. There are too much of this information for conscious mind to process, but pre-conscious mind could come up with pretty good guess.
3) problem-solving skills or at least one ingredient of it is based on our ability to recall experiences.
Problem: confidence level unknown and ways to increase it.
In conclusion I would like to add some negative aspect. Pre-conscious mind’s “criteria for relevance do not always seem logical to the conscious mind, and therefore the ego learns to censor certain kinds of information from the sub-conscious, preventing them from rising higher into full consciousness” .
The issue is that pre-conscious mind actually never tells you what are the criteria of relevance and your conscious mind wouldn’t always understand it anyway. So you never know how confident you could be about the search results, especially when empty results are received: do you need to refine your search criteria or give up?