The Illusion of software testing

Magic tricks and optical illusions deceive us. We often see what we are looking for, even when it is not there. And we miss what we are not looking for, even when it is there. There are a myriad reasons for this.

Edward de Bono pointed out that stage magicians lead their audience “to make assumptions or elaborations that are perfectly reasonable, but do not, in fact, match what is being done in front of them”. So we are not seeing the world as it is, we are seeing the world as our brain chooses to see it.

With illusions, we are amazed by the conflict between what we expect and what we observe and this is entertaining. We would not find it so entertaining if we were deceived in testing. But since software testing is all about observation, it is clearly possible. Take a look at this well-known image by Ludimar Hermann:
optical illusion 1.jpg
Do you see any black dots? Yes? But if you stare at a black dot it will disappear and become a white dot. So in this simple example, we are seeing something that is not there.

Imagine the test was that there should be black dots on this page and you glanced at the image saw the dots and passed the test. Passing a test that should have failed is surely one of the cardinal sins of software testing.

Not seeing what is there is also easy, especially if we are concentrating on observing something specific. The invisible gorilla is a fantastic demonstration of this. Even if you have seen the invisible gorilla video before, you will probably enjoy this video, which is slightly different from the original. As the commentary points out, those who had seen the video before were looking for the gorilla, so they saw it. But they probably missed the other unusual changes in the video. The ones they weren’t looking for.

If we transpose this situation into testing what should we do? Imagine the requirements or specifications had no mention of a gorilla. Should we report it? Many functional testers working for suppliers would not, citing the mandate to make sure the product or service meets the specification, not to raise defects which are out of scope for that phase of testing. This means that the best-outcome scenario is finding the defect in a later phase of testing (and the worst outcome results in the users seeing the defect).

This opens up a whole new topic of the context of the observer and their relationship and interaction with the environment they are observing. We’ll save this for another post.

The question we want to pose in this post is, because we are aware that we can be deceived by what we observe can we take steps to reduce the risk of this occurring? We will be exploring this further in a series of posts in this blog which we will run under the heading of “The illusion of software testing”, paying homage to Myers .

Contact acutest