Some Experience

Over the past years working with product developers I've asked, "Why do you test your products?" The overwhelming response has been, "We want to give our customers a reliable and high quality product."

Well, this seems quite reasonable - the eventual goal we all seek. Then I've asked, "what methods do you use to show that your products are reliable and of high quality?" Again, the overwhelming answer has been, "We test it thoroughly enough to know that it works as we expect." This makes sense; but, when asked, "What do you expect?" - I have heard things such as…

• "We know what our clients want."
• "The Boss or President OKs the results."
• "The system boots-up and runs just fine!"
• "We (engineers) have built 20 units and they all work."
• "We build good products; we are very experienced" (i.e. why do you even need to ask, Pal?)
• "We have a product spec."

All of these responses are stock answers, which do not prove what is claimed or even establish a path to a proof.

When further asked about testing: "How do you know that your tested product is of the high quality and reliability you say you want?" - the responses rapidly become ethereal or spiritual. Unfortunately, too many product developers believe that 'product reliability and quality' is only mantra2:

• "The totality of features and characteristics of a product or service that bears on its ability to meet a stated or implied need." (ISO 8402 3.1, 1986)
• The ability of your product to satisfy your customers.
• Consistency of Implementation.
• Quality engenders trust.
• "Better than you would or could, have ever expected."
• The customers' expressed and implied requirements are met fully.
• "Quality is doing things right the first time."
• "Quality is not perfection."
• "Quality is not negotiable; the product either meets or does not meet customer needs."
• "Quality is composed of those characteristics or features of a product, service or process which are needed to satisfy customer needs, or achieve fitness for use."
• "Quality characteristics need to be specified and their achievement controlled, assured, improved, managed and demonstrated."

Saying what must be so… does not make it so.

Therefore, where can you start and what can you do?