MQM.book Page 1 Wednesday, May 7, 2008 11:29 AM AL Chapter 1 MA TE RI What Is the Big Deal About Testing? GH TE D “Time waste differs from material waste in that there can be no salvage. The easiest of all wastes and the hardest to correct is the waste of time, because wasted time does not litter the floor like wasted material.” -Henry Ford CO PY RI This book is about saving time.
MQM.book Page 2 Wednesday, May 7, 2008 11:29 AM 2 ■ Optimize Quality for Business Outcomes Any person on the manufacturing floor can stop the entire manufacturing process if they notice something is not right. Everybody on the assembly line is a quality checker. When a problem is spotted early, it’s a lot cheaper to fix at that point, instead of cranking out a bunch of flawed cars and later having to rework the design, or possibly even do a recall.
MQM.book Page 3 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 3 Figure 1: IT project success rate has changed over the past 10 years To learn why the success rate is so low, we wondered: Why were the successful projects successful? The Standish group also collected data on success factors, asking respondents what contributed to their project’s success. Executive support and user involvement were high on the list. Here are the other factors that figured in project success.
MQM.book Page 4 Wednesday, May 7, 2008 11:29 AM 4 ■ Optimize Quality for Business Outcomes Reliable estimates (5%) Other criteria (5%) While we expected to see executive support and experienced project management, we were interested in other noteworthy elements like user involvement, clear business objectives, and firm basic requirements on the list.
MQM.book Page 5 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 5 Defects in this phase happen mostly because the requirements themselves are incomplete, ambiguous, or contradictory. When unclear requirements are translated into developers’ technical specifications, misalignment of the original ambiguous business requirement and misinterpreted technical specification results in significant rework.
MQM.book Page 6 Wednesday, May 7, 2008 11:29 AM 6 ■ Optimize Quality for Business Outcomes The whole process begins when somebody needs something — in this case a means of transport (the product) — and expresses an expectation (the requirement). The contractor asks no questions because he has a picture in his mind of what the customer wants. Now, the contractor has made an assumption without checking with the customer. We’re clearly missing some communication — a requirement proof point here.
MQM.book Page 7 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 7 At this point, people rarely go back to their initial assumption (in this case, that the customer needs a bicycle) to find out whether the additional information (the customer doesn’t want to get wet and needs to carry a briefcase) means the whole design should be reconsidered.
MQM.book Page 8 Wednesday, May 7, 2008 11:29 AM 8 ■ Optimize Quality for Business Outcomes Despite all efforts to the contrary, we’ve got an unhappy customer along with a wasted investment into something that doesn’t solve the problem. After more iterations, the solution that the contractor comes up with will be more what the customer originally wanted and the problem is solved.
MQM.book Page 9 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 9 The answer lies in the cost of fixing defects in various stages of the SDLC (Figure 3). Figure 3: The cost of correcting defects increases dramatically as we move toward the end of the SDLC If we are complacent, neglecting to identify defects introduced in the requirements phase until we get to user acceptance testing, we lose approximately $7,000 per defect. This quickly adds up to significant numbers.
MQM.book Page 10 Wednesday, May 7, 2008 11:29 AM 10 ■ Optimize Quality for Business Outcomes To answer that question let’s first look at some testing definitions: Testing is about finding defects, bugs, or flaws. This is true for software, as well as for all other products built to suit specific needs.
MQM.book Page 11 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 11 The HP Quality Model is a pragmatic approach to software testing that aligns IT with business outcomes. Quality Goals and Test Phases To build meaningful and concise tests, we need to define requirements that describe the system behavior we anticipate. But that alone would not be enough to specify the intensity or the coverage needed for the tests to meet the end users’ quality goals.
MQM.book Page 12 Wednesday, May 7, 2008 11:29 AM 12 ■ Optimize Quality for Business Outcomes how well a component meets specified requirements or user needs and expectations. And, it’s the ability of a component to produce specified outputs when given specified inputs, and the extent to which they match or satisfy the requirements. Completeness — How well the component implements all required capabilities.
MQM.book Page 13 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 13 This phased approach is a common practice in engineering and is done to assure the quality of a single unit, before it is verified as an integrated unit to other units. The positive effects are obvious: When you discover an error before getting too far down the production line, it pays off in tremendous cost savings.
MQM.book Page 14 Wednesday, May 7, 2008 11:29 AM 14 ■ Optimize Quality for Business Outcomes Figure 4: HP Quality Model In the HP Quality Model, we recommend the following test phases: Requirements Verification In the Requirements Verification phase, the end user adds acceptance criteria to each requirement in order to answer the question, “What would make me sign off on this requirement?” The acceptance criteria needs to: Validate the completeness and correctness of the requirements.
MQM.book Page 15 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 15 Unit Test The Unit Test is a white-box test conducted by the developers to find out whether the application is stable and completely implemented from a developer’s perspective. (We look into white-box and black-box testing later in this chapter.) Unit testing validates that a particular module of source code is working as the developer intended. A developer conducts the testing, not an end user.
MQM.book Page 16 Wednesday, May 7, 2008 11:29 AM 16 ■ Optimize Quality for Business Outcomes User Acceptance Test The User Acceptance Test (UAT) is done by users or their representatives on a new or changed information system. If the system behaves according to their expectations, the user acceptance testers approve deploying the system. Cosmetic and other small changes may still be required as a result of the test, but the system is considered stable and behaving according to requirements.
MQM.book Page 17 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 17 Figure 5: The software test lifecycle is shown running parallel to and integrated with the SDLC Case Study: Flight Application We’ll illustrate how the phased approach is applied by examining the operation of a web-based portal that allows customers to book airline flights. The portal is an easy-to-use, straightforward solution that allows customers to search, select, and book flights in the flight database.
MQM.book Page 18 Wednesday, May 7, 2008 11:29 AM 18 ■ Optimize Quality for Business Outcomes The following functions are needed: Log in Log out Register Search flight Book flight Requirements Verification The prerequisite for the Requirements Verification phase is broken down into logical business functions (Figure 6).
MQM.book Page 19 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 19 a button or ENTER on the keyboard, the validation of username and password should start.” The acceptance criteria for that requirement could be: If a username without a password is entered, an error message is displayed. If the username and the password are a correct combination, the system shows the flight-search screen.
MQM.
MQM.book Page 21 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 21 All identified defects have been resolved. (“Resolved” means they’re either closed or a decision has been made to delay the closing.) Business Process Test When it comes to the Business Process Test, we want to make sure to test the various possible business function sequences. In our flight-booking example, the possible process is shown in Figure 7.
MQM.book Page 22 Wednesday, May 7, 2008 11:29 AM 22 ■ Optimize Quality for Business Outcomes System Test During the System Test, each individual application should have passed the quality gates for the Integration Test. The System Test often involves large and complex environments with multiple independent applications working together to support a business process (Figure 8).
MQM.book Page 23 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 23 Accessibility — A system is usable by as many people as possible with modification. Possible quality gates for the System Test are: All end-to-end processes can be executed. No severe defects exist. Performance Test The Performance Test helps us: Demonstrate that the system meets performance criteria. Compare two systems to find which performs better.
MQM.book Page 24 Wednesday, May 7, 2008 11:29 AM 24 ■ Optimize Quality for Business Outcomes User Acceptance Test When users are satisfied that the business functional specifications and business requirements have been met, system signoff and release can take place. The User Acceptance Test (UAT) phase involves turning loose a number of real end users to try the system, using it as they would in real life to accomplish whatever task the system is designed for.
MQM.book Page 25 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 25 This test phase is the last check of the production system for operational readiness before the system go-live point. This phase should be a part of the Project Implementation Plan.
MQM.book Page 26 Wednesday, May 7, 2008 11:29 AM 26 ■ Optimize Quality for Business Outcomes In software testing, it’s usually a programmer that performs white-box tests. Often, multiple programmers will write tests based on certain code to gain various perspectives on possible outcomes.
MQM.book Page 27 Wednesday, May 7, 2008 11:29 AM What Is the Big Deal About Testing? ■ 27 The most common techniques and approaches used in blackbox testing are the smoke test, equivalence partitioning, boundary value analysis, and user input validation. (For detailed descriptions of these tests techniques, see Appendix A.
MQM.