Common Testing Pitfalls & Ways to Prevent & Mitigate Them Descriptions Symptoms Consequences Causes & Recommendations

–Capers Jones, VP and CTO, Namcook Analytics LLC

Software and system testers repeatedly fall victim to the same pitfalls. Think of them as “anti-patterns”: mistakes that make testing far less effective and efficient than it ought to be. In Common System and Software Testing Pitfalls, Donald G. Firesmith catalogs 92 of these pitfalls. Drawing on his 35 years of software and system engineering experience, Firesmith shows testers and technical managers and other stakeholders how to avoid falling into these pitfalls, recognize when they have already fallen in, and escape while minimizing their negative consequences.

Firesmith writes for testing professionals and other stakeholders involved in large or medium-sized projects. His anti-patterns and solutions address both “pure software” applications and “software-reliant systems,” encompassing heterogeneous subsystems, hardware, software, data, facilities, material, and personnel. For each pitfall, he identifies its applicability, characteristic symptoms, potential negative consequences and causes, and offers specific actionable recommendations for avoiding it or limiting its consequences.

This guide will help you

• Pinpoint testing processes that need improvement–before, during, and after the project
• Improve shared understanding and collaboration among all project participants
• Develop, review, and optimize future project testing programs
• Make your test documentation far more useful
• Identify testing risks and appropriate risk-mitigation strategies
• Categorize testing problems for metrics collection, analysis, and reporting
• Train new testers, QA specialists, and other project stakeholders

With 92 common testing pitfalls organized into 14 categories, this taxonomy of testing pitfalls should be relatively complete. However, in spite of its comprehensiveness, it is also quite likely that additional pitfalls and even missing categories of pitfalls will be identified over time as testers read this book and compare it to their personal experiences. As an enhancement to the print edition, the author has provided the following location on the web where readers can find major additions and modifications to this taxonomy of pitfalls: http://donald.firesmith.net/home/common-testing-pitfalls

Please send any recommended changes and additions to dgf (at) sei (dot) cmu (dot) edu, and the author will consider them for publication both on the website and in future editions of this book.

–Robert V. Binder, robertvbinder.com

“Don’s compilation of real-world testing problems, symptoms, and solutions is the most comprehensive available. You can use it early in your project to prevent these problems. Or you can use it at the end, as a ready list of costly lessons you could have avoided, had you used it early on. I’m afraid this book’s publication will undermine a lot of excuses for repeating these mistakes.”

–Vince Alcalde, National Australia Bank

“Excellent, Excellent, Excellent! This book should be mandatory reading for anyone involved in product development. Donald’s book addresses the pitfalls that need to be understood and allowed for in all product development verification and validation planning. While the focus of the book is on software projects, most of the pitfalls are equally applicable to any size project that involves both hardware and software.”

–Louis S. Wheatcraft, Requirements Experts Inc.

“The potential impact of this book cannot be overstressed. Software systems that are not adequately tested do not adequately evolve. I highly recommend this book as a must-read for people directly involved in the development and management of software- intensive systems.”

–Dr. Kenneth E. Nidiffer, Director of Strategic Plans for Government Programs, Software Engineering Institute, Carnegie Mellon University

“Common System and Software Testing Pitfalls identifies realistic testing pitfalls. More importantly, it also identifies solutions for avoiding them on your next project. Every manager should read this book and follow the recommendations.”

–Barry Stanly, Enterprise Technology Alliance

“Whether you are a novice tester or a seasoned professional, you will find this book to be a valuable resource. The information on how to identify and prevent problem areas is clear, concise and, most importantly, actionable.”

–Allison Yeager, Blackbaud

“First of all, this is great material! It contains probably all of the testing problems I have faced in my career and some that I wasn’t aware of. . . . Thank you for the opportunity to read this superb material!”

–Alexandru Cosma, Frequentis

“As a tester, I consider Common System and Software Testing Pitfalls by Donald Firesmith to be a must-read book for all testers and QA engineers.”

–Thanh Huynh, LogiGear

“Your book provides very good insight and knowledge. After working in IT for over thirty years, and focusing on software testing the past thirteen years, I still learned more tips and best practices in software testing.”

–Tom Zalewski, Texas State Government

“This book is essential for the people in the cyber security business . . . I can see it becoming a classic. Don has done a great job.”

–Michael Hom, Compass360 Consulting

“Awesome work. Very mature.”

–Alejandro Salado, Kaiser, Threde Gmbh

“All in all, a great document.”

–Peter Bolin, Revolution IT Pty Ltd.

Testing system/software today can take 30-50% of a product's schedule and budget. This expensive and inefficient process leaves many residual defects to be delivered with the operational system/software. This book is about the testing-related pitfalls that projects commonly fall into – and more importantly, how to avoid them.

Common Testing Pitfalls and Ways to Prevent and Mitigate Them goes beyond naming, describing, and categorizing pitfalls. Not a how-to book, but a how-not-to, Common Testing Pitfalls teaches you to recognize the symptoms of an at-risk project, describes the negative effects that can result from falling into each particular pitfall, alerts you to the causes of these pitfalls, and recommends steps for dealing with them. With the help of this book, you will be enabled to avoid pitfalls as you prepare and perform a project, mitigate problems as they occur, and verify that the pitfalls have been dealt with properly.

Common Testing Pitfalls can be used as a training material for testers and stakeholders of testing. This book will help your team improve communication regarding the commonly occurring testing pitfalls. Through the development and review process of project testing, you will be equipped to identify areas that will potentially need improvements and ensure that your project won’t fall into the common testing pitfalls.

Foreword           xiii

Preface             xvii

About the Author            xxiii

Chapter 1: Overview               1

1.1 What Is Testing?        1

1.2 Testing and the V Models     2

1.3 What Is a Defect?    5

1.4 Why Is Testing Critical?   7

1.5 The Limitations of Testing    9

1.6 What Is a Testing Pitfall?    10

1.7 Categorizing Pitfalls    11

1.8 Pitfall Specifications    11

Chapter 2: Brief Overviews of the Testing Pitfalls       13

2.1 General Testing Pitfalls    13

   2.1.1 Test Planning and Scheduling Pitfalls    13

   2.1.2 Stakeholder Involvement and Commitment Pitfalls    14

   2.1.3 Management-Related Testing Pitfalls   14

   2.1.4 Staffing Pitfalls   15

   2.1.5 Test-Process Pitfalls    16

   2.1.6 Test Tools and Environments Pitfalls    17

   2.1.7 Test Communication Pitfalls   18

   2.1.8 Requirements-Related Testing Pitfalls   19

2.2 Test-Type-Specific Pitfalls   20

   2.2.1 Unit Testing Pitfalls   20

   2.2.2 Integration Testing Pitfalls   20

   2.2.3 Specialty Engineering Testing Pitfalls   21

   2.2.4 System Testing Pitfalls   22

   2.2.5 System of Systems (SoS) Testing Pitfalls   22

   2.2.6 Regression Testing Pitfalls    23

Chapter 3: Detailed Descriptions of the Testing Pitfalls           25

3.1 Common Negative Consequences    25

3.2 General Recommendations    26

3.3 General Testing Pitfalls    28

   3.3.1 Test Planning and Scheduling Pitfalls    28

   3.3.2 Stakeholder Involvement and Commitment Pitfalls    44

   3.3.3 Management-Related Testing Pitfalls    51

   3.3.4 Staffing Pitfalls    65

   3.3.5 Test Process Pitfalls   75

   3.3.6 Test Tools and Environments Pitfalls    106

   3.3.7 Test Communication Pitfalls    131

   3.3.8 Requirements-Related Testing Pitfalls   143

3.4 Test-Type-Specific Pitfalls   164

   3.4.1 Unit Testing Pitfalls    164

   3.4.2 Integration Testing Pitfalls    169

   3.4.3 Specialty Engineering Testing Pitfalls    177

   3.4.4 System Testing Pitfalls    206

3.4.5 System of Systems (SoS) Testing Pitfalls    211

3.4.6 Regression Testing Pitfalls    225

Chapter 4: Conclusion             241

4.1 Future Work    241

4.2 Maintaining the Lists of Pitfalls    242

Appendix A: Glossary         243

Appendix B: Acronyms          253

Appendix C: Notes             255

Appendix D: REFERENCES           269

Appendix E: Planning Checklist          271

Index             279