The Art of Software Security Assessment: Identifying and Avoiding Software Vulnerabilities

—Halvar Flake, CEO and head of research, SABRE Security GmbH

The Definitive Insider’s Guide to Auditing Software Security

This is one of the most detailed, sophisticated, and useful guides to software security auditing ever written. The authors are leading security consultants and researchers who have personally uncovered vulnerabilities in applications ranging from sendmail to Microsoft Exchange, Check Point VPN to Internet Explorer. Drawing on their extraordinary experience, they introduce a start-to-finish methodology for “ripping apart” applications to reveal even the most subtle and well-hidden security flaws.

The Art of Software Security Assessment covers the full spectrum of software vulnerabilities in both UNIX/Linux and Windows environments. It demonstrates how to audit security in applications of all sizes and functions, including network and Web software. Moreover, it teaches using extensive examples of real code drawn from past flaws in many of the industry's highest-profile applications.

Coverage includes

• Code auditing: theory, practice, proven methodologies, and secrets of the trade

• Bridging the gap between secure software design and post-implementation review

• Performing architectural assessment: design review, threat modeling, and operational review

• Identifying vulnerabilities related to memory management, data types, and malformed data

• UNIX/Linux assessment: privileges, files, and processes

• Windows-specific issues, including objects and the filesystem

• Auditing interprocess communication, synchronization, and state

• Evaluating network software: IP stacks, firewalls, and common application protocols

• Auditing Web applications and technologies

This book is an unprecedented resource for everyone who must deliver secure software or assure the safety of existing software: consultants, security specialists, developers, QA staff, testers, and administrators alike.

Contents

ABOUT THE AUTHORS     xv

PREFACE     xvii

ACKNOWLEDGMENTS    xxi

I Introduction to Software Security Assessment

1 SOFTWARE VULNERABILITY FUNDAMENTALS    3

2 DESIGN REVIEW     25

3 OPERATIONAL REVIEW    67

4 APPLICATION REVIEW PROCESS    91

II Software Vulnerabilities

5 MEMORY CORRUPTION    167

6 C LANGUAGE ISSUES     203

7 PROGRAM BUILDING BLOCKS     297

8 STRINGS ANDMETACHARACTERS    387

9 UNIX I: PRIVILEGES AND FILES     459

10 UNIX II: PROCESSES     559

11 WINDOWS I: OBJECTS AND THE FILE SYSTEM     625

12 WINDOWS II: INTERPROCESS COMMUNICATION     685

13 SYNCHRONIZATION AND STATE    755

III Software Vulnerabilities in Practice

14 NETWORK PROTOCOLS    829

15 FIREWALLS    891

16 NETWORK APPLICATION PROTOCOLS    921

17 WEB APPLICATIONS    1007

18 WEB TECHNOLOGIES     1083

BIBLIOGRAPHY     1125

INDEX     1129

Mark Dowd is a principal security architect at McAfee, Inc. and an established expert in the field of application security. His professional experience includes several years as a senior researcher at Internet Security Systems (ISS) X-Force, and the discovery of a number of high-profile vulnerabilities in ubiquitous Internet software. He is responsible for identifying and helping to address critical flaws in Sendmail, Microsoft Exchange Server, OpenSSH, Internet Explorer, Mozilla (Firefox), Checkpoint VPN, and Microsoft’s SSL implementation. In addition to his research work, Mark presents at industry conferences, including Black Hat and RUXCON.

John McDonald is a senior consultant with Neohapsis, where he specializes in advanced application security assessment across a broad range of technologies and platforms. He has an established reputation in software security, including work in security architecture and vulnerability research for NAI (now McAfee), Data Protect GmbH, and Citibank. As a vulnerability researcher, John has identified and helped resolve numerous critical vulnerabilities, including issues in Solaris, BSD, Checkpoint FireWall-1, OpenSSL, and BIND.

Justin Schuh is a senior consultant with Neohapsis, where he leads the Application Security Practice. As a senior consultant and practice lead, he performs software security assessments across a range of systems, from embedded device firmware to distributed enterprise web applications. Prior to his employment with Neohapsis, Justin spent nearly a decade in computer security activities at the Department of Defense (DoD) and related agencies. His government service includes a role as a lead researcher with the National Security Agency (NSA) penetration testing team–the Red Team.

ABOUT THE AUTHORS     xv

PREFACE     xvii

ACKNOWLEDGMENTS    xxi

I Introduction to Software Security Assessment

1 SOFTWARE VULNERABILITY FUNDAMENTALS    3

2 DESIGN REVIEW     25

3 OPERATIONAL REVIEW    67

4 APPLICATION REVIEW PROCESS    91

II Software Vulnerabilities

5 MEMORY CORRUPTION    167

6 C LANGUAGE ISSUES     203

7 PROGRAM BUILDING BLOCKS     297

8 STRINGS ANDMETACHARACTERS    387

9 UNIX I: PRIVILEGES AND FILES     459

10 UNIX II: PROCESSES     559

11 WINDOWS I: OBJECTS AND THE FILE SYSTEM     625

12 WINDOWS II: INTERPROCESS COMMUNICATION     685

13 SYNCHRONIZATION AND STATE    755

III Software Vulnerabilities in Practice

14 NETWORK PROTOCOLS    829

15 FIREWALLS    891

16 NETWORK APPLICATION PROTOCOLS    921

17 WEB APPLICATIONS    1007

18 WEB TECHNOLOGIES     1083

BIBLIOGRAPHY     1125

INDEX     1129