Synopses & Reviews
It's simple: you need to know how to work with the bash shell if you want to get to the heart of Mac OS X, Linux, and other Unix systems. Updated for the most recent version of bash, this concise little book puts all of the essential information about bash at your fingertips. You'll quickly find answers to annoying questions that always come up when you're writing shell scripts -- What characters do you need to quote? How do you get variable substitution to do exactly what you want? How do you use arrays? -- and much more.
If you're a user or programmer of any Unix variant, or if you're using bash on Windows, you'll find this pocket reference indispensable. This book covers:
- Invoking the Shell
- Arithmetic Expressions
- Command History
- Programmable Completion
- Job Control
- Shell Options
- Command Execution
- Restricted Shells
- Built-in Commands
The key to mastering any Unix system, especially Linux and Mac OS X, is a thorough knowledge of shell scripting. Scripting is a way to harness and customize the power of any Unix system, and it's an essential skill for any Unix users, including system administrators and professional OS X developers. But beneath this simple promise lies a treacherous ocean of variations in Unix commands and standards.
bash Cookbook teaches shell scripting the way Unix masters practice the craft. It presents a variety of recipes and tricks for all levels of shell programmers so that anyone can become a proficient user of the most common Unix shell -- the bash shell -- and cygwin or other popular Unix emulation packages. Packed full of useful scripts, along with examples that explain how to create better scripts, this new cookbook gives professionals and power users everything they need to automate routine tasks and enable them to truly manage their systems -- rather than have their systems manage them.
This guide teaches shell scripting the way Unix masters practice the craft. It presents a variety of recipes and tricks for all levels of shell programmers so that anyone can become a proficient user of the most common Unix shell--the bash shell--and cygwin or other popular Unix emulation packages.
About the Author
Carl Albing is currently a Distinguished Visiting Professor in the Department of Computer Science at the U.S. Naval Academy where he is teaching courses on programming languages and on High Performance Computing. Prior to this he was writing software for some of the biggest and fastest computers in the world as a software engineer for Cray, Inc. As an independent consultant, he is comfortable programming with C, Java, bash and much more. Carl is the coauthor of two books, one on Java development on Linux and his latest, the O'Reilly "bash Cookbook". A former software consultant, manager, analyst and programmer with an amazing breadth of software experience, Carl has worked with companies in the US, Canada and Europe. He has worked for large companies and small startups, in technical as well as in managerial and marketing roles. Carl's software projects, past and present, involve the design and development of distributed computing software, medical image processing applications, compilers, medical devices, web-based factory floor automation, and more. Carl's education includes a Ph.D. in Computer Science as well as a B.A. degree in Mathematics and an International MBA. He has spoken at conferences and training seminars in the US, Canada and Europe as well as local high schools and colleges. Carl enjoys speaking at user groups and seminars on Linux, C, Java, and bash topics.
JP Vossen has been working with computers since the early 80s and has been in the IT industry since the early 90s, specializing in Information Security since the late 90s. He's been fascinated with scripting and automation since he first understood what an autoexec.bat was, and was delighted to discover the power and flexibility of bash and GNU on Linux in the mid-90s. He has previously written for Information Security Magazine and SearchSecurity.com, among others. On those few occasion when he's not in front of a computer, he is usually taking something apart, putting something together, or both.
Cameron Newham lives in Perth, Western Australia. After completing a Bachelor of Science majoring in information technology and geography at the University of Western Australia, Cameron joined Universal Defence Systems (later to become Australian Defence Industries) as a software engineer. He has been with ADI for six years, working on various aspects of command and control systems. In his spare time Cameron can be found surfing the Internet, ballroom dancing, or driving his sports car. He also has more than a passing interest in space science, 3D graphics, synthesiser music, and Depeche Mode.
Table of Contents
Preface; Who Should Read This Book; About This Book; Conventions Used in This Book; Using Code Examples; We'd Like to Hear from You; Safari® Enabled; Acknowledgments; Chapter 1: Beginning bash; 0 Why bash?; 0 The bash Shell; 1.1 Decoding the Prompt; 1.2 Showing Where You Are; 1.3 Finding and Running Commands; 1.4 Getting Information About Files; 1.5 Showing All Hidden (dot) Files in the Current Directory; 1.6 Using Shell Quoting; 1.7 Using or Replacing Built-ins and External Commands; 1.8 Determining If You Are Running Interactively; 1.9 Setting bash As Your Default Shell; 1.10 Getting bash for Linux; 1.11 Getting bash for xBSD; 1.12 Getting bash for Mac OS X; 1.13 Getting bash for Unix; 1.14 Getting bash for Windows; 1.15 Getting bash Without Getting bash; 1.16 Learning More About bash Documentation; Chapter 2: Standard Output; 2.1 Writing Output to the Terminal/Window; 2.2 Writing Output but Preserving Spacing; 2.3 Writing Output with More Formatting Control; 2.4 Writing Output Without the Newline; 2.5 Saving Output from a Command; 2.6 Saving Output to Other Files; 2.7 Saving Output from the ls Command; 2.8 Sending Both Output and Error Messages to Different Files; 2.9 Sending Both Output and Error Messages to the Same File; 2.10 Appending Rather Than Clobbering Output; 2.11 Using Just the Beginning or End of a File; 2.12 Skipping a Header in a File; 2.13 Throwing Output Away; 2.14 Saving or Grouping Output from Several Commands; 2.15 Connecting Two Programs by Using Output As Input; 2.16 Saving a Copy of Output Even While Using It As Input; 2.17 Connecting Two Programs by Using Output As Arguments; 2.18 Using Multiple Redirects on One Line; 2.19 Saving Output When Redirect Doesn't Seem to Work; 2.20 Swapping STDERR and STDOUT; 2.21 Keeping Files Safe from Accidental Overwriting; 2.22 Clobbering a File on Purpose; Chapter 3: Standard Input; 3.1 Getting Input from a File; 3.2 Keeping Your Data with Your Script; 3.3 Preventing Weird Behavior in a Here-Document; 3.4 Indenting Here-Documents; 3.5 Getting User Input; 3.6 Getting Yes or No Input; 3.7 Selecting from a List of Options; 3.8 Prompting for a Password; Chapter 4: Executing Commands; 4.1 Running Any Executable; 4.2 Telling If a Command Succeeded or Not; 4.3 Running Several Commands in Sequence; 4.4 Running Several Commands All at Once; 4.5 Deciding Whether a Command Succeeds; 4.6 Using Fewer if Statements; 4.7 Running Long Jobs Unattended; 4.8 Displaying Error Messages When Failures Occur; 4.9 Running Commands from a Variable; 4.10 Running All Scripts in a Directory; Chapter 5: Basic Scripting: Shell Variables; 5.1 Documenting Your Script; 5.2 Embedding Documentation in Shell Scripts; 5.3 Promoting Script Readability; 5.4 Separating Variable Names from Surrounding Text; 5.5 Exporting Variables; 5.6 Seeing All Variable Values; 5.7 Using Parameters in a Shell Script; 5.8 Looping Over Arguments Passed to a Script; 5.9 Handling Parameters with Blanks; 5.10 Handling Lists of Parameters with Blanks; 5.11 Counting Arguments; 5.12 Consuming Arguments; 5.13 Getting Default Values; 5.14 Setting Default Values; 5.15 Using null As a Valid Default Value; 5.16 Using More Than Just a Constant String for Default; 5.17 Giving an Error Message for Unset Parameters; 5.18 Changing Pieces of a String; 5.19 Using Array Variables; Chapter 6: Shell Logic and Arithmetic; 6.1 Doing Arithmetic in Your Shell Script; 6.2 Branching on Conditions; 6.3 Testing for File Characteristics; 6.4 Testing for More Than One Thing; 6.5 Testing for String Characteristics; 6.6 Testing for Equal; 6.7 Testing with Pattern Matches; 6.8 Testing with Regular Expressions; 6.9 Changing Behavior with Redirections; 6.10 Looping for a While; 6.11 Looping with a read; 6.12 Looping with a Count; 6.13 Looping with Floating-Point Values; 6.14 Branching Many Ways; 6.15 Parsing Command-Line Arguments; 6.16 Creating Simple Menus; 6.17 Changing the Prompt on Simple Menus; 6.18 Creating a Simple RPN Calculator; 6.19 Creating a Command-Line Calculator; Chapter 7: Intermediate Shell Tools I; 7.1 Sifting Through Files for a String; 7.2 Getting Just the Filename from a Search; 7.3 Getting a Simple True/False from a Search; 7.4 Searching for Text While Ignoring Case; 7.5 Doing a Search in a Pipeline; 7.6 Paring Down What the Search Finds; 7.7 Searching with More Complex Patterns; 7.8 Searching for an SSN; 7.9 Grepping Compressed Files; 7.10 Keeping Some Output, Discarding the Rest; 7.11 Keeping Only a Portion of a Line of Output; 7.12 Reversing the Words on Each Line; 7.13 Summing a List of Numbers; 7.14 Counting String Values; 7.15 Showing Data As a Quick and Easy Histogram; 7.16 Showing a Paragraph of Text After a Found Phrase; Chapter 8: Intermediate Shell Tools II; 8.1 Sorting Your Output; 8.2 Sorting Numbers; 8.3 Sorting IP Addresses; 8.4 Cutting Out Parts of Your Output; 8.5 Removing Duplicate Lines; 8.6 Compressing Files; 8.7 Uncompressing Files; 8.8 Checking a tar Archive for Unique Directories; 8.9 Translating Characters; 8.10 Converting Uppercase to Lowercase; 8.11 Converting DOS Files to Linux Format; 8.12 Removing Smart Quotes; 8.13 Counting Lines, Words, or Characters in a File; 8.14 Rewrapping Paragraphs; 8.15 Doing More with less; Chapter 9: Finding Files: find, locate, slocate; 9.1 Finding All Your MP3 Files; 9.2 Handling Filenames Containing Odd Characters; 9.3 Speeding Up Operations on Found Files; 9.4 Finding Files Across Symbolic Links; 9.5 Finding Files Irrespective of Case; 9.6 Finding Files by Date; 9.7 Finding Files by Type; 9.8 Finding Files by Size; 9.9 Finding Files by Content; 9.10 Finding Existing Files and Content Fast; 9.11 Finding a File Using a List of Possible Locations; Chapter 10: Additional Features for Scripting; 10.1 "Daemon-izing" Your Script; 10.2 Reusing Code with Includes and Sourcing; 10.3 Using Configuration Files in a Script; 10.4 Defining Functions; 10.5 Using Functions: Parameters and Return Values; 10.6 Trapping Interrupts; 10.7 Redefining Commands with alias; 10.8 Avoiding Aliases, Functions; Chapter 11: Working with Dates and Times; 11.1 Formatting Dates for Display; 11.2 Supplying a Default Date; 11.3 Automating Date Ranges; 11.4 Converting Dates and Times to Epoch Seconds; 11.5 Converting Epoch Seconds to Dates and Times; 11.6 Getting Yesterday or Tomorrow with Perl; 11.7 Figuring Out Date and Time Arithmetic; 11.8 Handling Time Zones, Daylight Saving Time, and Leap Years; 11.9 Using date and cron to Run a Script on the Nth Day; Chapter 12: End-User Tasks As Shell Scripts; 12.1 Starting Simple by Printing Dashes; 12.2 Viewing Photos in an Album; 12.3 Loading Your MP3 Player; 12.4 Burning a CD; 12.5 Comparing Two Documents; Chapter 13: Parsing and Similar Tasks; 13.1 Parsing Arguments for Your Shell Script; 13.2 Parsing Arguments with Your Own Error Messages; 13.3 Parsing Some HTML; 13.4 Parsing Output into an Array; 13.5 Parsing Output with a Function Call; 13.6 Parsing Text with a read Statement; 13.7 Parsing with read into an Array; 13.8 Getting Your Plurals Right; 13.9 Taking It One Character at a Time; 13.10 Cleaning Up an SVN Source Tree; 13.11 Setting Up a Database with MySQL; 13.12 Isolating Specific Fields in Data; 13.13 Updating Specific Fields in Data Files; 13.14 Trimming Whitespace; 13.15 Compressing Whitespace; 13.16 Processing Fixed-Length Records; 13.17 Processing Files with No Line Breaks; 13.18 Converting a Data File to CSV; 13.19 Parsing a CSV Data File; Chapter 14: Writing Secure Shell Scripts; 14.1 Avoiding Common Security Problems; 14.2 Avoiding Interpreter Spoofing; 14.3 Setting a Secure $PATH; 14.4 Clearing All Aliases; 14.5 Clearing the Command Hash; 14.6 Preventing Core Dumps; 14.7 Setting a Secure $IFS; 14.8 Setting a Secure umask; 14.9 Finding World-Writable Directories in Your $PATH; 14.10 Adding the Current Directory to the $PATH; 14.11 Using Secure Temporary Files; 14.12 Validating Input; 14.13 Setting Permissions; 14.14 Leaking Passwords into the Process List; 14.15 Writing setuid or setgid Scripts; 14.16 Restricting Guest Users; 14.17 Using chroot Jails; 14.18 Running As a Non-root User; 14.19 Using sudo More Securely; 14.20 Using Passwords in Scripts; 14.21 Using SSH Without a Password; 14.22 Restricting SSH Commands; 14.23 Disconnecting Inactive Sessions; Chapter 15: Advanced Scripting; 15.1 Finding bash Portably for #!; 15.2 Setting a POSIX $PATH; 15.3 Developing Portable Shell Scripts; 15.4 Testing Scripts in VMware; 15.5 Using for Loops Portably; 15.6 Using echo Portably; 15.7 Splitting Output Only When Necessary; 15.8 Viewing Output in Hex; 15.9 Using bash Net-Redirection; 15.10 Finding My IP Address; 15.11 Getting Input from Another Machine; 15.12 Redirecting Output for the Life of a Script; 15.13 Working Around "argument list too long" Errors; 15.14 Logging to syslog from Your Script; 15.15 Sending Email from Your Script; 15.16 Automating a Process Using Phases; Chapter 16: Configuring and Customizing bash; 16.1 bash Startup Options; 16.2 Customizing Your Prompt; 16.3 Change Your $PATH Permanently; 16.4 Change Your $PATH Temporarily; 16.5 Setting Your $CDPATH; 16.6 Shortening or Changing Command Names; 16.7 Adjusting Shell Behavior and Environment; 16.8 Adjusting readline Behavior Using .inputrc; 16.9 Keeping a Private Stash of Utilities by Adding ~/bin; 16.10 Using Secondary Prompts: $PS2, $PS3, $PS4; 16.11 Synchronizing Shell History Between Sessions; 16.12 Setting Shell History Options; 13 Creating a Better cd Command; 16.14 Creating and Changing into a New Directory in One Step; 16.15 Getting to the Bottom of Things; 16.16 Adding New Features to bash Using Loadable Built-ins; 16.17 Improving Programmable Completion; 16.18 Using Initialization Files Correctly; 16.19 Creating Self-Contained, Portable RC Files; 16.20 Getting Started with a Custom Configuration; Chapter 17: Housekeeping and Administrative Tasks; 17.1 Renaming Many Files; 17.2 Using GNU Texinfo and Info on Linux; 17.3 Unzipping Many ZIP Files; 17.4 Recovering Disconnected Sessions Using screen; 17.5 Sharing a Single bash Session; 17.6 Logging an Entire Session or Batch Job; 17.7 Clearing the Screen When You Log Out; 17.8 Capturing File Metadata for Recovery; 17.9 Creating an Index of Many Files; 17.10 Using diff and patch; 17.11 Counting Differences in Files; 17.12 Removing or Renaming Files Named with Special Characters; 17.13 Prepending Data to a File; 17.14 Editing a File in Place; 17.15 Using sudo on a Group of Commands; 17.16 Finding Lines in One File But Not in the Other; 17.17 Keeping the Most Recent N Objects; 17.18 Grepping ps Output Without Also Getting the grep Process Itself; 17.19 Finding Out Whether a Process Is Running; 17.20 Adding a Prefix or Suffix to Output; 17.21 Numbering Lines; 17.22 Writing Sequences; 17.23 Emulating the DOS Pause Command; 17.24 Commifying Numbers; Chapter 18: Working Faster by Typing Less; 18.1 Moving Quickly Among Arbitrary Directories; 18.2 Repeating the Last Command; 18.3 Running Almost the Same Command; 18.4 Quick Substitution; 18.5 Reusing Arguments; 18.6 Finishing Names for You; 18.7 Playing It Safe; Chapter 19: Tips and Traps: Common Goofs for Novices; 19.1 Forgetting to Set Execute Permissions; 19.2 Fixing "No such file or directory" Errors; 19.3 Forgetting That the Current Directory Is Not in the $PATH; 19.4 Naming Your Script Test; 19.5 Expecting to Change Exported Variables; 19.6 Forgetting Quotes Leads to "command not found" on Assignments; 19.7 Forgetting That Pattern Matching Alphabetizes; 19.8 Forgetting That Pipelines Make Subshells; 19.9 Making Your Terminal Sane Again; 19.10 Deleting Files Using an Empty Variable; 19.11 Seeing Odd Behavior from printf; 19.12 Testing bash Script Syntax; 19.13 Debugging Scripts; 19.14 Avoiding "command not found" When Using Functions; 19.15 Confusing Shell Wildcards and Regular Expressions; Appendix A: Reference Lists; A.1 bash Invocation; A.2 Prompt String Customizations; A.3 ANSI Color Escape Sequences; A.4 Built-in Commands and Reserved Words; A.5 Built-in Shell Variables; A.6 set Options; A.7 shopt options; A.8 Adjusting Shell Behavior Using set, shopt, and Environment Variables; A.9 Test Operators; A.10 I/O Redirection; A.11 echo Options and Escape Sequences; A.12 printf; A.13 Date and Time String Formatting with strftime; A.14 Pattern-Matching Characters; A.15 extglob Extended Pattern-Matching Operators; A.16 tr Escape Sequences; A.17 Readline Init File Syntax; A.18 emacs Mode Commands; A.19 vi Control Mode Commands; A.20 Table of ASCII Values; Appendix B: Examples Included with bash; B.1 Startup-Files Directory Examples; Appendix C: Command-Line Processing; C.1 Command-Line Processing Steps; Appendix D: Revision Control; D.1 CVS; D.2 Subversion; D.3 RCS; D.4 Other; Appendix E: Building bash from Source; E.1 Obtaining bash; E.2 Unpacking the Archive; E.3 What's in the Archive; E.4 Who Do I Turn To?; Colophon;