iii
Preface v
1. Document Conventions ................................................................................................... v
1.1. Typographic Conventions ...................................................................................... v
1.2. Pull-quote Conventions ........................................................................................ vi
1.3. Notes and Warnings ........................................................................................... vii
2. We Need Feedback! ..................................................................................................... viii
1. Introduction 1
1.1. Documentation Goals ................................................................................................... 1
1.2. SystemTap Capabilities ................................................................................................ 1
2. Using SystemTap 3
2.1. Installation and Setup ................................................................................................... 3
2.1.1. Installing SystemTap .......................................................................................... 3
2.1.2. Installing Required Kernel Information RPMs ....................................................... 3
2.1.3. Initial Testing ..................................................................................................... 4
2.2. Generating Instrumentation for Other Computers ........................................................... 5
2.3. Running SystemTap Scripts .......................................................................................... 7
3. Understanding How SystemTap Works 9
3.1. Architecture .................................................................................................................. 9
3.2. SystemTap Scripts ....................................................................................................... 9
3.2.1. Event .............................................................................................................. 11
3.2.2. Systemtap Handler/Body .................................................................................. 13
3.3. Basic SystemTap Handler Constructs .......................................................................... 16
3.3.1. Variables ......................................................................................................... 16
3.3.2. Conditional Statements .................................................................................... 17
3.3.3. Command-Line Arguments ............................................................................... 19
3.4. Associative Arrays ...................................................................................................... 19
3.5. Array Operations in SystemTap ................................................................................... 20
3.5.1. Assigning an Associated Value ......................................................................... 20
3.5.2. Reading Values From Arrays ........................................................................... 20
3.5.3. Incrementing Associated Values ....................................................................... 21
3.5.4. Processing Multiple Elements in an Array .......................................................... 21
3.5.5. Clearing/Deleting Arrays and Array Elements .................................................... 22
3.5.6. Using Arrays in Conditional Statements ............................................................ 23
3.5.7. Computing for Statistical Aggregates ................................................................ 24
3.6. Tapsets ...................................................................................................................... 26
4. Useful SystemTap Scripts 27
4.1. Network ..................................................................................................................... 27
4.1.1. Network Profiling ............................................................................................. 27
4.1.2. Tracing Functions Called in Network Socket Code ............................................. 29
4.1.3. Monitoring Incoming TCP Connections .............................................................. 30
4.1.4. Monitoring Network Packets Drops in Kernel ..................................................... 30
4.2. Disk ........................................................................................................................... 32
4.2.1. Summarizing Disk Read/Write Traffic ................................................................ 32
4.2.2. Tracking I/O Time For Each File Read or Write ................................................. 34
4.2.3. Track Cumulative IO ........................................................................................ 36
4.2.4. I/O Monitoring (By Device) ............................................................................... 38
4.2.5. Monitoring Reads and Writes to a File .............................................................. 39
4.2.6. Monitoring Changes to File Attributes ................................................................ 40
4.3. Profiling ..................................................................................................................... 40