BPF Performance Tools（英文版）：洞悉Linux系统和应用性能

作者: [美] Brendan Gregg(布伦丹·格雷格

出版社: 电子工业出版社

出版时间: 2020-12

版次: 1

ISBN: 9787121386947

定价: 219.00

装帧: 其他

开本: 其他

纸张: 胶版纸

页数: 840页

分类: 计算机与互联网

1 张插图图片

21人买过

本书作为全面介绍 BPF 技术的图书，从 BPF 技术的起源到未来发展方向都有涵盖，不仅系统介绍了 BPF 的编程模型，还完整介绍了两个主要的 BPF 前端编程框架—BCC 和 bpftrace，更给出了一系列实现范例，生动展示了 BPF 技术的实际能力和未来发展前景。本书的另一个关注方向是 Linux 系统性能和应用程序性能的调优，内容涉及系统性能调优的策略、工具与实践案例，不仅介绍了对应的 BPF 工具，还着重介绍了这些工具如何与 Linux 传统性能工具配合使用，这样读者可以选择*方案。本书介绍的工具小巧精致，并提供了简单易读的源代码，它们充分展现了 BPF 技术的魅力：安全、高效、快捷的系统扩展力。未来 BPF 技术在 Linux 中的应用场景会越来越多、越来越重要。希望本书能在大家学习 BPF 技术并关注它的发展时提供帮助。 Brendan Gregg 是Joyent公司的首席性能工程师，通过软件栈分析性能和扩展。在Sun Microsystem公司（之后为Oracle）作为首席性能和内核工程师期间，他的工作包括开发ZFS L2ARC，这是一个利用闪速存储器提升性能的文件系统。他还开发了许许多多的性能工具，部分工具收录在Mac OS X和Oracle Solaris 11的发行版中。他最近从事的工作覆盖针对Linux和illumos内核分析的性能可视化。他还是《DTrace》（Prentice Hall出版社，2011年）和《Solaris Performance and Tools》的两书合著者。

Netflix 资深性能工程师 Brendan Gregg 是 BPF（eBPF）的主要贡献者，他帮助开发和维护了两个主要的 BPF 前端编程框架，开创了 BPF 用于可观测性的先河，并创建了数十种基于 BPF 的性能分析工具。他还编著有畅销书《性能之巅：洞悉系统、企业与云计算》。 Part I: Technologies

1 Introduction 1

1.1 What Are BPF and eBPF? 1

1.2 What Are Tracing, Snooping, Sampling, Profiling, and

Observability? 2

1.3 What Are BCC, bpftrace, and IO Visor? 3

1.4 A First Look at BCC: Quick Wins 4

1.5 BPF Tracing Visibility 6

1.6 Dynamic Instrumentation: kprobes and uprobes 8

1.7 Static Instrumentation: Tracepoints and USDT 9

1.8 A First Look at bpftrace: Tracing open() 10

1.9 Back to BCC: Tracing open() 12

1.10 Summary 14

2 Technology Background 15

2.1 BPF Illustrated 15

2.2 BPF 16

2.3 Extended BPF (eBPF) 17

2.3.1 Why Performance Tools Need BPF 19

2.3.2 BPF Versus Kernel Modules 21

2.3.3 Writing BPF Programs 22

2.3.4 Viewing BPF Instructions: bpftool 23

2.3.5 Viewing BPF Instructions: bpftrace 30

2.3.6 BPF API 31

2.3.7 BPF Concurrency Controls 35

2.3.8 BPF sysfs Interface 36

2.3.9 BPF Type Format (BTF) 37

2.3.10 BPF CO-RE 37

2.3.11 BPF Limitations 38

2.3.12 BPF Additional Reading 38

2.4 Stack Trace Walking 39

2.4.1 Frame Pointer?CBased Stacks 39

2.4.2 debuginfo 40

2.4.3 Last Branch Record (LBR) 40

2.4.4 ORC 40

2.4.5 Symbols 41

2.4.6 More Reading 41

2.5 Flame Graphs 41

2.5.1 Stack Trace 41

2.5.2 Profiling Stack Traces 41

2.5.3 Flame Graph 42

2.5.4 Flame Graph Features 44

2.5.5 Variations 44

2.6 Event Sources 45

2.7 kprobes 46

2.7.1 How kprobes Work 46

2.7.2 kprobes Interfaces 47

2.7.3 BPF and kprobes 48

2.7.4 kprobes Additional Reading 49

2.8 uprobes 49

2.8.1 How uprobes Work 49

2.8.2 Uprobes Interfaces 51

2.8.3 BPF and uprobes 51

2.8.4 uprobes Overhead and Future Work 52

2.8.5 uprobes Additional Reading 52

2.9 Tracepoints 53

2.9.1 Adding Tracepoint Instrumentation 53

2.9.2 How Tracepoints Work 55

2.9.3 Tracepoint Interfaces 56

2.9.4 Tracepoints and BPF 56

2.9.5 BPF Raw Tracepoints 57

2.9.6 Additional Reading 58

2.10 USDT 58

2.10.1 Adding USDT Instrumentation 58

2.10.2 How USDT Works 60

2.10.3 BPF and USDT 61

2.10.4 USDT Additional Reading 61

2.11 Dynamic USDT 61

2.12 PMCs 63

2.12.1 PMC Modes 63

2.12.2 PEBS 64

2.12.3 Cloud Computing 64

2.13 perf_events 64

2.14 Summary 65

3 Performance Analysis 67

3.1 Overview 67

3.1.1 Goals 68

3.1.2 Activities 68

3.1.3 Mulitple Performance Issues 69

3.2 Performance Methodologies 69

3.2.1 Workload Characterization 70

3.2.2 Drill-Down Analysis 71

3.2.3 USE Method 72

3.2.4 Checklists 72

3.3 Linux 60-Second Analysis 73

3.3.1 uptime 73

3.3.2 dmesg | tail 74

3.3.3 vmstat 1 74

3.3.4 mpstat -P ALL 1 75

3.3.5 pidstat 1 75

3.3.6 iostat -xz 1 76

3.3.7 free -m 77

3.3.8 sar -n DEV 1 77

3.3.9 sar -n TCP,ETCP 1 78

3.3.10 top 78

3.4 BCC Tool Checklist 79

3.4.1 execsnoop 80

3.4.2 opensnoop 80

3.4.3 ext4slower 80

3.4.4 biolatency 81

3.4.5 biosnoop 81

3.4.6 cachestat 82

3.4.7 tcpconnect 82

3.4.8 tcpaccept 82

3.4.9 tcpretrans 83

3.4.10 runqlat 83

3.4.11 profile 84

3.5 Summary 84

4 BCC 85

4.1 BCC Components 86

4.2 BCC Features 86

4.2.1 Kernel-Level Features 87

4.2.2 BCC User-Level Features 87

4.3 BCC Installation 88

4.3.1 Kernel Requirements 88

4.3.2 Ubuntu 88

4.3.3 RHEL 89

4.3.4 Other Distributions 89

4.4 BCC Tools 89

4.4.1 Highlighted Tools 90

4.4.2 Tool Characteristics 91

4.4.3 Single-Purpose Tools 91

4.4.4 Multi-Purpose Tools 93

4.5 funccount 94

4.5.1 funccount Examples 94

4.5.2 funccount Syntax 97

4.5.3 funccount One-Liners 97

4.5.4 funccount Usage 98

4.6 stackcount 99

4.6.1 stackcount Example 99

4.6.2 stackcount Flame Graphs 100

4.6.3 stackcount Broken Stack Traces 101

4.6.4 stackcount Syntax 102

4.6.5 stackcount One-Liners 102

4.6.6 stackcount Usage 103

4.7 trace 104

4.7.1 trace Example 104

4.7.2 trace Syntax 105

4.7.3 trace One-Liners 106

4.7.4 trace Structs 107

4.7.5 trace Debugging File Descriptor Leaks 107

4.7.6 trace Usage 109

4.8 argdist 110

4.8.1 argdist Syntax 111

4.8.2 argdist One-Liners 112

4.8.3 argdist Usage 113

4.9 Tool Documentation 114

4.9.1 Man Page: opensnoop 115

4.9.2 Examples File: opensnoop 118

4.10 Developing BCC Tools 119

4.11 BCC Internals 120

4.12 BCC Debugging 121

4.12.1 printf() Debugging 122

4.12.2 BCC Debug Output 124

4.12.3 BCC Debug Flag 125

4.12.4 bpflist 126

4.12.5 bpftool 127

4.12.6 dmesg 127

4.12.7 Resetting Events 127

4.13 Summary 128

5 bpftrace 129

5.1 bpftrace Components 130

5.2 bpftrace Features 131

5.2.1 bpftrace Event Sources 131

5.2.2 bpftrace Actions 131

5.2.3 bpftrace General Features 132

5.2.4 bpftrace Compared to Other Observability Tools 132

5.3 bpftrace Installation 133

5.3.1 Kernel Requirements 133

5.3.2 Ubuntu 134

5.3.3 Fedora 134

5.3.4 Post-Build Steps 134

5.3.5 Other Distributions 135

5.4 bpftrace Tools 135

5.4.1 Highlighted Tools 136

5.4.2 Tool Characteristics 136

5.4.3 Tool Execution 137

5.5 bpftrace One-Liners 137

5.6 bpftrace Documentation 138

5.7 bpftrace Programming 138

5.7.1 Usage 139

5.7.2 Program Structure 140

5.7.3 Comments 140

5.7.4 Probe Format 141

5.7.5 Probe Wildcards 141

5.7.6 Filters 142

5.7.7 Actions 142

5.7.8 Hello, World! 142

5.7.9 Functions 143

5.7.10 Variables 143

5.7.11 Map Functions 144

5.7.12 Timing vfs_read() 145

5.8 bpftrace Usage 147

5.9 bpftrace Probe Types 148

5.9.1 tracepoint 148

5.9.2 usdt 150

5.9.3 kprobe and kretprobe 151

5.9.4 uprobe and uretprobe 151

5.9.5 software and hardware 152

5.9.6 profile and interval 153

5.10 bpftrace Flow Control 154

5.10.1 Filter 154

5.10.2 Ternary Operators 154

5.10.3 If Statements 155

5.10.4 Unrolled Loops 155

5.11 bpftrace Operators 155

5.12 bpftrace Variables 156

5.12.1 Built-in Variables 156

5.12.2 Built-ins: pid, comm, and uid 157

5.12.3 Built-ins: kstack and ustack 157

5.12.4 Built-ins: Positional Parameters 159

5.12.5 Scratch 160

5.12.6 Maps 160

5.13 bpftrace Functions 161

5.13.1 printf() 162

5.13.2 join() 163

5.13.3 str() 163

5.13.4 kstack() and ustack() 164

5.13.5 ksym() and usym() 165

5.13.6 kaddr() and uaddr() 166

5.13.7 system() 166

5.13.8 exit() 167

5.14 bpftrace Map Functions 167

5.14.1 count() 168

5.14.2 sum(), avg(), min(), and max() 169

5.14.3 hist() 170

5.14.4 lhist() 171

5.14.5 delete() 171

5.14.6 clear() and zero() 172

5.14.7 print() 172

5.15 bpftrace Future Work 173

5.15.1 Explicit Address Modes 173

5.15.2 Other Additions 174

5.15.3 ply 175

5.16 bpftrace Internals 175

5.17 bpftrace Debugging 176

5.17.1 printf() Debugging 177

5.17.2 Debug Mode 177

5.17.3 Verbose Mode 179

5.18 Summary 180

Part II: Using BPF Tools

6 CPUs 181

6.1 Background 181

6.1.1 CPU Fundamentals 182

6.1.2 BPF Capabilities 184

6.1.3 Strategy 185

6.2 Traditional Tools 186

6.2.1 Kernel Statistics 187

6.2.2 Hardware Statistics 189

6.2.3 Hardware Sampling 192

6.2.4 Timed Sampling 192

6.2.5 Event Statistics and Tracing 196

6.3 BPF Tools 198

6.3.1 execsnoop 200

6.3.2 exitsnoop 202

6.3.3 runqlat 203

6.3.4 runqlen 207

6.3.5 runqslower 210

6.3.6 cpudist 211

6.3.7 cpufreq 212

6.3.8 profile 215

6.3.9 offcputime 219

6.3.10 syscount 224

6.3.11 argdist and trace 226

6.3.12 funccount 229

6.3.13 softirqs 231

6.3.14 hardirqs 232

6.3.15 smpcalls 233

6.3.16 llcstat 237

6.3.17 Other Tools 238

6.4 BPF One-Liners 238

6.4.1 BCC 238

6.4.2 bpftrace 239

6.5
内容简介:
本书作为全面介绍 BPF 技术的图书，从 BPF 技术的起源到未来发展方向都有涵盖，不仅系统介绍了 BPF 的编程模型，还完整介绍了两个主要的 BPF 前端编程框架—BCC 和 bpftrace，更给出了一系列实现范例，生动展示了 BPF 技术的实际能力和未来发展前景。本书的另一个关注方向是 Linux 系统性能和应用程序性能的调优，内容涉及系统性能调优的策略、工具与实践案例，不仅介绍了对应的 BPF 工具，还着重介绍了这些工具如何与 Linux 传统性能工具配合使用，这样读者可以选择*方案。本书介绍的工具小巧精致，并提供了简单易读的源代码，它们充分展现了 BPF 技术的魅力：安全、高效、快捷的系统扩展力。未来 BPF 技术在 Linux 中的应用场景会越来越多、越来越重要。希望本书能在大家学习 BPF 技术并关注它的发展时提供帮助。
作者简介:
Brendan Gregg 是Joyent公司的首席性能工程师，通过软件栈分析性能和扩展。在Sun Microsystem公司（之后为Oracle）作为首席性能和内核工程师期间，他的工作包括开发ZFS L2ARC，这是一个利用闪速存储器提升性能的文件系统。他还开发了许许多多的性能工具，部分工具收录在Mac OS X和Oracle Solaris 11的发行版中。他最近从事的工作覆盖针对Linux和illumos内核分析的性能可视化。他还是《DTrace》（Prentice Hall出版社，2011年）和《Solaris Performance and Tools》的两书合著者。

Netflix 资深性能工程师 Brendan Gregg 是 BPF（eBPF）的主要贡献者，他帮助开发和维护了两个主要的 BPF 前端编程框架，开创了 BPF 用于可观测性的先河，并创建了数十种基于 BPF 的性能分析工具。他还编著有畅销书《性能之巅：洞悉系统、企业与云计算》。
目录:
Part I: Technologies

1 Introduction 1

1.1 What Are BPF and eBPF? 1

1.2 What Are Tracing, Snooping, Sampling, Profiling, and

Observability? 2

1.3 What Are BCC, bpftrace, and IO Visor? 3

1.4 A First Look at BCC: Quick Wins 4

1.5 BPF Tracing Visibility 6

1.6 Dynamic Instrumentation: kprobes and uprobes 8

1.7 Static Instrumentation: Tracepoints and USDT 9

1.8 A First Look at bpftrace: Tracing open() 10

1.9 Back to BCC: Tracing open() 12

1.10 Summary 14

2 Technology Background 15

2.1 BPF Illustrated 15

2.2 BPF 16

2.3 Extended BPF (eBPF) 17

2.3.1 Why Performance Tools Need BPF 19

2.3.2 BPF Versus Kernel Modules 21

2.3.3 Writing BPF Programs 22

2.3.4 Viewing BPF Instructions: bpftool 23

2.3.5 Viewing BPF Instructions: bpftrace 30

2.3.6 BPF API 31

2.3.7 BPF Concurrency Controls 35

2.3.8 BPF sysfs Interface 36

2.3.9 BPF Type Format (BTF) 37

2.3.10 BPF CO-RE 37

2.3.11 BPF Limitations 38

2.3.12 BPF Additional Reading 38

2.4 Stack Trace Walking 39

2.4.1 Frame Pointer?CBased Stacks 39

2.4.2 debuginfo 40

2.4.3 Last Branch Record (LBR) 40

2.4.4 ORC 40

2.4.5 Symbols 41

2.4.6 More Reading 41

2.5 Flame Graphs 41

2.5.1 Stack Trace 41

2.5.2 Profiling Stack Traces 41

2.5.3 Flame Graph 42

2.5.4 Flame Graph Features 44

2.5.5 Variations 44

2.6 Event Sources 45

2.7 kprobes 46

2.7.1 How kprobes Work 46

2.7.2 kprobes Interfaces 47

2.7.3 BPF and kprobes 48

2.7.4 kprobes Additional Reading 49

2.8 uprobes 49

2.8.1 How uprobes Work 49

2.8.2 Uprobes Interfaces 51

2.8.3 BPF and uprobes 51

2.8.4 uprobes Overhead and Future Work 52

2.8.5 uprobes Additional Reading 52

2.9 Tracepoints 53

2.9.1 Adding Tracepoint Instrumentation 53

2.9.2 How Tracepoints Work 55

2.9.3 Tracepoint Interfaces 56

2.9.4 Tracepoints and BPF 56

2.9.5 BPF Raw Tracepoints 57

2.9.6 Additional Reading 58

2.10 USDT 58

2.10.1 Adding USDT Instrumentation 58

2.10.2 How USDT Works 60

2.10.3 BPF and USDT 61

2.10.4 USDT Additional Reading 61

2.11 Dynamic USDT 61

2.12 PMCs 63

2.12.1 PMC Modes 63

2.12.2 PEBS 64

2.12.3 Cloud Computing 64

2.13 perf_events 64

2.14 Summary 65

3 Performance Analysis 67

3.1 Overview 67

3.1.1 Goals 68

3.1.2 Activities 68

3.1.3 Mulitple Performance Issues 69

3.2 Performance Methodologies 69

3.2.1 Workload Characterization 70

3.2.2 Drill-Down Analysis 71

3.2.3 USE Method 72

3.2.4 Checklists 72

3.3 Linux 60-Second Analysis 73

3.3.1 uptime 73

3.3.2 dmesg | tail 74

3.3.3 vmstat 1 74

3.3.4 mpstat -P ALL 1 75

3.3.5 pidstat 1 75

3.3.6 iostat -xz 1 76

3.3.7 free -m 77

3.3.8 sar -n DEV 1 77

3.3.9 sar -n TCP,ETCP 1 78

3.3.10 top 78

3.4 BCC Tool Checklist 79

3.4.1 execsnoop 80

3.4.2 opensnoop 80

3.4.3 ext4slower 80

3.4.4 biolatency 81

3.4.5 biosnoop 81

3.4.6 cachestat 82

3.4.7 tcpconnect 82

3.4.8 tcpaccept 82

3.4.9 tcpretrans 83

3.4.10 runqlat 83

3.4.11 profile 84

3.5 Summary 84

4 BCC 85

4.1 BCC Components 86

4.2 BCC Features 86

4.2.1 Kernel-Level Features 87

4.2.2 BCC User-Level Features 87

4.3 BCC Installation 88

4.3.1 Kernel Requirements 88

4.3.2 Ubuntu 88

4.3.3 RHEL 89

4.3.4 Other Distributions 89

4.4 BCC Tools 89

4.4.1 Highlighted Tools 90

4.4.2 Tool Characteristics 91

4.4.3 Single-Purpose Tools 91

4.4.4 Multi-Purpose Tools 93

4.5 funccount 94

4.5.1 funccount Examples 94

4.5.2 funccount Syntax 97

4.5.3 funccount One-Liners 97

4.5.4 funccount Usage 98

4.6 stackcount 99

4.6.1 stackcount Example 99

4.6.2 stackcount Flame Graphs 100

4.6.3 stackcount Broken Stack Traces 101

4.6.4 stackcount Syntax 102

4.6.5 stackcount One-Liners 102

4.6.6 stackcount Usage 103

4.7 trace 104

4.7.1 trace Example 104

4.7.2 trace Syntax 105

4.7.3 trace One-Liners 106

4.7.4 trace Structs 107

4.7.5 trace Debugging File Descriptor Leaks 107

4.7.6 trace Usage 109

4.8 argdist 110

4.8.1 argdist Syntax 111

4.8.2 argdist One-Liners 112

4.8.3 argdist Usage 113

4.9 Tool Documentation 114

4.9.1 Man Page: opensnoop 115

4.9.2 Examples File: opensnoop 118

4.10 Developing BCC Tools 119

4.11 BCC Internals 120

4.12 BCC Debugging 121

4.12.1 printf() Debugging 122

4.12.2 BCC Debug Output 124

4.12.3 BCC Debug Flag 125

4.12.4 bpflist 126

4.12.5 bpftool 127

4.12.6 dmesg 127

4.12.7 Resetting Events 127

4.13 Summary 128

5 bpftrace 129

5.1 bpftrace Components 130

5.2 bpftrace Features 131

5.2.1 bpftrace Event Sources 131

5.2.2 bpftrace Actions 131

5.2.3 bpftrace General Features 132

5.2.4 bpftrace Compared to Other Observability Tools 132

5.3 bpftrace Installation 133

5.3.1 Kernel Requirements 133

5.3.2 Ubuntu 134

5.3.3 Fedora 134

5.3.4 Post-Build Steps 134

5.3.5 Other Distributions 135

5.4 bpftrace Tools 135

5.4.1 Highlighted Tools 136

5.4.2 Tool Characteristics 136

5.4.3 Tool Execution 137

5.5 bpftrace One-Liners 137

5.6 bpftrace Documentation 138

5.7 bpftrace Programming 138

5.7.1 Usage 139

5.7.2 Program Structure 140

5.7.3 Comments 140

5.7.4 Probe Format 141

5.7.5 Probe Wildcards 141

5.7.6 Filters 142

5.7.7 Actions 142

5.7.8 Hello, World! 142

5.7.9 Functions 143

5.7.10 Variables 143

5.7.11 Map Functions 144

5.7.12 Timing vfs_read() 145

5.8 bpftrace Usage 147

5.9 bpftrace Probe Types 148

5.9.1 tracepoint 148

5.9.2 usdt 150

5.9.3 kprobe and kretprobe 151

5.9.4 uprobe and uretprobe 151

5.9.5 software and hardware 152

5.9.6 profile and interval 153

5.10 bpftrace Flow Control 154

5.10.1 Filter 154

5.10.2 Ternary Operators 154

5.10.3 If Statements 155

5.10.4 Unrolled Loops 155

5.11 bpftrace Operators 155

5.12 bpftrace Variables 156

5.12.1 Built-in Variables 156

5.12.2 Built-ins: pid, comm, and uid 157

5.12.3 Built-ins: kstack and ustack 157

5.12.4 Built-ins: Positional Parameters 159

5.12.5 Scratch 160

5.12.6 Maps 160

5.13 bpftrace Functions 161

5.13.1 printf() 162

5.13.2 join() 163

5.13.3 str() 163

5.13.4 kstack() and ustack() 164

5.13.5 ksym() and usym() 165

5.13.6 kaddr() and uaddr() 166

5.13.7 system() 166

5.13.8 exit() 167

5.14 bpftrace Map Functions 167

5.14.1 count() 168

5.14.2 sum(), avg(), min(), and max() 169

5.14.3 hist() 170

5.14.4 lhist() 171

5.14.5 delete() 171

5.14.6 clear() and zero() 172

5.14.7 print() 172

5.15 bpftrace Future Work 173

5.15.1 Explicit Address Modes 173

5.15.2 Other Additions 174

5.15.3 ply 175

5.16 bpftrace Internals 175

5.17 bpftrace Debugging 176

5.17.1 printf() Debugging 177

5.17.2 Debug Mode 177

5.17.3 Verbose Mode 179

5.18 Summary 180

Part II: Using BPF Tools

6 CPUs 181

6.1 Background 181

6.1.1 CPU Fundamentals 182

6.1.2 BPF Capabilities 184

6.1.3 Strategy 185

6.2 Traditional Tools 186

6.2.1 Kernel Statistics 187

6.2.2 Hardware Statistics 189

6.2.3 Hardware Sampling 192

6.2.4 Timed Sampling 192

6.2.5 Event Statistics and Tracing 196

6.3 BPF Tools 198

6.3.1 execsnoop 200

6.3.2 exitsnoop 202

6.3.3 runqlat 203

6.3.4 runqlen 207

6.3.5 runqslower 210

6.3.6 cpudist 211

6.3.7 cpufreq 212

6.3.8 profile 215

6.3.9 offcputime 219

6.3.10 syscount 224

6.3.11 argdist and trace 226

6.3.12 funccount 229

6.3.13 softirqs 231

6.3.14 hardirqs 232

6.3.15 smpcalls 233

6.3.16 llcstat 237

6.3.17 Other Tools 238

6.4 BPF One-Liners 238

6.4.1 BCC 238

6.4.2 bpftrace 239

6.5

查看详情

BPF Performance Tools（英文版）：洞悉Linux系统和应用性能

内容简介:

作者简介:

目录: