--- name: ebpf description: eBPF skill for Linux observability and networking. Use when writing eBPF programs with libbpf or bpftrace, attaching kprobes/tracepoints/XDP hooks, debugging verifier errors, working with eBPF maps, or achieving CO-RE portability across kernel versions. Activates on queries about eBPF, bpftool, bpftrace, XDP programs, libbpf, verifier errors, eBPF maps, or kernel tracing with BPF. --- # eBPF ## Purpose Guide agents through writing, loading, and debugging eBPF programs using libbpf, bpftrace, and bpftool. Covers map types, program types, verifier errors, XDP networking, and CO-RE portability. ## Triggers - "How do I write an eBPF program to trace system calls?" - "My eBPF program fails with a verifier error" - "How do I use bpftrace to trace kernel events?" - "How do I share data between kernel eBPF and userspace?" - "How do I write an XDP program for packet filtering?" - "How do I make my eBPF program portable across kernel versions (CO-RE)?" ## Workflow ### 1. Choose the right tool ``` Goal? ├── One-liner kernel tracing / scripting → bpftrace ├── Production eBPF program with userspace → libbpf (C) or aya (Rust) ├── Inspect loaded programs and maps → bpftool └── High-performance packet processing → XDP + libbpf ``` ### 2. bpftrace — quick kernel tracing ```bash # Trace all execve calls with comm and args bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s %s\n", comm, str(args->filename)); }' # Count syscalls by process bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }' # Latency histogram for read() syscall bpftrace -e ' tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; } tracepoint:syscalls:sys_exit_read { @us = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]); }' # List available tracepoints bpftrace -l 'tracepoint:syscalls:*' bpftrace -l 'kprobe:tcp_*' ``` ### 3. libbpf skeleton — minimal C program ```c // counter.bpf.c — kernel-side #include #include struct { __uint(type, BPF_MAP_TYPE_HASH); __type(key, u32); __type(value, u64); __uint(max_entries, 1024); } call_count SEC(".maps"); SEC("tracepoint/syscalls/sys_enter_read") int trace_read(struct trace_event_raw_sys_enter *ctx) { u32 pid = bpf_get_current_pid_tgid() >> 32; u64 *cnt = bpf_map_lookup_elem(&call_count, &pid); if (cnt) (*cnt)++; else { u64 one = 1; bpf_map_update_elem(&call_count, &pid, &one, BPF_ANY); } return 0; } char LICENSE[] SEC("license") = "GPL"; ``` ```c // counter.c — userspace loader #include "counter.skel.h" int main(void) { struct counter_bpf *skel = counter_bpf__open(); if (!skel || counter_bpf__load(skel)) return 1; counter_bpf__attach(skel); // read map, print results counter_bpf__destroy(skel); } ``` ```bash # Build with libbpf 1.x clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I/usr/include/bpf \ -c counter.bpf.c -o counter.bpf.o bpftool gen skeleton counter.bpf.o > counter.skel.h gcc -o counter counter.c -lbpf -lelf -lz ``` libbpf 1.x API changes: ```c // Open and load (replaces older bpf_object__open/load split patterns) struct counter_bpf *skel = counter_bpf__open(); counter_bpf__load(skel); counter_bpf__attach(skel); // Or explicit file open struct bpf_object *obj = bpf_object__open_file("counter.bpf.o", NULL); bpf_object__load(obj); // Skeleton generation (always via bpftool) // bpftool gen skeleton counter.bpf.o name counter > counter.skel.h ``` ### 4. eBPF map types | Map type | Key→Value | Use case | |----------|-----------|----------| | `BPF_MAP_TYPE_HASH` | arbitrary→arbitrary | Per-PID counters, state | | `BPF_MAP_TYPE_ARRAY` | u32→fixed | Config, metrics indexed by CPU | | `BPF_MAP_TYPE_PERCPU_HASH` | key→per-CPU val | High-frequency counters without locks | | `BPF_MAP_TYPE_RINGBUF` | — | Efficient kernel→userspace events | | `BPF_MAP_TYPE_PERF_EVENT_ARRAY` | — | Legacy perf event output | | `BPF_MAP_TYPE_LRU_HASH` | key→val | Connection tracking, limited size | | `BPF_MAP_TYPE_PROG_ARRAY` | u32→prog | Tail calls, program chaining | | `BPF_MAP_TYPE_XSKMAP` | — | AF_XDP socket redirection | Use `BPF_MAP_TYPE_RINGBUF` over `PERF_EVENT_ARRAY` for new code — lower overhead, variable-size records. ### 5. Verifier error triage | Error message | Root cause | Fix | |---------------|-----------|-----| | `invalid mem access 'scalar'` | Dereferencing unbounded pointer | Check pointer with null test before use | | `R0 !read_ok` | Return without setting R0 | Ensure all paths set a return value | | `jump out of range` | Branch target beyond program end | Restructure conditionals | | `back-edge detected` | Backward jump (loop) | Use `bpf_loop()` helper (kernel ≥5.17) or bounded loop | | `unreachable insn` | Dead code after return | Remove dead branches | | `invalid indirect read` | Stack read of uninitialised bytes | Zero-init structs: `struct foo x = {}` | | `misaligned stack access` | Pointer arithmetic off alignment | Align reads to `__u64` boundaries | ```bash # Get detailed verifier log bpftool prog load prog.bpf.o /sys/fs/bpf/prog type kprobe \ 2>&1 | head -100 # Check loaded programs bpftool prog list bpftool prog dump xlated id 42 ``` ### 6. XDP programs ```c // xdp_drop_icmp.bpf.c #include #include #include SEC("xdp") int xdp_filter(struct xdp_md *ctx) { void *data_end = (void *)(long)ctx->data_end; void *data = (void *)(long)ctx->data; struct ethhdr *eth = data; if ((void *)(eth + 1) > data_end) return XDP_PASS; if (bpf_ntohs(eth->h_proto) != ETH_P_IP) return XDP_PASS; struct iphdr *ip = (void *)(eth + 1); if ((void *)(ip + 1) > data_end) return XDP_PASS; if (ip->protocol == IPPROTO_ICMP) return XDP_DROP; return XDP_PASS; } char LICENSE[] SEC("license") = "GPL"; ``` ```bash # Attach XDP program to interface ip link set dev eth0 xdp obj xdp_drop_icmp.bpf.o sec xdp # Remove ip link set dev eth0 xdp off # Use native (driver) mode for best performance ip link set dev eth0 xdp obj prog.bpf.o sec xdp mode native ``` XDP return codes: `XDP_PASS`, `XDP_DROP`, `XDP_TX` (hairpin), `XDP_REDIRECT`. ### 7. CO-RE — compile once, run everywhere CO-RE (Compile Once - Run Everywhere) uses BTF type info to relocate field accesses at load time. ```c // Use BTF-based field access (CO-RE aware) #include // generated from running kernel's BTF #include SEC("kprobe/tcp_connect") int trace_connect(struct pt_regs *ctx) { struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx); u16 dport = BPF_CORE_READ(sk, __sk_common.skc_dport); // BPF_CORE_READ relocates the field offset at load time bpf_printk("connect to port %d\n", bpf_ntohs(dport)); return 0; } ``` ```bash # Generate vmlinux.h from running kernel bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h # Verify BTF is enabled ls /sys/kernel/btf/vmlinux ``` ### 8. BPF ring buffer vs perf buffer ```c // Ring buffer (preferred for new programs — better perf, no per-CPU loss) struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 256 * 1024); } rb SEC(".maps"); SEC("kprobe/sys_open") int handle_open(struct pt_regs *ctx) { struct event *e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0); if (!e) return 0; e->pid = bpf_get_current_pid_tgid() >> 32; bpf_ringbuf_submit(e, 0); // bpf_ringbuf_discard(e, 0) on error paths return 0; } ``` | Feature | Ring buffer | Perf buffer | |---------|-------------|-------------| | API | `bpf_ringbuf_reserve/submit` | `bpf_perf_event_output` | | Backpressure | Reserve fails if full | May drop events | | Userspace | `ring_buffer__poll` (libbpf) | `perf_buffer__poll` | | Multi-producer | Yes | Per-CPU buffers | ### 9. BPF iterators ```c struct { __uint(type, BPF_MAP_TYPE_PROG_ARRAY); __uint(max_entries, 1); } iter_prog SEC(".maps"); SEC("iter/task") int dump_tasks(struct bpf_iter__task *ctx) { struct task_struct *task = ctx->task; if (task) bpf_seq_printf(ctx->meta->seq, "%d %s\n", task->tgid, task->comm); return 0; } ``` ```bash # Read iterator output from userspace bpftool prog tracelog # or attach iter to seq_file reader cat /sys/kernel/debug/tracing/trace_pipe ``` Iterators walk kernel data structures (tasks, maps, TCP sockets) without kprobe overhead per element. ### 10. BPF atomics ```c // GCC/Clang atomic builtins in BPF programs (kernel 5.12+) static __always_inline void inc_counter(__u32 *counter) { __sync_fetch_and_add(counter, 1); } // Use for per-CPU or map-backed counters under concurrent probes ``` Prefer per-CPU array maps for high-frequency counters; use atomics when aggregating into a single map value. For the full map types reference, see [references/ebpf-map-types.md](references/ebpf-map-types.md). ## Related skills - Use `skills/observability/ebpf-rust` for Aya framework Rust eBPF programs - Use `skills/profilers/linux-perf` for perf-based tracing without eBPF - Use `skills/runtimes/binary-hardening` for seccomp-bpf syscall filtering - Use `skills/low-level-programming/linux-kernel-modules` for kernel module development - Use `skills/async-io/af-xdp` for XDP_REDIRECT to AF_XDP sockets