--- name: ebpf-test-harness description: Generate test infrastructure for eBPF CNF programs including network namespace setup, virtual interface creation, traffic generation, packet injection, and validation of packet processing logic. Use when implementing automated tests for CNFs. --- # eBPF Test Harness Skill This skill generates comprehensive test infrastructure for validating eBPF-based CNF programs. ## What This Skill Does Generates test code for: 1. Network namespace setup and teardown 2. Virtual interface (veth/netkit) creation 3. Traffic generation (ping, netcat, iperf) 4. Packet injection and capture 5. Map validation (checking counters, stats) 6. Event verification (ringbuf events) 7. Integration tests with real packet flows 8. Unit tests for eBPF helper functions ## When to Use - Writing automated tests for CNFs - Validating packet processing logic - Testing eBPF map operations - Verifying event generation - CI/CD integration - Performance benchmarking - Regression testing ## Test Types ### 1. Unit Tests - Test individual eBPF programs in isolation - Mock traffic patterns - Validate map operations ### 2. Integration Tests - Full CNF setup with real network interfaces - Multi-namespace scenarios - End-to-end packet flows ### 3. Performance Tests - Throughput benchmarks - Latency measurements - Resource usage validation ## Information to Gather Ask the user: 1. **CNF Type**: What does the CNF do? (filter, forward, monitor, etc.) 2. **Test Scope**: Unit tests, integration tests, or both? 3. **Network Setup**: Single namespace or multi-namespace? 4. **Traffic Type**: TCP, UDP, ICMP, or mixed? 5. **Validation**: Check maps, events, packet modifications? 6. **Performance**: Need benchmarks? ## Go Test Setup ### Basic Test Structure ```go package main import ( "testing" "github.com/cilium/ebpf" ) func TestMain(m *testing.M) { // Setup before all tests // Teardown after all tests os.Exit(m.Run()) } func TestCNFBasic(t *testing.T) { // Load eBPF program spec, err := LoadMyCNF() if err != nil { t.Fatalf("loading spec: %v", err) } objs := &MyCNFObjects{} if err := spec.LoadAndAssign(objs, nil); err != nil { t.Fatalf("loading objects: %v", err) } defer objs.Close() // Run test // Validate results } ``` **Tip:** For cleaner assertions, use stretchr/testify (`require.NoError`, `assert.Equal`). See [EXAMPLES.md](EXAMPLES.md) for patterns and when to use `require` vs `assert`. ## Network Namespace Setup ### Creating Test Namespaces ```go import ( "runtime" "github.com/vishvananda/netns" "github.com/vishvananda/netlink" ) type TestEnv struct { OrigNS netns.NsHandle TestNS netns.NsHandle Cleanup func() } func setupTestNamespace(t *testing.T) *TestEnv { t.Helper() // Lock to current OS thread runtime.LockOSThread() // Save original namespace origNS, err := netns.Get() if err != nil { t.Fatalf("getting original netns: %v", err) } // Create test namespace testNS, err := netns.New() if err != nil { origNS.Close() t.Fatalf("creating test netns: %v", err) } // Switch back to original for test setup if err := netns.Set(origNS); err != nil { testNS.Close() origNS.Close() t.Fatalf("switching back to original netns: %v", err) } env := &TestEnv{ OrigNS: origNS, TestNS: testNS, Cleanup: func() { testNS.Close() origNS.Close() runtime.UnlockOSThread() }, } return env } func TestWithNamespace(t *testing.T) { env := setupTestNamespace(t) defer env.Cleanup() // Switch to test namespace if err := netns.Set(env.TestNS); err != nil { t.Fatalf("switching to test netns: %v", err) } defer netns.Set(env.OrigNS) // Run tests in isolated namespace // Create interfaces, attach programs, etc. } ``` ## Virtual Interface Setup ### Creating veth Pairs ```go import ( "github.com/vishvananda/netlink" ) func createVethPair(t *testing.T, name1, name2 string) { t.Helper() veth := &netlink.Veth{ LinkAttrs: netlink.LinkAttrs{ Name: name1, }, PeerName: name2, } if err := netlink.LinkAdd(veth); err != nil { t.Fatalf("creating veth pair: %v", err) } // Cleanup t.Cleanup(func() { netlink.LinkDel(veth) }) // Bring up interfaces link1, _ := netlink.LinkByName(name1) link2, _ := netlink.LinkByName(name2) if err := netlink.LinkSetUp(link1); err != nil { t.Fatalf("bringing up %s: %v", name1, err) } if err := netlink.LinkSetUp(link2); err != nil { t.Fatalf("bringing up %s: %v", name2, err) } } func TestWithVethPair(t *testing.T) { createVethPair(t, "veth0", "veth1") // Use veth0 and veth1 in test } ``` ### Creating netkit Pairs ```go func createNetkitPair(t *testing.T, name1, name2 string) { t.Helper() // Note: Requires kernel 6.6+ and recent netlink library netkit := &netlink.Netkit{ LinkAttrs: netlink.LinkAttrs{ Name: name1, }, PeerName: name2, Mode: netlink.NETKIT_MODE_L3, } if err := netlink.LinkAdd(netkit); err != nil { t.Fatalf("creating netkit pair: %v", err) } t.Cleanup(func() { netlink.LinkDel(netkit) }) // Bring up interfaces link1, _ := netlink.LinkByName(name1) link2, _ := netlink.LinkByName(name2) netlink.LinkSetUp(link1) netlink.LinkSetUp(link2) } ``` ## Traffic Generation ### ICMP Ping ```go import ( "os/exec" "time" ) func sendPing(t *testing.T, target string, count int) { t.Helper() cmd := exec.Command("ping", "-c", fmt.Sprintf("%d", count), target) if err := cmd.Run(); err != nil { t.Logf("ping failed (may be expected): %v", err) } } func TestPingProcessing(t *testing.T) { // Setup namespace, interfaces, attach program... // Generate ICMP traffic sendPing(t, "10.0.0.2", 5) // Wait for processing time.Sleep(100 * time.Millisecond) // Validate results (check maps, counters, etc.) } ``` ### TCP Traffic ```go func sendTCPTraffic(t *testing.T, addr string, data []byte) { t.Helper() conn, err := net.DialTimeout("tcp", addr, 5*time.Second) if err != nil { t.Fatalf("dialing: %v", err) } defer conn.Close() if _, err := conn.Write(data); err != nil { t.Fatalf("writing: %v", err) } } func TestTCPProcessing(t *testing.T) { // Start TCP server in background listener, err := net.Listen("tcp", "127.0.0.1:8080") if err != nil { t.Fatalf("listening: %v", err) } defer listener.Close() go func() { conn, _ := listener.Accept() if conn != nil { defer conn.Close() io.ReadAll(conn) } }() // Setup CNF... // Send TCP traffic sendTCPTraffic(t, "127.0.0.1:8080", []byte("test data")) // Validate processing } ``` ### UDP Traffic ```go func sendUDPPacket(t *testing.T, addr string, data []byte) { t.Helper() conn, err := net.Dial("udp", addr) if err != nil { t.Fatalf("dialing: %v", err) } defer conn.Close() if _, err := conn.Write(data); err != nil { t.Fatalf("writing: %v", err) } } ``` ## Packet Injection with gopacket ```go import ( "github.com/google/gopacket" "github.com/google/gopacket/layers" "github.com/google/gopacket/pcap" ) func injectPacket(t *testing.T, iface string, packet []byte) { t.Helper() handle, err := pcap.OpenLive(iface, 1600, true, pcap.BlockForever) if err != nil { t.Fatalf("opening pcap: %v", err) } defer handle.Close() if err := handle.WritePacketData(packet); err != nil { t.Fatalf("writing packet: %v", err) } } func createTCPSYN(srcIP, dstIP string, srcPort, dstPort uint16) []byte { // Create Ethernet layer eth := layers.Ethernet{ SrcMAC: net.HardwareAddr{0x00, 0x00, 0x00, 0x00, 0x00, 0x01}, DstMAC: net.HardwareAddr{0x00, 0x00, 0x00, 0x00, 0x00, 0x02}, EthernetType: layers.EthernetTypeIPv4, } // Create IP layer ip := layers.IPv4{ Version: 4, TTL: 64, Protocol: layers.IPProtocolTCP, SrcIP: net.ParseIP(srcIP), DstIP: net.ParseIP(dstIP), } // Create TCP layer tcp := layers.TCP{ SrcPort: layers.TCPPort(srcPort), DstPort: layers.TCPPort(dstPort), SYN: true, Seq: 1000, Window: 65535, } tcp.SetNetworkLayerForChecksum(&ip) // Serialize buf := gopacket.NewSerializeBuffer() opts := gopacket.SerializeOptions{ ComputeChecksums: true, FixLengths: true, } gopacket.SerializeLayers(buf, opts, ð, &ip, &tcp) return buf.Bytes() } func TestTCPSYNProcessing(t *testing.T) { // Setup... // Inject TCP SYN packet packet := createTCPSYN("10.0.0.1", "10.0.0.2", 12345, 80) injectPacket(t, "veth0", packet) // Validate... } ``` ## Map Validation ### Reading and Validating Counters ```go func validateCounter(t *testing.T, m *ebpf.Map, expectedCount uint64) { t.Helper() var ( key uint32 = 0 value uint64 ) if err := m.Lookup(&key, &value); err != nil { t.Fatalf("looking up counter: %v", err) } if value != expectedCount { t.Errorf("counter mismatch: got %d, want %d", value, expectedCount) } } func TestPacketCounting(t *testing.T) { // Load program spec, _ := LoadMyCNF() objs := &MyCNFObjects{} spec.LoadAndAssign(objs, nil) defer objs.Close() // Attach program... // Send 10 packets for i := 0; i < 10; i++ { sendPing(t, "10.0.0.2", 1) } time.Sleep(100 * time.Millisecond) // Validate counter validateCounter(t, objs.PacketCounter, 10) } ``` ### Validating Flow Table ```go func validateFlowExists(t *testing.T, flowMap *ebpf.Map, key FlowKey) { t.Helper() var stats FlowStats if err := flowMap.Lookup(&key, &stats); err != nil { t.Fatalf("flow not found: %v", err) } if stats.Packets == 0 { t.Error("flow has zero packets") } t.Logf("Flow stats: packets=%d, bytes=%d", stats.Packets, stats.Bytes) } ``` ## Event Validation (Ringbuf) ```go import ( "github.com/cilium/ebpf/ringbuf" ) func collectEvents(t *testing.T, eventsMap *ebpf.Map, expected int, timeout time.Duration) []PacketEvent { t.Helper() rd, err := ringbuf.NewReader(eventsMap) if err != nil { t.Fatalf("opening ringbuf: %v", err) } defer rd.Close() var events []PacketEvent deadline := time.After(timeout) for len(events) < expected { select { case <-deadline: t.Fatalf("timeout waiting for events: got %d, want %d", len(events), expected) default: record, err := rd.Read() if err != nil { if errors.Is(err, ringbuf.ErrClosed) { return events } continue } var event PacketEvent if err := binary.Read(bytes.NewReader(record.RawSample), binary.LittleEndian, &event); err != nil { t.Logf("parsing event: %v", err) continue } events = append(events, event) } } return events } func TestEventGeneration(t *testing.T) { // Setup... // Start event collection in background eventsChan := make(chan []PacketEvent) go func() { events := collectEvents(t, objs.Events, 5, 5*time.Second) eventsChan <- events }() // Generate traffic time.Sleep(100 * time.Millisecond) // Let reader start for i := 0; i < 5; i++ { sendPing(t, "10.0.0.2", 1) } // Wait for events events := <-eventsChan if len(events) != 5 { t.Errorf("event count mismatch: got %d, want 5", len(events)) } } ``` ## Complete Integration Test Example ```go package main import ( "testing" "time" "net" "github.com/vishvananda/netlink" "github.com/vishvananda/netns" "github.com/cilium/ebpf/link" ) func TestCNFIntegration(t *testing.T) { // Skip if not root if os.Getuid() != 0 { t.Skip("test requires root") } // Create test namespaces ns1, err := netns.New() if err != nil { t.Fatalf("creating ns1: %v", err) } defer ns1.Close() ns2, err := netns.New() if err != nil { t.Fatalf("creating ns2: %v", err) } defer ns2.Close() // Get original namespace origNS, _ := netns.Get() defer origNS.Close() defer netns.Set(origNS) // Create veth pair in host namespace veth := &netlink.Veth{ LinkAttrs: netlink.LinkAttrs{Name: "veth0"}, PeerName: "veth1", } if err := netlink.LinkAdd(veth); err != nil { t.Fatalf("creating veth: %v", err) } defer netlink.LinkDel(veth) // Move veth1 to ns1 veth1, _ := netlink.LinkByName("veth1") if err := netlink.LinkSetNsFd(veth1, int(ns1)); err != nil { t.Fatalf("moving veth1 to ns1: %v", err) } // Configure veth0 in host veth0, _ := netlink.LinkByName("veth0") addr, _ := netlink.ParseAddr("10.0.0.1/24") netlink.AddrAdd(veth0, addr) netlink.LinkSetUp(veth0) // Configure veth1 in ns1 netns.Set(ns1) veth1, _ = netlink.LinkByName("veth1") addr, _ = netlink.ParseAddr("10.0.0.2/24") netlink.AddrAdd(veth1, addr) netlink.LinkSetUp(veth1) netns.Set(origNS) // Load eBPF program spec, err := LoadMyCNF() if err != nil { t.Fatalf("loading spec: %v", err) } objs := &MyCNFObjects{} if err := spec.LoadAndAssign(objs, nil); err != nil { t.Fatalf("loading objects: %v", err) } defer objs.Close() // Attach to veth0 l, err := link.AttachXDP(link.XDPOptions{ Program: objs.XdpCnf, Interface: veth0.Attrs().Index, Flags: link.XDPGenericMode, }) if err != nil { t.Fatalf("attaching XDP: %v", err) } defer l.Close() // Generate traffic from ns1 netns.Set(ns1) sendPing(t, "10.0.0.1", 10) netns.Set(origNS) // Wait for processing time.Sleep(500 * time.Millisecond) // Validate results validateCounter(t, objs.PacketCounter, 10) } ``` ## Multi-Container Test Environments For more realistic testing scenarios, use Docker Compose to create multi-container topologies with the CNF acting as a router or gateway between containers. ### Why Multi-Container Testing **Benefits:** - Tests real packet forwarding behavior - Validates routing and redirection logic - Simulates production network topologies - Tests asymmetric routing scenarios - Verifies end-to-end connectivity - Isolates test environment from host **Use Cases:** - CNF routers between networks - Service mesh sidecar testing - Load balancer validation - VPN/tunnel endpoint testing - Multi-homing scenarios ### Example: 3-Tier Router Topology **Scenario:** Test a CNF router that solves asymmetric routing ``` Client (10.0.2.2) ↓ Router (10.0.2.1 / 10.0.1.1) [CNF with eBPF] ↓ Server (10.0.1.2) - Virtual IP: 192.168.100.5 on lo - Problem: Return traffic would go wrong way - Solution: eBPF redirect based on source IP ``` ### Docker Compose Setup **docker-compose.yml:** ```yaml version: '3' services: server: image: ubuntu:22.04 privileged: true # Required for eBPF cap_add: - NET_ADMIN - SYS_ADMIN volumes: - ./:/app entrypoint: /app/scripts/server-entrypoint.sh networks: primary: ipv4_address: 10.111.220.11 server_to_router: ipv4_address: 10.111.221.11 router: image: ubuntu:22.04 privileged: true cap_add: - NET_ADMIN - SYS_ADMIN - BPF volumes: - ./:/app - /sys/kernel/debug:/sys/kernel/debug:rw entrypoint: /app/scripts/router-entrypoint.sh networks: server_to_router: ipv4_address: 10.111.221.21 router_to_client: ipv4_address: 10.111.222.21 depends_on: - server client: image: ubuntu:22.04 privileged: true cap_add: - NET_ADMIN volumes: - ./:/app entrypoint: /app/scripts/client-entrypoint.sh networks: router_to_client: ipv4_address: 10.111.222.22 depends_on: - router networks: primary: driver: bridge ipam: config: - subnet: 10.111.220.0/24 server_to_router: driver: bridge ipam: config: - subnet: 10.111.221.0/24 router_to_client: driver: bridge ipam: config: - subnet: 10.111.222.0/24 ``` ### Entrypoint Scripts **scripts/server-entrypoint.sh:** ```bash #!/bin/bash set -e # Add virtual IP to loopback ip addr add 192.168.100.5/32 dev lo # Enable IP forwarding sysctl -w net.ipv4.ip_forward=1 # Add route to reach client via router ip route add 10.111.222.0/24 via 10.111.221.21 # Start listener echo "Server ready - listening on port 8080" nc -l -p 8080 -k & # Keep container running tail -f /dev/null ``` **scripts/router-entrypoint.sh:** ```bash #!/bin/bash set -e # Enable IP forwarding sysctl -w net.ipv4.ip_forward=1 # Add routes ip route add 192.168.100.0/24 via 10.111.221.11 echo "Router ready - starting CNF..." # Build and run CNF with eBPF program cd /app go build -o cnf-router ./cnf-router & # Keep container running tail -f /dev/null ``` **scripts/client-entrypoint.sh:** ```bash #!/bin/bash set -e # Add route to server via router ip route add 192.168.100.0/24 via 10.111.222.21 echo "Client ready" # Keep container running tail -f /dev/null ``` ### Running Tests **Start environment:** ```bash docker-compose up -d ``` **Test without eBPF (should fail):** ```bash # From client container docker-compose exec client nc -v 192.168.100.5 8080 # Expected: Connection timeout (asymmetric routing) ``` **Test with eBPF (should work):** ```bash # Router is already running CNF with eBPF # From client container docker-compose exec client nc -v 192.168.100.5 8080 # Expected: Connection successful ``` **Manual validation:** ```bash # Check eBPF program loaded docker-compose exec router bpftool prog list # Check packet counters docker-compose exec router bpftool map dump name stats # View trace logs docker-compose exec router cat /sys/kernel/debug/tracing/trace_pipe ``` **Cleanup:** ```bash docker-compose down -v ``` ### Automated Test Integration **Go test that uses Docker Compose:** ```go package main_test import ( "fmt" "os" "os/exec" "testing" "time" ) func TestCNFRouterWithDockerCompose(t *testing.T) { // Skip if Docker not available if _, err := exec.LookPath("docker-compose"); err != nil { t.Skip("docker-compose not available") } // Start containers cmd := exec.Command("docker-compose", "up", "-d") if err := cmd.Run(); err != nil { t.Fatalf("starting containers: %v", err) } // Cleanup defer func() { exec.Command("docker-compose", "down", "-v").Run() }() // Wait for containers to be ready time.Sleep(5 * time.Second) // Test: ping from client to server's virtual IP t.Run("ping_virtual_ip", func(t *testing.T) { cmd := exec.Command("docker-compose", "exec", "-T", "client", "ping", "-c", "3", "192.168.100.5") output, err := cmd.CombinedOutput() if err != nil { t.Errorf("ping failed: %v\nOutput: %s", err, output) } t.Logf("Ping output:\n%s", output) }) // Test: TCP connection from client to server t.Run("tcp_connection", func(t *testing.T) { // Start server listener serverCmd := exec.Command("docker-compose", "exec", "-T", "server", "sh", "-c", "echo 'hello' | nc -l -p 9000 &") serverCmd.Run() time.Sleep(1 * time.Second) // Connect from client clientCmd := exec.Command("docker-compose", "exec", "-T", "client", "sh", "-c", "nc -w 2 192.168.100.5 9000") output, err := clientCmd.CombinedOutput() if err != nil { t.Errorf("TCP connection failed: %v", err) } if string(output) != "hello\n" { t.Errorf("unexpected response: got %q, want %q", output, "hello\n") } }) // Test: verify eBPF statistics t.Run("ebpf_statistics", func(t *testing.T) { cmd := exec.Command("docker-compose", "exec", "-T", "router", "bpftool", "map", "dump", "name", "stats") output, err := cmd.CombinedOutput() if err != nil { t.Logf("Warning: could not read stats: %v", err) return // Don't fail test } t.Logf("eBPF statistics:\n%s", output) // Could parse output and check counters here }) } ``` ### Alternative: Makefile-Based Testing **Makefile:** ```makefile .PHONY: test-up test-validate test-down test-all test-up: docker-compose up -d @echo "Waiting for containers..." @sleep 5 test-validate: test-up @echo "Testing connectivity..." docker-compose exec -T client ping -c 3 192.168.100.5 docker-compose exec -T client nc -zv 192.168.100.5 8080 test-down: docker-compose down -v test-all: test-validate test-down ``` Usage: ```bash make test-all ``` ### Advanced Multi-Container Patterns #### Pattern 1: Chain of CNFs ```yaml services: cnf1: # First CNF - packet classifier cnf2: # Second CNF - rate limiter depends_on: [cnf1] cnf3: # Third CNF - load balancer depends_on: [cnf2] ``` #### Pattern 2: Service Mesh Topology ```yaml services: app: # Application container sidecar: # eBPF sidecar CNF network_mode: "service:app" # Share network namespace ``` #### Pattern 3: Multi-Path Testing ```yaml services: router: networks: - isp1 - isp2 - internal # Test multiple paths/ISPs ``` ### Debugging Multi-Container Tests **View logs:** ```bash docker-compose logs -f router docker-compose logs -f server ``` **Enter container:** ```bash docker-compose exec router /bin/bash ``` **Check routing:** ```bash docker-compose exec router ip route docker-compose exec server ip route docker-compose exec client ip route ``` **Capture packets:** ```bash docker-compose exec router tcpdump -i eth0 -n ``` **View eBPF programs:** ```bash docker-compose exec router bpftool prog list docker-compose exec router bpftool map list ``` ### Best Practices for Multi-Container Tests 1. **Use specific container images** with version tags (not `latest`) 2. **Set resource limits** to prevent test interference 3. **Add health checks** to wait for readiness 4. **Use named networks** for clarity 5. **Volume mount your CNF binary** for faster iteration 6. **Add cleanup in defer/trap** to avoid orphaned containers 7. **Check prerequisites** (Docker, docker-compose) in test setup 8. **Use meaningful container names** for debugging 9. **Log extensively** during setup and validation 10. **Test both success and failure** scenarios ### Common Issues **Problem:** eBPF program won't load in container **Solution:** Ensure `privileged: true` and correct capabilities **Problem:** Asymmetric routing still occurs **Solution:** Verify CNF attached to correct interface and direction (egress) **Problem:** Containers can't resolve each other **Solution:** Use IP addresses or configure DNS in docker-compose **Problem:** tc/bpftool commands fail **Solution:** Mount `/sys/kernel/debug` and ensure BPF capability ## Benchmarking ```go func BenchmarkCNFThroughput(b *testing.B) { // Setup CNF... packet := createTCPSYN("10.0.0.1", "10.0.0.2", 12345, 80) b.ResetTimer() for i := 0; i < b.N; i++ { injectPacket(b, "veth0", packet) } } func BenchmarkMapLookup(b *testing.B) { // Load program... var ( key uint32 = 0 value uint64 ) b.ResetTimer() for i := 0; i < b.N; i++ { objs.CounterMap.Lookup(&key, &value) } } ``` ## Best Practices 1. **Use t.Helper()**: Mark helper functions 2. **Use t.Cleanup()**: Automatic resource cleanup 3. **Skip non-root tests**: Check `os.Getuid()` 4. **Isolate with namespaces**: Avoid interfering with host 5. **Add timeouts**: Prevent hanging tests 6. **Validate incrementally**: Test small pieces first 7. **Use table-driven tests**: Test multiple scenarios 8. **Mock when possible**: Unit test eBPF logic separately 9. **Clean up resources**: Use defer or t.Cleanup() 10. **Log important info**: Use t.Logf() for debugging ## Common Test Patterns ```go // Table-driven test func TestPacketFiltering(t *testing.T) { tests := []struct { name string srcIP string dstIP string wantDrop bool }{ {"allow local", "10.0.0.1", "10.0.0.2", false}, {"drop external", "1.1.1.1", "10.0.0.2", true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { // Test logic... }) } } ```