eBPF in Production: Superpowers for Linux Observability and Security

eBPF is one of the most consequential Linux technologies of the last decade. It lets you run sandboxed programs inside the kernel in response to events, without writing or loading kernel modules. In production, that means deep observability, programmable networking, runtime security, and continuous profiling with far less overhead and far fewer application changes than traditional agents. But eBPF is not a magic switch. It is an execution model, an ecosystem, and an operational discipline. Teams that adopt it successfully treat it as critical infrastructure, not as a debugging toy.

What eBPF Actually Is

eBPF stands for extended Berkeley Packet Filter, but the name is historical. Modern eBPF is a general-purpose in-kernel virtual machine. It runs bytecode that has been verified for safety, JIT-compiled for speed, and attached to hooks across the kernel and userspace. It is not a single tool. It is a collection of capabilities: a verifier, a JIT compiler, maps, ring buffers, helper functions, program types, and attach points.

• Kernel-side execution: eBPF programs run in kernel context, close to the data they observe or control.
• Event-driven: Programs fire on syscalls, network packets, tracepoints, function entry and exit, cgroup events, and more.
• Safe by construction: The verifier rejects unsafe memory access, unbounded loops, and invalid helper calls before a program is loaded.
• Programmable: Maps and ring buffers let eBPF programs share state with userspace and with each other.
• Not kernel modules: eBPF programs cannot call arbitrary kernel functions. They operate within a constrained instruction set and helper API.

The safety model is the key difference from kernel modules. A bug in a kernel module can crash the host. A bug in an eBPF program is usually caught by the verifier or contained by the runtime. That does not make eBPF risk-free, but it makes it far more practical for production agents.

Why eBPF Matters in Cloud Native and Production

Traditional observability and security rely on agents, sidecars, service meshes, and log pipelines. Each layer adds overhead, deployment complexity, and blind spots. eBPF changes the economics by observing kernel events directly. It can see syscalls, network flows, process executions, file access, DNS queries, and scheduler behavior without modifying application code.

• No application instrumentation: Works for compiled, interpreted, and legacy workloads.
• Low overhead when designed well: Aggregation in kernel space and sampling keep event volume manageable.
• Kubernetes-aware: Agents can enrich kernel events with pod, namespace, service, and container identity.
• Security telemetry: Captures process, file, network, and privilege activity for detection and enforcement.
• Network dataplane: Powers high-performance load balancing, policy enforcement, and DDoS mitigation.
• Profiling: Samples CPU stacks and off-CPU time with less overhead than many userspace profilers.

In cloud-native environments, eBPF often replaces or augments iptables, sidecar proxies, node agents, and kernel modules. It can reduce the number of moving parts per node while increasing the depth of telemetry.

The eBPF Architecture: From Bytecode to Verifier

A typical eBPF workflow looks like this: source code in C, Rust, or a tracing language is compiled to eBPF bytecode. The bytecode is loaded through the bpf system call. The kernel verifier checks safety, then the JIT compiler translates it to native machine code. The program is attached to a hook. When the hook fires, the program executes, reads context, updates maps, and optionally emits events to userspace through a ring buffer or perf buffer.

Userspace plays a critical role. It loads programs, configures maps, consumes events, and enriches data with metadata such as Kubernetes pod names. The split between kernel-side collection and userspace-side processing is where most production design decisions happen.

Program Types and Attach Points

eBPF program types determine what a program can do and where it can attach. Choosing the right hook is a balance between stability, performance, and capability.

• kprobe and kretprobe: Attach to kernel function entry and return. Powerful but brittle across kernel versions.
• tracepoint: Attach to stable kernel tracepoints. Preferred for long-term production tracing.
• raw tracepoint: Lower-level tracepoint access with less overhead but more complexity.
• fentry and fexit: BTF-based function entry and exit hooks. Safer and more portable than kprobes when BTF is available.
• uprobe and uretprobe: Attach to userspace function entry and return. Useful for language runtimes and OpenSSL tracing.
• USDT: User statically defined tracepoints for applications that expose them.
• XDP: Earliest packet processing at the driver level. Ideal for DDoS mitigation and high-performance networking.
• tc and classifier: Traffic control hooks for shaping, redirecting, and inspecting packets.
• cgroup, sock, and sk_msg: Cgroup and socket-level hooks for policy, load balancing, and per-pod controls.
• LSM: Linux Security Module hooks for enforcement, not just detection.
• perf_event: Performance counters and sampling for profiling.

Production tip: prefer stable hooks. Tracepoints, fentry, fexit, and LSM hooks tend to be more maintainable than raw kprobes. If you must use kprobes, isolate them behind feature flags and test across every kernel version in your fleet.

Maps, Ring Buffers, and CO-RE

eBPF maps are kernel-resident key-value stores shared between eBPF programs and userspace. They are the backbone of stateful eBPF applications. Common map types include hash maps, arrays, LRU hash maps, per-CPU hash maps, ring buffers, perf event arrays, and sockmaps.

• Ring buffer: The preferred event streaming mechanism on modern kernels. It uses a single shared buffer and handles variable-length records efficiently.
• Perf buffer: Older but widely supported. Uses per-CPU buffers and is still common in existing tools.
• Per-CPU maps: Reduce contention by giving each CPU its own map instance. Ideal for counters and per-CPU aggregation.
• LRU maps: Automatically evict old entries when full. Useful for bounded-cardinality tracking.
• CO-RE: Compile Once, Run Everywhere. Uses BTF to relocate field offsets at load time so one binary can run across multiple kernel versions.
• BTF: BPF Type Format. Exposes kernel and module type information. Essential for CO-RE and modern tracing.

Production tip: pin maps with bpffs when agents restart. Pinning lets state survive restarts and allows multiple components to share maps, but it also requires careful ownership and cleanup to avoid stale data.

Production Use Cases

Observability Without Sidecars

eBPF can trace HTTP and gRPC at the socket layer, DNS queries, TCP retransmits, process execution, file access, and scheduler latency. Tools can auto-discover services and build service maps without a sidecar per pod. This is especially valuable for polyglot environments and legacy workloads that cannot be restarted with new instrumentation.

Limitations matter. Encrypted traffic requires uprobes on TLS libraries or integration with a service mesh. Protocol parsing at the socket layer is harder than application-level instrumentation. For deep request context, eBPF is often one signal among several, not the only one.

Network Performance and Service Maps

eBPF powers Cilium’s dataplane: load balancing, network policy, transparent encryption, and observability. XDP can drop DDoS traffic at line rate before it reaches the kernel network stack. tc eBPF programs can shape, redirect, and inspect packets. For platform teams, this means Kubernetes NetworkPolicy enforcement without the scale problems of iptables.

Service maps built from eBPF data are useful for dependency discovery and incident response. They show which services talk to which, with latency and error rates, without requiring distributed tracing headers from every application.

Runtime Security and Policy Enforcement

Falco and Tetragon use eBPF to detect suspicious syscalls, process executions, file writes, and network connections. eBPF LSM enables enforcement, not just detection. Examples include blocking execution from /tmp, restricting ptrace, enforcing file integrity, and preventing privilege escalation.

Security teams must balance false positives, kernel compatibility, and operational risk. Enforcement should be rolled out in audit mode first, with allowlists, canary nodes, and a fast rollback path.

Profiling and Capacity Planning

Continuous profilers such as Parca and Pyroscope use eBPF to sample CPU stacks with low overhead. eBPF can also measure off-CPU time, lock contention, scheduling delays, and I/O latency. This helps answer questions that metrics alone cannot: Why is p99 latency high? Is the bottleneck CPU, locks, disk, network, or the scheduler?

In production, profiling data should be aggregated and sampled. Storing every stack trace from every host is expensive and rarely necessary. Flame graphs over time windows are usually enough to find regressions.

Tooling Landscape

• bpftrace: High-level tracing for one-liners and ad hoc debugging. Excellent for SREs and incident responders.
• BCC: Python and Lua frontends with prebuilt tools. Useful for exploration, but heavier than libbpf-based agents.
• libbpf: C library for production eBPF programs. Strong CO-RE support and a common foundation for modern tools.
• libbpf-rs and Aya: Rust options for safer userspace and eBPF development.
• Cilium: Kubernetes networking, security, and observability platform built on eBPF.
• Falco: Runtime security detection with eBPF and other drivers.
• Tetragon: eBPF-based security observability and enforcement.
• Pixie: Kubernetes observability with eBPF and auto-telemetry.
• Parca, Pyroscope, and Coroot: Profiling and observability platforms that use eBPF.

Tool choice depends on whether you need ad hoc debugging, a production agent, a security product, or a platform dataplane. Many teams use bpftrace for debugging and a libbpf-based agent for long-running production collection.

Designing an eBPF Production Rollout

1. Start with questions, not tools. Define the exact signals you need: latency attribution, policy violations, DNS failures, or file integrity events.
2. Validate kernel support. Check kernel version, BTF availability, CONFIG_DEBUG_INFO_BTF, and required program types. Older kernels may need backports or fallback paths.
3. Prefer stable hooks. Use tracepoints, fentry, fexit, and LSM where possible. Avoid raw kprobes unless there is no alternative.
4. Build with CO-RE. Compile once against BTF and let libbpf relocate at load time. Test across every kernel version in your fleet.
5. Control overhead. Use sampling, filters, per-CPU maps, and ring buffers. Never emit an event for every packet or syscall unless you have measured capacity.
6. Design for failure. eBPF agents should degrade gracefully. If the verifier rejects a program or a kernel lacks a feature, the platform should continue with reduced visibility.
7. Secure the agent. eBPF programs run with kernel privileges. Sign releases, limit capabilities, audit map access, and restrict who can load programs.
8. Observe the observer. Track lost events, ring buffer drops, verifier failures, CPU usage, and memory. Meta-monitoring is mandatory.

A phased rollout works best. Start with a canary node pool. Enable one probe at a time. Measure overhead and event loss. Expand only after the agent is stable and the data is actionable.

Performance and Overhead: What to Measure

eBPF is not free. Overhead depends on hook frequency, program complexity, event volume, and map contention.

• Hook frequency: Tracing every syscall on a busy host is expensive. Filter early in the program.
• Event volume: Sending every event to userspace can saturate CPUs. Aggregate in kernel maps whenever possible.
• Map contention: Shared hash maps can become bottlenecks across CPUs. Use per-CPU maps or LRU maps.
• Ring buffer pressure: Monitor drops. If drops occur, sample or reduce event size.
• JIT and verifier: JIT usually improves performance, but complex programs may hit verifier limits.
• Memory: Maps can grow. Set max entries and use LRU eviction for bounded memory.

Measure with production traffic. Benchmark agent CPU, memory, event loss, and latency impact. A good rollout starts with a feature flag that can disable each probe independently.

Security and Safety Considerations

eBPF improves security but also expands the attack surface if mismanaged.

• Privilege: Loading eBPF programs typically requires CAP_BPF and CAP_PERFMON on newer kernels, or CAP_SYS_ADMIN on older ones. Restrict tightly.
• Verifier: The verifier prevents unsafe memory access and unbounded loops, but logic bugs can still cause data leaks or performance issues.
• Supply chain: Treat eBPF bytecode as executable kernel code. Sign, scan, and review it like any privileged binary.
• Data privacy: eBPF can capture filenames, process arguments, and network metadata. Redact sensitive fields before storage.
• Kernel compatibility: A program that works on one kernel may fail on another. Use CO-RE and CI testing across kernel versions.
• Enforcement risk: Blocking syscalls or network connections can cause outages. Start in audit mode and build rollback paths.

Common Pitfalls and Failure Modes

• Assuming all kernels are equal. Cloud providers, distributions, and managed Kubernetes versions vary widely.
• Over-tracing. Starting with too many probes causes performance regressions and alert fatigue.
• Ignoring event loss. Silent drops create false confidence in observability.
• Brittle kprobes. Kernel function names change. Use tracepoints or fentry and fexit when possible.
• Unbounded cardinality. Labels from process IDs, container IDs, or filenames can explode metrics and storage.
• No fallback. When eBPF cannot load, teams lose visibility unless another telemetry path exists.
• Confusing detection with enforcement. Enforcing security policies with eBPF requires careful testing to avoid production outages.
• Neglecting userspace. Kernel-side collection is only half the system. Userspace enrichment, buffering, and export must be designed for scale.

Hands-on Example: Tracing File Opens with bpftrace

This one-liner counts file opens by process name using a tracepoint. It is a safe starting point because tracepoints are stable and aggregation happens in kernel space.

bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'

Run it on a test host for 10 seconds, then press Ctrl-C. The output shows counts by process. For production, move this logic into a libbpf CO-RE program, add filters for specific containers, and emit only aggregated metrics or sampled events.

Another useful pattern is measuring TCP retransmits:

bpftrace -e 'tracepoint:tcp:tcp_retransmit_skb { @[comm] = count(); }'

These examples show the strength of eBPF: quick, deep answers without changing applications. Turning them into reliable platform features requires packaging, testing, and observability around the agent itself.

eBPF and Kubernetes: Practical Integration Patterns

In Kubernetes, eBPF agents typically run as a DaemonSet with hostPID, hostNetwork, and access to /sys/kernel/debug or /sys/fs/bpf. They enrich events with pod metadata by watching the Kubernetes API or using container runtime hooks.

• DaemonSet per node: Standard for node-level observability and security agents.
• eBPF-based CNI: Cilium can replace kube-proxy and enforce network policy with better scale characteristics.
• Sidecarless service mesh: eBPF can handle mTLS and L7 policy in some architectures, reducing per-pod overhead.
• Multi-tenant isolation: Cgroup eBPF programs can apply per-pod network and security policies.

The key challenge is mapping kernel events to Kubernetes identity. Process IDs and network namespaces change. Pod IPs are reused. Agents must maintain a cache with retries, fallbacks, and TTLs. Without accurate identity, observability becomes noisy and security alerts become unreliable.

Build vs Buy: When to Write Your Own eBPF

Buy or adopt open source when the problem is common: Kubernetes network policy, runtime security, continuous profiling, and service maps. Build custom eBPF when you have a unique kernel-level question: a specific protocol, a custom scheduler, a proprietary runtime, or a latency path that no vendor instruments.

• Adopt: Cilium for networking and policy; Falco or Tetragon for security; Parca for profiling.
• Extend: Use existing agents if they expose custom probe hooks or OpenTelemetry export.
• Build: If the signal is core to your product or competitive advantage, invest in libbpf, CO-RE, CI across kernels, and a small expert team.

Writing production eBPF is not just writing a probe. It is building a privileged agent with lifecycle management, upgrade safety, crash recovery, and compatibility testing. Budget for that work before committing to a custom implementation.

The Future of eBPF in Production

eBPF is expanding from tracing and networking into scheduling, memory management, and security enforcement. Projects like sched_ext allow custom CPU schedulers in eBPF. BPF LSM is maturing for mandatory access control. eBPF is also becoming a standard telemetry source for OpenTelemetry pipelines.

Expect three shifts:

• Standardization: More stable program types and cross-kernel portability through CO-RE and BTF.
• Consolidation: Fewer agents per node as networking, security, and observability converge on eBPF dataplanes.
• Governance: Stronger controls around who can load eBPF, what it can observe, and how data is redacted.

The most successful teams will treat eBPF as a platform capability, not a collection of scripts. They will standardize on a small set of agents, enforce CO-RE, monitor overhead, and integrate eBPF data into existing observability and security workflows.

Conclusion

eBPF gives production teams a rare capability: deep, programmable visibility into the Linux kernel without the fragility of kernel modules or the overhead of application instrumentation. It powers networking, security, observability, and profiling in modern cloud-native platforms. But eBPF is not magic. It demands careful hook selection, CO-RE discipline, overhead measurement, failure handling, and security controls.

Start small. Pick one high-value question, answer it with a stable tracepoint or bpftrace, then productionize with a CO-RE agent and meta-monitoring. When done well, eBPF becomes a quiet foundation for reliability and security across the entire fleet.