eBPF in Production: Kernel-Level Observability and Security

by

Every platform team that has bolted a service mesh sidecar onto every pod eventually asks the same question: why run a full proxy per pod just to see and control traffic the kernel already handles? That question is what pushed a lot of us toward eBPF observability and security. Rather than stack another layer on top, you move visibility and enforcement into the kernel — at the same hook points the traffic and system calls already pass through.

This is the architecture I reach for when a client’s sidecar mesh is eating double-digit percentages of fleet-wide CPU and memory, or when a security team needs process-level enforcement that a network-only agent can’t provide. It assumes you already know basic Linux networking and container runtime concepts — this is about applying eBPF specifically to production observability and security, not an introduction to what eBPF is.

The Problem: Why Sidecars and Host Agents Miss What Production Needs

A sidecar-based service mesh solves a real problem — consistent mTLS, retries, and L7 routing without touching application code — but it does so by inserting a full userspace proxy into the request path of every pod. That has three costs that compound at scale.

Overhead is per-pod, not per-node. A typical Envoy sidecar adds 50–100MB of memory and a meaningful CPU tax per pod, and every request now makes two extra hops (client-side proxy, server-side proxy) instead of zero. On a fleet running thousands of pods, that’s not a rounding error — it’s a standing tax you pay whether or not the mesh’s advanced routing features are in use that hour. Latency-sensitive services feel it directly: 1–2ms per proxy hop doesn’t sound like much until it’s stacked twice on every call in a deep service graph.

Sidecars only see what passes through them. A proxy can only inspect traffic that’s actually routed through it. Traffic that bypasses the proxy — raw sockets, UDP flows some meshes don’t intercept, traffic between processes on the same host outside the mesh’s iptables/nftables redirect rules — is invisible. And a sidecar has zero visibility into what’s happening inside the container’s process tree: which binary executed, which file got opened, whether a shell spawned unexpectedly inside a database pod. That’s a security blind spot, not just an observability gap.

Traditional host agents trade depth for compatibility, and usually lose both. Kernel modules give deep visibility but require per-kernel-version builds and carry real crash risk if buggy — a bad kernel module panics the box. Userspace agents that poll /proc, tail logs, or sample syscalls trade accuracy for safety: you get eventual, incomplete signal instead of an accurate event stream, and high-frequency polling itself adds load.

eBPF sidesteps this trilemma. It runs small, kernel-verified programs directly at stable hook points — kprobes, tracepoints, XDP, LSM hooks — without a kernel module, without a proxy in the data path, and without polling. The kernel verifies every program before it loads, so a buggy eBPF program fails to load rather than crashing the box. That combination — kernel-native placement, safety guarantees, and CO-RE (Compile Once – Run Everywhere) portability — is why teams outgrowing both sidecars and legacy agents keep landing on eBPF.

Reference Architecture: An eBPF Data Plane Without Sidecars

The setup I keep landing on: eBPF programs attached directly to kernel hook points on every node, feeding both a network policy path and an observability/security event path, with a single node agent (Cilium for networking, Tetragon for runtime security — often deployed together) doing the userspace coordination. No per-pod proxy anywhere in the diagram.

flowchart TB
    subgraph Workloads["Pod / Process Workloads — no sidecar"]
        P1[Pod A - app container]
        P2[Pod B - app container]
        P3[Host process]
    end

    subgraph Kernel["Linux Kernel — Hook Points"]
        direction LR
        KP[kprobes/kretprobes - syscalls]
        TP[Tracepoints - stable ABI events]
        XDP[XDP - NIC driver ingress]
        TC[tc/eBPF - egress, L3/L4 policy]
        LSM[LSM hooks - exec, file, socket]
    end

    subgraph BPFProg["eBPF Programs — verified, JIT-compiled"]
        direction LR
        B1[Network policy program]
        B2[Process/exec tracing program]
        B3[Socket-level LB/redirect program]
    end

    subgraph Maps["BPF Maps"]
        M1[Ring buffer - events]
        M2[Hash maps - identity, policy, conn state]
    end

    subgraph Control["Userspace Control + Observability"]
        Agent[Node Agent - Cilium/Tetragon]
        Policy[Policy Engine - identity-aware L3-L7]
        Obs[Observability Backend - Hubble/Prometheus/Grafana]
        SIEM[SIEM / Alerting]
    end

    P1 --> KP
    P2 --> TP
    P3 --> LSM
    KP --> B2
    TP --> B2
    LSM --> B2
    XDP --> B1
    TC --> B1
    B1 --> M2
    B2 --> M1
    B3 --> M2
    M1 --> Agent
    M2 --> Agent
    Agent --> Policy
    Agent --> Obs
    Obs --> SIEM
    Policy -.->|updates maps, no program reload| M2

ASCII fallback for docs, terminals, or anywhere Mermaid doesn’t render:

   Pods/processes (no sidecar proxy in the data path)
          |               |                |
     syscalls/exec    network I/O      file/socket ops
          |               |                |
   +------v------+  +-----v------+   +-----v------+
   | kprobes/    |  | XDP (NIC)  |   | LSM hooks  |
   | tracepoints |  | tc (egress)|   | (exec/file)|
   +------|------+  +-----|------+   +-----|------+
          |                |                |
   +------v----------------v----------------v------+
   |         eBPF programs (verified, JIT'd)        |
   |   policy program | tracing program | LB/redir  |
   +------|-----------------------------------|-----+
          |                                    |
    ring buffer (events)                hash maps (policy/identity/conn state)
          |                                    |
          +----------------+-------------------+
                           |
                  Node agent (Cilium/Tetragon)
                    |                |
             Policy engine    Observability backend
            (identity-aware   (Hubble/Prometheus/Grafana)
             L3-L7 policy,          |
             updates maps      SIEM / alerting
             live, no reload)

   Everything left of "node agent" runs in kernel space, on every node,
   with zero per-pod proxy processes.

The key design choice: policy and identity live in BPF maps that the kernel programs read on every packet or syscall, and the userspace agent updates those maps live. Changing a network policy or a security rule doesn’t mean recompiling or reloading the eBPF program — it means writing a new map entry, which is why policy changes propagate in milliseconds instead of requiring a rolling proxy restart.

Decision Criteria: eBPF-Native vs Sidecar Service Mesh vs Traditional Host Agents

Criteria eBPF-Native (Cilium/Tetragon) Sidecar Service Mesh (Istio/Linkerd) Traditional Host Agents
Performance overhead Low — in-kernel processing, no extra hops, shared per-node cost instead of per-pod High — proxy per pod, two extra hops per request, meaningful memory/CPU tax at scale Medium — depends on polling frequency; kernel modules add near-zero runtime overhead but real crash risk
Visibility depth Deep — syscalls, process exec, file access, and network, correlated by kernel-verified identity Network/L7 only, and only for traffic actually routed through the proxy Shallow to deep depending on tooling; usually sampled or delayed, rarely both broad and real-time
Security enforcement In-kernel enforcement (drop packets, kill processes) before the action completes, not just after-the-fact alerting L7 policy (mTLS, authz) only for meshed traffic; no process-level enforcement Detection-oriented; enforcement usually requires a separate LSM or firewall layer
Operational complexity Medium-high — requires kernel/BPF literacy on the team, but a single agent replaces mesh + several point tools Medium — well-documented, but adds a proxy lifecycle (upgrades, cert rotation, resource tuning) to manage per workload Low to medium — familiar tooling, but often means stitching together separate network, process, and file monitoring products
Kernel/version constraints Real — needs a reasonably modern kernel (5.x+ recommended) with BTF for CO-RE; older or heavily customized kernels need work None specific to the kernel — runs in userspace, portable across kernel versions Kernel modules are tightly kernel-version coupled; userspace agents are portable but lose depth
Best fit Platforms already on Kubernetes/cloud-native stacks with kernel expertise in-house, needing both network policy and runtime security from one data plane Teams that need mesh features (traffic shaping, canary routing, mTLS) more than raw performance, and don’t have in-house kernel expertise Simple environments, legacy (non-containerized) hosts, or teams unwilling to take on kernel-level tooling yet

Where I draw the line: if you’re running Kubernetes at meaningful scale and already fighting sidecar tax or gaps in runtime security visibility, eBPF-native tooling is worth the ramp-up. If your primary need is mesh-style traffic management (canaries, retries, fine-grained L7 routing) and your fleet is small enough that the sidecar tax doesn’t hurt yet, a service mesh is still the pragmatic choice — several mesh projects are themselves adopting eBPF for the data plane while keeping the control plane familiar, which is a reasonable middle path.

A Concrete Example: bpftrace for Ad Hoc Observability, Tetragon for Enforcement

Two levels of “real code,” because production eBPF work spans both: quick ad hoc tracing during an incident, and durable, declarative policies that run continuously.

For ad hoc observability, a single bpftrace line attached to a kernel function, with no agent, no daemon, and no code deployed to the application:

# Count TCP retransmissions per process command, every 10 seconds.
# Useful for correlating network degradation with a specific service
# without touching application code or adding a sidecar.
bpftrace -e '
kprobe:tcp_retransmit_skb { @retransmits[comm] = count(); }
interval:s:10 { print(@retransmits); clear(@retransmits); }
'

This attaches a kprobe to tcp_retransmit_skb, the kernel function invoked on every TCP retransmission, and aggregates a count per process name (comm) in a BPF map, printing and clearing it every ten seconds. During an incident, this is the fastest way to answer “which service is actually seeing retransmits right now” — no log shipping, no dashboards to build, no sidecar to inspect.

For durable enforcement, a Tetragon TracingPolicy that hooks an LSM-instrumented kernel function to detect — and kill — any process that touches /etc/shadow, regardless of which syscall it uses to get there:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "block-shadow-read"
spec:
  kprobes:
  - call: "security_file_permission"
    syscall: false
    args:
    - index: 0
      type: "file"
    - index: 1
      type: "int"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Prefix"
        values:
        - "/etc/shadow"
      matchActions:
      - action: Sigkill

security_file_permission is an LSM hook the kernel calls on file access regardless of the entry point (open, openat, memory-mapped reads), which makes it a more robust match point than hooking individual syscalls one by one. The matchArgs selector filters to paths prefixed with /etc/shadow, and matchActions sends SIGKILL to the offending process before the read completes. This runs in the kernel, on every node, with no application changes and no proxy — it’s enforcement, not just an alert that fires after the fact.

Pitfalls and What I’d Do in Production

Kernel version and CO-RE portability aren’t automatic. CO-RE relies on BTF (BPF Type Format) being present in the running kernel so a program compiled once can adjust to that kernel’s actual struct layouts at load time. Most distros ship BTF by default on kernels 5.x+, but older kernels, minimal/embedded images, or heavily customized kernels sometimes strip it. What I’d do: verify /sys/kernel/btf/vmlinux exists on your actual target hosts before committing to an eBPF-based rollout, not just in a dev VM — and have a fallback plan (bundling BTF, or building per-kernel) for the hosts that don’t have it.

The verifier will reject programs that look fine to you. The BPF verifier enforces a bounded stack (512 bytes), limits on program complexity, and restrictions on loops and pointer arithmetic — all necessary to guarantee a program terminates and can’t read arbitrary memory. Programs that need to do more get split across tail calls (up to 32 chained programs) or use BPF-to-BPF function calls. What I’d do: budget verifier iteration into the timeline for anything non-trivial — expect to restructure logic around the verifier’s rules rather than around what’s algorithmically simplest, and lean on existing map-based patterns from mature projects instead of reinventing state-tracking logic.

Observability cardinality gets expensive fast. eBPF makes it trivially easy to trace every syscall, every packet, or every function call — which means it’s just as easy to generate more events than any backend can ingest or you can afford to store. What I’d do: aggregate in-kernel using BPF maps (counts, histograms, top-N) before anything crosses into userspace, and treat full per-event export as a targeted, time-boxed debugging tool rather than a standing collection default.

Loading eBPF programs is a privileged operation — treat it like one. Loading a program has historically required CAP_SYS_ADMIN; kernels 5.8+ split this into more granular capabilities (CAP_BPF, CAP_PERFMON, CAP_NET_ADMIN), but it’s still meaningful kernel access, and kernel lockdown mode can block bpf() entirely in confidentiality mode. What I’d do: restrict which nodes and service accounts can load programs, use signed/vetted images from the project you’re standardizing on rather than ad hoc custom programs in production, and audit bpf() syscall usage itself — an attacker who can load their own eBPF program has kernel-level access.

Measure real overhead, not the vendor’s benchmark. XDP’s performance profile changes enormously between native mode (running in the NIC driver) and generic mode (running later in the stack, when the driver doesn’t support native XDP) — the difference can be an order of magnitude in packets-per-second. What I’d do: benchmark on your actual NIC/driver combination under your actual traffic shape before trusting any published number, and re-verify after driver or kernel upgrades since native XDP support is driver-specific and can regress silently.

When I wouldn’t reach for eBPF at all. Rolling your own eBPF programs for problems mature projects already solve (Cilium, Tetragon, Falco, Pixie) is rarely worth it without in-house kernel engineering capacity — you inherit verifier debugging, BTF/CO-RE edge cases, and upgrade risk that a maintained project has already absorbed. It’s also a poor fit on kernels too old for BTF/CO-RE, on managed platforms that restrict the capabilities needed to load programs, or when the actual need is L7 application routing logic — that’s still better served by a proxy that understands the protocol semantics than by kernel-level packet inspection.

Related Reading

Let’s Talk eBPF

If you’re weighing eBPF-native tooling against a sidecar mesh or legacy host agents and want a second opinion on the architecture, I’m happy to compare notes. Connect with me on LinkedIn or reach out via Contact.