Service Mesh vs eBPF-Native Data Planes: How to Choose

by

Every platform team that runs Kubernetes past a few dozen services eventually has the service mesh vs eBPF argument, usually in a design review where one engineer points at fleet-wide CPU graphs and says the Envoy sidecars are costing too much, and another engineer points at a canary rollout ticket and says none of that matters if you can’t shift 10% of traffic to a new version without touching application code. Both of them are right. That’s the actual problem — not “sidecars bad” or “eBPF hype,” but a genuine trade-off between how much a data plane costs to run and how much it can do with a single request once it’s in flight.

I wrote about eBPF for kernel-level observability and security in an earlier piece — Tetragon watching process exec, Cilium enforcing identity-aware policy in the kernel. This one is narrower and more contested: what should actually carry your service-to-service traffic, terminate mTLS, and make routing decisions on a live request. That’s a different question from “should I run a kernel agent instead of a log-tailing daemon,” and it deserves its own answer.

The Problem: What a Traffic Data Plane Actually Has to Do

Strip away the marketing and a service mesh data plane is answering four questions on every request: who is this caller, are they allowed to reach this destination, where should this specific request go, and what happens if the destination is slow or down. The first two are identity and policy. The last two are traffic management, and that’s where the real disagreement lives.

Identity and policy are, at this point, a solved problem at the L3/L4 level — mutual TLS between workloads, identity derived from something stronger than a source IP, deny-by-default network policy. Sidecar meshes solved it with per-workload proxies terminating TLS on both ends. eBPF-native platforms solve the same problem in the kernel, with no proxy in the path at all. Neither camp disputes that this part works.

The argument starts at layer 7. A canary rollout that shifts 10% of traffic to a new version based on a request header. A retry policy that gives a flaky downstream two attempts with a two-second per-try timeout, but only for idempotent methods. A circuit breaker that stops sending traffic to an instance once it starts erroring, without a human paging in to do it. These are per-request, protocol-aware decisions. Something has to parse HTTP framing, read headers, count errors per upstream, and hold that state — a raw packet or a TCP segment doesn’t carry any of it. That something is either a proxy sitting in the request path that understands the protocol, or nothing.

That’s the trade-off in one sentence: every L7 feature you want costs you a proxy hop somewhere, and every proxy hop you remove costs you an L7 feature. What’s changed in the last few years isn’t the trade-off itself — it’s how much middle ground now exists between “proxy on every pod” and “no proxy anywhere.”

Reference Architecture: Sidecar Mesh vs Ambient vs eBPF-Native L3/L4

Three shapes are actually running in production, not two. It helps to draw them side by side rather than pick a side first.

The classic sidecar model — Istio with Envoy, Linkerd with its own Rust micro-proxy — injects a proxy into every pod’s network namespace. Every request leaves the app container, hits the local sidecar, crosses the network to the destination’s sidecar, and only then reaches the destination app container. Two extra hops and two TLS terminations, per pod, for every request, whether or not that request needed any of the L7 features the proxy provides.

The ambient model — Istio’s ztunnel plus optional waypoint proxies, structurally similar to what Cilium’s own service mesh does — moves the always-on part down to one proxy per node instead of one per pod. That per-node proxy (ztunnel) handles mTLS and L4 authorization for every pod on the node. It does not parse HTTP. When a namespace or service account actually needs L7 behavior — retries, header routing, canary weighting — you deploy a waypoint proxy for just that workload, and only that traffic takes the extra hop through it. Everything else gets identity and encryption without an L7 proxy anywhere near it.

The eBPF-native model, in its purest form, skips the proxy entirely for L3/L4. Cilium’s agent programs the kernel data path — tc/eBPF and socket-level load balancing — to enforce identity-based policy and, optionally, transparent WireGuard or IPsec encryption between nodes. No per-pod or per-node proxy process handles this traffic; the kernel does. L7 policy, where Cilium supports it, runs through a shared per-node Envoy instance invoked only for the specific traffic that needs it — architecturally close to the waypoint idea, just approached from the CNI side of the fence instead of the mesh side.

flowchart TB
    subgraph SidecarModel["Sidecar Mesh — Istio classic / Linkerd"]
        direction LR
        SP1[Pod A - app container] --> SS1[Sidecar proxy]
        SS1 -->|mTLS, per request| SS2[Sidecar proxy]
        SS2 --> SP2[Pod B - app container]
    end

    subgraph AmbientModel["Ambient / Sidecarless — Istio ambient, Cilium Service Mesh"]
        direction LR
        AP1[Pod C - app container] --> AZ1[ztunnel - per node]
        AZ1 -->|mTLS + L4 authz| AZ2[ztunnel - per node]
        AZ2 --> AP2[Pod D - app container]
        AZ1 -.->|opt-in L7 only| WP[Waypoint proxy - per namespace/SA]
        WP -.-> AZ2
    end

    subgraph EbpfModel["eBPF-Native L3/L4 — Cilium core, no L7 proxy"]
        direction LR
        EP1[Pod E - app container] --> EK1[eBPF - tc/socket LB, node]
        EK1 -->|WireGuard/IPsec + identity policy| EK2[eBPF - tc/socket LB, node]
        EK2 --> EP2[Pod F - app container]
    end

ASCII fallback for docs, terminals, or anywhere Mermaid doesn’t render:

SIDECAR MESH (Istio classic / Linkerd)
  Pod A app --> [sidecar proxy] --mTLS, per-request L7--> [sidecar proxy] --> Pod B app
  Cost: 2 proxy hops, 2 TLS terminations, per pod, every request.

AMBIENT / SIDECARLESS (Istio ambient, Cilium Service Mesh)
  Pod C app --> [ztunnel, per node] --mTLS + L4 authz--> [ztunnel, per node] --> Pod D app
                       \                                    /
                        \--(opt-in only)--> [waypoint proxy] --/
  Cost: 1 node-level hop for everyone; L7 hop only for workloads with a waypoint.

eBPF-NATIVE L3/L4 (Cilium core, no L7 proxy)
  Pod E app --> [eBPF: tc/socket LB] --WireGuard/IPsec + identity policy--> [eBPF: tc/socket LB] --> Pod F app
  Cost: kernel data path only. No proxy process in the path for L3/L4 traffic.

The design choice that actually matters across all three: where does the always-on cost live, and how much of your traffic pays it. Sidecar mesh — every request, every pod, pays for a full proxy whether it needs L7 or not. Ambient and eBPF-native — the cheap path (identity, mTLS, L3/L4 policy) is the default, and the expensive path (L7 parsing) is something you opt specific workloads into.

Service Mesh vs eBPF: Decision Criteria for the Data Plane

Criteria Sidecar Mesh (Istio/Linkerd classic) Ambient / Sidecarless (Istio ambient, Cilium Service Mesh) eBPF-Native L3/L4 (Cilium core, no L7 proxy)
L7 feature richness Full — retries with budgets, timeouts, outlier detection, weighted/canary routing, header and path routing, fault injection, on every meshed pod Full, but opt-in — same feature set, available only where you deploy a waypoint (or Cilium’s per-node Envoy) for a given workload None at the pure L3/L4 layer; Cilium’s embedded L7 support covers method/path/header authorization but not weighted traffic splitting or retry budgets
mTLS / identity model SPIFFE-based workload identity, cert issued and rotated by the mesh CA, TLS terminated per-connection on both sidecars Same SPIFFE identity model, terminated at ztunnel (node-level) instead of per-pod; waypoint adds L7-aware authorization where deployed Node-to-node tunnel encryption (WireGuard/IPsec) by default; policy identity comes from Cilium security identities, not certs — true per-workload mutual auth is a newer, narrower feature, not the multi-year default
Per-request overhead / latency Highest — two proxy hops and two TLS terminations per request, per pod, regardless of features used Lower — one node-level hop for L4; an extra hop only for workloads with a waypoint Lowest — kernel data path, no userspace proxy hop for L3/L4 traffic
Operational maturity Very high — years of production use, large operator community, well-understood failure modes Newer — same control-plane maturity underneath (Istio), smaller deployed base, fewer war stories at scale Cilium as a CNI/L3-L4 enforcement layer is mature and widely run; the L7 mesh story built on top of it is younger
Debuggability Well-trodden — Envoy admin API, access logs, kubectl exec into a specific pod’s sidecar Harder to attribute — one ztunnel serves many pods, so per-flow correlation takes more work; waypoint adds per-workload logs back where deployed Different skill set entirely — Hubble flow logs, cilium monitor, eBPF map dumps instead of proxy logs; no L7 retry/circuit-breaker state to inspect because it doesn’t exist
Kernel requirements None — runs in userspace, portable across kernel versions ztunnel/waypoint run in userspace; the underlying CNI may still want a modern kernel depending on choice Real — wants a modern kernel with BTF for CO-RE, the same constraint covered in depth in the eBPF observability piece linked below
Best fit Teams that need L7 traffic shaping broadly and already operate Envoy comfortably Teams that want mTLS and L4 policy everywhere cheaply, but only need L7 traffic shaping on a handful of services Teams whose actual requirement is identity-aware network policy and encryption, not per-request traffic shaping

The way I’d sort a real decision: if more than a handful of your services need canary weighting, retry budgets, or header-based routing, you’re going to end up running an L7 proxy somewhere no matter which camp you start from — the only question is whether it sits on every pod or on the handful of workloads that actually need it. If your services mostly need to prove who’s calling and enforce who’s allowed to reach what, that’s an L3/L4 problem, and paying for a full L7 proxy on every pod to solve it is overhead you don’t need to carry.

Traffic Policy in Practice: An Istio Canary vs a Cilium Identity Policy

Nothing makes the capability gap concrete like putting the two config models side by side for the same rollout: shifting traffic to a new version of a payments service, with retries, while still restricting who can reach it at all.

Here’s the Istio side — a canary that sends 10% of traffic to v2, routes anyone with an x-beta-user header straight to v2 regardless of the split, and retries idempotent failures twice with a two-second per-try timeout:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payments-svc
spec:
  hosts:
  - payments.prod.svc.cluster.local
  http:
  - match:
    - headers:
        x-beta-user:
          exact: "true"
    route:
    - destination:
        host: payments.prod.svc.cluster.local
        subset: v2
  - route:
    - destination:
        host: payments.prod.svc.cluster.local
        subset: v1
      weight: 90
    - destination:
        host: payments.prod.svc.cluster.local
        subset: v2
      weight: 10
    retries:
      attempts: 2
      perTryTimeout: 2s
      retryOn: 5xx,reset,connect-failure
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payments-svc
spec:
  host: payments.prod.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

That’s roughly fifteen lines expressing three separate per-request decisions, and it works identically whether the proxy executing it is a classic sidecar or an Istio ambient waypoint — the config model doesn’t change, only where the proxy that runs it happens to live.

Now the identity side, expressed as a CiliumNetworkPolicy — only pods labeled app: checkout may reach payments on port 8080, enforced by identity rather than IP, in the kernel:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payments-allow-checkout
spec:
  endpointSelector:
    matchLabels:
      app: payments
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: checkout
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP

This policy is enforced by the kernel data path on every packet, survives pod IP churn because it matches on Cilium’s security identity rather than a source address, and adds essentially no per-packet latency. It also can’t do one thing the VirtualService above does in three different ways: it has no concept of a request. No header to match, no percentage to weight, no retry counter. You can extend it with Cilium’s own toPorts.rules.http block to match an HTTP method or path for authorization purposes — that’s real and useful — but it still won’t give you a 90/10 weighted split or a per-try timeout. That’s not a missing feature waiting on a future release. It’s a different layer of the stack answering a different question. Weighted canary and retry budgets are inherently request-level concerns, and something has to hold per-request state to answer them — a stateless, per-packet kernel policy structurally can’t.

Pitfalls, Trade-offs, and What I’d Actually Ship

Don’t migrate off a working mesh because eBPF is the interesting conference talk this year. I’ve seen teams burn a quarter ripping out Istio because sidecar CPU showed up as a line item, then spend the next two quarters rebuilding canary rollouts and retry policies that used to be free. If your mesh works, your team can operate it, and the overhead isn’t actually blocking anything, that’s not a broken system — it’s a working one with a cost you already understand. Measure the actual sidecar tax on your own fleet before committing to a rewrite, not the number from someone else’s blog post.

Know exactly which L7 features you’re giving up before you go sidecarless. Retry budgets, outlier detection that ejects a bad instance automatically, fault injection for chaos testing, header-based routing for staged rollouts — none of that exists at the pure eBPF L3/L4 layer, and it only exists in ambient mode wherever a waypoint has been deployed. Audit which services actually use these features today (your mesh’s own telemetry will tell you) before assuming you can drop them fleet-wide.

The hybrid is usually the right answer, not a compromise. Run eBPF for L3/L4 identity and encryption everywhere — it’s cheap, it survives IP churn, and every workload benefits regardless of whether it needs L7 anything. Then add an L7 proxy, whether that’s an Istio waypoint or Cilium’s per-node Envoy, only for the services that actually do canaries, retries, or header routing. That’s a handful of services in most fleets, not all of them, and it’s exactly the architecture ambient mode and Cilium Service Mesh are both converging on from opposite directions.

Get precise about what “mTLS” means for the option you pick. A sidecar mesh gives you per-workload SPIFFE identity, verified on every connection, as the default, and has for years. eBPF-native encryption by default is a node-to-node tunnel — real encryption, but it isn’t verifying the calling workload’s identity the way a TLS handshake does. Cilium’s own mutual authentication closes that gap, but it’s a narrower, newer feature than the sidecar mesh’s default behavior. Don’t let “we have encryption” on a slide stand in for “we have the same identity guarantee we had before.”

Budget real time for learning to debug a kernel data path. There’s no Envoy admin endpoint to curl, no access log with a request ID to grep. You’re in cilium monitor, Hubble flow logs, and eBPF map dumps, and tracing a slow request back to a specific policy decision takes a different set of habits than reading a sidecar’s access log. Send at least one engineer through this before the migration, not during the first incident after it.

Treat the migration itself as the risky part, not the destination. The config models don’t translate — a VirtualService doesn’t map onto a CiliumNetworkPolicy, and running both control planes at once means two identity systems and two sources of truth for who’s allowed to talk to whom. Pick one low-traffic, non-critical service as the pilot, run it in parallel with the old path for a full deployment cycle, and confirm policy parity against real traffic before touching anything customer-facing.

Related Reading

Let’s Talk Data Planes

If you’re weighing a sidecar mesh against ambient mode or an eBPF-native data plane and want a second opinion on the architecture, I’m happy to compare notes. Connect with me on LinkedIn or reach out via Contact.