Val Kafedzhy

The AI Inference Networking Maturity Model

Wed, 15 Jul 2026 00:00:00 GMT

A vendor-neutral way to judge how ready your network is to serve AI inference at scale. Five levels, six dimensions. Find where you actually are, then decide the one investment that moves latency, resilience, or cost.

Why this model exists

Most “AI infrastructure” conversations stop at GPUs and model serving. The network underneath, the thing that actually carries inference traffic across regions, into clusters, and between services, gets treated as plumbing right up until it causes an outage or blows a latency SLO.

Inference is unusually unforgiving of lazy networking. The p99 budgets are tight. A request stuck in the wrong place idles a $30k accelerator. Regional capacity is lumpy, and an inference session doesn’t retry cleanly the way a stateless web GET does. I kept watching teams solve the GPU problem beautifully and then lose the latency they’d paid for somewhere between the load balancer and the pod.

So I wrote this down to answer three questions without flinching:

Where are we, honestly?
What breaks next at this level?
What’s the next dollar actually worth spending on?

It’s deliberately vendor-neutral. You can climb it on a hyperscaler’s managed stack, on your own AS with BGP and eBPF, or some hybrid of the two. Each level names a capability, not a product you can buy your way into.

The six dimensions

Score every level across the same six axes:

Topology: how regions and clusters sit relative to where demand is.
Traffic steering: how a request finds the right region or cluster.
Failover: how fast, and how cleanly, traffic leaves a degraded region.
Observability: what you can genuinely see about network and inference health.
Security: how east-west and north-south traffic gets authorized (your Zero Trust posture).
Capacity and cost awareness: whether routing reflects GPU capacity and dollars per request.

The five levels

L1: single region, best effort

One region, one entry point. “Healthy” means the process is up. No capacity or cost signal anywhere.

Failure mode: any regional incident is a full outage, and everyone outside that region eats cross-continent tail latency the rest of the time.
Advance by: standing up a second region and some way to send traffic to it.

L2: multi-region, manual

More than one region, but steering is GeoDNS or a human, failover is TTL-bound (minutes, if you’re lucky), and metrics are a few dashboards. Capacity gets balanced by hand.

Failure mode: failover is far too slow for an inference SLO, clients cache and ignore your TTLs, and capacity hotspots stay invisible until they page someone at 2am.
Advance by: wiring real health into steering so failover happens automatically, in seconds.

L3: health-aware routing

Anycast plus BGP, or a global load balancer, with SLO-based health integration. Routes get withdrawn on real latency or queue-depth breaches, failover lands in seconds, and observability is per-region rather than global-average. (A well-run BGP multi-region design lives here.)

Failure mode: routing knows healthy from unhealthy, but nothing about how much capacity is left or what it costs. It will happily steer a flood into a healthy region that is one request from saturation, or into your most expensive one.
Advance by: feeding capacity and cost signals into steering, and closing the east-west security gaps you have been ignoring.

L4: capacity- and cost-aware, secured

Steering weighs GPU capacity, queue depth, and dollars per request, not just latency and liveness. East-west traffic runs under Zero Trust enforcement, and eBPF-level observability gives you kernel-accurate visibility without a sidecar tax. Withdrawal and rebalancing are automated.

Failure mode: the optimization is still reactive and rule-based. Humans own the thresholds, and the ugly cases (correlated demand spikes across regions at once) still need someone to step in.
Advance by: closing the loop, so the system optimizes against SLO and cost objectives on its own.

L5: autonomous, self-optimizing

A closed control loop continuously tunes traffic, capacity, and placement against latency SLOs and cost per token. Active-active across regions with predictive scaling. When something goes wrong it degrades gracefully, shedding load or downshifting the model tier, instead of falling over. Security is policy-as-code with a full audit trail, and capacity and cost are first-class routing inputs.

Failure mode: the risks are second-order now. Control-loop stability, telemetry and model drift, over-automation. These are the problems of a mature system, not a fragile one.
Advance by: hardening the control loop itself. This is the frontier, and there is no tidy next rung.

Summary matrix

Dimension	L1 Best-Effort	L2 Manual Multi-Region	L3 Health-Aware	L4 Capacity/Cost-Aware + Secure	L5 Autonomous
Topology	Single region	Few regions, static	Regions near demand	Demand-driven placement	Predictive, active-active
Steering	Single entry	GeoDNS / manual	Anycast+BGP / global LB, SLO health	+ capacity and cost weighting	Closed-loop optimization
Failover	None (outage)	Minutes (TTL-bound)	Seconds (health-triggered)	Seconds + auto-rebalance	Predictive / pre-emptive
Observability	Liveness only	Basic dashboards	Per-region SLO metrics	eBPF kernel-level, no blind spots	Telemetry drives control loop
Security	Perimeter/flat	Perimeter + some segmentation	Segmented, TLS	Zero-Trust east-west enforcement	Policy-as-code, full audit
Capacity/Cost	None	Manual	Health only (not capacity)	Capacity + $/req in routing	Continuous SLO/$ optimization

Quick self-assessment

Score each axis 1 to 5 and take the lowest, not the average. An inference network behaves like its weakest dimension, not its best one.

Can we lose a whole region without a user-visible outage? (Failover at L3 or better)
Does traffic leave a degraded region in seconds, on its own? (Steering at L3 or better)
Do we route on real inference SLOs, not just “the port is open”? (Observability at L3 or better)
Does routing know each region’s GPU capacity and cost? (Capacity at L4 or better)
Is east-west inference traffic authorized, rather than implicitly trusted? (Security at L4 or better)

Most teams running AI in production right now sit at L2 to L3. Getting from L2 to L3 is what removes the outages. L3 to L4 is what removes the waste and closes the security gaps. Neither is a research project; both are things you can scope, fund, and ship this quarter.

How to use this

Architects and leads: locate your level per dimension, find the single weakest axis, and fund that one. Do not spread the budget evenly.
Interviews and reviews: a shared vocabulary for “how mature is our inference network,” instead of everyone arguing from a different definition.
Vendors and tools: map what a product claims to the capability it actually delivers, not the tier its marketing puts it in.

Cite as: Kafedzhy, V. “The AI Inference Networking Maturity Model.” vkafed.com.

Background references

The five levels and scoring approach are my own assessment framework. These sources cover the underlying reliability, routing, and security concepts.

Google SRE: Service Level Objectives. Choosing indicators and objectives that reflect the service users receive.
RFC 4271: BGP-4. The routing protocol used by the anycast design.
NIST SP 800-207: Zero Trust Architecture. The access-control principles behind the security dimension.

Networking: the pillar this anchors
BGP for the AI Era: the L3 routing layer in depth
Zero Trust Networking Reference Architecture: the L4 security dimension

BGP for the AI Era: Multi-Region Routing for Inference Workloads

Wed, 17 Jun 2026 00:00:00 GMT

Every network team that has run global web traffic thinks it already understands multi-region routing. Then it puts a GPU inference cluster behind that same design and watches its assumptions fail, one at a time. BGP multi-region routing for inference workloads looks like the CDN playbook at first glance. It isn’t. The failure modes are different, and the cost of getting it wrong is a lot higher, so the design has to account for both.

This is the architecture I reach for when a team wants inference latency predictable across regions without turning the network team into a full-time incident desk. I’m assuming you already know BGP basics here: AS numbers, path attributes, eBGP/iBGP. What follows is about applying that knowledge to a workload that won’t forgive sloppy convergence.

The problem: why inference traffic breaks web routing assumptions

Generic web traffic is forgiving. An extra 200ms because a request landed in the wrong region barely registers, and a stateless HTTP GET can be retried anywhere for free. Inference traffic doesn’t get that slack. Four reasons why:

Latency SLOs are tighter and less forgiving. A p99 target of 300-500ms end-to-end for a real-time inference call leaves almost no room for a routing detour. Cross-continental round trips alone can burn half that budget before a single token comes back. A web app hides 150ms behind a spinner. An inference API can’t hide it at all; it lands directly in the user-perceived response time.

The cost of a stall is a GPU, not a thread. Queue a request behind a slow or overloaded node and you’re not blocking a cheap CPU worker. You’re sitting a $30,000+ accelerator idle while it could be serving other requests. Misroute traffic into a saturated region and you’re wasting the most expensive unit of compute in the stack, latency aside. At that price point, capacity-aware routing is a cost control measure, full stop.

Regional capacity is lumpy, not elastic. You can autoscale a web tier from 10 to 200 pods in minutes. GPU capacity doesn’t work that way. It comes in expensive, supply-constrained chunks, and a region running hot often can’t spin up more on demand. Routing has to steer load toward where capacity actually exists, not just where latency is lowest.

Failover has to be fast and clean, not eventually consistent. DNS-based failover leans on clients respecting TTLs. They routinely don’t. Resolver caching, sticky client pools, and long-lived streaming connections all get in the way. An inference session mid-token-stream doesn’t get a graceful retry the way a stateless page load does. If a region degrades, the network layer needs to redirect new connections in single-digit seconds, not minutes.

The natural response is anycast plus BGP: announce the same service IP from multiple regions and let the internet’s routing fabric pick the closest, healthiest path. Right instinct. It only works, though, if you engineer the failure paths as carefully as the happy path, and that’s where most first attempts fall apart.

Reference architecture: BGP multi-region routing for inference clusters

Here’s the shape that’s worked for me across several inference platforms: a shared anycast VIP announced from edge routers in each region, backed by per-region GPU clusters, with a health agent that ties BGP advertisement to real inference SLO metrics rather than process liveness alone.

The design choice that matters here: health state feeds directly into route advertisement. A region doesn’t go down because a ping failed. It goes down when the health agent decides the actual inference SLO is breached, and that decision is what withdraws the route.

Decision criteria: anycast + BGP vs GeoDNS vs global load balancer

Before you commit to BGP, be honest about when it’s overkill.

Criteria	Anycast + BGP	GeoDNS	Managed Global L7 Load Balancer
Steady-state latency	Best, routed at the network layer, no resolver hop	Good, depends on resolver geo-accuracy	Good, comparable if the provider has dense PoP coverage
Failover speed	Seconds, once health integration and BFD are tuned	Minutes, bounded by TTL and resolver caching behavior	Seconds to tens of seconds, provider-dependent
Blast radius of a bad regional deploy	Contained, if health checks and withdrawal logic are correct; a misconfigured route-map can leak globally	Contained, bounded by TTL	Contained, provider manages isolation
Operational complexity	High: own AS, peering relationships, route-map/community discipline, BFD tuning	Low to medium	Low: you trade control for simplicity
Cost	Predictable transit/peering + engineering time	Low, mostly DNS query cost	Pay-per-request/data processed; can get expensive at inference-scale egress volumes
Client behavior dependency	None, works regardless of resolver or client caching	High, depends on clients respecting TTLs	Low, provider anycast, but still DNS-fronted in most implementations
Best fit	High-QPS, strict p99 SLOs, teams with real network engineering capacity	Simple regional failover, small teams, no in-house BGP expertise	Teams that want managed simplicity and can tolerate the cost/control trade-off

My rule of thumb: if you don’t already run BGP for something else in production, don’t stand up your first AS just for an inference product. Start with a managed global load balancer or GeoDNS, and graduate to anycast + BGP once query volume and SLO tightness justify the operational overhead. If you already own an AS and edge peering, though, folding inference traffic into that fabric is usually the right call. The marginal cost is low and the failover speed is hard to beat elsewhere.

BGP configuration: communities, health-triggered withdrawal, and BFD

Here’s a representative FRR config for an edge router in us-east-1. It announces the shared anycast VIP conditionally on local health, with BFD for fast peer failure detection and communities for regional signaling to upstream/transit.

! FRR bgpd.conf - us-east-1 edge router router bgp 65001 bgp router-id 10.10.1.1 no bgp ebgp-requires-policy neighbor 203.0.113.1 remote-as 64500 neighbor 203.0.113.1 description TRANSIT-1 neighbor 203.0.113.1 bfd neighbor 203.0.113.1 bfd profile fast neighbor 203.0.113.5 remote-as 64500 neighbor 203.0.113.5 description TRANSIT-2 neighbor 203.0.113.5 bfd neighbor 203.0.113.5 bfd profile fast

address-family ipv4 unicast redistribute connected route-map ANYCAST-HEALTH neighbor 203.0.113.1 route-map SET-REGION-COMMUNITY out neighbor 203.0.113.5 route-map SET-REGION-COMMUNITY out exit-address-family ! bfd profile fast detect-multiplier 3 transmit-interval 150 receive-interval 150 exit ! ! redistribute connected advertises the VIP only while its loopback ! (connected) address exists. The health agent removes that loopback ! address on SLO breach, so bgpd withdraws the prefix from every eBGP peer. route-map ANYCAST-HEALTH permit 10 match ip address prefix-list ANYCAST-VIP ! route-map SET-REGION-COMMUNITY permit 10 set community 65001:100 65001:910 additive ! ip prefix-list ANYCAST-VIP seq 5 permit 198.51.100.10/32

The 65001:910 community is an internal signal. In this scheme, 9xx communities mark region priority tiers that a route-reflector or upstream policy can act on, say de-preferring a region running at reduced GPU capacity without withdrawing it entirely. BFD at 150ms/×3 gives sub-second peer failure detection. That matters more than people expect: default BGP hold timers (180s) are far too slow for an inference SLO measured in hundreds of milliseconds.

The health agent itself is a small, boring script, deliberately so. This is not where you want cleverness:

# health-agent.sh: runs every 2s via systemd timer on each edge router.
# Ties the BGP-advertised anycast VIP to actual inference SLO health,
# not just "is bgpd running."

P99_MS=$(curl -s http://localhost:9100/metrics/p99_inference_latency_ms)
QUEUE_DEPTH=$(curl -s http://localhost:9100/metrics/gpu_queue_depth)

if (( $(echo "$P99_MS > 250" | bc -l) )) || (( QUEUE_DEPTH > 500 )); then
  ip addr del 198.51.100.10/32 dev lo 2>/dev/null
  logger -t health-agent "WITHDRAWN region=us-east-1 p99=${P99_MS}ms queue=${QUEUE_DEPTH}"
else
  ip addr add 198.51.100.10/32 dev lo 2>/dev/null
  logger -t health-agent "HEALTHY region=us-east-1 p99=${P99_MS}ms queue=${QUEUE_DEPTH}"
fi

This pattern (redistribute a connected loopback address into BGP, then let a health agent add or remove that address) is deliberately simple. It keeps the failure domain small. If the health agent crashes, the route just stays in whatever state it was last in, instead of flapping unpredictably.

Pitfalls and what I’d do in production

BGP dampening will punish a flapping health check. If your health agent oscillates, withdrawing and re-announcing every time a metric crosses a threshold, upstream route flap dampening (RFC 2439-style) can suppress your prefix for 30-60 minutes, long after the region actually recovered. What I’d do: require N consecutive bad probes before withdrawal and a longer streak of good probes before re-announcing (I typically use 3 bad / 5 good over a rolling window), and confirm with your transit providers whether they still apply dampening to customer-originated prefixes. Many disabled it after RFC 7196, but not all.

Global convergence is not instant, even when your local withdrawal is. Your edge router can pull the route in milliseconds, but propagation across the internet’s BGP mesh can take tens of seconds in the worst case, especially during path hunting. If your client-facing timeout is 2-5 seconds, some in-flight and newly-initiated requests will still land on the degraded region during that window. What I’d do: treat BGP withdrawal as capacity steering, not a request-level correctness guarantee. Pair it with LB-level connection draining and a client SDK that retries against a secondary regional endpoint on failure. Don’t make BGP the only line of defense.

MED and AS-path prepending are weaker levers than they look. MED is only compared between routes from the same neighboring AS and is non-transitive, and plenty of transit providers ignore or reset it entirely, so don’t rely on it for cross-provider steering. AS-path prepending is coarse, and because you can’t guarantee how every downstream AS weighs path length against local policy, “symmetric” traffic engineering built on prepending often breaks asymmetrically during partial outages. One direction reroutes cleanly, the other doesn’t, and you get latency degradation on returning traffic that’s hard to diagnose. What I’d do: use LOCAL_PREF for steering when you control both ends (iBGP between owned regions or DCs) where the result is deterministic, and treat inter-AS prepending/communities as advisory hints, not guarantees.

Health checks need to understand inference, not just liveness. A GPU node can be “up” (process healthy, port open) while still failing its actual job: model not loaded, request queue backed up from a traffic burst, thermal throttling reducing throughput. What I’d do: feed the health agent real SLO telemetry (p99 latency, queue depth, GPU utilization and temperature, model load state), not a synthetic ping.

Withdrawal order matters, or you strand live sessions. Pulling the BGP route before draining the load balancer cuts in-flight streaming inference sessions mid-token. What I’d do: drain new connections at the LB first, let existing sessions finish or hit their natural timeout, then withdraw the route, and verify the withdrawal actually propagated (check a couple of external looking-glass vantage points) before marking the region fully out of rotation. Also audit graceful-restart / GR-helper configuration on your edge routers. It exists to survive planned bgpd restarts without dropping sessions, but misconfigured GR timers can leave a withdrawn region’s routes looking valid to peers longer than intended.

References

The protocol specifications and FRRouting manual describe the mechanisms behind this design. Validate timers, route policy, and health-agent behavior against your deployed release.

RFC 4271: BGP-4. Reachability advertisements, path attributes, and route withdrawal.
RFC 5880: Bidirectional Forwarding Detection. Detecting forwarding-path failures independently of application health.
FRRouting BGP manual. Neighbor configuration, route maps, redistribution, and BGP policy controls.

Networking: the full pillar on routing architecture for modern infrastructure
Networking: networking design specifically for AI/ML workloads
Multi-Region Connectivity for Distributed GPU Clusters: coming soon, private backbone and inter-region transport design for training and inference
IPv6 at Enterprise Scale: A Migration Playbook: dual-stack, a real address plan, and NAT64/DNS64 for IPv6-only GPU fabrics

Let’s talk routing

If you’re designing multi-region routing for an inference platform and want a second opinion on the architecture, I’m happy to compare notes. Connect with me on LinkedIn or reach out via Contact.

Service Mesh vs eBPF-Native Data Planes: How to Choose

Wed, 13 May 2026 00:00:00 GMT

Every platform team that runs Kubernetes past a few dozen services eventually has the service mesh vs eBPF argument, usually in a design review where one engineer points at fleet-wide CPU graphs and says the Envoy sidecars are costing too much, and another engineer points at a canary rollout ticket and says none of that matters if you can’t shift 10% of traffic to a new version without touching application code. Both of them are right. That’s the actual problem. The choice isn’t “sidecars bad” versus “eBPF hype.” It’s a real trade-off between how much a data plane costs to run and how much it can do with a single request once it’s in flight.

I wrote about eBPF for kernel-level observability and security in an earlier piece: Tetragon watching process exec, Cilium enforcing identity-aware policy in the kernel. This one is narrower and more contested: what should actually carry your service-to-service traffic, terminate mTLS, and make routing decisions on a live request. That’s a different question from “should I run a kernel agent instead of a log-tailing daemon,” and it deserves its own answer.

The problem: what a traffic data plane actually has to do

Strip away the marketing and a service mesh data plane answers four questions on every request: who is this caller, are they allowed to reach this destination, where should this specific request go, and what happens if the destination is slow or down. The first two are identity and policy. The last two are traffic management, and that’s where the real disagreement lives.

Identity-aware network policy and transport protection are available in both approaches, but the guarantees differ. Sidecar meshes use proxies for workload mutual TLS. Cilium can enforce L3/L4 policy in the kernel and offers separately configured encryption mechanisms. Check workload authentication, encryption coverage, and policy enforcement as separate requirements for the release you run.

The argument starts at layer 7. A canary rollout that shifts 10% of traffic to a new version based on a request header. A retry policy that gives a flaky downstream two attempts with a two-second per-try timeout, but only for idempotent methods. A circuit breaker that stops sending traffic to an instance once it starts erroring, without a human paging in to do it. These are per-request, protocol-aware decisions. Something has to parse HTTP framing, read headers, count errors per upstream, and hold that state. A raw packet or a TCP segment doesn’t carry any of it. That something has to be a proxy in the request path that understands the protocol. Without one, those decisions don’t get made.

That’s the trade-off in one sentence: every L7 feature you want costs you a proxy hop somewhere, and every proxy hop you remove costs you an L7 feature. The trade-off itself hasn’t changed. What has changed is how much middle ground now exists between “proxy on every pod” and “no proxy anywhere.”

Reference architecture: sidecar mesh vs ambient vs eBPF-native L3/L4

Three shapes run in production today, not the two that the usual framing assumes. It helps to draw them side by side rather than pick a side first.

The classic sidecar model (Istio with Envoy, Linkerd with its own Rust micro-proxy) injects a proxy into every pod’s network namespace. Every request leaves the app container, hits the local sidecar, crosses the network to the destination’s sidecar, and only then reaches the destination app container. Two extra hops and two TLS terminations, per pod, for every request, whether or not that request needed any of the L7 features the proxy provides.

The ambient model (Istio’s ztunnel plus optional waypoint proxies, structurally similar to what Cilium’s own service mesh does) moves the always-on part down to one proxy per node instead of one per pod. That per-node proxy (ztunnel) handles mTLS and L4 authorization for every pod on the node. It does not parse HTTP. When a namespace or service account actually needs L7 behavior (retries, header routing, canary weighting), you deploy a waypoint proxy for just that workload, and only that traffic takes the extra hop through it. Everything else gets identity and encryption without an L7 proxy anywhere near it.

The eBPF-native model, in its purest form, skips the proxy entirely for L3/L4. Cilium’s agent programs the kernel data path (tc/eBPF and socket-level load balancing) to enforce identity-based policy and, optionally, transparent WireGuard or IPsec encryption between nodes. No per-pod or per-node proxy process handles this traffic; the kernel does. L7 policy, where Cilium supports it, runs through a shared per-node Envoy instance invoked only for the specific traffic that needs it. That’s architecturally close to the waypoint idea, just approached from the CNI side of the fence instead of the mesh side.

The design choice that matters across all three: where does the always-on cost live, and how much of your traffic pays it. Sidecar, ambient, and eBPF-based deployments place that work in different parts of the traffic path. Istio ambient separates its L4 tunnel from optional L7 waypoints. For Cilium, check the configured policy, encryption, and proxy paths together before comparing their cost with a mesh.

Service mesh vs eBPF: decision criteria for the data plane

Criteria	Sidecar Mesh (Istio/Linkerd classic)	Ambient / Sidecarless (Istio ambient, Cilium Service Mesh)	eBPF-Native L3/L4 (Cilium core, no L7 proxy)
L7 feature richness	Full: retries with budgets, timeouts, outlier detection, weighted/canary routing, header and path routing, fault injection, on every meshed pod	Full, but opt-in: same feature set, available only where you deploy a waypoint (or Cilium’s per-node Envoy) for a given workload	None at the pure L3/L4 layer; Cilium’s embedded L7 support covers method/path/header authorization but not weighted traffic splitting or retry budgets
mTLS / identity model	SPIFFE-based workload identity, cert issued and rotated by the mesh CA, TLS terminated per-connection on both sidecars	Same SPIFFE identity model, terminated at ztunnel (node-level) instead of per-pod; waypoint adds L7-aware authorization where deployed	WireGuard/IPsec encryption when configured; policy uses Cilium security identities. Workload authentication and encryption coverage depend on the enabled mechanism and release
Per-request overhead / latency	Highest: two proxy hops and two TLS terminations per request, per pod, regardless of features used	Lower: one node-level hop for L4; an extra hop only for workloads with a waypoint	Lowest: kernel data path, no userspace proxy hop for L3/L4 traffic
Operational maturity	Very high: years of production use, large operator community, well-understood failure modes	Newer: same control-plane maturity underneath (Istio), smaller deployed base, fewer war stories at scale	Cilium as a CNI/L3-L4 enforcement layer is mature and widely run; the L7 mesh story built on top of it is younger
Debuggability	Well-trodden: Envoy admin API, access logs, `kubectl exec` into a specific pod’s sidecar	Harder to attribute: one ztunnel serves many pods, so per-flow correlation takes more work; waypoint adds per-workload logs back where deployed	Different skill set entirely: Hubble flow logs, `cilium monitor`, eBPF map dumps instead of proxy logs; no L7 retry/circuit-breaker state to inspect because it doesn’t exist
Kernel requirements	None: runs in userspace, portable across kernel versions	ztunnel/waypoint run in userspace; the underlying CNI may still want a modern kernel depending on choice	Real: wants a modern kernel with BTF for CO-RE, the same constraint covered in depth in the eBPF observability piece linked below
Best fit	Teams that need L7 traffic shaping broadly and already operate Envoy comfortably	Teams that want mTLS and L4 policy everywhere cheaply, but only need L7 traffic shaping on a handful of services	Teams whose actual requirement is identity-aware network policy and encryption, not per-request traffic shaping

The way I’d sort a real decision: if more than a handful of your services need canary weighting, retry budgets, or header-based routing, you’re going to end up running an L7 proxy somewhere no matter which camp you start from. The only question is whether it sits on every pod or on the handful of workloads that actually need it. If your services mostly need to prove who’s calling and enforce who’s allowed to reach what, that’s an L3/L4 problem, and paying for a full L7 proxy on every pod to solve it is overhead you don’t need to carry.

Traffic policy in practice: an Istio canary vs a Cilium identity policy

Nothing makes the capability gap concrete like putting the two config models side by side for the same rollout: shifting traffic to a new version of a payments service, with retries, while still restricting who can reach it at all.

Here’s the Istio side, a canary that sends 10% of traffic to v2, routes anyone with an x-beta-user header straight to v2 regardless of the split, and retries idempotent failures twice with a two-second per-try timeout:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payments-svc
spec:
  hosts:
  - payments.prod.svc.cluster.local
  http:
  - match:
    - headers:
        x-beta-user:
          exact: "true"
    route:
    - destination:
        host: payments.prod.svc.cluster.local
        subset: v2
  - route:
    - destination:
        host: payments.prod.svc.cluster.local
        subset: v1
      weight: 90
    - destination:
        host: payments.prod.svc.cluster.local
        subset: v2
      weight: 10
    retries:
      attempts: 2
      perTryTimeout: 2s
      retryOn: 5xx,reset,connect-failure
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payments-svc
spec:
  host: payments.prod.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

That’s roughly fifteen lines expressing three separate per-request decisions, and it works identically whether the proxy executing it is a classic sidecar or an Istio ambient waypoint. The config model doesn’t change, only where the proxy that runs it happens to live.

Now the identity side, expressed as a CiliumNetworkPolicy, where only pods labeled app: checkout may reach payments on port 8080, enforced by identity rather than IP, in the kernel:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payments-allow-checkout
spec:
  endpointSelector:
    matchLabels:
      app: payments
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: checkout
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP

This policy is enforced by the kernel data path on every packet, survives pod IP churn because it matches on Cilium’s security identity rather than a source address, and adds essentially no per-packet latency. It also can’t do one thing the VirtualService above does in three different ways: it has no concept of a request, so there is no header to match, no percentage to weight, and no retry counter. You can extend it with Cilium’s own toPorts.rules.http block to match an HTTP method or path for authorization purposes (that’s real and useful), but it still won’t give you a 90/10 weighted split or a per-try timeout. That’s not a missing feature waiting on a future release. It’s a different layer of the stack answering a different question. Weighted canary and retry budgets are inherently request-level concerns, and something has to hold per-request state to answer them. A stateless, per-packet kernel policy structurally can’t.

Pitfalls, trade-offs, and what I’d actually ship

Don’t migrate off a working mesh because eBPF is the interesting conference talk this year. I’ve seen teams burn a quarter ripping out Istio because sidecar CPU showed up as a line item, then spend the next two quarters rebuilding canary rollouts and retry policies that used to be free. If your mesh works, your team can operate it, and the overhead isn’t blocking anything, that’s not a broken system. It’s a working one with a cost you already understand. Measure the sidecar tax on your own fleet before committing to a rewrite, not the number from someone else’s blog post.

Know exactly which L7 features you’re giving up before you go sidecarless. Retry budgets, outlier detection that ejects a bad instance automatically, fault injection for chaos testing, header-based routing for staged rollouts: none of that exists at the pure eBPF L3/L4 layer, and it only exists in ambient mode wherever a waypoint has been deployed. Audit which services actually use these features today (your mesh’s own telemetry will tell you) before assuming you can drop them fleet-wide.

The hybrid is usually the right answer, not a compromise. Run eBPF for L3/L4 identity and encryption everywhere: it’s cheap, it survives IP churn, and every workload benefits regardless of whether it needs L7 anything. Then add an L7 proxy, whether that’s an Istio waypoint or Cilium’s per-node Envoy, only for the services that actually do canaries, retries, or header routing. That’s a handful of services in most fleets, not all of them, and it’s exactly the architecture ambient mode and Cilium Service Mesh are both converging on from opposite directions.

Get precise about what “mTLS” means for the option you pick. Check where peer identities are verified, where traffic is encrypted, and which connections are covered. Cilium's WireGuard/IPsec modes require configuration, and a node tunnel provides different guarantees from a workload TLS handshake. Check the authentication mechanism supported by your release, then test the coverage you need. An encryption setting alone does not establish policy parity with an existing mesh.

Budget real time for learning to debug a kernel data path. There’s no Envoy admin endpoint to curl, no access log with a request ID to grep. You’re in cilium monitor, Hubble flow logs, and eBPF map dumps, and tracing a slow request back to a specific policy decision takes a different set of habits than reading a sidecar’s access log. Send at least one engineer through this before the migration, not during the first incident after it.

Treat the migration itself as the risky part, not the destination. The config models don’t translate: a VirtualService doesn’t map onto a CiliumNetworkPolicy, and running both control planes at once means two identity systems and two sources of truth for who’s allowed to talk to whom. Pick one low-traffic, non-critical service as the pilot, run it in parallel with the old path for a full deployment cycle, and confirm policy parity against real traffic before touching anything customer-facing.

References

Use these project documents to check which traffic paths and policy features your deployed releases support.

Istio ambient data plane. The roles of ztunnel and waypoint proxies.
Istio VirtualService reference. Request matching, weighted routes, retries, and timeouts.
Cilium Layer 7 policies. HTTP authorization and application-aware policy handling.
Cilium transparent encryption. Encryption mechanisms, configuration, and coverage limitations.

Networking: the full pillar on service meshes, eBPF-native data planes, and cloud networking trade-offs
Zero Trust: identity-aware segmentation and workload authentication patterns
eBPF in Production: Kernel-Level Observability and Security: the companion piece on eBPF for runtime security and observability, not the traffic data plane
Gateway API Migration: Retiring Ingress Without Breaking Traffic Management (coming soon): moving L7 routing off legacy Ingress controllers and onto Gateway API, sidecar or not

Let’s talk data planes

If you’re weighing a sidecar mesh against ambient mode or an eBPF-native data plane and want a second opinion on the architecture, I’m happy to compare notes. Connect with me on LinkedIn or reach out via Contact.

Zero Trust Networking Beyond the Buzzword: An Enterprise Reference Architecture

Wed, 08 Apr 2026 00:00:00 GMT

Every vendor booth at RSA sells “Zero Trust” like it’s a SKU. It isn’t. A zero trust network architecture is a set of design principles: identity-aware access, least privilege by default, policy enforced in the data path, and an assume-breach posture. It is not a box you rack, a license you renew, or an agent you install once and forget. The vendors aren’t wrong that the components matter; they’re wrong about the sequencing. Architecture first, product selection second.

This is the reference architecture I use when an organization wants to move off a flat, VPN-fronted network toward something that survives a compromised laptop or a leaked service-account credential without turning into a two-year segmentation rewrite. It assumes you already know the vocabulary (mTLS, RBAC, network segmentation). This is about how the pieces actually fit together in production, not a vendor’s marketing diagram.

The problem: why the perimeter model fails

The perimeter model rests on an assumption that stopped being true roughly a decade ago: that everything inside the network boundary is trustworthy and everything outside is not. A VPN concentrator, a firewall ring, and an implicit trust grant to anything that gets past both. It worked reasonably well when “inside the network” meant an office building and a data center you controlled end to end.

It breaks down for four concrete reasons:

Lateral movement is free once you’re in. A phished credential or an unpatched VPN appliance (a recurring theme in ransomware incident reports for years now) gives an attacker the same implicit trust as an employee at their desk. Once inside, most enterprise networks have almost no internal friction. A compromised host reaches a database three hops away because nothing in the data path checks whether it should.

Identity got disconnected from network location. Employees, contractors, and service-to-service SaaS integrations connect from home networks, coffee shops, and third-party clouds that never touch a corporate LAN. “Is this traffic coming from inside the network” stopped being a meaningful security signal long before most enterprises admitted it.

Workloads move faster than firewall rules. Autoscaling groups, ephemeral containers, and multi-cloud deployments mean an IP address is not a stable identity. It can be reused by an entirely different service ten minutes later. IP-based ACLs and subnet segmentation are chasing a target that no longer holds still.

Assume-breach is now a compliance expectation. Frameworks enterprises are already accountable to (NIST SP 800-207, CISA’s Zero Trust Maturity Model, most cyber insurance questionnaires) now explicitly ask how you contain a compromised identity or workload, not just how you keep attackers off the network.

What actually has to change, architecturally:

Access decisions are identity-aware. Every request (user-to-app or service-to-service) is authorized on verified identity and context, not source IP or network segment.
Least privilege is the default. Access is scoped to the specific resource and action needed, time-bounded, and re-evaluated continuously rather than granted once at login and trusted forever.
Policy enforcement lives in the data path. Enforcement points sit between services, not only between the internet and your network.
Design for blast-radius containment. The design question shifts from “how do we keep attackers out” to “when something is compromised, how little can it actually reach.”

None of this requires ripping out your identity provider or rebuilding the network from scratch. It requires being deliberate about where policy decisions get made, where they get enforced, and what identity means for a workload versus a human. Most half-finished zero trust projects never separate those three things cleanly.

Reference architecture: a zero trust network architecture for the enterprise

Here’s the layout I keep coming back to in enterprise environments: a single policy decision point consulted by multiple, purpose-placed enforcement points, backed by workload identity that’s independent of IP address, with microsegmentation as the default network posture rather than an add-on.

The distinction most half-finished rollouts get muddled: policy decision and policy enforcement are different responsibilities. Conflating them is why so many zero trust projects turn into an unmaintainable pile of duplicated, drifting rules across a dozen products.

Identity Provider (IdP): Okta, Azure AD/Entra, Ping, etc. Authenticates humans and, increasingly, is the source of truth for device posture via EDR/MDM integration.
Policy Decision Point (PDP): the brain. Given an identity, a posture signal, a requested resource, and context (time, risk score, request type), it returns allow or deny. Implemented as an OPA-based service, a cloud provider’s policy engine, or a ZTNA vendor’s control plane.
Policy Enforcement Points (PEPs) are the muscle, placed everywhere traffic actually flows: an identity-aware proxy at the edge for user-to-app traffic, and eBPF- or sidecar-based enforcement between services for east-west traffic.
Workload identity: SPIFFE/SPIRE (or a mesh’s built-in equivalent) issues short-lived cryptographic identities (SVIDs) to workloads, independent of IP address. East-west PEPs check that identity, not source subnet.
Microsegmentation: workloads are grouped by identity and label into segments where communication is deny-by-default and explicitly allow-listed, so a compromised pod in one segment can’t reach a database in another without a policy that says it can.

Decision criteria: perimeter/VPN vs managed ZTNA/SASE vs self-hosted microsegmentation

Before picking a product, be honest about which model actually fits your team’s operational reality.

Criteria	Perimeter/VPN	Managed ZTNA/SASE	Self-Hosted Microsegmentation (IAP + mesh/eBPF)
Security model	Implicit trust once connected; network location = trust	Identity-aware per-app access, brokered by vendor control plane	Identity-aware, policy enforced in-house at edge and east-west
Blast radius	Large: flat network behind the VPN, broad lateral movement	Contained per-app, but bounded by how granular the vendor’s policy model actually gets	Smallest, if microsegmentation and workload identity are implemented correctly
Latency	Low once connected, but concentrator hairpin adds overhead	Depends on vendor PoP density; extra hop through vendor cloud	Lowest for east-west (in-cluster/eBPF); edge adds one proxy hop
Operational complexity	Low to maintain, but rules rot into an unauditable mess over years	Low: vendor manages PDP, PEP, and posture integration	High: you own PDP uptime, PEP rollout, identity issuance, and policy lifecycle
Cost	Low license cost, high hidden cost of breaches and audit findings	Per-user/per-app subscription; scales with headcount and app count	Engineering time is the real cost; infrastructure itself is often open source
Vendor lock-in	Low, but the security debt is its own lock-in	High: policy logic, posture signals, and often identity live in the vendor’s control plane	Low: SPIFFE, OPA, and Cilium/Envoy are portable, standards-based building blocks
Best fit	Legacy environments not yet ready to change, as a bridge, not a destination	Teams that want managed simplicity and don’t have platform security headcount to spare	Teams with platform engineering capacity who need fine-grained east-west control and want to avoid control-plane lock-in

The heuristic I use: if you don’t have a platform team that already owns Kubernetes networking or service mesh operations, don’t self-host microsegmentation as your first move. You’ll spend a year building PDP/PEP plumbing instead of shipping policy. Start with managed ZTNA/SASE for user-to-app access, and reserve self-hosted microsegmentation for east-west traffic between your own services, where you already have the operational muscle and the vendor’s edge-only model doesn’t reach anyway. The two are not mutually exclusive. Most mature environments I’ve built end up running both, with a shared PDP contract between them.

Policy in practice: identity-aware microsegmentation with Cilium

Here’s a representative CiliumNetworkPolicy enforcing microsegmentation by workload identity rather than IP or subnet. The orders service can only be reached by the payments service, on a specific port, at L7, and only when Cilium has verified the caller’s identity via its own security-identity mechanism (derived from Kubernetes labels/ServiceAccount, not source address).

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: orders-allow-from-payments
  namespace: orders
spec:
  endpointSelector:
    matchLabels:
      app: orders-service
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: payments-service
            io.kubernetes.pod.namespace: payments
      toPorts:
        - ports:
            - port: "8443"
              protocol: TCP
          rules:
            http:
              - method: "POST"
                path: "/v1/orders/reserve"
  # Deny everything else implicitly: Cilium network policies are
  # default-deny once any policy selects an endpoint.

Cilium derives a security identity from workload labels and uses it for network policy. Its L3/L4 checks run in the eBPF datapath. The HTTP method and path in this example require Cilium's Layer 7 proxy handling. Port 8443 alone does not configure TLS or make encrypted HTTP visible to that proxy; configure a supported TLS visibility path when inspecting HTTPS. Workload authentication and transport encryption need their own configuration. For example, services can use SPIFFE/SPIRE-issued X.509 SVIDs to establish mutual TLS. Check the authentication and encryption guarantees of your deployed Cilium release separately from this authorization policy.

For the PDP layer itself (the decision that gates edge, user-facing access), a short Rego rule shows the shape of the logic an OPA-based PDP evaluates on every request:

package authz

default allow = false

# Allow if the user's role can access the resource, their device posture
# is compliant, and the request isn't a stale/expired session.
allow {
    input.user.role == "finance-analyst"
    input.resource.name == "billing-api"
    input.device.posture == "compliant"
    time.now_ns() < input.session.expires_at_ns
}

Nothing exotic. That’s the point. The PDP should be boring, testable, and versioned like application code, because it is the single place every access decision in the system ultimately traces back to.

Pitfalls and what I’d do in production

Don’t try to boil the ocean. A big-bang “segment everything, enforce mTLS everywhere, migrate the whole IdP” rollout is how zero trust programs die in year two. What I’d do: phase it. Start with edge, user-to-app access behind an identity-aware proxy for your highest-risk applications, then extend to east-west for your most sensitive service-to-service paths (payments, PII stores), then broaden. Six months of narrow, working zero trust beats eighteen months of broad, half-finished zero trust.

Your identity provider is now tier-0 infrastructure. When every access decision (human and increasingly workload) traces back to the IdP and PDP, an IdP outage doesn’t just block logins, it can block your entire access-control fabric. What I’d do: treat IdP and PDP availability like you’d treat DNS or your BGP edge: redundant, monitored with real SLOs, and with a documented behavior for what happens to in-flight requests during an outage (fail closed for new requests is usually correct; fail open is rarely acceptable once you’ve made this claim to auditors).

Policy sprawl is the silent killer. Policies accumulate across the IAP, the PDP, the mesh, and half-remembered legacy firewall rules nobody dares delete. Six months in, nobody can answer “who can reach the billing database” with confidence. What I’d do: policy-as-code from day one, in version control, with CI checks that diff policy changes and a quarterly access review that actually revokes what it finds unused, not just a report that gets filed away.

Break-glass access needs to exist and be audited harder than normal access. Zero trust done rigidly enough will lock out the on-call engineer during an actual incident. What I’d do: build an explicit, time-boxed break-glass path: separate credentials, mandatory second-approver or post-hoc review, and an alert that fires the moment it’s used. The absence of a break-glass path isn’t more secure, it just means someone will improvise one under pressure, unaudited.

Log every PDP decision, both allow and deny. Denies get logged reflexively; allows often don’t, and allows are exactly what you need during an incident to reconstruct what a compromised identity actually touched. What I’d do: every PDP decision, both directions, streamed to your SIEM with enough context (identity, resource, posture, policy version) to reconstruct a full access timeline without guessing.

Enforcement adds latency. Measure it, don’t assume it away. A PDP round-trip or proxy processing adds work to the request path. What I’d do: cache PDP decisions at the PEP for short, bounded windows where staleness risk is acceptable, and benchmark the actual enforcement path. Cilium's L3/L4 path and the HTTP proxy path used by the example above have different costs. Include TLS handling and application traffic in that measurement.

Legacy apps that can’t do mTLS or modern auth are your long pole, not an excuse to skip them. What I’d do: front them with a gateway or proxy that terminates modern auth and mTLS on their behalf, place them in their own tightly-scoped, elevated-monitoring segment, and treat that segment as a known, bounded exception, not a silent gap in the architecture diagram.

References

NIST SP 800-207: Zero Trust Architecture. Resource-focused access decisions and policy decision and enforcement components.
SPIFFE overview. Workload identities, SVIDs, and authentication across environments.
Cilium Layer 7 policies. HTTP policy semantics and proxy requirements.
Cilium TLS visibility. Inspecting encrypted traffic for application-layer policy.

Zero Trust: the full pillar on identity-aware security architecture
Networking: networking design for modern, multi-cloud platforms
SPIFFE/SPIRE for Multi-Cluster Workload Identity: coming soon, issuing and rotating workload identity across Kubernetes clusters
Service Mesh vs eBPF-Native Data Planes: How to Choose: the head-to-head on sidecar vs sidecar-less policy enforcement

Let’s talk zero trust

If you’re designing a zero trust rollout and want a second opinion on where to draw the PDP/PEP boundaries, or how much to self-host versus buy, I’m happy to compare notes. Connect with me on LinkedIn or reach out via Contact.

eBPF in Production: Kernel-Level Observability and Security

Wed, 25 Feb 2026 00:00:00 GMT

Every platform team that has bolted a service mesh sidecar onto every pod eventually asks the same question: why run a full proxy per pod just to see and control traffic the kernel already handles? That question is what pushed a lot of us toward eBPF observability and security. Rather than stack another layer on top, you move visibility and enforcement into the kernel, at the same hook points the traffic and system calls already pass through.

This is the architecture I reach for when a team’s sidecar mesh is eating double-digit percentages of fleet-wide CPU and memory, or when a security team needs process-level enforcement that a network-only agent can’t provide. It assumes you already know basic Linux networking and container runtime concepts. This is about applying eBPF to production observability and security, not an introduction to eBPF itself.

The problem: why sidecars and host agents miss what production needs

A sidecar-based service mesh solves a real problem (consistent mTLS, retries, and L7 routing without touching application code), but it does so by inserting a full userspace proxy into the request path of every pod. That has three costs that compound at scale.

Overhead is per-pod, not per-node. A typical Envoy sidecar adds 50-100MB of memory and a meaningful CPU tax per pod, and every request now makes two extra hops (client-side proxy, server-side proxy) instead of zero. On a fleet running thousands of pods, that’s not a rounding error. It’s a standing tax you pay whether or not the mesh’s advanced routing features are in use that hour. Latency-sensitive services feel it directly: 1-2ms per proxy hop doesn’t sound like much until it’s stacked twice on every call in a deep service graph.

Sidecars only see what passes through them. A proxy can only inspect traffic that’s actually routed through it. Traffic that bypasses the proxy (raw sockets, UDP flows some meshes don’t intercept, traffic between processes on the same host outside the mesh’s iptables/nftables redirect rules) is invisible. And a sidecar has zero visibility into what’s happening inside the container’s process tree: which binary executed, which file got opened, whether a shell spawned unexpectedly inside a database pod. That’s a security blind spot, worse than a mere observability gap.

Traditional host agents trade depth for compatibility, and usually lose both. Kernel modules give deep visibility but require per-kernel-version builds and carry real crash risk if buggy: a bad kernel module panics the box. Userspace agents that poll /proc, tail logs, or sample syscalls trade accuracy for safety: you get eventual, incomplete signal instead of an accurate event stream, and high-frequency polling itself adds load.

eBPF runs small programs at kernel hook points such as kprobes, tracepoints, XDP, and LSM hooks. The verifier checks program state and memory access before loading and rejects programs that violate its rules. Those checks reduce risk; they do not replace kernel patching or testing. CO-RE helps compiled programs adapt to kernel data layouts. Kernel support, hook availability, and the behavior of the chosen tools still need validation on the hosts where they will run.

Reference architecture: an eBPF data plane without sidecars

The setup I keep landing on: eBPF programs attached directly to kernel hook points on every node, feeding both a network policy path and an observability/security event path, with a single node agent (Cilium for networking, Tetragon for runtime security, often deployed together) doing the userspace coordination. No per-pod proxy anywhere in the diagram.

The design choice that matters: policy and identity live in BPF maps that the kernel programs read on every packet or syscall, and the userspace agent updates those maps live. Changing a network policy or a security rule doesn’t mean recompiling or reloading the eBPF program. It means writing a new map entry, which is why policy changes propagate in milliseconds instead of requiring a rolling proxy restart.

The diagram focuses on kernel paths. Cilium's HTTP policy uses a shared Envoy proxy for the traffic selected for Layer 7 inspection. Include that proxy path when assessing overhead and encryption coverage.

Decision criteria: eBPF-native vs sidecar service mesh vs traditional host agents

Criteria	eBPF-Native (Cilium/Tetragon)	Sidecar Service Mesh (Istio/Linkerd)	Traditional Host Agents
Performance overhead	Low: in-kernel processing, no extra hops, shared per-node cost instead of per-pod	High: proxy per pod, two extra hops per request, meaningful memory/CPU tax at scale	Medium: depends on polling frequency; kernel modules add near-zero runtime overhead but real crash risk
Visibility depth	Deep: syscalls, process exec, file access, and network, correlated by kernel-verified identity	Network/L7 only, and only for traffic actually routed through the proxy	Shallow to deep depending on tooling; usually sampled or delayed, rarely both broad and real-time
Security enforcement	In-kernel packet filtering, process termination, and supported return-value overrides; prevention guarantees depend on the hook and action	L7 policy (mTLS, authz) only for meshed traffic; no process-level enforcement	Detection-oriented; enforcement usually requires a separate LSM or firewall layer
Operational complexity	Medium-high: requires kernel/BPF literacy on the team, but a single agent replaces mesh + several point tools	Medium: well-documented, but adds a proxy lifecycle (upgrades, cert rotation, resource tuning) to manage per workload	Low to medium: familiar tooling, but often means stitching together separate network, process, and file monitoring products
Kernel/version constraints	Real: needs a reasonably modern kernel (5.x+ recommended) with BTF for CO-RE; older or heavily customized kernels need work	None specific to the kernel: runs in userspace, portable across kernel versions	Kernel modules are tightly kernel-version coupled; userspace agents are portable but lose depth
Best fit	Platforms already on Kubernetes/cloud-native stacks with kernel expertise in-house, needing both network policy and runtime security from one data plane	Teams that need mesh features (traffic shaping, canary routing, mTLS) more than raw performance, and don’t have in-house kernel expertise	Simple environments, legacy (non-containerized) hosts, or teams unwilling to take on kernel-level tooling yet

Where I draw the line: if you’re running Kubernetes at meaningful scale and already fighting sidecar tax or gaps in runtime security visibility, eBPF-native tooling is worth the ramp-up. If your primary need is mesh-style traffic management (canaries, retries, fine-grained L7 routing) and your fleet is small enough that the sidecar tax doesn’t hurt yet, a service mesh is still the pragmatic choice. Several mesh projects are themselves adopting eBPF for the data plane while keeping the control plane familiar, which is a reasonable middle path.

A concrete example: bpftrace for ad hoc observability, Tetragon for enforcement

Two levels of “real code,” because production eBPF work spans both: quick ad hoc tracing during an incident, and durable, declarative policies that run continuously.

For ad hoc observability, a single bpftrace line attached to a kernel function, with no agent or daemon, and no code deployed to the application:

# Count TCP retransmissions per process command, every 10 seconds.
# Useful for correlating network degradation with a specific service
# without touching application code or adding a sidecar.
bpftrace -e '
kprobe:tcp_retransmit_skb { @retransmits[comm] = count(); }
interval:s:10 { print(@retransmits); clear(@retransmits); }
'

This attaches a kprobe to tcp_retransmit_skb, the kernel function invoked on every TCP retransmission, and aggregates a count per process name (comm) in a BPF map, printing and clearing it every ten seconds. During an incident, this is the fastest way to answer “which service is actually seeing retransmits right now.” No log shipping, no dashboards to build, no sidecar to inspect.

For a process-termination example, this Tetragon TracingPolicy matches the file path at security_file_permission and sends SIGKILL. Check hook availability and coverage on your target kernel before using it:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "block-shadow-read"
spec:
  kprobes:
  - call: "security_file_permission"
    syscall: false
    args:
    - index: 0
      type: "file"
    - index: 1
      type: "int"
    selectors:
    - matchArgs:
      - index: 0
        operator: "Prefix"
        values:
        - "/etc/shadow"
      matchActions:
      - action: Sigkill

The selector matches paths prefixed with /etc/shadow. Sigkill terminates the matching process, but does not guarantee that the triggering operation is denied. For access prevention, Tetragon documents combining a signal with a supported return-value override. This example contains no override. Validate both the returned error and whether data was accessed when testing a prevention policy.

Pitfalls and what I’d do in production

Kernel and CO-RE portability. This isn’t automatic. CO-RE relies on BTF (BPF Type Format) being present in the running kernel so a program compiled once can adjust to that kernel’s actual struct layouts at load time. Most distros ship BTF by default on kernels 5.x+, but older kernels, minimal/embedded images, or heavily customized kernels sometimes strip it. What I’d do: verify /sys/kernel/btf/vmlinux exists on your actual target hosts before committing to an eBPF-based rollout, not just in a dev VM. Keep a fallback plan (bundling BTF, or building per-kernel) for the hosts that don’t have it.

The verifier. It will reject programs that look fine to you. The BPF verifier enforces a bounded stack (512 bytes), limits on program complexity, and restrictions on loops and pointer arithmetic, all necessary to guarantee a program terminates and can’t read arbitrary memory. Programs that need to do more get split across tail calls (up to 32 chained programs) or use BPF-to-BPF function calls. What I’d do: budget verifier iteration into the timeline for anything non-trivial. Expect to restructure logic around the verifier’s rules rather than around what’s algorithmically simplest, and lean on existing map-based patterns from mature projects instead of reinventing state-tracking logic.

Observability cardinality. It gets expensive fast. eBPF makes it trivially easy to trace every syscall, every packet, or every function call, which means it’s just as easy to generate more events than any backend can ingest or you can afford to store. What I’d do: aggregate in-kernel using BPF maps (counts, histograms, top-N) before anything crosses into userspace, and treat full per-event export as a targeted, time-boxed debugging tool rather than a standing collection default.

Loading eBPF programs is privileged. Treat it that way. Loading a program has historically required CAP_SYS_ADMIN; kernels 5.8+ split this into more granular capabilities (CAP_BPF, CAP_PERFMON, CAP_NET_ADMIN), but it’s still meaningful kernel access, and kernel lockdown mode can block bpf() entirely in confidentiality mode. What I’d do: restrict which nodes and service accounts can load programs, use signed/vetted images from the project you’re standardizing on rather than ad hoc custom programs in production, and audit bpf() syscall usage itself. An attacker who can load their own eBPF program has kernel-level access.

Measure real overhead. XDP’s performance profile changes enormously between native mode (running in the NIC driver) and generic mode (running later in the stack, when the driver doesn’t support native XDP). The difference can be an order of magnitude in packets-per-second. What I’d do: benchmark on your actual NIC/driver combination under your actual traffic shape before trusting any published number, and re-verify after driver or kernel upgrades since native XDP support is driver-specific and can regress silently.

When I wouldn’t reach for eBPF at all. Rolling your own eBPF programs for problems mature projects already solve (Cilium, Tetragon, Falco, Pixie) is rarely worth it without in-house kernel engineering capacity. You inherit verifier debugging, BTF/CO-RE edge cases, and upgrade risk that a maintained project has already absorbed. It’s also a poor fit on kernels too old for BTF/CO-RE, on managed platforms that restrict the capabilities needed to load programs, or when the actual need is L7 application routing logic. That’s still better served by a proxy that understands the protocol semantics than by kernel-level packet inspection.

References

These project documents explain the kernel checks, tracing language, and enforcement actions discussed here. Use the documentation for the versions deployed on your hosts.

Linux kernel: eBPF verifier. Load-time checks on program state and memory access.
bpftrace 0.24 language reference. Probe syntax and aggregation used by the tracing example.
Tetragon enforcement. The distinction between terminating a process and overriding an operation's return value.
Cilium Layer 7 policies. Application-aware policy and the proxy path it requires.

ebpf-observability-samples. The companion repo: the bpftrace one-liner, the Tetragon block-shadow-read policy, and Cilium L3/L4 + L7 policies from this article, plus BTF/CO-RE and XDP-mode readiness checks. All runnable.
Networking. The full pillar on service meshes, eBPF-native data planes, and cloud networking trade-offs.
Zero Trust. Identity-aware segmentation and kernel-level policy enforcement patterns.
Service Mesh vs eBPF-Native Data Planes: How to Choose. The head-to-head on when sidecars still win.
Zero-Trust Segmentation with Cilium Network Policies. Coming soon: identity-aware L3-L7 policy design for multi-tenant clusters.

Let’s talk eBPF

If you’re weighing eBPF-native tooling against a sidecar mesh or legacy host agents and want a second opinion on the architecture, I’m happy to compare notes. Connect with me on LinkedIn or reach out via Contact.

IPv6 at Enterprise Scale: A Migration Playbook

Wed, 14 Jan 2026 00:00:00 GMT

Ask a network architect why they’re finally doing an enterprise IPv6 migration and “we’re running out of IPv4 addresses” is almost never the honest answer. Nobody sitting in a data center cares about the global IPv4 free pool. That argument was settled a decade ago, and either way it was never their problem. The trigger that actually shows up on a change ticket is that 10.0.0.0/8 is spoken for twice inside the same routing domain, or that this quarter’s GPU cluster can’t get a subnet large enough for its back-end fabric without carving into a block someone else already owns.

This is the playbook I reach for when RFC1918 space has actually run out (not in theory, in the IPAM tool), usually after a run of acquisitions, a few years of multi-cluster Kubernetes sprawl, and a GPU buildout that eats subnets faster than anyone budgeted for. I’m assuming you already know IPv6 basics: SLAAC, the address format, that /64 is the standard subnet size. What follows is about sequencing a migration a large organization can survive, not another argument for why IPv6 is good for you.

Why enterprise IPv6 migration stops being optional

RFC1918 was never sized for GPU fabrics. A single GPU node in a modern training or inference cluster carries multiple NICs for the compute fabric, plus a separate management/BMC network, plus a storage network. One rack of eight GPU servers can burn a few hundred addresses before a workload ever runs. Multiply that across dozens of racks and several clusters (training, inference, staging, DR) and you’re allocating subnets by the thousand. Private space handles that arithmetic fine in isolation. It stops working the moment every business unit, region, and cluster is drawing from the same finite 10.0.0.0/8.

Kubernetes multiplies the allocation problem instead of solving it. Every cluster wants its own pod CIDR and service CIDR, and the easy default (carve out a /16 per cluster) burns through address space you can’t easily reclaim once workloads are scheduled against it. Cross-cluster service mesh or federation needs those CIDRs to be globally unique, or NAT’d at every cluster boundary. A platform team running fifty-plus clusters across regions runs out of non-overlapping /16s inside a /8 far sooner than the raw host count would suggest.

Mergers and acquisitions guarantee overlap. Two companies have almost certainly both standardized on 10.0.0.0/8, or the same handful of common /12 and /16 blocks, entirely independently. Day one of any integration starts with subnet collisions rather than free space. The standard fix, 1:1 NAT or a renumbering project, is expensive either way: hundreds of manual mapping entries, monitoring and alerting that now has to key off the NAT’d address instead of the real one, and one missed mapping entry causes packet loss nobody can explain from the logs alone.

NAT and overlapping-CIDR VPNs are a standing tax rather than a one-time cost. Every site-to-site tunnel between two overlapping networks needs static NAT or a policy-routing workaround to disambiguate traffic. Every new acquisition or partner integration adds another set of translation rules to a table that’s already hard to reason about. A packet capture during an incident needs a NAT lookup before anyone can even identify which host originated the traffic. This is the part that never makes it into the original NAT decision: it doesn’t cost you once, it costs your on-call rotation every single day you keep running it.

Reference architecture: dual-stack core, IPv6-only islands, and NAT64/DNS64 at the edge

The target I aim for, phased over twelve to twenty-four months depending on estate size, looks like this: a dual-stack core and WAN backbone that everything still routes through, new capacity built as IPv6-only islands from day one, and a NAT64/DNS64 gateway sitting at the boundary so those islands can still reach the IPv4-only world that isn’t going anywhere soon.

The address plan underneath that topology is the part people underinvest in, and it’s the part that actually determines whether the migration is boring or miserable two years in. Flat is the enemy. Assign a /48 or /56 per site with a fixed hierarchy and you can summarize routes at every level and delegate blocks to regional teams without a spreadsheet fight every quarter.

Layer	Prefix	Example	Notes
Enterprise allocation	/32	`2001:db8::/32`	One block from your RIR (or provider PA space if you’re not running your own ASN yet). Sub-allocate internally, don’t request more than one
Region	/36	`2001:db8:1000::/36` (AMER), `2001:db8:2000::/36` (EMEA)	4 bits per region = 16 regions with room to spare
Site / data center	/48	`2001:db8:1210::/48` (us-east-dc2)	Roughly the guidance in RFC 6177; a small branch office doesn’t need a full /48, so I use /56 there and save /48s for real DCs and campuses
Segment / VLAN / pod CIDR	/64	`2001:db8:1210:a3::/64`	One per L2 segment, or the block a Kubernetes cluster carves per-node pod ranges out of
Router point-to-point link	/127	`2001:db8:1210:ffff::1/127`	Per RFC 6164. Don’t burn a /64 on a two-node link
Loopback	/128	`2001:db8:1210:ffff::1/128`	Router ID, anycast source

For the Kubernetes piece specifically, dual-stack has to be declared at the cluster level before it shows up at the Service level. You can’t retrofit it onto a running cluster’s pod network without a CIDR migration, so get this right at cluster build time:

# kubeadm ClusterConfiguration fragment - dual-stack cluster networking.
# This is what actually gets you dual-stack pods and services, not just
# a dual-stack NIC on the node.
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
networking:
  podSubnet: "10.244.0.0/16,2001:db8:1210:a300::/56"
  serviceSubnet: "10.96.0.0/12,2001:db8:1210:a3ff::/108"
---
apiVersion: v1
kind: Service
metadata:
  name: inference-api
spec:
  ipFamilyPolicy: PreferDualStack
  ipFamilies:
    - IPv6
    - IPv4
  selector:
    app: inference-api
  ports:
    - port: 443
      targetPort: 8443

The order in ipFamilies isn’t cosmetic. The first entry becomes the primary family and gets .spec.clusterIP, which matters for anything that only reads that one field instead of clusterIPs. And confirm your CNI and cloud load-balancer integration actually support dual-stack before you flip this in production; Calico and Cilium do, but plenty of managed load-balancer controllers still only provision a v4 frontend even when the Service behind it is dual-stack.

For the IPv6-only islands, DNS64 and NAT64 are the seam that lets them reach anything that hasn’t moved yet:

# BIND9 dns64 clause - synthesizes AAAA records for IPv4-only legacy
# hosts, so an IPv6-only client gets a routable address instead of NXDOMAIN
dns64 64:ff9b::/96 {
    clients { ipv6-only-islands; };
    mapped { !10.0.0.0/8; any; };
    exclude { 64:ff9b::/96; ::ffff:0:0/96; };
};

# Jool NAT64 instance (syntax varies by version - illustrative) - translates
# 64:ff9b::/96 traffic back to a real routable IPv4 address at the edge
jool instance add nat64-edge --netfilter --pool6 64:ff9b::/96
jool -i nat64-edge global update pool4 --tcp --add 203.0.113.50/32

Neither of these is meant to be permanent architecture. Treat NAT64/DNS64 as a transition device with a removal date attached to each legacy service it’s covering. Don’t build new dependencies on it.

Decision criteria: dual-stack vs. IPv6-only + NAT64/DNS64 vs. IPv6-mostly

Criteria	Dual-Stack Everywhere	IPv6-Only + NAT64/DNS64	IPv6-Mostly (Dual-Stack Edge, IPv6-Only Core)
App/library compatibility	Highest. Every legacy IPv4-only app and library keeps working untouched	Lowest. Anything with a hardcoded IPv4 literal or a v4-only library breaks until fixed	Good. Legacy stays dual-stack at the edge; only new/internal services need to be v6-clean
Address-exhaustion relief	Minimal short-term. Every host still consumes RFC1918 space on its v4 side	Complete. New fabric and clusters never touch private v4 space again	Strong. The growth that was actually eating your address space (new clusters, new GPU fabric) stops doing so
Operational complexity (running two stacks)	Highest, indefinitely. Every ACL, monitoring rule, and runbook exists twice, forever	Lower once cut over. One internal stack; NAT64/DNS64 is one well-understood edge component	Moderate. Complexity concentrated at the edge boundary instead of spread through the whole estate
Security parity effort	High but at least symmetric. One policy engine usually covers both families equally	High and asymmetric. The v6-only interior is simpler to secure, but the NAT64 gateway becomes a chokepoint that needs its own hardening and logging	Moderate. The edge needs full dual-stack parity; the core only needs v6
Rollout risk	Low per-step, high cumulative cost. Safe, but you never actually finish	Highest per-step. A broken v6-only cutover has no v4 fallback	Balanced. New builds carry the risk, legacy is untouched until its own retirement
Best fit	Orgs early in migration, or with a hard requirement that specific legacy IPv4-only integrations keep working indefinitely	Greenfield GPU fabrics and new Kubernetes clusters where you control every workload going in	Most large enterprises mid-migration. New capacity built v6-only, legacy kept dual-stack until it’s decommissioned on its own schedule

If I had to compress this into one line for a steering committee: don’t run dual-stack forever, and don’t flag-day your way into IPv6-only either. Build new capacity as IPv6-only from day one, keep legacy dual-stack until it’s decommissioned anyway, and let NAT64/DNS64 be the seam between the two. Keep it a transition device with a removal date, not a permanent piece of the architecture.

Pitfalls and what I’d do in production

Don’t flag-day it, and don’t let “eventually” pass for a plan either. Flipping a whole estate to dual-stack or IPv6-only in one maintenance window guarantees you discover every broken assumption at the same time, usually at 2am. What I’d do: sequence by blast radius. Greenfield workloads first, since they carry no legacy baggage, then internal east-west traffic between teams who can coordinate fixes together, then anything client-facing or regulated last, with a hard decommission date on the old path so dual-stack doesn’t quietly become permanent.

Hardcoded IPv4 literals and v4-only libraries don’t announce themselves until traffic actually hits them. A config file with a literal dotted-quad, a monitoring agent that only opens v4 sockets, a vendor appliance whose management API has no v6 support at all: none of this surfaces in a design review. It surfaces in production, as a silent timeout from a v6-only pod. What I’d do: grep every config repo for IPv4 literals before cutover, and, more useful than any static scan, actually run each candidate service in an IPv6-only test namespace before anyone calls it “IPv6 ready.” A code review can’t catch what a real socket call will.

Security tooling parity lags the network layer, and that gap is where the real exposure lives. Firewalls, IDS/IPS signatures, NetFlow/sFlow collectors, and SIEM parsers were tuned against years of v4 traffic and get bolted-on v6 support nobody has stress-tested. An ACL that’s airtight on v4 and default-permit on v6 because someone forgot to mirror the rule is an exploitable hole, not a theoretical one. What I’d do: audit every control on the path (firewall, ACL, IDS, flow logging) for explicit v6 parity before v6 goes live anywhere that matters, and treat “does the SIEM actually parse this v6 log format” as a launch blocker rather than a backlog ticket.

Dual-stack doubles your operational surface, and it stays doubled for as long as you run it. Every ACL, monitoring rule, capacity plan, and incident runbook now has a v4 branch and a v6 branch, and they drift apart the moment one gets updated without the other. That’s the real cost of dual-stack: not a performance tax but an ongoing tax on whoever’s holding the pager. What I’d do: attach a real decommission date to dual-stack for each workload at migration time, not just an intention. No date means no plan, and you’ll be running both stacks longer than you think.

Happy Eyeballs will quietly hide a broken IPv6 path from you, which is worse than an outright outage. Client OSes race both address families per RFC 8305 and use whichever connects first, so a broken v6 route on your egress path can sit broken for weeks. Traffic just falls back to v4 every time, and nothing pages anyone. What I’d do: monitor v6 connection success rate as its own signal, separate from the blended success rate, and alert on it independently. Watch only the combined number and a 100%-failing v6 path with working v4 fallback looks identical to a fully healthy dual-stack service.

If your monitoring and logging stack doesn’t understand v6, you’re flying blind the moment you turn it on. Flow exporters that only emit v4 records, log parsers with a regex that matches dotted-quad and nothing else, dashboards that silently drop or mis-bucket v6 addresses: you find out during the first v6 incident, which is the worst possible time to learn it. What I’d do: validate end-to-end v6 visibility (flow collection, log parsing, alerting) before the network change goes out, using synthetic v6 traffic to confirm every hop in the observability pipeline actually sees it.

ULA vs. GUA is a real decision, not a formality. Get it wrong and you re-architect later. Unique Local Addresses (RFC 4193, fc00::/7) feel like the IPv6 answer to RFC1918: private, not globally routed. But they don’t get you out of NAT for anything that needs to reach the internet or a partner network, which defeats half the point of migrating in the first place. What I’d do: use GUA for anything that might ever need direct external reachability (which, once you count SaaS integrations and partner VPNs, is most things in a large enterprise), and reserve ULA narrowly, for genuinely isolated management or out-of-band networks that will never route externally. Don’t let nostalgia for RFC1918’s “private by default” model drag you into ULA everywhere; that just rebuilds a smaller version of the exhaustion problem you started this migration to escape.

References

RFC 7381: Enterprise IPv6 Deployment Guidelines. Planning and sequencing an enterprise migration.
RFC 6146: Stateful NAT64. Translation between IPv6 clients and IPv4 servers.
RFC 6147: DNS64. Synthesizing IPv6 DNS answers for use with translation.
RFC 8305: Happy Eyeballs Version 2. Connection attempts across address families and why fallback can hide a broken IPv6 path.

Networking. The full pillar on routing architecture for modern infrastructure
Networking. Networking design specifically for AI/ML workloads
BGP for the AI Era: Multi-Region Routing for Inference Workloads. The anycast and health-triggered withdrawal patterns there apply just as directly once your VIPs are dual-stack
Zero-Downtime Renumbering: Merging Two Overlapping RFC1918 Networks After an Acquisition. Coming soon: the tactical playbook for the integration-week collision this article assumes you’ll eventually hit

Planning your own migration?

If you’re mapping out an enterprise IPv6 migration and want a second opinion on the address plan or the NAT64 cutover sequence, I’m happy to compare notes. Connect with me on LinkedIn or reach out via Contact.

Creating a New CA (SHA-512, 4096-bit) with PKCS #11 Smart Card Storage and Subordinate CA

Tue, 18 Feb 2025 00:00:00 GMT

Today, I will walk through the process of creating a new Certificate Authority (CA) using a 4096-bit RSA key and SHA-512 as the signature hash algorithm. I will also store the private key securely on a PKCS #11–compatible smart card, and then create a subordinate (sub) CA using the root CA. Throughout this guide, I will use recommended security controls that you should follow to protect your keys and certificates.

Why Use a Smart Card (PKCS #11)?

Storing your private key on a smart card (or Hardware Security Module, HSM) significantly enhances security by:

Preventing key extraction: Private keys generated on or imported into smart cards cannot be easily exported in plain text.
Hardware-based security: Many smart cards implement protections against physical tampering and side-channel attacks.
Secure key usage: Sign and decrypt operations occur on the card, which reduces exposure of the private key.

Recommended Security Controls

Before diving into commands, keep these best practices in mind:

Offline Root CA:
- Maintain your root CA offline (on a laptop or computer not connected to any network).
- Only bring it online in a secure environment when issuing or renewing certificates.
Physical Security:
- Store the smart card containing the root CA key in a locked safe or a secure location.
- Use tamper-evident bags or containers for added protection.
Strong Passphrase/PIN:
- Use a complex PIN/passphrase for the smart card (or HSM).
- Never store the PIN/passphrase on the same device or in the same physical location as the smart card.
Dedicated Machine for CA Operations:
- Limit software installed on the CA machine to the bare minimum (e.g., OpenSSL, PKCS #11 drivers).
- Keep the system fully patched and hardened.
Regular Audits and Logging:
- Maintain logs of issuance, revocation, and key usage.
- Periodically review who has access to the CA environment.
Multi-factor Authentication (MFA):
- If possible, layer an additional form of authentication (e.g., a separate password or token) to access CA resources.

With these controls in place, let’s move on to the technical steps.

Step-by-Step: Creating a Root CA (SHA-512, 4096-bit) on a Smart Card

3.1 Install Required Packages

On most Linux systems, you will need:

sudo apt-get update
sudo apt-get install opensc openssl libengine-pkcs11-openssl pkcs11-tools

OpenSSL: Core cryptographic toolkit.
OpenSC: Contains drivers and utilities for several PKCS #11 smart cards.
pkcs11-tools: Utility set for managing PKCS #11 tokens.
libengine-pkcs11-openssl: Engine plugin allowing OpenSSL to communicate with PKCS #11–compatible devices.

3.2 Initialize and Prepare Your Smart Card

Insert the smart card into your reader.
Initialize the card (careful—this may erase existing keys!). For example, with pkcs15-init from OpenSC:

pkcs15-init --create-pkcs15 --profile pkcs15+onepin --pin 123456 --puk 654321

--pin 123456 – sets your user PIN (example value; use a stronger PIN in reality).
--puk 654321 – sets an unblock code if the PIN is entered incorrectly too many times.

Alternatively, you can use pkcs11-tool commands depending on your card’s vendor and management software.

3.3 Generate or Import the Root CA Key onto the Smart Card

There are two main approaches to get the key onto the card:

Generate directly on the card (best practice, more secure).
Generate externally and import (some cards may not support this, or you might prefer external generation).

3.3.1 Generating the Key on the Card

pkcs11-tool --module /usr/lib/opensc-pkcs11.so \
            --login \
            --keypairgen \
            --key-type RSA:4096 \
            --label "RootCA-Key" \
            --id 01 \
            --pin 123456

--module points to your PKCS #11 driver (path may differ by system).
--keypairgen performs the on-card key generation.
--label "RootCA-Key" helps identify the key.
--id 01 sets a hex identifier for the key, which you can reference later.

3.3.2 Creating the Root CA Certificate Request

Create a configuration file for OpenSSL (let’s call it root-ca.conf) specifying default settings, like so:

[ req ]
default_bits       = 4096
default_keyfile    = rootCA.key
distinguished_name = req_distinguished_name
req_extensions     = v3_req
prompt             = no

[ req_distinguished_name ]
C  = US
ST = YourState
L  = YourCity
O  = YourOrganization
OU = Root CA
CN = vkafed.com Root CA

[ v3_req ]
keyUsage = critical, keyCertSign, cRLSign
basicConstraints = critical, CA:true

Use OpenSSL’s PKCS #11 engine to create a certificate request that references the on-card key

openssl req -new -engine pkcs11 \
    -keyform engine \
    -key "pkcs11:object=RootCA-Key;type=private;pin-value=123456" \
    -config root-ca.conf \
    -sha512 \
    -out rootCA.csr

3.3.3 Self-Sign the Root CA Certificate

Because this is a root CA, we are self-signing:

openssl x509 -req -in rootCA.csr -sha512 \
    -extensions v3_req \
    -signkey "pkcs11:object=RootCA-Key;type=private;pin-value=123456" \
    -days 3650 \
    -out rootCA.crt

-days 3650 (10 years) is an example. Adjust it if needed.
The resulting file rootCA.crt is your self-signed root CA certificate.

Store rootCA.crt in a secure offline location (and keep backups!).

4. Creating a Subordinate CA (SubCA)

Next, we create a subordinate CA signed by the root CA.

4.1 Generate SubCA Keys and CSR

You may generate this subordinate CA key in software or on another smart card, depending on your security policies. Below is a software-based example:

Generate the key pair:

openssl genrsa -out subCA.key 4096

2. Create a configuration file for the SubCA (e.g., sub-ca.conf):

[ req ]
default_bits       = 4096
default_keyfile    = subCA.key
distinguished_name = req_distinguished_name
req_extensions     = v3_req
prompt             = no

[ req_distinguished_name ]
C  = US
ST = YourState
L  = YourCity
O  = YourOrganization
OU = Sub CA
CN = vkafed.com Sub CA

[ v3_req ]
keyUsage = critical, keyCertSign, cRLSign
basicConstraints = critical, CA:true, pathlen=0

3. Generate the CSR:

openssl req -new -sha512 -key subCA.key -config sub-ca.conf -out subCA.csr

4.2 Sign the SubCA CSR Using the Offline Root CA

On your offline root CA machine (where rootCA.crt and root CA smart card are stored), sign the SubCA CSR:

openssl x509 -req -in subCA.csr \
    -CA rootCA.crt \
    -CAkey "pkcs11:object=RootCA-Key;type=private;pin-value=123456" \
    -CAcreateserial \
    -days 1825 \
    -sha512 \
    -extensions v3_req \
    -extfile sub-ca.conf \
    -out subCA.crt

-CAcreateserial will create a file rootCA.srl that tracks certificate serial numbers.
-days 1825 sets the SubCA certificate to be valid for 5 years (example; adjust if needed).

5. Final Security Considerations

Secure Storage:

Keep the root CA’s smart card physically secured and offline.
Maintain secure backups (preferably in another physical location).

Revocation Mechanisms:

Publish a Certificate Revocation List (CRL) or use Online Certificate Status Protocol (OCSP) so end-users can check certificate status.
Host the CRL or OCSP service in a reliable location.

Certificate Policies:

Draft a Certificate Policy (CP) and a Certification Practice Statement (CPS) if this CA will be used for a production or public trust environment.

Regular Renewal and Key Rollover:

Plan certificate renewal schedules and ensure that sub CA and end-entity certificates are renewed before they expire.
Periodically rotate (reissue) keys to reduce long-term exposure.

Logging and Monitoring:

Log all CA operations (issuance, revocation, etc.).
Monitor your environment for unauthorized issuance attempts or suspicious access.

Running PostgreSQL in Docker on my Mac

Mon, 13 Apr 2020 00:00:00 GMT

It’s extremely convenient to run your local tools, like RDMS, development environments and so on on Docker containers on Mac. To be honest with you, I hate installing a bunch of software on my Mac, and that’s the main reason why do I use containers for my local development.

Before doing that manual, make sure your Docker is installed on Mac. To do so, you can go to https://www.docker.com/products/docker-desktop and download the latest version of Docker for your Mac.

So, how do I can use PostgreSQL on Mac in a Docker container? Let’s create a new Project folder, or you may use your own where docker-compose files.

mkdir ~/Projects
cd ~/Projects

And creating a docker-compose.yml file with the following code:

#File: docker-compose.yml 
version: "3"
services:
  db:
    image: "postgres:11"
    container_name: "local_postgres"
    ports:
      - "54320:5432"
    volumes:
      - dbdata:/var/lib/postgresql/data
    environment:
      - POSTGRES_HOST_AUTH_METHOD=trust
      - POSTGRES_PASSWORD=somesecretpassword
volumes:
  dbdata:

So, a quick clarification on what we do here. We use postgres:11 image, naming containers as local_postgres, expose port 54320, store data on dbdata volume, use passwordless access to database (as you we do everything locally), and use somesecretpassword as password. Pretty straight forward, ha?

Ok, let’s build it and run it:

docker-compose up -d

You shouldn’t see any errors, and let’s see logs with the following command:

docker logs -f local_postgres

And if we can connect to database, like this:

docker exec -it local_postgres psql -U postgres

Hit CTRL+D to exit from the prompt.

Create a database

So, to create a database, you can run the following command, where new_database is database new you want to create.

docker exec -it my_postgres psql -U postgres -c "create database new_database"

You can use a database connection now with the following login details:

host='localhost',
port=54320,
dbname='hew_database',
user='postgres',
password='somesecretpassword'

That’s pretty much it. Easy, right?

Please comment below if you have any issues or questions!

PostgreSQL Notes

Originally published April 20, 2020.

How to drop all tables sharing the same prefix in PostgreSQL?

DO
$do$
DECLARE
   _tbl text;
BEGIN
FOR _tbl  IN
    SELECT quote_ident(table_schema) || '.'
        || quote_ident(table_name)
    FROM   information_schema.tables
    WHERE  table_name LIKE '[prefix]' || '%'
    AND    table_schema NOT LIKE 'pg\_%'
LOOP
-- RAISE NOTICE '%',
  EXECUTE 'DROP TABLE ' || _tbl || ' CASCADE';
END LOOP;
END
$do$;

Non-Cascade version:

EXECUTE 'DROP TABLE ' || _tbl;

How to drop all functions in PostgreSQL?

DO
$do$
DECLARE
   _sql text;
BEGIN
   SELECT INTO _sql
          string_agg(format('DROP %s %s;'
                          , CASE WHEN proisagg THEN 'AGGREGATE' ELSE 'FUNCTION' END
                          , oid::regprocedure)
                   , E'\n')
   FROM   pg_proc
   WHERE  pronamespace = 'public'::regnamespace;

   IF _sql IS NOT NULL THEN
      RAISE NOTICE '%', _sql;
      -- EXECUTE _sql;
   ELSE 
      RAISE NOTICE 'No fuctions found in schema %', quote_ident(_schema);
   END IF;
END
$do$  LANGUAGE plpgsql;

ElementaryOS: Resolution fix for 2560×1440

Tue, 03 Dec 2019 00:00:00 GMT

Edit /usr/share/X11/xorg.conf.d/10-monitor.conf and paste the following:

Section "Monitor"
  Identifier "Monitor0"
  Section "Monitor"
  Identifier "Monitor0"
  Modeline "2560x1440_60.00"  661.25  2560 2784 3064 3568 1440 1443 1448 1545 -hsync +vsync
EndSection
Section "Screen"
  Identifier "Screen0"
  Device "Virtual1"
  Monitor "Monitor0"
  DefaultDepth 24
  SubSection "Display"
    Depth 24
    Modes "2560x1440_60.00"
  EndSubSection
EndSection

And reboot system with sudo reboot

# sudo reboot

Thanks!

SendMail function in Golang

Mon, 28 Oct 2019 00:00:00 GMT

Easy-to-use function to send HTML and plain-text emails in Go

func SendMail(addr, from, subject, body string, to []string) error {
	r := strings.NewReplacer("\r\n", "", "\r", "", "\n", "", "%0a", "", "%0d", "")
	c, err := smtp.Dial(addr)
	if err != nil {
		return err
	}
	defer c.Close()
	if err = c.Mail(r.Replace(from)); err != nil {
		return err
	}
	for i := range to {
		to[i] = r.Replace(to[i])
		if err = c.Rcpt(to[i]); err != nil {
			return err
		}
	}
	w, err := c.Data()
	if err != nil {
		return err
	}
	msg := "To: " + strings.Join(to, ",") + "\r\n" +
		"From: " + from + "\r\n" +
		"Subject: " + subject + "\r\n" +
		"Content-Type: text/html; charset=\"UTF-8\"\r\n" +
		"Content-Transfer-Encoding: base64\r\n" +
		"\r\n" + base64.StdEncoding.EncodeToString([]byte(body))
	_, err = w.Write([]byte(msg))
	if err != nil {
		return err
	}
	err = w.Close()
	if err != nil {
		return err
	}
	return c.Quit()
}

Usage:

SendMail("127.0.0.1:25", (&amp;mail.Address{"from name", "from@example.com"}).String(), "Email Subject", "message body", []string{(&amp;mail.Address{"to name", "to@example.com"}).String()})

Apache 2.4.6 + mod_wsgi + Python 3.6 + Django 2.2.6 on CentOS 7

Mon, 14 Oct 2019 00:00:00 GMT

There are many how-to’s on installing Apache with mod_wsgi, python 3.x and Django, but nothing worked for me, looks like mod_wsgi libraries installed by default with yum use Python 2.x, and I keep getting the following error message:

[Mon Oct 14 09:42:07.713418 2019] [:error] [pid 18066] [remote 10.30.40.31:0] mod_wsgi (pid=18066): Target WSGI script '/usr/local/val/app/app/wsgi.py' cannot be loaded as Python module.
[Mon Oct 14 09:42:07.713497 2019] [:error] [pid 18066] [remote 10.30.40.31:0] mod_wsgi (pid=18066): Exception occurred processing WSGI script '/usr/local/val/app/wsgi.py'.
[Mon Oct 14 09:42:07.713560 2019] [:error] [pid 18066] [remote 10.30.40.31:0] Traceback (most recent call last):
[Mon Oct 14 09:42:07.713596 2019] [:error] [pid 18066] [remote 10.30.40.31:0]   File "/usr/local/val/app/app/wsgi.py", line 12, in <module>
[Mon Oct 14 09:42:07.713701 2019] [:error] [pid 18066] [remote 10.30.40.31:0]     from django.core.wsgi import get_wsgi_application
[Mon Oct 14 09:42:07.713718 2019] [:error] [pid 18066] [remote 10.30.40.31:0]   File "/usr/local/val/venv/lib/python3.6/site-packages/django/__init__.py", line 1, in <module>
[Mon Oct 14 09:42:07.713789 2019] [:error] [pid 18066] [remote 10.30.40.31:0]     from django.utils.version import get_version
[Mon Oct 14 09:42:07.713804 2019] [:error] [pid 18066] [remote 10.30.40.31:0]   File "/usr/local/val/venv/lib/python3.6/site-packages/django/utils/version.py", line 71, in <module>
[Mon Oct 14 09:42:07.713886 2019] [:error] [pid 18066] [remote 10.30.40.31:0]     @functools.lru_cache()
[Mon Oct 14 09:42:07.713914 2019] [:error] [pid 18066] [remote 10.30.40.31:0] AttributeError: 'module' object has no attribute 'lru_cache'

Installing Apache and devel package

$ yum install httpd httpd-devel wget

So, I had to compile mod_wsgi library from sources, which actually helped me to fix the error above with running Django app with wsgi.

Let’s make sure we have Python 3.6 or 3.7 installed or actually it would work with any Python 3 minor version.

$ python3
Python 3.6.8 (default, Aug  7 2019, 17:28:10) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.

If we have Python 3 installed, we need to make sure we got mod_wsgi removed, which was installed by yum.

$ yum remove mod_wsgi

Downloading, compiling and installing a mod_wsgi package by doing this:

$ cd /opt
$ wget https://files.pythonhosted.org/packages/25/d8/1df4ba9c051cd88e02971814f0867274a8ac821baf983b6778dacd6e31f7/mod_wsgi-4.6.8.tar.gz
$ tar -zxvf mod_wsgi-4.6.8.tar.gz
$ cd mod_wsgi-4.6.8
$ ./configure --with-python=/usr/bin/python3.6
$ LD_RUN_PATH=/usr/local/lib make
$ sudo make install

Add a new entry to /etc/ld.so.conf

include ld.so.conf.d/*.conf
include /usr/local/lib

Run:

ldconfig

Make sure mod_wsgi linked properly:

# ldd usr/lib64/httpd/modules/mod_wsgi.so
	linux-vdso.so.1 =>  (0x00007ffd5bdd7000)
	libpython3.6m.so.1.0 => /lib64/libpython3.6m.so.1.0 (0x00007fe5ab02b000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe5aae0f000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fe5aac0b000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007fe5aaa08000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fe5aa706000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fe5aa338000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe5ab78f000)

Install Django:

$ pip3.6 install --upgrade pip
$ pip3.6 install Django

Verify Django installed properly:

$ django-admin version
2.2.6

My httpd configuration file looks like this:

Alias /static /usr/local/val/app/static
<Directory /usr/local/val/app/static>
 Require all granted
</Directory>

<Directory /usr/local/val/app/app>
   <Files wsgi.py>
       Require all granted
   </Files>
</Directory>

WSGIDaemonProcess app python-path=/usr/local/val/app:/usr/local/val/venv/lib/python3.6/site-packages python-home=/usr/local/val/env
WSGIProcessGroup app
WSGIScriptAlias / /usr/local/val/app/app/wsgi.py
WSGIPassAuthorization On

Where app is the name of my django app, and val is directory where app is located.

Before that I created virtual environment by running this:

$ python3 -m venv /usr/local/val/env

Test if everything works correctly, you may probably also want to add your hosts to ALLOWED_HOSTS in your settings.py file and setup static files by specifying the following paths in your settings.py

STATIC_ROOT = '/usr/local/val/app/static/'
STATIC_URL = '/static/'

Let me know if you have any questions

Thanks!

Nagios environment variables

Thu, 05 Sep 2019 00:00:00 GMT

As soon as Nagios is managed by systemd, you have to change the passing environment variables from /etc/sysconfig/nagios to systemd

Edit the file

$ vi /etc/systemd/system/multi-user.target.wants/nagios.service

Add Environment=”” or EnvironmentFile=”” to [service] section

[Service]
Type=forking
ExecStartPre=/usr/sbin/nagios -v /etc/nagios/nagios.cfg
ExecStart=/usr/sbin/nagios -d /etc/nagios/nagios.cfg
ExecStop=/usr/bin/kill -s TERM ${MAINPID}
ExecStopPost=/usr/bin/rm -f /var/spool/nagios/cmd/nagios.cmd
ExecReload=/usr/bin/kill -s HUP ${MAINPID}
Environment="PYTHONHTTPSVERIFY=1"

Restart systemd and nagios

$ sudo systemctl daemon-reload
$ sudo systemctl restart nagios

Check the process environment

$ cat /proc/30195/environ
LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binPYTHONHTTPSVERIFY=1
$

That will pass all variables into Nagios daemon

Nagios, the Nagios logo, and Nagios graphics are the servicemarks, trademarks, or registered trademarks owned by Nagios Enterprises.

Let me know if you have any issues with monitoring. Will be glad to help 🙂

Happy Independence Day!

Thu, 04 Jul 2019 00:00:00 GMT

Wishing everybody a wonderful time on this special day in American history. Happy Fourth of July! God bless America!

Open-Xchange IMAP: STARTTLS failure

Thu, 02 May 2019 00:00:00 GMT

During some experiments with open-sources code, found an issue with STARTTLS on the latest open-xchange appsuite 7.10.1-Rev9

Something like this:

com.openexchange.exception.OXException: LGI-0003 Categories=ERROR Message='Unknown problem: "STARTTLS failure".' exceptionID=-1484511495-7
	at com.openexchange.exception.OXExceptionFactory.create(OXExceptionFactory.java:175)
	at com.openexchange.exception.OXExceptionFactory.create(OXExceptionFactory.java:165)
	at com.openexchange.exception.OXExceptionFactory.create(OXExceptionFactory.java:138)
	at com.openexchange.authentication.LoginExceptionCodes.create(LoginExceptionCodes.java:267)
	at com.openexchange.authentication.imap.impl.IMAPAuthentication.handleLoginInfo(IMAPAuthentication.java:373)
	at com.openexchange.authentication.service.Authentication.login(Authentication.java:111)
	at com.openexchange.authentication.service.Authentication.login(Authentication.java:98)
	at com.openexchange.login.internal.NormalLoginMethod.doAuthentication(NormalLoginMethod.java:83)
	at com.openexchange.login.internal.LoginPerformer.doLogin(LoginPerformer.java:216)
	at com.openexchange.login.internal.LoginPerformer.doLogin(LoginPerformer.java:157)
	at com.openexchange.login.internal.LoginPerformer.doLogin(LoginPerformer.java:145)
	at com.openexchange.ajax.login.Login$1.doLogin(Login.java:109)
	at com.openexchange.ajax.login.AbstractLoginRequestHandler.loginOperation(AbstractLoginRequestHandler.java:226)
	at com.openexchange.ajax.login.AbstractLoginRequestHandler.loginOperation(AbstractLoginRequestHandler.java:184)
	at com.openexchange.ajax.login.Login.doLogin(Login.java:97)
	at com.openexchange.ajax.login.Login.handleRequest(Login.java:90)
	at com.openexchange.ajax.LoginServlet.doJSONAuth(LoginServlet.java:793)
	at com.openexchange.ajax.LoginServlet.doGet(LoginServlet.java:758)
	at com.openexchange.ajax.LoginServlet.doPost(LoginServlet.java:878)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
	at com.openexchange.ajax.AJAXServlet.doService(AJAXServlet.java:566)
	at com.openexchange.ajax.LoginServlet.service(LoginServlet.java:738)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:791)
	at org.glassfish.grizzly.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:147)
	at com.openexchange.http.grizzly.servletfilter.RequestReportingFilter.doFilter(RequestReportingFilter.java:138)
	at org.glassfish.grizzly.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:137)
	at com.openexchange.http.grizzly.servletfilter.WrappingFilter.doFilter(WrappingFilter.java:222)
	at org.glassfish.grizzly.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:137)
	at com.openexchange.http.grizzly.service.http.OSGiAuthFilter.doFilter(OSGiAuthFilter.java:139)
	at org.glassfish.grizzly.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:137)
	at org.glassfish.grizzly.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:106)
	at org.glassfish.grizzly.servlet.ServletHandler.doServletService(ServletHandler.java:226)
	at org.glassfish.grizzly.servlet.ServletHandler.service(ServletHandler.java:178)
	at com.openexchange.http.grizzly.service.http.OSGiMainHandler.service(OSGiMainHandler.java:301)
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224)
	at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$MDCProvidingRunnable.run(CustomThreadPoolExecutor.java:2575)
	at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.runTask(CustomThreadPoolExecutor.java:841)
	at com.openexchange.threadpool.internal.CustomThreadPoolExecutor$Worker.run(CustomThreadPoolExecutor.java:868)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.mail.MessagingException: STARTTLS failure
	at com.sun.mail.imap.IMAPStore.protocolConnect(IMAPStore.java:954)
	at javax.mail.Service.connect(Service.java:369)
	at com.openexchange.authentication.imap.impl.IMAPAuthentication.handleLoginInfo(IMAPAuthentication.java:357)
	... 34 common frames omitted

Looks like empty vars for SSL protocol and SSL ciphersuites are not working properly, if specify them everything works just fine.

# File /opt/open-xchange/etc/imap.properties

# Specifies the SSL protocols that will be enabled for SSL connections. The property value is a whitespace separated list of tokens.
# Default is empty
com.openexchange.imap.ssl.protocols=TLSv1 TLSv1.1 TLSv1.2

# Specifies the SSL cipher suites that will be enabled for SSL connections. The property value is a whitespace separated list of tokens.
#
# Check "http://<ox-grizzly-hostname>:<ox-grizzly-port>/stats/diagnostic?param=ciphersuites" to check available cipher suites.
#
# Default value is empty (fall-back to current JVM's default SSL cipher suite)
com.openexchange.imap.ssl.ciphersuites=SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA SSL_RSA_WITH_3DES_EDE_CBC_SHA TLS_DHE_DSS_WITH_AES_128_CBC_SHA TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 TLS_DHE_DSS_WITH_AES_256_CBC_SHA TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA TLS_ECDH_RSA_WITH_AES_128_CBC_SHA TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDH_RSA_WITH_AES_256_CBC_SHA TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 TLS_EMPTY_RENEGOTIATION_INFO_SCSV TLS_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA256

Hope that helps, thanks!

Paperkey: How to backup your GnuPG keys on paper

Mon, 13 Aug 2018 00:00:00 GMT

To create a backup of your GPG Key, you may use one of the following commands

paperkey --secret-key my-secret-key.gpg --output to-be-printed.txt

Or using this if you have exported (not armored) GPG Key in file:

paperkey --secret-key my-secret-key.gpg --output to-be-printed.txt

To restore it you will need a paperkey data in file and you public key. The following command will take public key from public-key.gpg file and paperkey data from secret-paperkey.gpg file and will import it to ~/.gnupg.

$ paperkey --pubring public-key.gpg --secrets secret-paperkey.gpg | gpg --import

If you have armored gpg public key, you will need to dearmor it first by doing this:

$ gpg --dearmor public-key.gpg

To install it on your mac, you may use brew:

$ brew install paperkey

Paperkey by David Shaw

Thanks!

Reset your YubiKey to factory defaults using gpg-connect-agent

Fri, 06 Jul 2018 00:00:00 GMT

I’m using MacOS for my work, so here is how to reset your YubiKey using gpg-connect-agent. This is not a mine solution, I found this a while ago on github, can’t remember who’s is author, so I saved to my notes, just wanted to share it with you.

# Install gnupg if it's not installed yet
VKAFEDZH-M-2R3C:~ vkafedzh$ brew install gnupg

Insert your YubiKey to USB and run the following command:

gpg-connect-agent <<EOF
/hex
scd serialno
scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
scd apdu 00 e6 00 00
scd apdu 00 44 00 00
/echo Yubikey has been successfully reset.
/echo The factory default PINs are 123456 (user) and 12345678 (admin).
EOF

Here is what I have:

VKAFEDZH-M-2R3C:~ vkafedzh$ gpg-connect-agent <<EOF
> /hex
> scd serialno
> scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
> scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
> scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
> scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
> scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
> scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
> scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
> scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
> scd apdu 00 e6 00 00
> scd apdu 00 44 00 00
> /echo Yubikey has been successfully reset.
> /echo The factory default PINs are 123456 (user) and 12345678 (admin).
> EOF
gpg-connect-agent: no running gpg-agent - starting '/usr/local/Cellar/gnupg/2.2.8/bin/gpg-agent'
gpg-connect-agent: waiting for the agent to come up ... (5s)
gpg-connect-agent: connection to agent established
S SERIALNO XXXXXXXXXXXXXX24010201000607XXXXXXX
OK
D[0000]  69 82                                              i.
OK
D[0000]  69 82                                              i.
OK
D[0000]  69 82                                              i.
OK
D[0000]  69 83                                              i.
OK
D[0000]  69 82                                              i.
OK
D[0000]  69 82                                              i.
OK
D[0000]  69 82                                              i.
OK
D[0000]  69 83                                              i.
OK
D[0000]  90 00                                              ..
OK
D[0000]  90 00                                              ..
OK
Yubikey has been successfully reset.
The factory default PINs are 123456 (user) and 12345678 (admin).

Thanks!

Using YubiKey with GPG

Sun, 28 Jan 2018 00:00:00 GMT

Using Yubikey with GPG encryption

Let me try to show my experience in using Yubikey as a Smartcard for storing signing and GPG encryption keys. There are a lot of information in the internet about that, but that my first try in configuring yubikey for such purposes.

Firstly we need to install necessary packages on our mac by doing the following command, I’m using brew:

$ brew install gnupg yubikey-personalization

This will allow us to program our Yubikey.

The next step is we need to create a new keys for further usage.

Generate a key

Let’s do a temporary directory:

export GNUPGHOME=$(mktemp -d)

And create a GPG configuration:

$ cat << EOF > $GNUPGHOME/gpg.conf
use-agent
personal-cipher-preferences AES256 AES192 AES CAST5
personal-digest-preferences SHA512 SHA384 SHA256 SHA224
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed
cert-digest-algo SHA512
s2k-digest-algo SHA512
s2k-cipher-algo AES256
charset utf-8
fixed-list-mode
no-comments
no-emit-version
keyid-format 0xlong
list-options show-uid-validity
verify-options show-uid-validity
with-fingerprint
EOF

Generate a master key

During creation a new master key, we will need to choose RSA (sign only) key and I would go with 4096 bits. And you’ll be asked to enter a passphrase, try to enter something unique and strong 🙂

VKAFEDZH-M-2R3C:~ val$ gpg --full-generate-key
gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
gpg: keybox ‘/var/folders/yy/bp5hkvxs1px1_f1q_10kzgk40000gn/T/tmp.SRHTb4gb/pubring.kbx’ created
Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
Your selection? 4
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
0 = key does not expire
<n>  = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y
GnuPG needs to construct a user ID to identify your key.
Real name: Valerii
Email address: vkafed@gmail.com
Comment:
You selected this USER-ID:
“Valerii <vkafed@gmail.com>”
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
public and secret key created and signed.
Note that this key cannot be used for encryption.  You may want to use
the command “--edit-key” to generate a subkey for this purpose.
pub   rsa4096/0xA2B71234247579BE 2018-01-28 [SC]
Key fingerprint = 1A28 A73B A841 4311 FA0A BA28 A4A7 1383 1171 7A99
uid                              Valerii <vkafed@gmail.com>

Export a new key

export KEYID=0xA2B71234247579BE

Create subkeys

VKAFEDZH-M-2R3C:~ val$ gpg --expert --edit-key $KEYID gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

gpg: checking the trustdb gpg: marginals needed: 3 completes needed: 1 trust model: pgp gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u sec rsa4096/0xA2B71234247579BE created: 2018-01-28 expires: never usage: SC trust: ultimate validity: ultimate [ultimate] (1). Valerii <vkafed@gmail.com>

During a creating a new subkeys enter the passphrase you entered during making a master key. In the following example I selected to generate a key with expiration in 1 year.

gpg> addkey
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (12) ECC (encrypt only)
  (13) Existing key
Your selection? 4
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Mon Jan 28 14:13:38 2019 EST
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
sec  rsa4096/0xA1289348BA3879AE
created: 2018-01-28  expires: never       usage: SC
trust: ultimate      validity: ultimate
ssb  rsa4096/0xA1CDBEFBA21203A4
created: 2018-01-28  expires: 2019-01-28  usage: S
[ultimate] (1). Valerii <vkafed@gmail.com>

Next, let’s create n encryption key by selecting RSA (encrypt only) – number 6

gpg> addkey
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (12) ECC (encrypt only)
  (13) Existing key
Your selection? 6
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Mon Jan 28 14:16:29 2019 EST
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
sec  rsa4096/0xA1289348BA3879AE
created: 2018-01-28  expires: never       usage: SC
trust: ultimate      validity: ultimate
ssb  rsa4096/0xA1CDBEFBA21203A4
created: 2018-01-28  expires: 2019-01-28  usage: S
ssb  rsa4096/0x181FFBA2120AA342
created: 2018-01-28  expires: 2019-01-28  usage: E
[ultimate] (1). Valerii <vkafed@gmail.com>

I selected to create 4096 bit RSA encrypt only key which is going to be valid for 365 days only.

And let’s create an authentification key now.

Export and backup our keys

$ gpg --armor --export-secret-keys $KEYID > $GNUPGHOME/master.key
$ gpg --armor --export-secret-subkeys $KEYID > $GNUPGHOME/sub.key

That will require to enter your passphrase for master key again.

Configure Yubikey

VKAFEDZH-M-2R3C:~ val$ gpg --card-edit

Reader …: Yubico Yubikey 4 OTP U2F CCID Application ID …: F1760095270102331226071222510019 Version …: 2.1 Manufacturer …: Yubico Serial number …: 12720751 Name of cardholder: [not set] Language prefs …: [not set] Sex …: unspecified URL of public key : [not set] Login data …: [not set] Signature PIN …: not forced Key attributes …: rsa2048 rsa2048 rsa2048 Max. PIN lengths .: 127 127 127 PIN retry counter : 3 0 3 Signature counter : 0 Signature key …: [none] Encryption key…: [none] Authentication key: [none] General key info..: [none]

gpg/card> admin Admin commands are allowed

gpg/card> passwd 1 - change PIN 2 - unblock PIN 3 - change Admin PIN 4 - set the Reset Code Q - quit

Your selection? 3 PIN changed.

1 - change PIN 2 - unblock PIN 3 - change Admin PIN 4 - set the Reset Code Q - quit

Your selection? 1 PIN changed.

1 - change PIN 2 - unblock PIN 3 - change Admin PIN 4 - set the Reset Code Q - quit

Your selection? q

gpg/card> quit

Let’s transfer keys from our mac to yubikey

Move keys to yubikey

VKAFEDZH-M-2R3C:~ val$ gpg --edit-key $KEYID gpg (GnuPG) 2.2.4; Copyright (C) 2017 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

sec rsa4096/0xA1289348BA3879AE created: 2018-01-28 expires: never usage: SC trust: ultimate validity: ultimate ssb rsa4096/0xA1CDBEFBA21203A4 created: 2018-01-28 expires: 2019-01-28 usage: S ssb rsa4096/0x181FFBA2120AA342 created: 2018-01-28 expires: 2019-01-28 usage: E ssb rsa4096/0x181FFBA2120AA342 created: 2018-01-28 expires: 2019-01-28 usage: A [ultimate] (1). Valerii <vkafed@gmail.com>

gpg> key 1

sec rsa4096/0xA1289348BA3879AE created: 2018-01-28 expires: never usage: SC trust: ultimate validity: ultimate ssb* rsa4096/0xA1CDBEFBA21203A4 created: 2018-01-28 expires: 2019-01-28 usage: S ssb rsa4096/0x181FAAA2120AA342 created: 2018-01-28 expires: 2019-01-28 usage: E ssb rsa4096/0x181FFBA2120AA342 created: 2018-01-28 expires: 2019-01-28 usage: A [ultimate] (1). Valerii <vkafed@gmail.com>

gpg> keytocard Please select where to store the key: (1) Signature key (3) Authentication key Your selection? 1

Now for each key do *keytocard* command, before proceeding with a new key, you will need to deselect previous key by entering the same command, eg. key 1 again will deselect the key 1, then entry key 2 to select it.

Verify

VKAFEDZH-M-2R3C:~ val$ gpg --list-secret-keys
/var/folders/yy/0gn/T/tmp.Tb4gb/pubring.kbx
-------------------------------------------------------------------------
sec   rsa4096/0xA1289348BA3879AE 2018-01-28 [SC]
      Key fingerprint =1A28 A73B A841 4311 FA0A BA28 A4A7 1383 1171 7A99
uid                   [ultimate] Valerii <vkafed@gmail.com>
ssb>  rsa4096/0xA1289348BA3879AE 2018-01-28 [S] [expires: 2019-01-28]
ssb>  rsa4096/0xA1CDBEFBA21203A4 2018-01-28 [E] [expires: 2019-01-28]
ssb>  rsa4096/0x181FFBA2120AA342 2018-01-28 [A] [expires: 2019-01-28]

Export public key

$ gpg --armor --export $KEYID > ~/pubkey.txt

You may want to upload your public key to public keyserver:

$ gpg --send-key $KEYID
gpg: sending key0xA2B71234247579BE to hkps://hkps.pool.sks-keyservers.net

In next posts I’ll try to use those keys for encryption/decryption and signing.

Happy Halloween!

Tue, 31 Oct 2017 00:00:00 GMT

Wishing Happy Halloween to everyone!

Happy Halloween!

Icinga2: LiveStatus Activation/Installation

Mon, 30 Oct 2017 00:00:00 GMT

Hey all,

Here is a quick note how to install and activate MK Livestatus (Check_MK) API for your Icinga2 installation.

More info about livestatus on the official website: https://mathias-kettner.de/checkmk_livestatus.html

So, check if it’s already activated:

# icinga2 feature list
Disabled features: compatlog debuglog gelf graphite influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql mainlog notification

As we can see livestatus is currently disabled

Let’s enable it using the following command:

# icinga2 feature enable livestatus
Enabling feature livestatus. Make sure to restart Icinga 2 for these changes to take effect.

Now we need to add one more LivestatusListener object and fix the existing one to support livestatus on local unix socket and on TCP socket.

Open /etc/icinga2/features-enabled/livestatus.conf and edit it as following:

library "livestatus"
object LivestatusListener “livestatus-tcp” {
socket_type = “tcp”
bind_host = “0.0.0.0”
bind_port = “6666”
}
object LivestatusListener “livestatus-unix” {
socket_type = “unix”
socket_path = “/var/run/icinga2/cmd/livestatus”
}

You can bind it to some specific IP and/or change port to another one if you need, also make sure that your socket_path is exists

# ls -la /var/run/icinga2/cmd/livestatus
srw-rw---- 1 icinga icingacmd 0 Oct 30 10:51 /var/run/icinga2/cmd/livestatus

Let me know if you have any questions

Thanks!

Icinga2 Director: How to import a server IP from local file

Fri, 27 Oct 2017 00:00:00 GMT

Hey all,

#shortstory

Today I had a pretty interesting task with Icinga2 and Director. The issue was with accessibility to DNS server which was installed in AWS without any public access, so basically it was used only for internal requests. To make story short, the task was to get IP address from local file, hostname from SQL database and assign it to each other.

Here is the changes that I made for ‘PropertyModifierGetHostByName.php’ file.

<?php
// - /usr/share/icingaweb2/modules/director/library/Director/PropertyModifier/PropertyModifierGetHostByName.php
namespace Icinga\Module\Director\PropertyModifier;
use Icinga\Exception\InvalidPropertyException;
use Icinga\Module\Director\Hook\PropertyModifierHook;
use Icinga\Module\Director\Web\Form\QuickForm;
class PropertyModifierGetHostByName extends PropertyModifierHook
{
public static function addSettingsFormFields(QuickForm $form)
{
$form->addElement(‘select’, ‘on_failure’, array(
‘label’        => ‘On failure’,
‘description’  => $form->translate(‘What should we do if the host (DNS) lookup fails?’),
‘multiOptions’ => $form->optionalEnum(array(
‘null’ => $form->translate(‘Set no value (null)’),
‘keep’ => $form->translate(‘Keep the property (hostname) as is’),
‘fail’ => $form->translate(‘Let the whole import run fail’),
)),
‘required’    => true,
));
}
public function getName()
{
    return 'Get host by name (DNS lookup)';
}

public function transform($value)
{
    //$host = gethostbyname($value);
    $lines_array = file("/usr/share/icingaweb2/dns.txt"); //comma-separated txt file
    
    foreach($lines_array as $line) {
        if(strpos($line, $value) !== false) {
        list(, $new_str) = explode(",", $line);
        $host = trim($new_str); }
    }

    if (strlen(@inet_pton($host)) !== 4) {
        switch ($this-&gt;getSetting('on_failure')) {
            case 'null':
                return null;
            case 'keep':
                return $value;
            case 'fail':
            default:
                throw new InvalidPropertyException(
                    'Host lookup failed for "%s"',
                    $value
                );
        }
    }

    return $host;
}
}

The file dns.txt contains IP and domain names, which are comma-separated. You may change the location of the file, use many of them, or even use key-value database for such queries, it’s pretty easy.

So, basically it search for IP address in dns.txt file for provided hostname in Import function. It will return null or fail (based on your settings), if returned IP is not IPv4.

Extra module will be created and pushed to the github on next Monday.

Happy monitoring!

Garage Sale – new or like-new books on Sale!

Tue, 01 Aug 2017 00:00:00 GMT

Learning Python, 5th Edition, by Mark Lutz

Get a comprehensive, in-depth introduction to the core Python language with this hands-on book. Based on author Mark Lutz’s popular training course, this updated fifth edition will help you quickly write efficient, high-quality code with Python. It’s an ideal way to begin, whether youâ€™re new to programming or a professional developer versed in other languages.

Complete with quizzes, exercises, and helpful illustrations, this easy-to-follow, self-paced tutorial gets you started with both Python 2.7 and 3.3 the latest releases in the 3.X and 2.X lines plus all other releases in common use today. Youâ€™ll also learn some advanced language features that recently have become more common in Python code.

Explore Python’s major built-in object types such as numbers, lists, and dictionaries
Create and process objects with Python statements, and learn Python’s general syntax model
Use functions to avoid code redundancy and package code for reuse
Organize statements, functions, and other tools into larger components with modules
Dive into classes: Python’s object-oriented programming tool for structuring code
Write large programs with Pythonâ€™s exception-handling model and development tools
Learn advanced Python tools, including decorators, descriptors, metaclasses, and Unicode processing

Price: $60

Amazon:Â https://www.amazon.com/Learning-Python-5th-Mark-Lutz/dp/1449355730/

Learning Python

Read more under cut…

The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices

This book is about different techniques that help us architect software in a better and more efficient way with microservices packed as immutable containers, tested and deployed continuously to servers that are automatically provisioned with configuration management tools. It’s about fast, reliable and continuous deployments with zero-downtime and ability to roll-back. It’s about scaling to any number of servers, design of self-healing systems capable of recuperation from both hardware and software failures and about centralized logging and monitoring of the cluster.In other words, this book envelops the whole microservices development and deployment lifecycle using some of the latest and greatest practices and tools. We’ll use Docker, Kubernetes, Ansible, Ubuntu, Docker Swarm and Docker Compose, Consul, etcd, Registrator, confd, and so on. We’ll go through many practices and even more tools. Finally, while there will be a lot of theory, this is a hands-on book. You won’t be able to complete it by reading it in a metro on a way to work. You’ll have to read this book while in front of the computer and get your hands dirty.

Price: $35

Amazon: https://www.amazon.com/DevOps-2-0-Toolkit-Containerized-Microservices/dp/152391744X/

The Go Programming Language (Addison-Wesley Professional Computing Series)

The Go Programming Language is the authoritative resource for any programmer who wants to learn Go. It shows how to write clear and idiomatic Go to solve real-world problems. The book does not assume prior knowledge of Go nor experience with any specific language, so you’ll find it accessible whether you’re most comfortable with JavaScript, Ruby, Python, Java, or C++.

The first chapter is a tutorial on the basic concepts of Go, introduced through programs for file I/O and text processing, simple graphics, and web clients and servers.
Early chapters cover the structural elements of Go programs: syntax, control flow, data types, and the organization of a program into packages, files, and functions. The examples illustrate many packages from the standard library and show how to create new ones of your own. Later chapters explain the package mechanism in more detail, and how to build, test, and maintain projects using the go tool.
The chapters on methods and interfaces introduce Goâ€™s unconventional approach to object-oriented programming, in which methods can be declared on any type and interfaces are implicitly satisfied. They explain the key principles of encapsulation, composition, and substitutability using realistic examples.
Two chapters on concurrency present in-depth approaches to this increasingly important topic. The first, which covers the basic mechanisms of goroutines and channels, illustrates the style known as communicating sequential processes for which Go is renowned. The second covers more traditional aspects of concurrency with shared variables. These chapters provide a solid foundation for programmers encountering concurrency for the first time.
The final two chapters explore lower-level features of Go. One covers the art of metaprogramming using reflection. The other shows how to use the unsafe package to step outside the type system for special situations, and how to use the cgo tool to create Go bindings for C libraries.

The book features hundreds of interesting and practical examples of well-written Go code that cover the whole language, its most important packages, and a wide range of applications. Each chapter has exercises to test your understanding and explore extensions and alternatives. Source code is freely available for download from http://gopl.io/ and may be conveniently fetched, built, and installed using the go get command.

Price: $30

Amazon: https://www.amazon.com/Programming-Language-Addison-Wesley-Professional-Computing/dp/0134190440/

RHCSA/RHCE Red Hat Linux Certification Study Guide, Seventh Edition (Exams EX200 & EX300) (Certification & Career – OMG) – 7th edition

Based on Red Hat Enterprise Linux 7, the new edition of this bestselling study guide covers the updated Red Hat Certified System Administrator (RHCSA) and Red Hat Certified Engineer (RHCE) exams.

RHCSA/RHCE Red Hat Linux Certification Study Guide, 7th Edition is fully revised to cover the recently released Red Hat Enterprise Linux 7 and the corresponding RHCSA and RHCE certification exams. This new edition provides complete coverage of all official exam objectives for the new exams.

An integrated study system based on proven pedagogy, this revised bestseller features special elements that reinforce and teach practical skills while preparing candidates for the exam. Each chapter includes step-by-step exercises, Exam Watch and On-the-Job sidebars, Two-Minute Drills, end-of-chapter self tests, and hands-on lab questions. Electronic content includes four complete lab-based practice exams to ensure you’re ready to sit for the live exams.

Complete coverage of all exam objectives and performance-based requirements related to the exams, including difficult lab-based scenarios
Electronic content includes four complete lab-based practice exams, two for RHCSA and two for RHCE
A proven study system for RHCSA and RHCE candidates

Price: $40

Amazon: https://www.amazon.com/RHCSA-Linux-Certification-Seventh-Career/dp/0071841962/

Ansible for DevOps: Server and configuration management for humans

Covers Ansible 2.0!

Ansible is a simple, but powerful, server and configuration management tool (with a few other tricks up its sleeve). This book helps those familiar with the command line and basic shell scripting start using Ansible to provision and manage anywhere from one to thousands of servers.

The book begins with fundamentals, like installing Ansible, setting up a basic inventory file, and basic concepts, then guides you through Ansible’s many uses, including ad-hoc commands, basic and advanced playbooks, application deployments, multiple-provider server provisioning, and even Docker orchestration! Everything is explained with pertinent real-world examples, often using Vagrant-managed virtual machines.

Price: $20

Amazon: https://www.amazon.com/Ansible-DevOps-Server-configuration-management/dp/098639341X/

Ansible for DevOps

Quick update

Sat, 09 Jul 2016 00:00:00 GMT

I didn’t have enough time actually to work on the website, just installed SSL onto www.vkafed.com and start uploading my rpms to yum.www.vkafed.com

Updates to follow

Thanks!

New blog released (english version)

Sun, 26 Jun 2016 00:00:00 GMT

Welcome! I just starting a new blog here. I will not update my oldÂ RussianÂ one anymore at http://www.valroot.com, and this will be English one.

Val Kafedzhy

The AI Inference Networking Maturity Model

Why this model exists

The six dimensions

The five levels

L1: single region, best effort

L2: multi-region, manual

L3: health-aware routing

L4: capacity- and cost-aware, secured

L5: autonomous, self-optimizing

Summary matrix

Quick self-assessment

How to use this

Background references

Related

BGP for the AI Era: Multi-Region Routing for Inference Workloads

The problem: why inference traffic breaks web routing assumptions

Reference architecture: BGP multi-region routing for inference clusters

Decision criteria: anycast + BGP vs GeoDNS vs global load balancer

BGP configuration: communities, health-triggered withdrawal, and BFD

Pitfalls and what I’d do in production

References

Related reading

Let’s talk routing

Service Mesh vs eBPF-Native Data Planes: How to Choose

The problem: what a traffic data plane actually has to do

Reference architecture: sidecar mesh vs ambient vs eBPF-native L3/L4

Service mesh vs eBPF: decision criteria for the data plane

Traffic policy in practice: an Istio canary vs a Cilium identity policy

Pitfalls, trade-offs, and what I’d actually ship

References

Related reading

Let’s talk data planes

Zero Trust Networking Beyond the Buzzword: An Enterprise Reference Architecture

The problem: why the perimeter model fails

Reference architecture: a zero trust network architecture for the enterprise

Decision criteria: perimeter/VPN vs managed ZTNA/SASE vs self-hosted microsegmentation

Policy in practice: identity-aware microsegmentation with Cilium

Pitfalls and what I’d do in production

References

Related reading

Let’s talk zero trust

eBPF in Production: Kernel-Level Observability and Security

The problem: why sidecars and host agents miss what production needs

Reference architecture: an eBPF data plane without sidecars

Decision criteria: eBPF-native vs sidecar service mesh vs traditional host agents

A concrete example: bpftrace for ad hoc observability, Tetragon for enforcement

Pitfalls and what I’d do in production

References

Related reading

Let’s talk eBPF

IPv6 at Enterprise Scale: A Migration Playbook

Why enterprise IPv6 migration stops being optional

Reference architecture: dual-stack core, IPv6-only islands, and NAT64/DNS64 at the edge

Decision criteria: dual-stack vs. IPv6-only + NAT64/DNS64 vs. IPv6-mostly

Pitfalls and what I’d do in production

References

Related reading

Planning your own migration?

Creating a New CA (SHA-512, 4096-bit) with PKCS #11 Smart Card Storage and Subordinate CA

Why Use a Smart Card (PKCS #11)?

Recommended Security Controls

Step-by-Step: Creating a Root CA (SHA-512, 4096-bit) on a Smart Card

3.1 Install Required Packages

3.2 Initialize and Prepare Your Smart Card

3.3 Generate or Import the Root CA Key onto the Smart Card

3.3.1 Generating the Key on the Card

3.3.2 Creating the Root CA Certificate Request

3.3.3 Self-Sign the Root CA Certificate

4. Creating a Subordinate CA (SubCA)

4.1 Generate SubCA Keys and CSR

Running PostgreSQL in Docker on my Mac

PostgreSQL Notes

ElementaryOS: Resolution fix for 2560×1440

SendMail function in Golang

Apache 2.4.6 + mod_wsgi + Python 3.6 + Django 2.2.6 on CentOS 7

Nagios environment variables

Happy Independence Day!

Open-Xchange IMAP: STARTTLS failure

Paperkey: How to backup your GnuPG keys on paper