Every network team that has run global web traffic thinks it already understands multi-region routing. Then it puts a GPU inference cluster behind that same design and watches its assumptions fail, one at a time. BGP multi-region routing for inference workloads looks like the CDN playbook at first glance. It isn’t. The failure modes are different, and the cost of getting it wrong is a lot higher — the design has to account for both.
This is the architecture I reach for when a client wants inference latency predictable across regions without turning the network team into a full-time incident desk. I’m assuming you already know BGP basics here — AS numbers, path attributes, eBGP/iBGP. What follows is about applying that knowledge to a workload that won’t forgive sloppy convergence.
The Problem: Why Inference Traffic Breaks Web Routing Assumptions
Generic web traffic is forgiving. An extra 200ms because a request landed in the wrong region barely registers, and a stateless HTTP GET can be retried anywhere for free. Inference traffic doesn’t get that slack. Four reasons why:
Latency SLOs are tighter and less forgiving. A p99 target of 300–500ms end-to-end for a real-time inference call leaves almost no room for a routing detour — cross-continental round trips alone can burn half that budget before a single token comes back. A web app hides 150ms behind a spinner. An inference API can’t hide it at all; it lands directly in the user-perceived response time.
The cost of a stall is a GPU, not a thread. Queue a request behind a slow or overloaded node and you’re not blocking a cheap CPU worker — you’re sitting a $30,000+ accelerator idle while it could be serving other requests. Misroute traffic into a saturated region and you’re wasting the most expensive unit of compute in the stack, latency aside. At that price point, capacity-aware routing is a cost control measure, full stop.
Regional capacity is lumpy, not elastic. You can autoscale a web tier from 10 to 200 pods in minutes. GPU capacity doesn’t work that way — it comes in expensive, supply-constrained chunks, and a region running hot often can’t spin up more on demand. Routing has to steer load toward where capacity actually exists, not just where latency is lowest.
Failover has to be fast and clean, not eventually consistent. DNS-based failover leans on clients respecting TTLs. They routinely don’t — resolver caching, sticky client pools, and long-lived streaming connections all get in the way. An inference session mid-token-stream doesn’t get a graceful retry the way a stateless page load does. If a region degrades, the network layer needs to redirect new connections in single-digit seconds. Not minutes.
The natural response is anycast plus BGP — announce the same service IP from multiple regions and let the internet’s routing fabric pick the closest, healthiest path. Right instinct. It only works, though, if you engineer the failure paths as carefully as the happy path, and that’s where most first attempts fall apart.
Reference Architecture: BGP Multi-Region Routing for Inference Clusters
Here’s the shape that’s worked for me across several inference platforms: a shared anycast VIP announced from edge routers in each region, backed by per-region GPU clusters, with a health agent that ties BGP advertisement to real inference SLO metrics — not just process liveness.
flowchart TB
subgraph Internet["Global Internet"]
C1[Client - Americas]
C2[Client - Europe]
C3[Client - APAC]
end
subgraph Anycast["Anycast Front Door — shared VIP 198.51.100.10/32"]
direction LR
A1[Edge PoP - IAD]
A2[Edge PoP - FRA]
A3[Edge PoP - SIN]
end
subgraph RegionUS["Region: us-east-1"]
BGP1[Edge Router - eBGP to Transit/IX]
LB1[L4/L7 Load Balancer]
GPU1[GPU Inference Cluster]
Health1[Health Probe Agent]
end
subgraph RegionEU["Region: eu-central-1"]
BGP2[Edge Router - eBGP to Transit/IX]
LB2[L4/L7 Load Balancer]
GPU2[GPU Inference Cluster]
Health2[Health Probe Agent]
end
subgraph RegionAPAC["Region: ap-southeast-1"]
BGP3[Edge Router - eBGP to Transit/IX]
LB3[L4/L7 Load Balancer]
GPU3[GPU Inference Cluster]
Health3[Health Probe Agent]
end
C1 -->|BGP best path| A1
C2 -->|BGP best path| A2
C3 -->|BGP best path| A3
A1 --> BGP1 --> LB1 --> GPU1
A2 --> BGP2 --> LB2 --> GPU2
A3 --> BGP3 --> LB3 --> GPU3
Health1 -.->|withdraw route on SLO breach| BGP1
Health2 -.->|withdraw route on SLO breach| BGP2
Health3 -.->|withdraw route on SLO breach| BGP3
ASCII fallback for docs, terminals, or anywhere Mermaid doesn’t render:
Global Internet (Anycast VIP 198.51.100.10/32)
|
+------------------------------+------------------------------+
| | |
Edge PoP IAD Edge PoP FRA Edge PoP SIN
(announces VIP) (announces VIP) (announces VIP)
| | |
eBGP to Transit/IX eBGP to Transit/IX eBGP to Transit/IX
| | |
+-----v------+ +------v-----+ +------v------+
| us-east-1 | |eu-central-1| |ap-southeast-1|
| Edge Router| | Edge Router| | Edge Router |
+-----|------+ +------|-----+ +------|------+
| | |
L4/L7 LB <--health probe--> L4/L7 LB <--health probe--> L4/L7 LB
| | |
GPU Inference Cluster GPU Inference Cluster GPU Inference Cluster
(H100/A100 pool) (H100/A100 pool) (H100/A100 pool)
Health agent detects SLO breach (p99 latency, queue depth) -> triggers BGP
route withdrawal for that region's VIP -> anycast convergence redirects new
connections to the next-closest healthy PoP within one BGP update cycle,
accelerated by BFD.
The key design choice here: health state feeds directly into route advertisement. A region doesn’t go down because a ping failed. It goes down when the health agent decides the actual inference SLO is breached, and that decision is what withdraws the route.
Decision Criteria: Anycast + BGP vs GeoDNS vs Global Load Balancer
Before you commit to BGP, be honest about when it’s overkill.
| Criteria | Anycast + BGP | GeoDNS | Managed Global L7 Load Balancer |
|---|---|---|---|
| Steady-state latency | Best — routed at the network layer, no resolver hop | Good, depends on resolver geo-accuracy | Good, comparable if the provider has dense PoP coverage |
| Failover speed | Seconds, once health integration and BFD are tuned | Minutes — bounded by TTL and resolver caching behavior | Seconds to tens of seconds, provider-dependent |
| Blast radius of a bad regional deploy | Contained, if health checks and withdrawal logic are correct; a misconfigured route-map can leak globally | Contained, bounded by TTL | Contained, provider manages isolation |
| Operational complexity | High — own AS, peering relationships, route-map/community discipline, BFD tuning | Low to medium | Low — you trade control for simplicity |
| Cost | Predictable transit/peering + engineering time | Low, mostly DNS query cost | Pay-per-request/data processed; can get expensive at inference-scale egress volumes |
| Client behavior dependency | None — works regardless of resolver or client caching | High — depends on clients respecting TTLs | Low — provider anycast, but still DNS-fronted in most implementations |
| Best fit | High-QPS, strict p99 SLOs, teams with real network engineering capacity | Simple regional failover, small teams, no in-house BGP expertise | Teams that want managed simplicity and can tolerate the cost/control trade-off |
My rule of thumb: if you don’t already run BGP for something else in production, don’t stand up your first AS just for an inference product. Start with a managed global load balancer or GeoDNS, and graduate to anycast + BGP once query volume and SLO tightness justify the operational overhead. If you already own an AS and edge peering, though, folding inference traffic into that fabric is usually the right call — the marginal cost is low and the failover speed is hard to beat elsewhere.
BGP Configuration: Communities, Health-Triggered Withdrawal, and BFD
Here’s a representative FRR config for an edge router in us-east-1 — it announces the shared anycast VIP conditionally on local health, with BFD for fast peer failure detection and communities for regional signaling to upstream/transit.
! FRR bgpd.conf - us-east-1 edge router
router bgp 65001
bgp router-id 10.10.1.1
no bgp ebgp-requires-policy
neighbor 203.0.113.1 remote-as 64500
neighbor 203.0.113.1 description TRANSIT-1
neighbor 203.0.113.1 bfd
neighbor 203.0.113.1 bfd profile fast
neighbor 203.0.113.5 remote-as 64500
neighbor 203.0.113.5 description TRANSIT-2
neighbor 203.0.113.5 bfd
neighbor 203.0.113.5 bfd profile fast
address-family ipv4 unicast
redistribute connected route-map ANYCAST-HEALTH
neighbor 203.0.113.1 route-map SET-REGION-COMMUNITY out
neighbor 203.0.113.5 route-map SET-REGION-COMMUNITY out
exit-address-family
!
bfd
profile fast
detect-multiplier 3
transmit-interval 150
receive-interval 150
exit
!
! redistribute connected advertises the VIP only while its loopback
! (connected) address exists. The health agent removes that loopback
! address on SLO breach, so bgpd withdraws the prefix from every eBGP peer.
route-map ANYCAST-HEALTH permit 10
match ip address prefix-list ANYCAST-VIP
!
route-map SET-REGION-COMMUNITY permit 10
set community 65001:100 65001:910 additive
!
ip prefix-list ANYCAST-VIP seq 5 permit 198.51.100.10/32
The 65001:910 community is an internal signal — in this scheme, 9xx communities mark region priority tiers that a route-reflector or upstream policy can act on, say de-preferring a region running at reduced GPU capacity without withdrawing it entirely. BFD at 150ms/×3 gives sub-second peer failure detection. That matters more than people expect: default BGP hold timers (180s) are far too slow for an inference SLO measured in hundreds of milliseconds.
The health agent itself is a small, boring script, deliberately so — this is not where you want cleverness:
# health-agent.sh — runs every 2s via systemd timer on each edge router.
# Ties the BGP-advertised anycast VIP to actual inference SLO health,
# not just "is bgpd running."
P99_MS=$(curl -s http://localhost:9100/metrics/p99_inference_latency_ms)
QUEUE_DEPTH=$(curl -s http://localhost:9100/metrics/gpu_queue_depth)
if (( $(echo "$P99_MS > 250" | bc -l) )) || (( QUEUE_DEPTH > 500 )); then
ip addr del 198.51.100.10/32 dev lo 2>/dev/null
logger -t health-agent "WITHDRAWN region=us-east-1 p99=${P99_MS}ms queue=${QUEUE_DEPTH}"
else
ip addr add 198.51.100.10/32 dev lo 2>/dev/null
logger -t health-agent "HEALTHY region=us-east-1 p99=${P99_MS}ms queue=${QUEUE_DEPTH}"
fi
This pattern — redistribute a connected loopback address into BGP, then let a health agent add or remove that address — is deliberately simple. It keeps the failure domain small. If the health agent crashes, the route just stays in whatever state it was last in, instead of flapping unpredictably.
Pitfalls and What I’d Do in Production
BGP dampening will punish a flapping health check. If your health agent oscillates — withdrawing and re-announcing every time a metric crosses a threshold — upstream route flap dampening (RFC 2439-style) can suppress your prefix for 30–60 minutes, long after the region actually recovered. What I’d do: require N consecutive bad probes before withdrawal and a longer streak of good probes before re-announcing (I typically use 3 bad / 5 good over a rolling window), and confirm with your transit providers whether they still apply dampening to customer-originated prefixes — many disabled it after RFC 7196, but not all.
Global convergence is not instant, even when your local withdrawal is. Your edge router can pull the route in milliseconds, but propagation across the internet’s BGP mesh can take tens of seconds in the worst case, especially during path hunting. If your client-facing timeout is 2–5 seconds, some in-flight and newly-initiated requests will still land on the degraded region during that window. What I’d do: treat BGP withdrawal as capacity steering, not a request-level correctness guarantee. Pair it with LB-level connection draining and a client SDK that retries against a secondary regional endpoint on failure — don’t make BGP the only line of defense.
MED and AS-path prepending are weaker levers than they look. MED is only compared between routes from the same neighboring AS and is non-transitive — plenty of transit providers ignore or reset it entirely, so don’t rely on it for cross-provider steering. AS-path prepending is coarse, and because you can’t guarantee how every downstream AS weighs path length against local policy, “symmetric” traffic engineering built on prepending often breaks asymmetrically during partial outages — one direction reroutes cleanly, the other doesn’t, and you get latency degradation on returning traffic that’s hard to diagnose. What I’d do: use LOCAL_PREF for steering when you control both ends (iBGP between owned regions or DCs) where the result is deterministic, and treat inter-AS prepending/communities as advisory hints, not guarantees.
Health checks need to understand inference, not just liveness. A GPU node can be “up” — process healthy, port open — while still failing its actual job: model not loaded, request queue backed up from a traffic burst, thermal throttling reducing throughput. What I’d do: feed the health agent real SLO telemetry (p99 latency, queue depth, GPU utilization and temperature, model load state), not a synthetic ping.
Withdraw regions in the right order, or you strand live sessions. Pulling the BGP route before draining the load balancer cuts in-flight streaming inference sessions mid-token. What I’d do: drain new connections at the LB first, let existing sessions finish or hit their natural timeout, then withdraw the route — and verify the withdrawal actually propagated (check a couple of external looking-glass vantage points) before marking the region fully out of rotation. Also audit graceful-restart / GR-helper configuration on your edge routers — it exists to survive planned bgpd restarts without dropping sessions, but misconfigured GR timers can leave a withdrawn region’s routes looking valid to peers longer than intended.
Related Reading
- Networking & Routing — the full pillar on routing architecture for modern infrastructure
- AI Infrastructure Networking — networking design specifically for AI/ML workloads
- Multi-Region Connectivity for Distributed GPU Clusters — coming soon: private backbone and inter-region transport design for training and inference
- IPv6 at Enterprise Scale: A Migration Playbook — dual-stack, a real address plan, and NAT64/DNS64 for IPv6-only GPU fabrics
Let’s Talk Routing
If you’re designing multi-region routing for an inference platform and want a second opinion on the architecture, I’m happy to compare notes. Connect with me on LinkedIn or reach out via Contact.