Gray-failure detection and failover for AI data-center fabrics

Bixder finds the fabric paths that are quietly dropping traffic, moves traffic off them and checks the fix, on the BGP fabric you already run.

The problem: failures your protocols can't see

Hard failures are handled

When a link goes dark, BFD notices within about 100–150 ms in our lab (50 ms × 3), BGP withdraws the route and ECMP sends traffic elsewhere. That part of the fabric already works well.

Gray failures are not

A path that silently drops traffic, for example a corrupted forwarding entry on a spine, keeps its BFD session and BGP routes while links stay up. ECMP keeps hashing flows onto it. Training jobs can stall on retransmits and timeouts, and nothing in the control plane says why.

How it works

1
Detect
Active path probes between leaves; a path is marked bad only when several leaves agree. Interface counters are a second, slower signal that is logged.
2
Drain
Traffic is moved off the bad path with an ordinary BGP policy change. No new protocol and no special switch hardware.
3
Verify, then restore
Every step is timestamped. Once the fault clears and the path checks out, the policy is withdrawn and the path rejoins ECMP.

Proof lab results, emulated GPUs

Headline metric: network outage time, from fault injection to the first kernel route change that moves traffic off the failed path¹. Ranges come from repeated runs. Single runs are examples. ON means Bixder acts. OFF means it only observes. BFD and BGP run in both.

Gray failure

A spine silently drops traffic for one destination; links, BFD and BGP stay up.

ON: 174.0–250.7 ms n=18
OFF: 9037.0–9046.8 ms n=5

With it OFF, BFD and BGP never react. Traffic keeps hitting the bad path until the fault is removed at 8 s (the outage runs about 1 s past that). This is where Bixder makes a difference.

Example runs: ON 250.7 ms (the slowest of the 18), OFF 9039.6 ms².

Fibre cut

A silent cut: the link stops carrying traffic.

ON: 186.7–222.2 ms n=11
OFF: 182.6–207.8 ms n=3

No measured advantage. The ranges overlap. In the controller-OFF runs BFD and BGP did the rerouting, and the outage was about the same as with it ON (OFF n=3, small sample).

Example run: ON 207.2 ms². One more ON run was left out of the range³.

Setup: a Fedora VM (4 vCPUs) running FRR 10.4.5, with 4 rails and 4 spines, veth links and emulated GPUs. BFD timers are 50 ms × 3. ON: controller in act mode. OFF: the same scenario with the controller in observe-only mode (it detects but takes no action). Outage time is measured by tools/outage.py. It is a lower bound and does not include client timeouts. If traffic is never rerouted, the outage ends when probes see the path clean again after the fault is removed (8 s), so the OFF figure reflects the lab's fault duration; it means 'not rerouted', not a measured time to recover. Lab runs on 29 Oct 2026. ↩
Example runs: cut-20261029-200401 (207.2 ms), gray-20261029-200606 (250.7 ms) and gray-nofiq-20261029-201137 (9039.6 ms). Source: metrics.json for each run. Run ids and hashes are in the lab's run archive. ↩
Run cut-20261029-185221 (10322.1 ms) is excluded: the controller log shows an avoid policy on spine3 still active when the fault was injected. The first reroute came at 218.9 ms, like the other cuts, but BGP then re-added the faulted path and a second loss segment of about 10 s was counted. Including it, the ON fibre-cut range is 186.7–10322.1 ms (n=12). ↩

Demo

A two-minute walkthrough of the lab: a fibre cut and a gray failure with the controller acting, a controller-off comparison for each, and the result that a plain cut is rerouted by BFD/BGP either way. The video and its narration use the working name Fabric IQ, which will change.

Screen recording of the lab dashboard (2 min 6 s, 1280×720) with narration.

Honest limits

Real in the lab

Linux kernel forwarding and ECMP
FRR BGP and BFD
Packet loss, TCP timeouts
BGP policy changes and FIB updates
Timestamps on every event

Emulated or not modelled

GPUs are emulated, with no NCCL
Everything runs on a single host: all routers and hosts share one machine's CPU
Switches are veth pairs and Linux, not real ASICs
A scaled-down fabric without 400G links, deep buffers or congestion
No claims about RoCE/RDMA, ECN or PFC
Adaptive routing is not modelled
Every comparison here is lab against lab, not production
n is small (3–18 per row)
Clean starts were verified by ECMP snapshot only for the later 26 runs; the earlier 57 only by controller log

Who it's for

Network teams running AI data centers and neoclouds on commodity BGP fabrics (the lab runs FRR; SONiC is not tested), who want gray failures found and routed around without changing hardware or protocols.

BGP / FRR Leaf–spine, multi-rail