Interconnect Choices for Multi-Node Inference: Ethernet vs RoCE vs InfiniBand
Measured performance of three interconnect options for tensor-parallel inference on a two-node 8-GPU cluster, and when InfiniBand is worth the premium.
Tensor parallelism is extremely sensitive to interconnect bandwidth. Once a model is sharded across GPUs, every layer performs an All-Reduce, and the traffic scales with layer count and hidden dimension. Pick the wrong interconnect and adding GPUs makes things slower.
Test setup
Two nodes, 4 GPUs each, model sharded at TP=8. Three interconnects:
- 25GbE Ethernet, switched
- 100Gb RoCE v2, with PFC lossless configuration
- 200Gb InfiniBand HDR, SHARP disabled
Workload: fixed 2K input / 512 output, batch 16.
Results
| Interconnect | Single-node TP=4 | Cross-node TP=8 | Cross-node efficiency |
|---|---|---|---|
| 25GbE | 100% | 41% | Poor |
| 100Gb RoCE | 100% | 78% | Acceptable |
| 200Gb IB | 100% | 94% | Good |
“Cross-node efficiency” is throughput relative to single-node TP=4. On 25GbE, crossing nodes nearly halves throughput — communication completely dominates any compute gain.
Why Ethernet loses
All-Reduce is synchronous; the slowest link sets the pace. Ethernet’s problem is not just bandwidth:
- No kernel bypass, every packet traverses the stack, so latency is high
- Congestion control relies on PFC, and misconfiguration causes head-of-line blocking
- No hardware collective offload
A properly tuned RoCE setup with PFC and ECN gets close to usable, but operational complexity rises noticeably. InfiniBand’s advantage comes mainly from hardware-level collectives and lower intrinsic latency.
When InfiniBand is worth it
My rule of thumb:
- Within a node: covered by NVLink, not a concern
- 2–4 nodes, TP across nodes: 100Gb RoCE is the price/performance knee
- 8+ nodes, or needing expert parallelism: IB gains become clear
- Large MoE models: EP’s all-to-all traffic pattern is harsher on the network; IB is essentially mandatory
If you only do pipeline parallelism (PP), nodes exchange activations, which is far less bandwidth-hungry — plain Ethernet suffices.
An easily missed detail
For cross-node TP, NCCL topology detection matters. Default configuration sometimes picks a suboptimal path, such as routing through a switch instead of a direct link. Before going live, confirm the actual path with nvidia-smi topo -m and NCCL debug logs. That beats blind parameter tuning.
Conclusion
With limited budget and a need for cross-node TP, 100Gb RoCE is the reasonable choice — provided you are willing to invest time tuning a lossless network. At 8+ nodes or for MoE, the IB premium is justified. 25GbE for cross-node tensor parallelism is essentially pointless.