Specs Reports Diagrams
Architecture Locked · Feb 2026

32-Node NVIDIA B300 GPU Training Farm

256 GPUs · 204.8 Tb/s InfiniBand NDR800 · 1:1 Non-Blocking Fat-Tree · Live Topology Visualization

Live simulation — 2-tier rail-optimized IB fat-tree · AllReduce traffic pattern
Farm Specifications
Total GPUs
0
NVIDIA B300 (HGX Blackwell Ultra)
GPU Memory
0
73.73 TB HBM3e (256 × 288 GB)
FP8 Peak Compute
0
PFLOPS dense
NVFP4 Peak
0
PFLOPS sparse
IB Bisection BW
0
Tb/s · NDR800 · 1:1 non-blocking
NVLink Farm BW
0
TB/s NVLink Gen 5 aggregate
IB Switches
0
Q3400-RA · 8 leaf + 4 spine
Total Racks
0
32 compute + 2 network tower
Farm Peak Power
0
kW · 53% of 1 MW allocation
Power Headroom
0
kW remaining (47%)
Safety Margin
0
kW burst margin below 20 kW/rack
Total Cabling
0
cable assemblies (~320 AOC)
Documentation & Reports
FULL REPORT · 15 SECTIONS

Full Network & Topology Report

Comprehensive 15-section reference — power audit, rail topology, switch specs, BOM, bandwidth summary, topology diagrams, design decisions & risk matrix.

→
ARCHITECTURE

Farm & Network Tower Architecture

Detailed rack-level layout for compute zone (C01–C32) and network tower (N1–N2). Covers IB fabric, Spectrum-4 Ethernet, UFM Appliance & OOB management.

→
SINGLE RACK

Single Rack Server Unit Architecture

Per-server deep-dive: ASUS XA NB3I-E12 BOM, port-by-port connectivity, CX8 rail mapping, BF3 DPU wiring, X710 OOB and power audit per component.

→
PLANNING SUMMARY

Farm Planning Summary

Complete planning overview consolidating the 20 kW wall-output envelope, zone layout, networking decisions, switch selection rationale and deployment checklist.

→
INFRASTRUCTURE

Infrastructure & Supporting Systems

Cooling (4× CRAC, N+1 with 3 active + 1 standby · 650+ kWth), dual-feed power distribution (80 rack PDUs · Row ATS · UPS), and IB-attached shared storage cluster (~590 TB usable · NVMe-oF / Lustre). Separate infrastructure layer covering all 40 rack positions.

→
Interactive Topology Diagrams
IDC Farm Topology
Diagram 1
IDC Farm Planning & Connectivity
End-to-end block flow: Internet → Router → Firewall → Border Leaf → Spectrum-4 → IB Fabric → 32× B300 servers
🔍 Click to open — scroll to zoom · drag to pan
Farm IB Topology
Diagram 2
Farm InfiniBand Rail Topology
NVIDIA-style full IB layout — 4 spines, 8 rail-colored leaves, 32 servers. 256 CX8→Leaf + 64 BF3→Spectrum-4 connections.
🔍 Click to open — scroll to zoom · drag to pan
Single Rack Topology
Diagram 3
Single Rack — Per-Port Connection Detail
C01 representative rack: all 8 CX8→Leaf, BF3→Spectrum-4, X710→OOB, UFM Agent, color-coded legends.
🔍 Click to open — scroll to zoom · drag to pan