Kedios Infrastructure Report
Kedios B300 Server Farm — Planning Summary
Kedios B300 Server Farm — Planning Summary
Date: March 30, 2026
Scope: 32-node deployed B300 compute farm with 72-node-standard-aligned network baseline
Purpose: High-level planning and review summary for the current source-of-truth markdown set
1. Executive Position
The Kedios package remains a 32-node NVIDIA B300 training farm. The change in this revision is not a move to 72 deployed servers. Instead, the network, management, border, and BOM basis now aligns to the 72-node B300 standard reference pattern, while the live compute population remains 32 nodes / 256 GPUs.
This summary therefore follows a two-layer rule:
- current deployment = 32 compute nodes, 256 GPUs
- standard-aligned network baseline = retained Spectrum-4 plus
SN5610,SN4700,SN2201, and dualUFM
2. Locked Planning Figures
| Item | Locked value |
|---|---|
| Current deployed compute nodes | 32 |
| Current deployed GPUs | 256 |
| Standard network reference capacity | 72 nodes / 576 GPUs |
| IB compute fabric | 8 × Q3400-RA leaf + 4 × Q3400-RA spine |
| Retained Ethernet side fabric | 2 × Spectrum-4 |
| Added standard-aligned layers | SN5610 ×6, SN4700 ×4, SN2201 ×17, UFM ×2 |
| Contracted package | 34 occupied racks |
| Facility allocation | 1.5 MW |
| Rack-constrained ceiling | 680 kW under the retained 20 kW/rack rule |
3. Power and Rack Framing
Validated Compute-Side Anchors
| Layer | Value |
|---|---|
| 32 compute racks sustained | ~464 kW |
| 32 compute racks burst peak | ~492 kW |
| 32 compute rack hard ceiling | 640 kW |
| 34-rack package hard ceiling | 680 kW |
Allocation Rule
Use these two facts together:
- 1.5 MW = facility-allocation envelope
- 680 kW = current rack-constrained hard ceiling
Because the refreshed network baseline adds SN5610, SN4700, SN2201, and a second UFM node, this planning summary does not publish a new all-in network watt total until the refreshed procurement BOM carries the missing per-model power entries.
4. Network Architecture Summary
Live Compute Fabric
| Component | Count | Planning note |
|---|---|---|
| Q3400-RA leaf | 8 | One per GPU rail |
| Q3400-RA spine | 4 | Keeps 1:1 non-blocking bisection |
| Total Q3400-RA | 12 | Live compute-fabric count remains unchanged |
| IB bisection bandwidth | 204.8 Tb/s | Live 32-node compute-fabric value |
Non-Compute Network Layers
| Component family | Count | Planning note |
|---|---|---|
| Spectrum-4 | 2 | Retained BF3-facing Ethernet side fabric |
| UFM nodes | 2 | Production + standby / HA pair |
| SN5610 | 6 | Locked owner formula: 2 + (2 + 2) |
| SN4700 | 4 | 2 border leaf + 2 C-spine / OOB |
| SN2201 | 17 | Locked owner formula: 9 + 8 |
Management Rule
The old single 96-port OOB switch story is withdrawn. The base management model is now the SN2201-based management layer.
The server-side port mapping remains:
- X710 Port 0 = OS management
- X710 Port 1 = BMC / IPMI
5. Split Baseline vs Current Deployment
| Component family | 72-node standard baseline | Current 32-node populated deployment |
|---|---|---|
| Compute nodes | 72 | 32 |
| GPUs | 576 | 256 |
| Q3400-RA leaf | 8 | 8 |
| Q3400-RA spine | 4 | 4 |
| Spectrum-4 | 2 | 2 |
| UFM nodes | 2 | 2 |
| SN5610 | 6 | 6 |
| SN4700 | 4 | 4 |
| SN2201 | 17 | 17 |
| UFM Agent (software) | 72 | 32 |
The practical meaning is simple:
- the compute population differs between baseline and deployment
- the network-side switch counts do not in the current working design
6. Placement and Cabling Rules
Locked Server-Side Cable Counts
| Link family | Count |
|---|---|
| CX8 to Q3400 leaf | 256 |
| BF3 to Spectrum-4 | 64 |
| Server management links | 64 |
| Q3400 leaf to Q3400 spine | 256 |
Placement Rule
The old assumption that only N1 and N2 are occupied is no longer authoritative. Treat N1–N6 as the logical network/services placement envelope until the refreshed draw.io diagrams lock the final physical placement of:
- Spectrum-4
- SN5610
- SN4700
- SN2201
- dual UFM nodes
7. Downstream Reporting Rules
Every downstream summary, questionnaire answer, or viewer card must preserve these distinctions:
- Never imply that 72 compute nodes are deployed today.
- Present 32 deployed nodes and 72-node standard baseline as separate labeled facts.
- Do not reuse the old
1 MW,1 UFM,1 generic OOB switch, orN3–N6 emptywording. - Do not publish a refreshed all-network watt total until the final BOM provides the missing switch power entries.
Glossary
- NDR
- Next Data Rate — InfiniBand generation at 400 Gb/s (NDR400) or 800 Gb/s (NDR800) per physical port.
- NDR400
- InfiniBand NDR at 400 Gb/s per port, used by the BlueField-3 DPU for side-fabric connections.
- NDR800
- InfiniBand NDR at 800 Gb/s per port, used by ConnectX-8 HCAs on the HGX B300 GPU-to-fabric links.
- ConnectX-8
- NVIDIA ConnectX-8 NDR800 InfiniBand HCA integrated on the HGX B300 tray — 8 per server, one per GPU rail.
- BlueField-3
- NVIDIA BF-3220 DPU — 400G NDR400 InfiniBand, provides side-fabric connectivity and in-network compute offload.
- Q3400-RA
- NVIDIA Quantum-X800 Q3400 Rail-Accelerated InfiniBand switch — 144 NDR ports; deployed as 8 leaf + 4 spine.
- Spectrum-4
- NVIDIA Spectrum-4 400GbE/InfiniBand Ethernet switch — 51.2 Tb/s; retained as active-active side-fabric pair.
- SN5610
- NVIDIA Spectrum-SN5610 converged 400G Ethernet switch — 6 units in the storage/converged service plane.
- SN4700
- NVIDIA Spectrum-SN4700 400G Ethernet switch — 4 units for border/WAN handoff and control-plane.
- SN2201
- NVIDIA Spectrum-SN2201 1G/10G management switch — 17 units covering the full OOB management layer.
- UFM
- Unified Fabric Manager — NVIDIA IB fabric management; deployed as 2-node HA pair (production + standby).
- SHARP
- Scalable Hierarchical Aggregation and Reduction Protocol — in-network collective offload on Q3400-RA.
- HGX B300
- NVIDIA HGX Blackwell Ultra B300 — 8-GPU tray with NVLink Gen 5 at 1.8 TB/s per GPU, 14.4 TB/s aggregate across the tray.
- B300 GPU
- NVIDIA Blackwell Ultra B300 — 288 GB HBM3e, 1.1 kW TDP; current report basis uses ~4.5 PFLOPS FP8 dense / ~9 PFLOPS FP8 sparse and ~15 PFLOPS NVFP4 dense / ~30 PFLOPS NVFP4 sparse per GPU.
- NVLink
- NVIDIA direct GPU interconnect — Gen 5 on Blackwell at 1.8 TB/s per GPU, yielding 14.4 TB/s across an 8-GPU HGX B300 tray.
- HBM3e
- High Bandwidth Memory 3e — stacked DRAM in B300 GPUs at 288 GB per GPU, 8 TB/s peak bandwidth.
- Fat-Tree
- Network topology providing non-blocking bisection bandwidth; IB compute fabric is a 2-tier rail-optimised fat-tree.
- Rail-Optimised
- IB fabric layout: each GPU rail maps to a dedicated leaf switch, keeping AllReduce traffic rail-local.
- AOC
- Active Optical Cable — fibre-based cable with integrated E/O conversion, used for all NDR800 IB inter-rack links.
- IPMI / BMC
- Intelligent Platform Management Interface / Baseboard Management Controller — out-of-band server management.
- PDU-A / PDU-B
- Dual-feed power distribution: each PSU bank pairs with one PDU, giving N+5 PSU + dual-feed facility redundancy.
- CRAC / CRAH
- Computer Room Air Conditioner / Air Handler — precision cooling units, N+1 target coverage in the Kedios facility.
- DPU
- Data Processing Unit — BlueField-3 Smart NIC providing network/storage offload and security isolation.
- XA NB3I-E12
- ASUS server SKU: 9U air-cooled, dual Xeon 6776P, 32 × 128 GB DDR5 (4 TB total), 10× NVMe, HGX B300 ×8, CX-8 ×8, BF-3 ×2.
- Xeon 6776P
- Intel Xeon 6 Granite Rapids-SP — 56-core, PCIe 5.0 host CPU in the XA NB3I-E12 server; current server power tables in this repo model ~350 W per socket.
- NVFP4
- NVIDIA FP4 format — current report basis uses ~15 PFLOPS dense / ~30 PFLOPS sparse per B300 GPU, reported in this repo as ~240 PFLOPS sparse per 8-GPU server.
- FP8
- 8-bit float — current report basis uses ~4.5 PFLOPS dense / ~9 PFLOPS sparse per B300 GPU, with the report itself citing ~36 PFLOPS dense per 8-GPU server.
- AllReduce
- Distributed-training collective operation across all GPUs; accelerated by IB fat-tree fabric and SHARP.
- Fat-Tree Bisection BW
- 204.8 Tb/s across the full 32-server farm — 1:1 non-blocking, no fabric oversubscription.
- 20 kW Rack Limit
- Hard power cap per rack in the Kedios facility; servers draw ~14.5 kW sustained, leaving 5.5 kW margin.